One of the easiest ways to tell whether the context you gave your agent is actually good enough is to look at the “apology” metric.
If your AI agent keeps saying things like “sorry” or “you’re absolutely right,” that is an indication that the context can be improved.
When an agent apologizes repeatedly on the same task, the problem is usually not the model - it is the context. Again, I said usually. Very likely, the problem would be solved by passing a better context rather than switching to a more expensive model.
So when you are deploying agents, instrument them - i.e., capture the responses and build a simple apology counter :) alongside your usual correctness metrics.
Some keywords and phrases to watch for are “sorry,” “my mistake,” or “you’re absolutely right.”
Depending on your use case, the root cause could be unclear documentation, an ambiguous API description, instructions that do not cover an edge case, or even a retrieval that is not bringing in the right context. These are all things that can be improved once you know where to look.
In a gist, track apologies as a metric and use them to debug your agent. It is simple, but it works.