A methodical approach to agent evaluation

AI is shifting from single-response models to complex, multi-step agents that can reason, use tools, and complete sophisticated tasks. This increased capability means you need an evolution in how you evaluate these systems. Metrics focused only on the final output are no longer enough for systems that make a sequence of decisions.

A core challenge is that an agent can produce a correct output through an inefficient or incorrect process—what we call a "silent failure". For instance, an agent tasked with reporting inventory might give the correct number but reference last year's report by mistake. The result looks right, but the execution failed. When an agent fails, a simple "wrong" or "right" doesn't provide the diagnostic information you need to determine where the system broke down.

To debug effectively and ensure quality, you must understand multiple aspects of the agent's actions:

• The trajectory—the sequence of ...