AI Agent Evaluation Metrics: Pass Rate and Coverage
Calculate agent pass rates without hiding skipped checks, missing traces, or coverage gaps. Includes worked examples and a release comparison checklist.
Calculate pass rates from evaluated checks and report skipped checks separately. Then measure how much captured traffic matches your scenarios. The worked examples below define the numerator, denominator, and exclusions so another engineer can reproduce each rate.
Choose the unit before calculating a rate
A trace is a recorded interaction. A test result evaluates a scenario against an execution. An assertion checks one expectation inside that test. One trace can match several scenarios, and one test can contain several assertions. These counts are not interchangeable.
The calculations below are synthetic examples for your evaluation report, not measured Mibo results or a promise that every metric appears as a dashboard widget. Keep behavior results, applicability, and evidence quality in separate columns.
Calculate assertion pass rate without counting skips as passes
Suppose a report contains 50 assertion records: 15 passed, 5 failed, and 30 skipped because a dependency prevented evaluation. The scored population contains 20 assertions.
- Pass rate: 15 / (15 + 5) = 75%.
- Failure rate: 5 / (15 + 5) = 25%.
- Skipped: 30 / 50 = 60% of the records. Investigate why these checks never evaluated.
Reporting 5 / 50 = 10% as the failure rate would dilute failures with unevaluated checks. Report “15 passed, 5 failed, 30 skipped” alongside the score. A skipped dependent check is also different from a scenario that was not applicable.
Mibo’s execution score divides passed assertions by scoreable assertions and excludes dependency-skipped assertions. Its aggregate quality metric averages recorded result scores. Those are different aggregations: a result with 1/1 passing and another with 1/9 passing average to 55.6%, while pooling their assertions gives 2/10 = 20%. State which aggregation you use.
When no checks can be scored, skips due to the AI evaluation allowance or unavailable n8n tool-call evidence can leave a result unevaluated with a score of 0.0. Check the passed, failed, and skipped counts when interpreting the score.
Define outcome and tool metrics around explicit contracts
- Scenario success rate: runs meeting every required outcome / applicable runs with sufficient evidence. Define the required outcomes in advance and report excluded runs separately. An average assertion score is not automatically this rate.
- Routing accuracy: correctly routed requests / labeled requests with observable routing. Label expected routes from the input; selecting only traces already on the correct route hides routing failures.
- Tool argument correctness: calls with the required validated arguments / inspected calls requiring those arguments. Also count runs where the required call never happened; those disappear from a call-only denominator.
- Unsupported completion rate: evaluated answers claiming an action succeeded without supporting evidence / evaluated answers making completion claims. Use explicit semantic criteria and review disputed judgments.
- Unauthorized-action count: observed forbidden mutations, grouped by scenario and severity. Keep this visible even when the overall pass rate is high.
Use the six-case testing workflow to define these contracts before collecting scores. An HTTP success status alone does not establish any of them.
Measure scenario coverage separately from quality
Imagine 100 captured production traces: 60 match at least one configured scenario, 25 match none, 10 lack usable input, and 5 cannot be classified. These are mutually exclusive trace groups for this example.
- Matched share of captured traces: 60 / 100 = 60%. This describes the captured sample, not all production traffic if export or sampling omitted interactions.
- Matched share of classifiable traces: 60 / (60 + 25) = 70.6%. Name this narrower denominator explicitly.
- Unmatched: inspect the 25 requests for missing scenarios or classification mistakes. They are not 25 successful tests.
- Unclassifiable: investigate the 10 missing-input and 5 unavailable-classification traces separately. Do not turn them into passes or silently remove them.
Mibo distinguishes matched, no_match, empty_input, and unavailable classification outcomes. Invariant checks can still run without a conditional scenario match. Since a trace can match multiple scenarios, adding scenario counts may double-count traces; deduplicate before reporting traffic coverage.
Treat missing evidence and evaluator mistakes as separate problems
A missing span may mean an action never happened, or that instrumentation lost it. Establish trace completeness before interpreting negative assertions. Missing-data behavior varies by check; an absent field does not always receive a separate instrumentation status.
For semantic checks, keep a small set of examples that engineers have labeled as acceptable or unacceptable. Compare the evaluator with those labels, inspect disagreements, and update the criteria when they are ambiguous. Recheck this calibration when the evaluator model or prompt changes.
Mibo’s hallucination rate is based on failures among configured hallucination_check assertions. With no such assertions, there is no measured rate. It does not measure every possible unsupported statement in all traffic.
Compare latency and cost on the same workload
Report end-to-end latency separately from model and tool durations. Choose a percentile convention, include the sample count, and disclose timeouts and missing timing. A lower p95 is not a quality improvement if the agent now skips required tools.
Tokens are usage, not a currency amount. Cost estimates need the applicable model prices, token categories, retries, and tool charges. Compare per completed task when task completion is your goal, and keep failed-attempt costs visible.
For a version comparison, hold the scenario mix, fixtures, evaluator, and measurement window constant where possible. Otherwise a shift toward easier requests can look like an improvement. Inspect per-scenario results before accepting an aggregate gain.
Write a release report that supports a decision
- Scope: agent version, suite version, evaluator version, scenarios, run count, and fixture conditions.
- Behavior: pass/fail counts, aggregation rule, critical failures, and links to the relevant trace evidence.
- Blind spots: skipped checks, incomplete traces, unmatched traffic, and unavailable classification.
- Operations: latency distribution, timeouts, usage or cost assumptions, and sample sizes.
- Decision: which scenario blocks release, what must be fixed, and what needs more runs or human review.
Five successful runs do not establish a 100% production success probability. Report “5/5 in this fixture” and expand testing around important failures. Select thresholds from the consequences of an error, not an arbitrary industry score.
Connect metrics to a test you can inspect
Start with one scenario from the agent test-case library, define its checks, and inspect both a passing and a failing execution. Mibo’s evaluation workflow connects configured checks to execution evidence so the result can lead to a concrete fix.
For implementation details, use the assertion reference and the OpenTelemetry ingestion guide. Choose metrics only after confirming that your instrumentation supplies the fields they need.