Semantic checks
Check whether the answer agrees with the recorded outcome. For an order-status request, a timed-out lookup does not support a claim that the order shipped.
An endpoint can be up while its agent chooses the wrong tool. Evaluate production traces to find behavior worth investigating.
Check whether the answer agrees with the recorded outcome. For an order-status request, a timed-out lookup does not support a claim that the order shipped.
Inspect the lookup result and timing behind the answer. Confirm that the trace captured the relevant tool call before interpreting a missing result.
Separate traces that matched a scenario from requests that need classification or a new test.
| Trace classification | Unique traces | What to investigate |
|---|---|---|
| Matched a scenario | 8 | Open the check results; a scenario match is not a passing verdict. |
| No scenario matched | 2 | Review the requests and decide whether to add or refine a scenario. |
| Missing input | 1 | Inspect the instrumentation before interpreting scenario coverage. |
| Classification unavailable | 1 | Investigate why classification was unavailable; do not count it as a pass. |
Cancellation matched 5 traces and order status matched 4; one trace matched both. That is 9 scenario matches across 8 unique matched traces. Refund review had no traffic. Counts describe this captured sample, not all production interactions.
Inspect scenario coverageThe latest result for each test shows where to start. It does not describe every run in the window.
| Scenario | Stage results | Reason to investigate |
|---|---|---|
| Cancellation |
| The tool received account 4821; the request concerned account 4812. |
| Order status |
| The lookup timed out, but the response claimed that the order shipped. |
| Refund review |
| No result is available for this test. Collect a relevant trace before judging this test. |
Open the underlying trace, compare the check with its evidence, and confirm that required fields were captured before deciding on a fix.
Synthetic production investigation
An agent can return a successful HTTP response while choosing the wrong tool, inventing an outcome, or giving an incomplete answer. Mibo evaluates configured behavior and trace evidence, not only whether the endpoint was reachable.
After an interaction, send its recorded execution over OTLP. Mibo evaluates applicable configured checks without calling the agent again.
No. Your system handles the customer request first and sends the trace to Mibo separately. The ingestion response is separate from the background evaluation.
No. Coverage is limited by configured checks, classification, and captured evidence. Mibo does not automatically fix regressions.
Trace payloads are encrypted at rest, and product access is scoped to the account that owns them. Send synthetic or appropriately governed data that fits your own privacy requirements.
Follow OTLP/HTTP protobuf or JSON exporter, service routing, authentication, and troubleshooting guidance.
Read OTLP documentationSee the span fields and attributes Mibo can use for semantic and procedural evaluation.
Read the trace referenceFind scenarios with traffic and those still idle.
Explore test coverageCompare agent behavior across changes.
Explore regression testing