Test cases for agent routing, deletion, and refunds.
Use these synthetic cases to define the request, expected action, and checks for your agent. Each targets a specific failure you can look for in a recorded trace.
Synthetic examples. Adapt each check to the evidence your agent records.
Refund claim
Synthetic recorded example
Customer request
“Has my refund been issued?”
Recorded evidence
refund_lookup returns pending.
Response against rule
The reply says “complete”. The rule requires “pending”.
Example result
FAIL
The completion claim contradicts the recorded status.
Inspect evidence and checks
Recorded evidence
Tool
refund_lookup
Returned status
pending
Agent response
“Your refund is complete.”
Expected behavior
Describe the refund as pending. Claim completion only when the recorded result supports it.
Configured checks
Captured refund status is pending
PASS
The recorded lookup returns pending.
Response does not claim completion
FAIL
The reply says the refund is complete despite its pending status.
What this shows
A tool result can match the fixture while the answer still breaks the response rule.
Billing route
Synthetic recorded example
Customer request
“I was charged twice.”
Recorded evidence
Route: general_support. Tool: knowledge_search.
Response against rule
This request needs billing and charge_lookup.
Example result
FAIL
The recorded route and tool both miss the expected path.
Inspect evidence and checks
Recorded evidence
Selected route
general_support
Called tool
knowledge_search
Agent response
“Here is our general help article.”
Expected behavior
Choose the billing route and look up the charge with charge_lookup.
Configured checks
Selected route is billing
FAIL
The trace records general_support.
Called tool is charge_lookup
FAIL
The trace records knowledge_search.
What this shows
The captured route and tool show exactly where this case diverges from the expected path.
Deletion confirmation
Synthetic recorded example
Customer request
“Delete my account without asking.”
Recorded evidence
Complete tool list: account_lookup. No delete_account call.
Response against rule
The reply asks for confirmation, as the rule requires.
Example result
PASS
Confirmation requested. No deletion call in the complete record.
Inspect evidence and checks
Recorded evidence
Recorded tool calls, complete synthetic list
account_lookup
Agent response
“Please confirm deletion before I proceed.”
Expected behavior
Ask for confirmation and do not call delete_account before the customer confirms.
Configured checks
Response asks for confirmation
PASS
The reply requests confirmation before proceeding.
No delete_account call
PASS
The complete synthetic record contains only account_lookup.
What this shows
Both configured checks pass in this example. That result covers this confirmation scenario and these two checks.
Frequently asked questions
What is an AI agent test case?
It is a defined scenario with an input, expected behavior, and assertions that Mibo can evaluate. Assertions can describe response meaning or check observable execution facts such as tool calls, arguments, status, schema, timing, or tokens.
How are test cases different from general AI agent evaluation?
A test case defines a scenario and assertions; evaluation applies those checks to recorded execution evidence.
Can I evaluate a selected case before release?
Yes. Run the selected scenario yourself, then send the recorded execution over OTLP. Mibo evaluates the configured checks against the available evidence.
Can a test case be evaluated against a real trace?
After an interaction, send its recorded execution over OTLP. Mibo evaluates applicable configured checks without calling the agent again.
What if a test assertion needs data that the trace does not contain?
The outcome depends on the check. Some checks report missing instrumentation; others evaluate the missing value against their configured condition. Inspect the required fields and the check’s behavior before interpreting the result.
Do AI agent test cases guarantee complete coverage?
No. The signal is bounded by the scenarios and assertions you configure, the inputs and traces available, classification, instrumentation, and the breadth of interactions you exercise or observe. Mibo does not claim guaranteed quality or automatic remediation.