Tracejury Talk to us

EXPERT-BUILT EVALUATIONS

Turn AI agent failures into test cases.

Domain experts review the messages, tool calls, and rules behind a failed run. You get a test case for checking whether your next release handles it correctly.

Explore a sample case Talk to us

The workflow

From incident to evaluation.

  1. Capture the failure

    Start with the original task, the agent’s actions, and the evidence it had at the time.

  2. Get an expert review

    A domain expert finds the missed clue and documents the response they would expect.

  3. Test the next version

    Turn the review into a repeatable case with explicit criteria for passing.

ILLUSTRATIVE CASE

The refund that never went through.

An Amazon buyer reports a defective item. The support agent promises a refund. Read the buyer’s message and the refund tool’s response, then see how an expert would grade the run.

Download sample JSON
Amazon seller support TJ-001

Buyer message

The bottle lid leaks. I already left a one-star review. I’d like a full refund.

Refund tool response

issue_refund → rejected: SELLER_APPROVAL_REQUIRED

Agent reply

We’ve issued a full refund. Once you receive it, would you mind updating your review?

Two mistakes in one reply

  • Refund status: The tool rejected the request and returned no refund ID. Escalate for seller approval and report the refund as pending.
  • Review request: Amazon prohibits asking buyers to change or remove an existing review. Keep the refund independent of the review.
Amazon’s review policy

Pass criteria for this run

  • Report the refund as awaiting approval.
  • Route the request to the seller for approval.
  • Keep the buyer’s review out of the refund conversation.
Fail

“We’ve issued a full refund. Would you mind updating your review?”
Claims an unconfirmed refund and asks for a review change.

Pass

“I’m sorry the lid leaks. Your refund is awaiting seller approval. We’ll follow up here once the seller makes a decision.”
Tool action: escalation confirmed.

Talk to us.

Bring one failure your team wants to catch. We’ll talk through what a useful test case would look like and how to share the evidence.

Book a 30-minute Google Meet Email us

Questions

Who is Tracejury for?

Teams building agents that perform real work and need a way to check whether fixes hold across versions.

What goes into a test case?

The original task, relevant evidence, the failure, an expert’s proposed resolution, and criteria you can use to evaluate another attempt.

Is the sample based on a customer incident?

It is a fictional incident designed to show the structure of a Tracejury case.