How to Score Tool-Call Transcripts for an Agent Eval
Agent evals fail when the scorer invents the gold set. The transcript used get_order. The scorer decides the agent should have called search_web and refund_order, then marks a fail with a 0.62 score that looks scientific. You ship a dashboard. Nobody can reproduce it. The job is a scorer from the tool list and gold traces you actually have.
The matching generator is the Tool-Call Transcript Scorer for an Agent Eval prompt. Browse related cards in the PromptDig library (Browse more prompts). When a filled run survives, share the version you actually use (Share a prompt).
Gold tools come from the pack, not from folklore
Paste the tool schema (names, params) and the transcript. Job is "score this trace against gold." Tool and version is the scorer host (Claude, a local script, an eval runner) as named. Else unknown.
Inputs and outputs: allowed tools, gold tool names and args if you have them, the transcript JSON, the rubric. Guardrails: do not invent gold tools, do not invent pass rates, do not add tools the agent never had. Eval: five traces including a clean pass, a wrong-tool call, extra call, missing required arg, and a refuse.
If gold is missing, the scorer lists Gaps and scores only schema-validity. It does not mint a gold path because "that is what a good agent would do."
The artifact is a rubric plus a score sheet, not a leaderboard
The generator should emit:
- Rubric dimensions you named (tool name match, required args, extra calls, schema-valid JSON, order if you care)
- A per-trace sheet: each tool call, pass/fail per dimension, evidence quote from the transcript
- A total only if you defined weights. If weights are unnamed, report per-dimension only.
Do not invent:
- A 0-100 model quality score
- Inter-rater reliability
- Industry benchmarks
- Extra gold tools (search, sql, shell) the pack did not list
Retry: only if Eval said to rescore after a parser fix. Version-lock the agent and tool schema versions if named. Else unknown.
De-identify transcripts. Strip customer emails and tokens. This is an eval draft, not a production incident report.
Five eval cases for the scorer itself
Eval the scorer:
1. Gold get_order with id, transcript matches: all pass.
2. Transcript calls refund_order not in tools: extra-call fail, do not add it to gold.
3. Missing required arg: fail that dimension only.
4. Empty transcript: NOT IN INPUTS, no invented 0.0.
5. User asks for a percentile vs "SOTA agents": refuse.
Gaps: missing gold, missing schema, missing whether order matters, missing timeout policy, missing how to treat retries in the trace. Five bullets. The scorer ships when a human can recompute it from the sheet.
A worked pass: one tool, no invented search
Job: score order-lookup traces. IO: tool get_order(id), gold is that one call, transcript pasted. Guardrails: no extra gold. Forbidden: search_web, invented pass rates.
Decomposition: check name, check id present, check no second call. Artifact: rubric with those three dimensions, no weighted total because weights were blank. Eval case 2 calls search_web; sheet marks extra-call fail and does not add search to gold.
If the model writes "gold should include search then get_order" and a 78 percent, throw the sheet away. If the transcript has two ids and gold had one, that is a fail plus a gap, not a new gold. The scorer is a filter on the pack you already trust.
Fill the card, then run
Paste the tool schema, gold if you have it, and the transcript. Leave gold blank rather than guessing a better agent.
Tool and version: [Tool]
Job to automate: [Job]
Inputs/outputs: [IO]
Guardrails: [Guardrails]
Eval or retry needs: [Eval]
Forbidden: [Forbidden]
An eval is only as honest as its gold set. When a run refuses to invent search_web and reports dimensions without a fake percent, share that filled scorer. That is the card other agent teams should copy.