Tool-Call Agent Prompt-to-Eval Rubric
Ppromptstudio·Aug 24, 2026
No rating
Turn an agent prompt into an eval rubric with tool-call gold traces, traps, and a pass/fail rule. No fake benchmark scores.
Act as an eval engineer. Turn an agent system prompt into a rubric plus cases. Distinct from the generic Prompt A/B harness and from RAG eval sets. Focus on tool calls: when to call, which args, when to refuse.
Inputs:
- Agent prompt: [Paste]
- Tools (names + args): [Tools]
- Failures we already saw: [Fails]
- Must-not: [PII, payments, live prod]
- Budget: [How many cases]
Generate:
1. Behaviors to score: tool choice, args, refusal, final answer groundedness. 0-2 scale each.
2. Gold traces: for each case, the tool sequence (or none).
3. Cases: Budget count, mix of happy path, trap (should not call), missing-arg, tool error.
4. Decision rule: pass if mean meets a threshold you state. Fail the run on any must-not hit. Do not invent a public leaderboard score.
5. Trap ideas from Fails only plus obvious PII.
6. Refused: fabricating tool JSON the schema does not have.
Constraints:
- If Tools is empty, say this prompt cannot be tool-eval'd yet.
- No fake 92 percent accuracy.