Promptfoo Eval Suite Ledger from Test Notes (No Invented Score Averages)

Prompt

Act as a Promptfoo LLM eval engineer who only uses pasted test notes. You compile an eval suite ledger the notes already support. You do not invent score averages, latency percentiles, leaderboard ranks, or token-cost invents. This is not a live promptfoo eval run and not a model-vendor sales comparison.
You work only from Inputs. Do not invent stats, citations, quotes, URLs, names, IDs, or records that are not in Inputs.

Inputs:
- Test notes I lock (suite cues, case cues, provider cues): [TestNotes]
- Promptfoo version I lock: [Version]
- Project label I may quote (or UNKNOWN): [ProjectLabel]
- Suite cues already present (or UNKNOWN): [SuiteCues]
- Case cues already present (or UNKNOWN): [CaseCues]
- Provider cues already present (or UNKNOWN): [ProviderCues]
- Assertion cues already present (or UNKNOWN): [AssertionCues]
- Words I must not use: [Banned]
- What I must never invent (score averages, latency percentiles, leaderboard ranks, token-cost invents): [Never]
- Output format: [Format]
- Language: [Lang]

Generate:
1. Honesty ledger: TestNotes nouns, Version, ProjectLabel, SuiteCues, CaseCues, ProviderCues, AssertionCues, Lang. Forbidden: invented score averages, latency percentiles, leaderboard ranks, token-cost invents.
2. Suite checklist: one checkbox row per SuiteCues entry. Mark case attach only when CaseCues says so. Missing assertion write NOT IN INPUTS.
3. Eval ledger table: for each SuiteCues entry, attach only CaseCues and ProviderCues named beside it in TestNotes. Do not invent an 0.91 score average if absent.
4. Assertion caution block: quote AssertionCues only. Red-team pack invents not in TestNotes stay NOT IN INPUTS.
5. Refuse list: inventing 0.91 score averages, inventing p95 420ms latency, inventing leaderboard rank #1, inventing token-cost invents.
6. Compliance pass: quote Banned and Never hits. Cut them. Print suite and case-cue counts from TestNotes only. Format as Format.

Constraints:
- Eval suite ledger from TestNotes only. No invented score averages.
- Honor Version. No emojis. Not a live promptfoo eval dashboard.

Instructions

Replace every [bracket] with your details before running. Works on ChatGPT, Claude, and Gemini.

Generated Output

This image was generated using the prompt above.

Promptfoo Eval Suite Ledger from Test Notes (No Invented Score Averages) - Result

Examples

Example Input

TestNotes: suite Quay Safety Suite as pasted; case cue Quay Jailbreak Refuse as pasted. Score invent NONE. Latency invent NONE.
Version: Promptfoo 0.89 as pasted (do not invent unreleased AI score coach)
ProjectLabel: Harbor Quay LLM evals as pasted
SuiteCues: Quay Safety Suite as pasted. Cedar Latency Suite UNKNOWN.
CaseCues: Quay Jailbreak Refuse as pasted beside Quay Safety Suite. Quay PII Leak UNKNOWN.
ProviderCues: provider gpt-4o-mini as pasted. Claude UNKNOWN.
AssertionCues: assert contains refuse as pasted. Red-team pack UNKNOWN.
Banned: 0.91 score average, p95 420ms, guaranteed leaderboard rank #1
Never: invent score averages, invent latency percentiles, invent leaderboard ranks, invent token-cost invents
Format: ledger + suite checklist + eval ledger + assertion caution + refuse + compliance
Lang: English

Example Output

1. Ledger. TestNotes: suite Quay Safety Suite; case cue Quay Jailbreak Refuse. Score invent NONE. Latency invent NONE. Version Promptfoo 0.89. ProjectLabel Harbor Quay LLM evals. SuiteCues Quay Safety Suite. CaseCues Quay Jailbreak Refuse. ProviderCues gpt-4o-mini. AssertionCues contains refuse. Red-team pack UNKNOWN. Lang English. Forbidden: invented score averages, latency percentiles, leaderboard ranks, token-cost invents, 0.91 score average, p95 420ms, guaranteed leaderboard rank #1.

2. Suite checklist.
[ ] Quay Safety Suite as pasted.
Case Quay Jailbreak Refuse as pasted. Cedar Latency Suite not listed.

3. Eval ledger table.
Quay Safety Suite | case Quay Jailbreak Refuse as pasted | provider gpt-4o-mini as pasted | score average NOT IN INPUTS so do not invent 0.91.
Cedar Latency Suite | NOT IN INPUTS.
Second Quay Safety Suite not invented.

4. Assertion caution. assert contains refuse as pasted. Red-team pack UNKNOWN. Token-cost pack NOT IN INPUTS. Do not invent leaderboard rank packs.

5. Refuse. 0.91 score averages: refused. p95 420ms latency: refused. leaderboard rank #1: refused. token-cost invents: refused. Unreleased AI score coach: refused.

6. Compliance. Banned hits none. Suite rows 1. Case-cue rows 1. Format ledger+suite checklist+eval ledger+assertion caution+refuse+compliance. Gaps: Cedar Latency Suite, Quay PII Leak, Claude provider, red-team pack, score average.

Missing-data policy: if a field was blank, write NOT IN INPUTS rather than guessing. Lock any tool version named in Inputs; if unnamed, write unknown. No invented testimonials, star ratings, or press logos. If legal, clinical, insurance, HR, education-plan, or veterinary content appears, add a one-line not-advice and de-identify banner. Quote banned-word hits and cut them. End with a gaps list of five bullets the user still owes you. Character and byte caps in the job are hard; print counts when relevant. Refuse to backfill DOIs, exam dumps, PHI, PII, or compensation promises not in Inputs.