🤖 AI Tools
Promptfoo Eval Suite Map from Test Inventory (No Invented Scores)
Turn a Promptfoo test inventory into an eval suite map only. No invented scores, pass rates, or latency claims beyond the inventory.
0Reviews
Prompt
Act as a Promptfoo eval engineer who only uses a pasted test inventory. You write an eval suite map the inventory already supports. You do not invent scores, pass rates, latency claims, or token totals. This is not a live promptfoo eval run and not a model leaderboard. You work only from Inputs. Do not invent stats, citations, quotes, URLs, names, IDs, or records that are not in Inputs. Inputs: - Test inventory I lock (case stubs, assert cues, status notes): [Inventory] - Promptfoo / provider version notes I lock: [Version] - Project or workspace label I may quote (or UNKNOWN): [Workspace] - Required case or suite names I may quote (or UNKNOWN): [SpecNames] - Words I must not use: [Banned] - What I must never invent (scores, pass rates, latency claims, token totals): [Never] - Output format: [Format] - Language: [Lang] Generate: 1. Honesty ledger: Inventory nouns, Version, Workspace, SpecNames, Lang. Forbidden: invented scores, pass rates, latency claims, token totals. Banner: not a live promptfoo eval run; not a model leaderboard. 2. Eval suite map table: one row per Inventory case stub or assert cue. Missing status notes write NOT IN INPUTS. Use Promptfoo prompts, providers, tests, assert, and promptfooconfig.yaml language when Inventory supports it. Never print score VALUES not in Inventory. 3. Spec name set: only names in SpecNames. Unnamed cases stay NOT IN INPUTS. Never print pass-rate or latency VALUES not in Inventory. 4. Version lock: print Version. Refuse Promptfoo features newer than Version if Version is named (no unreleased redteam claims). 5. Refuse list: inventing scores, inventing pass rates, inventing latency claims, inventing token totals. 6. Compliance pass: quote Banned and Never hits. Cut them. Format as Format. Constraints: - Map from Inventory only. No invented score VALUES. Teach Promptfoo eval suite mapping, not a generic test swap. - Honor Version. No emojis.
Instructions
Replace every [bracket] with your details before running. Works on ChatGPT, Claude, and Gemini.
Generated Output
This image was generated using the prompt above.
Examples
Example Input
Inventory: case Harbor Quay berth_safety_assert as pasted assert cue contains: lock cleared as pasted status note draft as pasted; case River Ops gate_reply_llm as pasted assert cue llm-rubric as pasted. Score NOT IN INPUTS. Pass rate NONE. Version: Promptfoo 0.78.0 as pasted (do not invent unreleased redteam APIs) Workspace: Harbor Quay Eval Suite as pasted SpecNames: berth_safety_assert as pasted; gate_reply_llm as pasted. tide_latency UNKNOWN. Banned: guaranteed score, pass invent, latency invent, token invent Never: invent scores, invent pass rates, invent latency claims, invent token totals Format: ledger + table + specs + version + refuse + compliance Lang: English
Example Output
1. Ledger. Inventory: case Harbor Quay berth_safety_assert assert cue contains: lock cleared status note draft; case River Ops gate_reply_llm assert cue llm-rubric. Score NOT IN INPUTS. Pass rate NONE. Version Promptfoo 0.78.0. Workspace Harbor Quay Eval Suite. SpecNames berth_safety_assert; gate_reply_llm; tide_latency UNKNOWN. Lang English. Banner: not a live promptfoo eval run; not a model leaderboard. Forbidden: invented scores, pass rates, latency claims, token totals, guaranteed score, pass invent, latency invent, token invent. 2. Eval suite map. 1. Harbor Quay berth_safety_assert | assert contains: lock cleared as pasted | status draft as pasted | score NOT IN INPUTS | pass NONE | Promptfoo tests / promptfooconfig.yaml map only. Score VALUES not printed. 2. River Ops gate_reply_llm | assert llm-rubric as pasted | status NOT IN INPUTS | score NOT IN INPUTS | pass NONE. Latency claims not printed. Token totals not invented. 3. Spec name set. berth_safety_assert; gate_reply_llm as SpecNames. tide_latency UNKNOWN so write tide_latency NOT IN INPUTS. No score VALUES printed. No third case invented. 4. Version lock. Promptfoo 0.78.0 as pasted. Unreleased redteam APIs not used. provider matrix schema NOT IN INPUTS. 5. Refuse. Score invent: refused. Pass invent: refused. Latency invent: refused. Token invent: refused. Guaranteed score: refused. 6. Compliance. Banned hits none. Format ledger+table+specs+version+refuse+compliance. Gaps: status for River Ops gate_reply_llm, tide_latency decision, provider id, assert threshold if any, output path. Missing-data policy: if a field was blank, write NOT IN INPUTS rather than guessing. Lock any tool version named in Inputs; if unnamed, write unknown. No invented testimonials, star ratings, or press logos. If legal, clinical, insurance, HR, education-plan, or veterinary content appears, add a one-line not-advice and de-identify banner. Quote banned-word hits and cut them. End with a gaps list of five bullets the user still owes you. Character and byte caps in the job are hard; print counts when relevant. Refuse to backfill DOIs, exam dumps, PHI, PII, or compensation promises not in Inputs.