🤖 AI Tools
Promptfoo Eval Config Map from Dataset Inventory (No Invented Scores)
Turn a dataset inventory into a Promptfoo eval config map only. No invented scores, pass rates, or API keys beyond the inventory.
0Reviews
Prompt
Act as a Promptfoo eval engineer who only uses a pasted dataset inventory. You write an eval config map the inventory already supports. You do not invent scores, pass rates, API keys, or user PII. This is not a live Promptfoo run and not an eval certification. You work only from Inputs. Do not invent stats, citations, quotes, URLs, names, IDs, or records that are not in Inputs. Inputs: - Dataset inventory I lock (case stubs, provider cues, assert notes): [Inventory] - Promptfoo / config version notes I lock: [Version] - Project or workspace label I may quote (or UNKNOWN): [Workspace] - Required case or provider names I may quote (or UNKNOWN): [SpecNames] - Words I must not use: [Banned] - What I must never invent (scores, pass rates, API keys, user PII): [Never] - Output format: [Format] - Language: [Lang] Generate: 1. Honesty ledger: Inventory nouns, Version, Workspace, SpecNames, Lang. Forbidden: invented scores, pass rates, API keys, user PII. Banner: not a live Promptfoo run; not an eval certification. 2. Eval config map table: one row per Inventory case stub or provider cue. Missing assert notes write NOT IN INPUTS. 3. Spec name set: only names in SpecNames. Unnamed cases stay NOT IN INPUTS. Never print numeric score or pass-rate VALUES not in Inventory. 4. Version lock: print Version. Refuse Promptfoo config keys newer than Version if Version is named. 5. Refuse list: inventing scores, inventing pass rates, inventing API keys, inventing user PII. 6. Compliance pass: quote Banned and Never hits. Cut them. Format as Format. Constraints: - Map from Inventory only. No invented score VALUES. - Honor Version. No emojis.
Instructions
Replace every [bracket] with your details before running. Works on ChatGPT, Claude, and Gemini.
Generated Output
This image was generated using the prompt above.
Examples
Example Input
Inventory: case Harbor Quay RAG Answer as pasted provider cue openai:gpt-4o-mini as pasted assert note contains-cite as pasted; case River Ops Tool Call as pasted provider cue anthropic:claude as pasted. Score VALUE NOT IN INPUTS. Pass rate NONE. Version: Promptfoo 0.103 as pasted (do not invent unreleased redteam APIs) Workspace: harbor-quay-evals as pasted SpecNames: RAG Answer as pasted; Tool Call as pasted. gradingRubric UNKNOWN. Banned: guaranteed score, pass-rate invent, key invent, PII invent Never: invent scores, invent pass rates, invent API keys, invent user PII Format: ledger + table + specs + version + refuse + compliance Lang: English
Example Output
1. Ledger. Inventory: case Harbor Quay RAG Answer provider cue openai:gpt-4o-mini assert note contains-cite; case River Ops Tool Call provider cue anthropic:claude. Score VALUE NOT IN INPUTS. Pass rate NONE. Version Promptfoo 0.103. Workspace harbor-quay-evals. SpecNames RAG Answer; Tool Call; gradingRubric UNKNOWN. Lang English. Banner: not a live Promptfoo run; not an eval certification. Forbidden: invented scores, pass rates, API keys, user PII, guaranteed score, pass-rate invent, key invent, PII invent. 2. Eval config map. 1. Harbor Quay RAG Answer | provider openai:gpt-4o-mini as pasted | assert contains-cite as pasted | VALUE NOT IN INPUTS | pass rate NONE. 2. River Ops Tool Call | provider anthropic:claude as pasted | assert NOT IN INPUTS | VALUE NOT IN INPUTS | pass rate NONE. API keys not printed. User PII not invented. 3. Spec name set. RAG Answer; Tool Call as SpecNames. gradingRubric UNKNOWN so write gradingRubric NOT IN INPUTS. No numeric score VALUES printed. No third case invented. 4. Version lock. Promptfoo 0.103 as pasted. Unreleased redteam APIs not used. Cache policy schema NOT IN INPUTS. 5. Refuse. Score invent: refused. Pass-rate invent: refused. Key invent: refused. PII invent: refused. Guaranteed score: refused. 6. Compliance. Banned hits none. Format ledger+table+specs+version+refuse+compliance. Gaps: assert for River Ops Tool Call, gradingRubric decision, concurrency, output path, threshold lock if any. Missing-data policy: if a field was blank, write NOT IN INPUTS rather than guessing. Lock any tool version named in Inputs; if unnamed, write unknown. No invented testimonials, star ratings, or press logos. If legal, clinical, insurance, HR, education-plan, or veterinary content appears, add a one-line not-advice and de-identify banner. Quote banned-word hits and cut them. End with a gaps list of five bullets the user still owes you. Character and byte caps in the job are hard; print counts when relevant. Refuse to backfill DOIs, exam dumps, PHI, PII, or compensation promises not in Inputs.