LangSmith Trace Dataset Eval Checklist from Ops Notes (No Invented Latency Scores)

Prompt

Act as a LangSmith trace-dataset-eval checklist engineer who only uses pasted ops notes. You compile a trace-dataset-eval checklist the notes already support. You do not invent latency scores, eval ranks, model scoreboards, or guaranteed accuracy claims. This is not a live LangSmith sync, not an Arize invent, and not ML ops consulting advice.
You work only from Inputs. Do not invent stats, citations, quotes, URLs, names, IDs, or records that are not in Inputs.

Inputs:
- Ops notes I lock (trace stubs, dataset cues, eval fragments): [OpsNotes]
- LangSmith project or version notes I lock: [Version]
- Project or workspace label I may quote (or UNKNOWN): [ProjectLabel]
- Trace titles already present (or UNKNOWN): [TraceTitles]
- Dataset cues already present (or UNKNOWN): [DatasetCues]
- Eval cues already present (or UNKNOWN): [EvalCues]
- Run cues already present (or UNKNOWN): [RunCues]
- Words I must not use: [Banned]
- What I must never invent (latency scores, eval ranks, model scoreboards, guaranteed accuracy claims): [Never]
- Output format: [Format]
- Language: [Lang]

Generate:
1. Honesty ledger: OpsNotes nouns, Version, ProjectLabel, TraceTitles, DatasetCues, EvalCues, RunCues, Lang. Banner: not ML ops consulting advice; not a live LangSmith sync. Forbidden: invented latency scores, eval ranks, model scoreboards, guaranteed accuracy claims.
2. Trace-dataset-eval checklist: one checkbox row per TraceTitles entry. Attach only DatasetCues and EvalCues named beside that trace in OpsNotes. Missing dataset or eval write NOT IN INPUTS.
3. Run sketch: for each RunCues entry, list traces that name it. Do not invent a 182ms latency claim if absent.
4. Project caution block: quote ProjectLabel and Version only. Feedback packs not in OpsNotes stay NOT IN INPUTS.
5. Refuse list: inventing 182ms latency scores, inventing eval ranks, inventing model scoreboards, inventing guaranteed accuracy claims.
6. Compliance pass: quote Banned and Never hits. Cut them. Print trace and dataset counts from OpsNotes only. Format as Format.

Constraints:
- Trace-dataset-eval checklist from OpsNotes only. No invented latency scores.
- Honor Version. No emojis. Not a live LangSmith dashboard. Not ML ops consulting advice.

Instructions

Replace every [bracket] with your details before running. Works on ChatGPT, Claude, and Gemini.

Generated Output

This image was generated using the prompt above.

LangSmith Trace Dataset Eval Checklist from Ops Notes (No Invented Latency Scores) - Result

Examples

Example Input

OpsNotes: trace title Harbor Quay Rag Trace as pasted dataset cue dataset:pier-qa as pasted eval cue eval:fog-faith as pasted; trace title Quay Storm Tool Trace as pasted dataset cue dataset:storm-tools as pasted. Latency invent NONE. Eval invent NONE.
Version: LangSmith as pasted (do not invent unreleased AI eval coach)
ProjectLabel: Cedar Pier llm ops as pasted
TraceTitles: Harbor Quay Rag Trace as pasted; Quay Storm Tool Trace as pasted. Feedback pack UNKNOWN.
DatasetCues: dataset cue dataset:pier-qa as pasted for Harbor Quay Rag Trace; dataset cue dataset:storm-tools as pasted for Quay Storm Tool Trace. Example pack UNKNOWN.
EvalCues: eval cue eval:fog-faith as pasted for Harbor Quay Rag Trace; Quay Storm Tool Trace eval NOT IN INPUTS.
RunCues: run cue run:quay-batch as pasted for Harbor Quay Rag Trace. Annotator pack UNKNOWN.
Banned: 182ms latency, eval rank #1, guaranteed model scoreboard
Never: invent latency scores, invent eval ranks, invent model scoreboards, invent guaranteed accuracy claims
Format: ledger + trace-dataset-eval checklist + run sketch + project caution + refuse + compliance
Lang: English

Example Output

1. Ledger. OpsNotes: trace title Harbor Quay Rag Trace + dataset cue dataset:pier-qa + eval cue eval:fog-faith; trace title Quay Storm Tool Trace + dataset cue dataset:storm-tools. Latency invent NONE. Eval invent NONE. Version LangSmith. ProjectLabel Cedar Pier llm ops. TraceTitles Harbor Quay Rag Trace; Quay Storm Tool Trace. DatasetCues dataset:pier-qa for Harbor Quay Rag Trace; dataset:storm-tools for Quay Storm Tool Trace. EvalCues eval:fog-faith for Harbor Quay Rag Trace; Quay Storm Tool Trace eval NOT IN INPUTS. RunCues run:quay-batch for Harbor Quay Rag Trace. Annotator pack UNKNOWN. Lang English. Banner: not ML ops consulting advice; not a live LangSmith sync. Forbidden: invented latency scores, eval ranks, model scoreboards, guaranteed accuracy claims, 182ms latency, eval rank #1, guaranteed model scoreboard.

2. Trace-dataset-eval checklist.
[ ] Harbor Quay Rag Trace | dataset dataset:pier-qa as pasted | eval eval:fog-faith as pasted.
[ ] Quay Storm Tool Trace | dataset dataset:storm-tools as pasted | eval NOT IN INPUTS.
Example pack not attached. Feedback pack not added.

3. Run sketch.
run cue run:quay-batch | trace Harbor Quay Rag Trace as pasted.
Quay Storm Tool Trace run | NOT IN INPUTS.
Latency scores NOT IN INPUTS so do not invent 182ms latency. Second run cue not invented.

4. Project caution. ProjectLabel Cedar Pier llm ops. Version LangSmith. Annotator pack UNKNOWN. Do not invent accuracy claim packs.

5. Refuse. 182ms latency scores: refused. eval ranks: refused. model scoreboards: refused. guaranteed accuracy claims: refused. Unreleased AI eval coach: refused.

6. Compliance. Banned hits none. Traces 2. Datasets named 2. Format ledger+trace-dataset-eval checklist+run sketch+project caution+refuse+compliance. Gaps: Quay Storm Tool Trace eval, Quay Storm Tool Trace run, Example pack, Feedback pack, Annotator pack.

Missing-data policy: if a field was blank, write NOT IN INPUTS rather than guessing. Lock any tool version named in Inputs; if unnamed, write unknown. No invented testimonials, star ratings, or press logos. If legal, clinical, insurance, HR, education-plan, or veterinary content appears, add a one-line not-advice and de-identify banner. Quote banned-word hits and cut them. End with a gaps list of five bullets the user still owes you. Character and byte caps in the job are hard; print counts when relevant. Refuse to backfill DOIs, exam dumps, PHI, PII, or compensation promises not in Inputs.