LangSmith Annotation Queue Checklist from Project Notes (No Invented Eval Scores)

Prompt

Act as a LangSmith annotation-queue librarian who only uses pasted notes. You compile a annotation-queue checklist the notes already support. You do not invent eval scores, queue ranks, latency scoreboards, accuracy percents. This is not a live LangSmith sync, not Phoenix merge, and not ML ops advice.
You work only from Inputs. Do not invent stats, citations, quotes, URLs, names, IDs, or records that are not in Inputs.

Inputs:
- Notes I lock (queue stubs, annotation cues, run fragments): [ProjectNotes]
- LangSmith version or project notes I lock: [Version]
- Project or dataset label I may quote (or UNKNOWN): [ProjectLabel]
- Queue titles already present (or UNKNOWN): [QueueTitles]
- Annotation cues already present (or UNKNOWN): [AnnotationCues]
- Run cues already present (or UNKNOWN): [RunCues]
- Dataset cues already present (or UNKNOWN): [DatasetCues]
- Words I must not use: [Banned]
- What I must never invent (eval scores, queue ranks, latency scoreboards, accuracy percents): [Never]
- Output format: [Format]
- Language: [Lang]

Generate:
1. Honesty ledger: ProjectNotes nouns, Version, ProjectLabel, QueueTitles, AnnotationCues, RunCues, DatasetCues, Lang. Banner: not ML ops advice; not a live LangSmith sync. Forbidden: invented eval scores, queue ranks, latency scoreboards, accuracy percents.
2. Annotation-queue checklist: one checkbox row per QueueTitles entry. Attach only AnnotationCues named beside that entry in ProjectNotes. Missing annotation write NOT IN INPUTS.
3. Run sketch: for each RunCues entry, list rows that name it. Do not invent a 0.91-eval claim if absent.
4. Dataset caution block: quote DatasetCues only. Extra packs not in ProjectNotes stay NOT IN INPUTS.
5. Refuse list: inventing 0.91-eval values for eval scores, inventing queue ranks, inventing latency scoreboards, inventing accuracy percents.
6. Compliance pass: quote Banned and Never hits. Cut them. Print counts from ProjectNotes only. Format as Format.

Constraints:
- Annotation-queue checklist from ProjectNotes only. No invented eval scores.
- Honor Version. No emojis. Not a live LangSmith dashboard. Not ML ops advice.

Instructions

Replace every [bracket] with your details before running. Works on ChatGPT, Claude, and Gemini.

Generated Output

This image was generated using the prompt above.

LangSmith Annotation Queue Checklist from Project Notes (No Invented Eval Scores) - Result

Examples

Example Input

ProjectNotes: queue title Harbor Quay Goldens as pasted annotation cue Correctness as pasted run cue run_harbor_12 as pasted; queue title Quay Storm Goldens as pasted annotation cue Faithfulness as pasted. Eval invent NONE. Rank invent NONE.
Version: LangSmith as pasted (do not invent unreleased AI eval coach)
ProjectLabel: Cedar Pier RAG eval as pasted
QueueTitles: Harbor Quay Goldens as pasted; Quay Storm Goldens as pasted. Latency queue UNKNOWN.
AnnotationCues: annotation cue Correctness as pasted for Harbor Quay Goldens; annotation cue Faithfulness as pasted for Quay Storm Goldens. Toxicity cue UNKNOWN.
RunCues: run cue run_harbor_12 as pasted for Harbor Quay Goldens; Quay Storm Goldens run NOT IN INPUTS.
DatasetCues: dataset cue pier_gold_v1 as pasted for Harbor Quay Goldens. Rubric pack UNKNOWN.
Banned: 0.91 eval, queue rank #1, guaranteed latency scoreboard
Never: invent queue ranks, invent eval scores, invent latency scoreboards, invent accuracy percents
Format: ledger + annotation-queue checklist + run sketch + dataset caution + refuse + compliance
Lang: English

Example Output

1. Ledger. ProjectNotes: queue title Harbor Quay Goldens as pasted annotation cue Correctness as pasted run cue run_harbor_12 as pasted; queue title Quay Storm Goldens as pasted annotation cue Faithfulness as pasted. Eval invent NONE. Rank invent NONE. Version LangSmith. ProjectLabel Cedar Pier RAG eval. QueueTitles Harbor Quay Goldens; Quay Storm Goldens. AnnotationCues Correctness for Harbor Quay Goldens; Faithfulness for Quay Storm Goldens. RunCues run_harbor_12 for Harbor Quay Goldens; Quay Storm Goldens run NOT IN INPUTS. DatasetCues pier_gold_v1 for Harbor Quay Goldens. Rubric pack UNKNOWN. Lang English. Banner: not ML ops advice; not a live LangSmith sync. Forbidden: invented eval scores, queue ranks, latency scoreboards, accuracy percents, 0.91 eval, queue rank #1, guaranteed latency scoreboard.

2. Annotation-queue checklist.
[ ] Harbor Quay Goldens | annotation Correctness as pasted.
[ ] Quay Storm Goldens | annotation Faithfulness as pasted.
Toxicity cue not attached. Latency queue not added.

3. Run sketch.
run cue run_harbor_12 | row Harbor Quay Goldens as pasted.
Quay Storm Goldens run | NOT IN INPUTS.
Eval scores NOT IN INPUTS so do not invent 0.91 eval. Second run_harbor_12 cue not invented.

4. Dataset caution. dataset cue pier_gold_v1 as pasted for Harbor Quay Goldens. Rubric pack UNKNOWN. Do not invent latency scoreboards packs.

5. Refuse. 0.91-eval eval scores: refused. queue ranks: refused. latency scoreboards: refused. accuracy percents: refused. Unreleased AI eval coach: refused.

6. Compliance. Banned hits none. Queues 2. Annotations named 2. Format ledger+annotation-queue checklist+run sketch+dataset caution+refuse+compliance. Gaps: Quay Storm Goldens run, Toxicity cue, Latency queue, Rubric pack, eval scores.

Missing-data policy: if a field was blank, write NOT IN INPUTS rather than guessing. Lock any tool version named in Inputs; if unnamed, write unknown. No invented testimonials, star ratings, or press logos. If legal, clinical, insurance, HR, education-plan, or veterinary content appears, add a one-line not-advice and de-identify banner. Quote banned-word hits and cut them. End with a gaps list of five bullets the user still owes you. Character and byte caps in the job are hard; print counts when relevant. Refuse to backfill DOIs, exam dumps, PHI, PII, or compensation promises not in Inputs.