How to Use the Label Studio Project Builder Prompt to Turn LLM Drafts Into Clean Training Labels

Letting a language model draft labels and having people correct them is one of the fastest ways to build a labeled dataset. It also fails in quiet ways. The import file uses a name that does not match the labeling config, so the drafts never show up. The model returns "Cancel Request" when your class is cancel_request. Annotators disagree on the same edge case for two weeks before anyone notices. The Label Studio Labeling Project Builder for LLM Assisted Annotation: Labeling Config XML with Choices, Labels, and TextArea Tags, Annotator Guidelines, Pre-annotation predictions JSON with model_version, Gold Task Review, and Export Checks prompt plans the whole Label Studio project around those failure points, from the config XML to the export you train on.

What the prompt produces

1. A labeling config XML with the right object tag for your data and control tags for every field in your schema, including required fields and hotkeys.
2. A name map table listing each control name, its target, its result type, and the value shape it expects. This is the contract your import file must follow.
3. Annotator guidelines built from the edge cases you have already seen, with a "use this, not that" line for each confusing pair.
4. A converter spec and a short Python script that turns raw model output into Label Studio predictions, normalizes labels, and flags anything outside the schema instead of guessing.
5. A gold task plan so you can spot drift in both people and the model.
6. An agreement plan that fits your edition, including a Community workaround.
7. Export checks to run before the labels reach training.

How to fill the inputs

DataSchema names what each task contains and the exact field names, such as chat and ticket_id. The config uses these as $chat and so on, so spelling matters.

LabelSchema is your list of classes, whether each field is single or multiple choice, and which fields are required. The prompt will not add classes you did not list.

EdgeCases is the most valuable input. Paste real examples where annotators split, for example "how do I cancel" questions. These become guideline lines.

LlmOutput should include a raw sample of what the model returns today. The converter is written against that shape.

TeamSetup covers team size, edition, and review process. Some features, such as overlap and agreement settings, depend on the edition, and the prompt marks anything it cannot confirm.

Reading the example output

The example labels support chats by intent and urgency, with a free text note only when the intent is other:

• The config uses two single choice Choices blocks and one TextArea, each pointed at the chat field, with hotkeys on the intent classes.
• The name map shows the value shapes, such as choices with a list of one value, which is exactly what the import JSON uses.
• The guidelines settle the cancel versus how to argument with one clear rule tied to whether the customer wants to stop paying.
• The converter normalizes "cancel request" and "High" into the schema values and imports anything invalid with no prediction, listed in a flagged file.
• Agreement on Community is handled by duplicating a sample, exporting, and computing Cohen's kappa offline.
• The export checks catch empty tasks, labels outside the schema, and gold tasks that should not be trained on.

Tips for better results

• Write the name map before you write any converter code. Most broken imports trace back to a mismatched from_name or to_name.
• Keep model_version on every prediction so you can compare how often people accepted or edited each model's drafts.
• Start with a small batch of a few dozen tasks, review it, then update the guidelines before importing the rest.
• Use hotkeys for the most common classes. They save real time on long queues.

Mistakes to avoid

• Do not let the converter guess a class when the model returns something unexpected. Flag it for a person.
• Do not mix gold tasks into the training export.
• Do not rely on prediction scores alone to skip review. A confident model can still be consistently wrong on one edge case.
• Do not change class names in the middle of a project without re-exporting and remapping earlier labels.

Who it is for

ML engineers and data operations leads setting up annotation projects, small teams building a first classifier or extraction dataset, and anyone who wants model drafted labels without losing control of quality.

Related PromptDig links

Open the Label Studio Labeling Project Builder for LLM Assisted Annotation: Labeling Config XML with Choices, Labels, and TextArea Tags, Annotator Guidelines, Pre-annotation predictions JSON with model_version, Gold Task Review, and Export Checks prompt and paste your label schema and a sample of model output to get started. For more prompts on AI workflows, Browse more prompts. If you have a labeling setup that works well, Share a prompt.