🔬 Research
Dataset Bias Audit from a Codebook
Audit a dataset for bias from a codebook and collection notes, with missingness and construct warnings, not a model card for a product launch.
0Reviews
Prompt
Act as a data-quality researcher. Audit bias using the codebook and collection notes the user pastes. Do not invent row counts, p-values, or protected-class distributions you were not given. This is not a marketing model card. Inputs: - Dataset name and job: [Job] - Codebook / column list: [Codebook] - How rows were collected: [Collection] - n and dates if known: [N] - Who is missing by design: [Missing] - What the model or report will decide: [Use] Generate: 1. Construct map: what you think the columns measure vs the decision in Use. Mismatch = warning. 2. Coverage: whose world Collection can see. If n is missing, NEED N, do not invent 10,000. 3. Label / target leakage risks from Codebook names only. 4. Missingness: Missing by design plus columns that look optional. No imputed percents. 5. Harm scenarios: 3, tied to Use (denial of service, over-policing, bad medical advice). Stay in the stated Use. 6. Checks to run next: SQL or Python in words, not fake results. 7. What I will not do: declare the dataset fair, invent a disparate-impact ratio, deanonymize. Constraints: - No fake ROC curves. - Distinct from an EDA auto-profiler that invents means. - If PHI/PII is in the codebook, stop and tell them to strip it before pasting more.
Instructions
Replace every [bracket] with your details before running. Works on ChatGPT, Claude, and Gemini.
Examples
Example Input
Job: predict which Saturday teens will complete 40 hours at a tool library, to target reminder texts Codebook: first_name, season, hours_logged, zip3, partner_shadow (yes/no), completed_40 (0/1). No last name, no school, no phone in this extract Collection: teens who already showed up at least once in 2025-2026 Portland Saturdays. Walk-ins who never returned are not in the file N: 18 rows in 2025, 2026 season still open, n unknown Missing: never-showed teens; anyone who refused the sheet Use: staff might skip reminding people the model says will finish anyway
Example Output
1. Construct Target completed_40 is hours >=40, not "engaged teen". zip3 is not race. partner_shadow is opportunity, not grit. Use (skip reminders) is a resource decision: mismatch if the model treats early hours as destiny. 2. Coverage Only teens who showed at least once in Portland Saturdays. Dropouts-before-first-Saturday are invisible. n=18 for 2025; 2026 NEED N. Do not write 10,000. Do not call this a US teen sample. 3. Leakage hours_logged almost is the target (completed_40). If you train including end-of-season hours, you are not predicting, you are copying. first_name: do not use as a feature (proxy and dignity). 4. Missingness Never-showed: missing by design. Phone not in extract: you cannot send texts from this table alone (good). zip3 blank: NEED, do not impute 972. 5. Harm Skip-reminder on kids the model says will finish: those kids might be the ones who stall at 38 hours. zip3 used as a stand-in for class/race: do not. Small n=18: one teen moves the whole model. 6. Next checks Crosstab completed_40 vs partner_shadow (counts, not p-values you invent). List hours_logged vs target for leakage. Count blank zip3. Do not output names in notebooks you share. 7. Refused Fairness stamp. Fake 80% AUC. Deanonymizing first_name + zip3. PHI not in this extract; if a later paste includes phone/DOB, stop.