How we check panel diversity against real Census data
Every panel's age distribution is compared against real U.S. adult population estimates — and we deliberately don't run the same comparison for income, because our income labels aren't real Census brackets.
Deliberately narrower than it could be.
Every panel's age distribution is compared against real U.S. adult population estimates — 18-24, 25-44, 45-64, and 65+ age brackets — using total variation distance, where 0 means the panel's age spread is identical to the real population baseline, and 1 means it's completely disjoint from it.
These four numbers — for the 18-24, 25-44, 45-64, and 65+ brackets respectively — are the actual reference distribution every generated panel's age spread gets measured against.
Why age, specifically, is checkable against Census data
Age is a real, externally-defined demographic category with an actual, publicly available population distribution behind it — which makes it possible to measure a panel's age spread against genuine ground truth, not just an internal consistency check.
Total variation distance is a standard, interpretable way to compare two distributions: a panel that mirrors the real population's age spread closely scores near 0, while one skewed entirely toward one bracket scores closer to 1.
Why we don't do the same for income
Our income categories are qualitative labels an LLM assigns, not tied to a real dollar figure or Census income bracket. There's no honest way to compare a qualitative label against a quantitative government distribution.
Why we say so directly instead of skipping the caveat
Pretending otherwise — implying an income comparison exists when it doesn't, or dressing up the qualitative labels as if they mapped cleanly to Census brackets — would be exactly the kind of overclaiming this whole exercise exists to avoid.
Common questions about this check
Does thepanelist check every demographic trait against Census data?
No — only age, because it's the trait with a genuine, externally-defined, quantitative population distribution to compare against. Income labels are qualitative and aren't checked the same way.
What does a total variation distance of 0 actually mean?
It means the panel's age distribution matches the real U.S. adult population baseline exactly across all four brackets — the closest possible match on this specific check.
Can a panel pass this age check and still be unrepresentative in other ways?
Yes — this check measures age distribution specifically. It says nothing about opinion diversity (covered by the separate mode-collapse check) or other traits not checked against external data.
See a panel's age-distribution score for yourself
Generate a panel and check its age spread against the real Census baseline.
Start a panelMore Methodology Reports
State of AI Persona Bias
A directional look at the default bias in unconstrained AI-generated personas — young, urban, agreeable — illustrated with the same real numbers already published on our methodology page, not a new live study.
We tested whether AI personas actually disagree with you. Most don't.
A panel can pass every demographic diversity check and still fail in a way that matters more: every persona gives functionally the same answer to the one question you actually asked.