When a founder should ignore the panel
A synthetic panel's answer isn't always the right input to weight heavily. Here are the specific situations where the honest move is to override it.
A tool that's never wrong to trust isn't being used honestly.
If we wanted thepanelist to feel maximally trustworthy, the easy move would be to never mention when to override it. Instead, here are the specific situations where a founder should treat a panel's answer as weak evidence at best, and act on other information instead.
When the panel is suspiciously unanimous
Real audiences rarely agree with total unanimity. A panel that produces the same answer from every persona, with no dissent at all, is more often a sign of mode collapse than genuine consensus — the diversity check exists to catch exactly this, and if it flags a panel, that panel's apparent agreement should be discounted, not celebrated.
When you have direct, contradicting evidence from real users
If actual paying customers are telling you something different from what the panel predicted, the real customers win, every time. A synthetic panel is a stand-in for information you don't have yet — it has no authority once the real information arrives.
When the question requires lived experience
Some questions depend on a texture of real experience a language model's training simply doesn't have access to — a specific regulatory environment, a niche technical failure mode, an emotional reaction tied to a very particular life circumstance.
When the decision is expensive to reverse
The more expensive and hard-to-reverse a decision is, the less appropriate it is to rely on a directional, fast-and-cheap instrument for it — that's exactly the situation a real, validated study exists for instead.
In each of these, the honest move is to weight the panel's answer lightly or ignore it outright — not because the tool failed, but because these are outside what it was ever built to answer well.
Common questions about overriding a panel
Doesn't recommending founders ignore the panel undercut the product?
No — a tool that's honest about when not to trust it is more useful, not less, because you can actually calibrate how much weight to give its answer in any given situation.
What should I trust instead, when I ignore the panel?
Direct evidence from real customers, domain expertise the model can't have, or — for expensive, hard-to-reverse decisions — a properly validated real-respondent study.
Is a unanimous panel always wrong?
Not always, but it's suspicious by default. Check the diversity score before trusting a unanimous result — real audiences rarely agree that cleanly.
Check whether your panel's agreement is genuine
See how the diversity check flags suspiciously unanimous panels.
Read the methodologyMore Critical Perspectives
Synthetic panels are a gut-check, not a replacement
It would be easy to let the marketing copy drift toward 'skip user research entirely.' Here's exactly what a synthetic panel can and can't tell you, and why we keep saying so even when it costs us a cleaner pitch.
AI market research: how accurate is it really?
Accuracy isn't one number. It's a set of separate, checkable claims — and most of what gets called 'accuracy' in AI research marketing is actually a claim about something else entirely.
Why we said no to fake respondent counts
It would be easy to show '10,000 simulated respondents' on a panel result. We don't, because a panel of five distinct personas and a panel of ten thousand near-identical copies would look the same on that number.