The short answer
AI agents can now complete online surveys convincingly: in a 2025 PNAS study, one passed 99.8% of attention checks. Combined with panel fraud rates of 21% to 38% and cleaning that catches only about 30% of fraud, a brand survey can be partly answered by people, or programs, that never saw the ad. The fix is to verify who answers, prove exposure and show every exclusion.
Key takeaways
- A simple AI agent passed 99.8% of attention checks across 6,000 trials.
- Fraud across major panels and exchanges ran at 21% to 38% in one study; standard cleaning catches about 30%.
- Pew’s 2026 test found no single screening method reliably removes bogus respondents.
- Recruit from the media and from consent, use decoy ads, and publish exclusions with reasons.
What changed.
Survey fraud used to mean speeders, straight-liners and crude bots that attention checks could catch. In 2025, Sean Westwood of Dartmouth published a paper in PNAS describing an autonomous AI respondent that kept a consistent persona, remembered its earlier answers and passed 99.8% of attention checks across 6,000 trials. NORC at the University of Chicago’s April 2026 literature review describes the explosion of automated bots as a danger to the integrity of online survey data.
How big the problem already is.
- 21% to 38% fraud across four large panels and two panel exchanges, averaging 29%, in a Rep Data study.
- About 30% of fraud caught by traditional data cleaning, according to Rep Data; the rest looks like good data.
- 12% of opt-in respondents under 30 told Pew they could operate a nuclear submarine.
- No single screening method reliably removed fake respondents in Pew’s 2026 test of 11,114 opt-in respondents.
Why it matters for brand lift.
Bogus respondents tend to say yes. They claim to recognise ads, to know brands and to intend to buy, because those answers keep them in the survey and earn the reward. In a brand lift study that inflates recall and awareness in both groups, blurs the real gap between exposed and control, and can produce a lift that belongs to nobody. Sub-groups that bogus respondents like to claim membership of, such as young adults, are hit hardest.
What to do about it.
- 1Recruit from the media and from consent. QR codes in or beside the media, publisher lists and opted-in panels tie respondents to real exposure and a real identity.
- 2Check exposure with a decoy ad. An ad that never ran shows how much the sample over-claims. Report the false-recall rate.
- 3Layer the checks. Fraud signals, duplicates, speed, attention and consistency together, not attention checks alone.
- 4Log every exclusion with its reason. If a vendor can’t tell you how many people were removed and why, you can’t judge what is left.
- 5Never report synthetic answers as results. Synthetic responses can pre-test wording; they are not people.
Questions
Can attention checks still catch bots?
They still catch careless humans, but they are no longer enough on their own. Westwood’s AI agent passed 99.8% of them. Use them as one layer alongside recruitment you can verify, fraud signals, a decoy ad and logged exclusions.
Are synthetic respondents ever useful?
They can help test survey wording before fieldwork. They should never be reported as results, because they are not people and did not see the advertising.
Sources
- Westwood (2025), PNAS: The potential existential threat of large language models to online survey research
- NORC at the University of Chicago: Fraudulent respondents and bots in nonprobability surveys (April 2026)
- Quantic Foundry: Rep Data’s fraud study across four panels and two exchanges
- Rep Data: State of survey fraud
- Pew Research Center: Online opt-in polls can produce misleading results
- GCN: Pew finds no screening method reliably filters fake respondents (2026)