The short answer
Most brand lift studies can’t be trusted for three reasons. The respondents may not be real, because online panels leak bots and fraud. Exposure is assumed rather than proven, and the company that sold the media often runs the test. And results arrive without the plan, the raw data or a trail back to the answers, so nobody can check them.
Key takeaways
- Fraud in online panels and AI agents that pass attention checks mean the answers may not come from real people.
- Platform and bundled vendor studies often mark their own test and assume exposure.
- Reports headline sub-groups and relative lift, and keep the statistics ‘available on request’.
- Ask for a locked KPI, the detectable lift, proof of exposure, exclusions and the raw data.
1. Untrusted sources: who answered?.
Most brand surveys run on online opt-in panels, and those panels have a fraud problem. A Rep Data study across four large panels and two panel exchanges found fraud rates from 21% to 38%, averaging 29%, and Rep Data says traditional data cleaning catches only about 30% of fraud. In a 2025 paper in PNAS, Sean Westwood built an AI agent that passed 99.8% of attention checks across 6,000 trials. Pew Research Center found that 12% of opt-in respondents under 30 claimed they were licensed to operate a nuclear submarine, a share that rounds to zero in reality.
A lift built on those answers isn’t a lift in real people. See AI bots are taking your brand surveys for what this means and what to do about it.
2. Unproven exposure: who saw the ad?.
Platform lift tools only measure inside their own platform, and the platform sets the holdout and reports the result. Bundled vendor studies often survey the test group where the ad ran and take the control from a separate panel, then offer ‘ad recallers’ as proof, even though people who remember an ad already lean towards the brand.
Marketers feel it. In TransUnion and EMARKETER’s 2025 survey of 196 US marketers, 41% named walled-garden reporting limits as a top barrier to accurate measurement, and 29% said up to a fifth of their budget had been moved or put at risk because of doubts about measurement.
3. Unevidenced results: can anyone check it?.
We reviewed three recent brand lift reports for national campaigns, 75 pages in total, each built on 200 people per cell. None of the headline lifts was significant on the full sample. One led with ‘+26% awareness’ from a sub-group of 39 against 32 people; on the full sample awareness fell 3.5 points. Another drew 85% against 91% purchase intent as 48% against 52%. Significance testing was ‘available upon request’.
| What the report did | Why it misleads |
|---|---|
| Headlined a sub-group chosen after the data came in | Small groups produce big swings by chance |
| Led with relative lift | Small moves on low baselines look large |
| Used 200 people per cell | Too few to detect lifts under about 13 points |
| Assumed exposure from where people were surveyed | Exposed and control groups differ in more than exposure |
| Kept statistics on request | Nobody can check the headline |
What to ask for instead.
- 1A primary KPI fixed before fieldwork, with proof of when it was fixed.
- 2The smallest lift the study could detect.
- 3How exposure was proven, including the false-recall rate on a decoy ad.
- 4How many people were excluded and why.
- 5The raw data, and a trace from each headline number to the rows behind it.
SpinProof answers all five in every study through the Evidence File.
Questions
Are platform brand lift studies wrong?
Not necessarily. They can be useful for a single digital channel. The problem is independence: the platform that sold the media also chooses the holdout and reports the result, and it can’t see your other channels.
How can I check a brand lift report I already have?
Find the primary KPI, check the sample size and detectable lift, read the full-sample result before any sub-group, convert relative lift to points, check the 95% range and ask for the raw data. Our step-by-step guide covers each check.
Sources
- Westwood (2025), PNAS: The potential existential threat of large language models to online survey research
- Quantic Foundry: Rep Data’s fraud study across four panels and two exchanges
- Rep Data: State of survey fraud
- Pew Research Center: Online opt-in polls can produce misleading results
- TransUnion and EMARKETER: The True Cost of Trust in Marketing Measurement