Treated arm
90%
Live Miss Selfridge creative
Same audience and investment logic, with the real ad.
Miss Selfridge: would they have bought anyway?
A display program built to find new customers, held to a causal standard: a randomized placebo holdout, a two-tailed test, and a success metric matched to the mission.
- My role
- Measurement & personalization specialist
- Team
- Qubit · for Miss Selfridge (Arcadia)
- Methods
- Holdout design · segmentation · significance testing
- Season
- Black Friday to Christmas
47%
of campaign-driven purchases came from customers new to the brand
01 · The question
The dashboard that’s always green
Every campaign claims the sales that follow it. The honest question is how many of those people would have bought with no ad at all.
Miss Selfridge’s display program had three jobs: find new customers, convert prospects, and grow the value of existing ones. The risk wasn’t a failing campaign. It was one that looked successful while mostly recycling demand that already existed.
1 · The attribution trap
Retargeting reaches people already close to buying, then takes credit when they do. Some would have bought anyway.
2 · One budget, three jobs
Acquisition, conversion and retention compete for the same spend. Optimising a blended number quietly starves acquisition.
3 · A calendar that never stops
Black Friday, Cyber Monday and the Christmas run-up changed the trade plan daily. The measurement couldn’t need a frozen campaign.
02 · Experiment design
Measured against doing nothing, not against zero
Every eligible person was randomly assigned to one of two arms before seeing anything. Both arms got the same targeting and the same spend logic. The only difference was the creative.
Step 1 · Audience
Eligible shoppers, segmented by funnel job
Step 2 · Random assignment
Each person is randomly placed in one arm
Holdout arm
10%
Blank placebo ad
Same audience and investment logic, but a blank ad. Any difference is caused by the advertising itself.
incremental lift = treated conversion rate − holdout conversion rate
Checked with a two-tailed significance test, because an ad can hurt as well as help.
The decisions behind the test
| Decision | Instead of | Why |
|---|---|---|
| Blank-ad placebo holdout | Comparing exposed vs unexposed people | Exposed people differ from unexposed ones. A placebo keeps both arms alike, so only the creative differs. |
| 10% holdout | A 50/50 split | Keeps 90% of spend working through peak season while still giving a clean comparison. |
| Two-tailed test | One-tailed (only looking for a gain) | A campaign can reduce sales. Testing both directions makes a positive result harder to fake. |
| New Customer Rate as the headline metric | Blended ROI | The mission was new customers. ROI alone rewards retargeting people who’d buy anyway. |
| ROI never reported alone | A single ROI figure | Incremental ROI was always shown with New Customer Rate, so the program was judged on its job. |
| Creative refreshed without pausing | Freezing the campaign for a clean read | Randomization holds even as creative changes daily, because both arms see the same calendar. |
03 · Budget by job
Fenced by funnel job, not by channel habit
Each job got its own share of spend and its own scorecard, so cheap retargeting conversions couldn’t quietly absorb the acquisition budget.
50%25%25%
Prospecting · 50%
Reach shoppers new to the brand. Segments exclude anyone already familiar with it.
Scored on: New Customer Rate
Conversion · 25%
Help engaged prospects buy, with its own scorecard.
Retention · 25%
Grow lifetime value from existing customers, with its own scorecard.
No blended number — each job kept its own scorecard.
04 · Results
What we could defend
47%
New Customer Rate
Nearly half of campaign-driven purchases came from people new to the brand.
Significant
Incremental lift
Incremental lift over the placebo holdout, on a two-tailed test.
50 / 25 / 25
Budget by job
Budget held by job through the peak trading season.
What 47% means
It describes who the campaign’s buyers were: 47% hadn’t bought from Miss Selfridge before. It is not the size of the lift. The lift itself was significant against the holdout; its size is confidential.
What changed
The program was judged on finding new customers, not on recycled demand. That made it possible to defend acquisition spend through peak season.
Proprietary metrics are generalised for confidentiality.
05 · Reflection
What I’d tell the next team
The metric must match the mission
A program meant to find new customers can’t be judged on blended ROI. Choosing the metric was the most important design decision in the study.
Test both directions
Admitting an ad could hurt, and testing for it, is what makes a positive result believable.
The holdout is the product
The most important number wasn’t ROI. It was the holdout’s conversion rate. Without it, retargeting flatters itself.
Placebos cost money
Blank ads are bought impressions. Today I’d look at ghost-ad methods, which log who would have seen an ad instead of paying to show a blank one.
If I ran it again
- Agree the minimum detectable lift and the sample size before launch, and write them down.
- Report the lift with a confidence interval, not just “significant”.
- Keep a small always-on holdout, so every season has its own baseline.
Further reading: Incrementality tests 101: intent-to-treat, PSA, ghost ads and ghost bids (opens in a new tab) (Remerge).