Would they have bought anyway? Measuring true incrementality for Miss Selfridge
Overview
How a lifecycle display program was held to a causal standard: randomized holdouts, two tailed significance testing, and a metric deliberately matched to the mission, delivering a 47% new customer rate that could actually be defended.
Measurement & Personalization Specialist
Qubit · Miss Selfridge
Experiment design, segmenting, significance testing
47% New Customer Rate
Proving what marketing actually causes
This project centered on a question most marketing programs never honestly answer: of all the conversions a campaign claims, how many would have bought anyway? For Miss Selfridge, a lifecycle display program was building to handle three tasks: find new customers, convert prospects, and grow lifetime value. Against a relentless daily trade calendar, we introduced an intervention: a holdout structure that proved causality, not attribution artifact.
Embedded in the Arcadia ecosystem, we relied on the measurement and audience architecture behind a creative led, data driven display strategy, one designed specifically to drive New Customer Rate rather than recycle existing demand.
01: The Tension
Three ways the numbers could have lied
1: The attribution trap
Retargeting is a customer acquisition trap. It shows ads to people already deep in the funnel, then claims credit when they convert. Incremental ROI is usually an illusion. Some fractions of those customers would have purchased with no ad at all. The dashboard was success, but the counterfactual was revealing.
2: One budget, three jobs
Acquisition, prospecting, conversion, and retention budgets compete for the same spend, and they do not share a success metric. Optimizing the blended number quietly starves acquisition. Proving genuinely new customers to feed the overall line meant converting the book.
3: A calendar that never sleeps
Miss Selfridge's trade plan changed daily: Black Friday, Cyber Monday, and the Christmas run up, with creative refreshed to match offline promotions. Analysis had to accommodate this volatility. Any framework that required a frozen campaign to produce a clean read was useless here.
The risk was not a failing campaign. It was a campaign that looked successful: one whose numbers could flatter without informing. The most dangerous dashboard is the one that is always green.
02: The Craft
A measurement first framework
Budget, audience, and metric were matched deliberately by funnel job, and everything was measured against a control holdout. Explore the four intelligence layers below.
1: Audience
Ensuring clean user segmentation so that acquisition campaigns target only genuinely new visitors, ignoring those already familiar with the brand.
Every claim of impact must survive a comparison against what if we did nothing.
- A 10% randomized holdout receives the audience and investment loop, but is served a blank ad.
- Incrementality is calculated as treated conversion minus holdout conversion.
03: Implementation
Rigorous execution: the placebo arm
We delivered a 10% control group in a true placebo arm: same audience intelligence, same investment logic, but a blank creative, so that the difference between arms could only be attributed to the advertising itself.
The lift was validated as statistically significant by a two tailed test. We chose a two tailed test because a campaign can plausibly hurt as well as help, and an honest test checks both directions. We held a hard reporting line: incremental ROI was never reported on its own. It was always paired with New Customer Rate, so the program was judged on its mission, not on recycled demand.
Audience intelligence and investment logic with a live creative, measured against the holdout arm.
Same audience and investment logic, but served a blank ad representing a true placebo.
Treated conversion rate minus holdout conversion rate. Validated by a two tailed significance test.
04: The Evidence
Measuring the lift
New Customer Rate: nearly half of campaign driven purchases came from customers entirely new to the brand.
randomized holdout: incrementality confirmed by a two tailed significance test.
disciplined allocation: budget fenced by funnel job, preventing retargeting from swallowing the budget.
creative alignment: displays refreshed against the trade calendar through peak season.
Our campaign was creative led, but data driven, and designed specifically to find completely new customers and increase existing customer lifetime value.
Key learnings
ROI can lie
The most targeted number in the program was not ROI; it was the holdout's conversion rate. Without the counterfactual, retargeting flatters itself. With it, we claim only what we cause. This is the study that made measured against the counterfactual, not against zero my operating standard for data decisions.
The metric must match the mission
A program whose mission is new customers cannot be judged on blended ROI. Liberating New Customer Rate from organic search changed what the program optimized for, and gave the team the courage to make hard decisions.
Two tailed humility
Testing both directions is a statement of intellectual honesty: an intervention can hurt as well as help. Building that risk into the design makes the result, when positive, that much more defensible.