This article explains how incremental conversions from an audience holdout experiment are measured, using sales from a hot dog cart as an analogy.
Imagine you’re running a hot dog cart on a busy street. Some people stop by naturally. Maybe they’re hungry, or they’re loyal fans of your food. In the spring, you hire someone to hand out flyers. Suddenly, you're attracting extra customers who wouldn’t have come otherwise. Sales are way up.
So you wonder, how many of those sales are because of the flyers?
That’s what incremental conversions measure: the net new sales that your advertising drives, above and beyond what would have happened naturally. Or, to return to the hot dog analogy, an incremental conversions report tells you how many hot dogs were sold because of the flyers.
First, we need to establish the counterfactual, which is the “what if” scenario. It's like asking: How many hot dogs would people have bought from your cart if they hadn’t seen your flyers?
This hypothetical scenario gives us a baseline for comparison. By estimating what sales (i.e. conversions) would have looked like without ads, we can measure the impact of advertising.
Imagine there's a group of people who walk past your hot dog cart but never see your flyers. This is your holdout group. Let's say that group has 10,000 people. Out of those 10,000 people, 100 stop and buy a hot dog.
With these numbers, we can calculate the natural baseline conversion rate, which is the hot dog sales rate for the holdout group (or CVRh in hot dog math).
In hot dog math, the formula looks like this:
Now, suppose a different group of people walk by your hot dog cart. They see the flyers. This is your exposed group. It has 90,000 people. If the people in the exposed group behave exactly like the folks in the holdout group, you could estimate total baseline hot dog sales during a testing period. We take the conversion rate from the smaller holdout group (i.e. 1%) and apply it to the larger exposed group.
In hot dog math, we'd write the formula like this:
Here's some good news. It seems like those flyers really did have an impact. At the end of the testing period, 5,040 people in the exposed group each bought one hot dog. But, did the flyers cause all those sales?
Nope. Remember the expected baseline? You would have sold 900 dogs without the flyers because that's your natural baseline of sales activity. So, subtract 900, and you get 4,140 extra hot dogs sold thanks to advertising.
In hot dog math, that's called the incremental conversion volume.
Sometimes, it's easier to talk about percentage changes. Of all the hot dogs sold (5,040), 82% were incremental. That means 82% of all sales wouldn’t have happened without the flyers. That's called the incremental conversion share.
Here's the hot dog math:
Finally, let's compare incremental hot dog sales (4,140) to the baseline expectation (900). In hot dog math, the flyers boosted sales by 460%. In other words, for every 1 natural hot dog sale, the flyers drove an extra 4.6 sales.
Lift is the measure of how much higher incremental conversions are compared to the expected baseline. It shows the percentage increase in sales caused by advertising relative to what would have happened naturally.
In hot dog math, that's called the incremental conversion lift.
Incremental conversions represent net new actions that advertising causes beyond natural behavior. The calculation compares exposed audiences to a holdout baseline to isolate conversions that ads create rather than those that would occur without ads.
A holdout group establishes the natural conversion rate without advertising. The experiment uses this rate as the counterfactual baseline, which estimates how many conversions would likely occur if the exposed audience never received ads.
Incremental conversion volume equals actual exposed conversions minus expected baseline conversions. The expected baseline number comes from applying the holdout conversion rate to the exposed audience size during the same measurement period.
Incremental conversion share shows the percentage of total exposed conversions that ads drive. The experiment divides incremental conversion volume by total exposed conversions to express how much performance depends on advertising.
Incremental conversion lift measures how much higher conversions are compared to baseline expectations. The experiment compares incremental volume to expected baseline conversions to show the relative percentage increase caused by advertising.