This article explains how incremental site engagement from an audience holdout experiment is measured, using museum visits as an analogy.
Imagine you manage a museum. Some people already love museums and some stop by on a whim. Let's say you put up posters around the city to promote a campaign called, "Museum Week." Later, ticket sales are way up.
So you wonder, how many of those visits are because of the posters?
That’s what incremental site engagements measure: the net new visits that your advertising drives, above and beyond what would have happened naturally. Or, to return to the museum analogy, how many visits happened because of the posters.
First, we need to define the counterfactual. This is the “what if” question. What if people had never seen your museum posters? How many visitors would have shown up just because they already like museums or happened to walk by?
This imagined scenario gives us a baseline for comparison. By estimating how many visits might have happened without advertising, we can measure what advertising actually adds.
Imagine a group of people who live in the city but never see the museum posters. This is the holdout group. Let’s say there are 10,000 people in this group. Out of those 10,000 people, 100 visit the museum on their own.
We use these numbers to calculate the natural baseline engagement rate for the holdout group. It shows how much interest exists without advertising. In simple terms, about 1% of people visit the museum naturally.
In museum math, the formula looks like this:
Now imagine a larger group of people who do see the museum posters. This is the exposed group, and it has 90,000 people. If these people behaved exactly like the holdout group, we could estimate how many visits might have happened without posters. This number, 900, represents how many museum visits would likely occur even if no one in the exposed group saw the advertising.
In museum math, the formula looks like this:
After the poster campaign runs, you observe that 5,040 people from the exposed group visit the museum. At first glance, it looks like the posters drove 5,040 visits. But that's not quite true. Based on expected baseline site engagement volume (i.e. number of natural visits calculated in step 2), about 900 of those total visits would likely have happened anyway.
So, we subtract the expected baseline from the actual visits to get the incremental total. This means advertising drove 4,140 extra museum visits that would not have happened naturally.
In museum math, we'd write the formula like this:
Sometimes, it's easier to describe museum visit changes as a percentage. Let's calculate the percent of total visits that came from advertising. Out of 5,040 total visits, 4,140 were incremental. That means 82% of museum visits during the poster campaign were caused by advertising.
The museum math formula looks like this:
Finally, we can compare the exposed group's extra (i.e. incremental) visits to the exposed group's expected baseline. Advertising generated 4,140 incremental visits. The museum expected 900 visits without ads. This means museum visits were 460% higher than normal. For every 1 visitor who would have come naturally, advertising drove an additional 4.6 visitors.
Lift is the measure of how much higher engagement is compared to the baseline. In other words, it shows the percentage increase in visits caused by advertising relative to what would have happened naturally.
In museum math, the incremental engagement lift formula looks like this:
Incremental site engagement measures site visits that advertising causes beyond natural behavior. The method compares exposed users to a holdout group to isolate visits that would not occur without ads. This approach helps marketers avoid overstating performance from users who would visit anyway.
A holdout group represents users who never see ads and shows natural engagement behavior. The engagement rate from this group defines the baseline expectation. Marketers use this rate to estimate how many visits exposed users would generate without advertising.
Incremental volume equals actual site engagements from exposed users minus expected baseline engagements. This subtraction removes visits that likely occur naturally. The result shows the net new site engagements that advertising truly drives.
Incremental lift shows the percentage increase in engagement compared to the expected baseline. The calculation divides incremental volume by expected baseline volume. This metric helps marketers compare impact across campaigns of different sizes.