Learn how to set up an audience holdout experiment. Marketers use these studies to measure what advertising actually changes. Instead of inferring performance from exposed users alone, an audience holdout experiment compares people who are served ads with a group that is not served the ads. The difference in tracked behavior reveals incremental impact.
The audience holdout tool works only with campaigns on the Canvas (new experience).
Prerequisites
Audience holdout experiments rely on data from live campaigns. You can configure an experiment before a campaign starts, but data collection begins only after the campaign goes live. If you have multiple published campaigns with different flights on the Canvas, the experiment launches on the earliest flight start date and stops collecting data on the last end date.
Steps to configure an experiment
- Open a campaign on the Canvas (new experience).
- Go to the action bar at the bottom of the canvas.
- Click the flask icon.
- Go to the Audience impact testing card.
- Click the Configure button.
Holdout size
On the next page, review the default holdout size. It's 10%. You can select a different percentage.
illumin recommends a 10% holdout because it balances delivery scale with measurement quality. A 5% holdout might work well for large or high-conversion campaigns. A 20% holdout improves measurement accuracy while removing more users from delivery, which can lower reach and reduce conversions, especially for smaller campaigns. Use larger holdouts only when measurement precision is a higher priority than scale.
Target campaigns
Go to the Target campaigns field. The default selection is All campaigns. You can limit the experiment to a specific campaign on the Canvas (new experience). This is useful if you want to measure incrementality for a single tactic, objective, or funnel stage.
Set up your conversion actions
You must enable at least one Primary conversion action (e.g. purchase or lead submission). A Site engagement conversion action (e.g. a landing page visit) is optional but recommended.
The page displays conversion trackers already configured on the Journey Settings > Open Web Trackers page. You can't add trackers here because the table pulls data from the Open Web Trackers page. To edit existing trackers or create new ones, click the Edit conversion actions button. This opens the Open Web Trackers page.
If no trackers exist, the section remains blank and the experiment can't launch. Create at least one primary conversion tracker.
When you finish adding the Primary action and the optional Site engagement tracker, go to the right-hand side of the page and review the checklist. The primary conversion action item must be checked. If not, you can't enable the experiment. If you add a site engagement acton, that item must be checked here as well.
Launch the experiment
When configuration is complete, go to the bottom-right corner of the page and click the Enable experiment button. Confirm your selection in the pop-up window to start the experiment.
Check experiment status
To review the status of the experiment, open the campaign, go to the action bar at the bottom of the Canvas, click the flask icon, and then click the gear icon. If the experiment is live and there's available data, click the View results button.
- If the experiment isn't active, the page displays a status message.
- Before the earliest campaign flight starts, you can stop the experiment, adjust settings such as holdout size, and enable it again. Once the first flight begins, the experiment can't be changed or restarted if you want to change the experiment's design. Once enabled, users can change the conversion actions (see below).
- Once the experiment goes live, click the View results button to review the charts and performance metrics.
Edit experiment settings
Once live, users can update conversion actions on the Open Web Trackers page.
- Switch the Conversion Label of a tracker from Primary to Secondary, and vice versa.
- Add Primary or Site engagement conversion actions.
illumin shows those changes in the audience holdout table and reporting.
Why stop an experiment
There are two main reasons.
First, the experiment data has reached statistical significance.
- If you don't expect any major changes to the campaign, it's unlikely you'll gain new insights by continuing to run the experiment. Instead, stop the experiment and start exposing the holdout audience to ads for the balance of the flight.
Second, the experiment can't scale.
- If campaign targeting is overly restricted, the experiment might not serve the number of impressions needed to generate a significant result. In this situation, stop the experiment, and create a new one with broader targeting.
FAQs
What types of campaigns work best for audience holdout experiments?
Campaigns with clear conversion goals and steady delivery perform best. High-volume or high-conversion campaigns reach statistical confidence faster.
Can I run a holdout experiment on any campaign?
No. Audience holdout experiments only work with Open Web campaigns built on the Canvas (new experience).
How long should a holdout experiment run?
The experiment should run long enough to collect meaningful conversion data. Short flights or low-volume campaigns may produce inconclusive results.
Does a larger holdout always mean better results?
Larger holdouts improve confidence but reduce reach and conversions. Smaller holdouts often work better when scale or pacing matters.
What happens if a targeting setting changes during an audience holdout experiment?
If a targeting setting changes, such as geographic scope, the experiment results lose reliability. The exposed and holdout groups no longer remain comparable, so measured lift may reflect targeting changes rather than true incremental impact. For accurate results, targeting settings should remain consistent for the full duration of the experiment.
Why would I exclude one or more campaigns from an audience holdout experiment?
You exclude campaigns when you want clearer, more controlled measurement. The most common reasons are listed below.
- You want to measure the impact of a specific campaign. Excluding other campaigns helps isolate incrementality for a single campaign, tactic, or strategy without interference from unrelated activity.
- The campaign is part of another test or experiment. If a campaign already runs in a brand lift study, creative test, or external experiment, excluding it prevents overlapping tests and conflicting results.
- You want to measure a specific funnel stage. Excluding upper, or lower-funnel, campaigns allows you to measure incrementality at a single funnel stage without influence from other stages.
Related articles
Review audience holdout results
How to interpret confidence and margin of error
How incremental site engagement is calculated