section 01
Use holdouts for journey value
Holdouts are most useful for onboarding, activation, trial conversion, retention, and winback journeys. They show whether the automation adds value beyond what users would have done anyway.
guide
Holdout Groups for Email
A split test compares message variants. A holdout compares sending against not sending. Growth teams need holdouts when the question is whether a journey changes behavior, not whether one subject line gets more clicks.
last updated 2026-06-12 4 sections
section 02
Pick the unit of randomization
B2B SaaS often needs account-level holdouts so teammates do not contaminate each other. Consumer and newsletter programs can usually randomize by user.
| unit | fit | risk |
|---|---|---|
| User | Individual lifecycle journeys. | Account teammates can influence each other. |
| Account | B2B SaaS activation and renewal. | Needs stable account membership. |
| Domain | Enterprise routing or sales-assist journeys. | Can be too coarse. |
section 03
Measure the business event
Do not stop at opens or clicks. Holdouts should measure activation, conversion, renewal, recovery, or usage. Secondary metrics should catch complaints and unsubscribes.
section 04
Keep the holdout stable
A holdout that changes membership mid-test is hard to trust. Store assignment, assignment time, journey ID, and eligibility state.
- ok Assign once per journey version.
- ok Exclude users who were already exposed.
- ok Keep the analysis window fixed before launch.
reading this as growth and lifecycle teams
Growth work lives or dies on targeting and measurement. What matters is whether behavioral triggers can be built without engineering time, whether the event stream supports attribution past the open, and whether deliverability holds when broadcast volume shares a reputation with product email.
Applied to holdout groups for email, that means weighing scheduling, event stream, deliverability, and operating track record ahead of the rest, against behavior-triggered journeys, broadcasts, and winback campaigns measured on revenue.