guide

A/B Testing Email

Email A/B testing is only useful when one variable changes, the audience split is valid, and the winning rule is chosen before the send. This guide keeps tests small enough to run and strict enough to trust.

last updated 2026-08-26 4 sections

section 01

What to test first

Start with variables that map to the metric being measured. Subject lines affect opens. CTA copy and offer framing affect clicks. Journey timing and audience rules affect conversion.

variable	primary metric	keep constant
Subject line	Open rate where opens are still useful.	Audience, send time, body, CTA.
CTA copy	Click rate.	Subject, audience, send time, layout.
Send time	Click or conversion by window.	Subject, body, audience.
Journey branch	Conversion or retention.	Entry criteria and attribution window.

section 02

Sample size limits

Small lists produce noisy tests. A test can still be useful, but it should be treated as directional unless the sample is large enough and the effect is clear.

ok Pick the success metric before sending.
ok Avoid testing more than one variable in the same split.
ok Do not stop a test early because one variant leads after a few clicks.
ok Use holdouts for journey-level impact, not only copy changes.

section 03

Holdouts vs split tests

A split test compares two versions of an email. A holdout compares sending the journey against not sending it, or sending a lighter version. Holdouts answer whether the email should exist.

test type	question answered	good fit
Subject split	Which wrapper earns more attention?	Broadcast and launch emails.
Content split	Which message drives more action?	Activation and upgrade prompts.
Journey holdout	Does this automation change behavior?	Lifecycle programs and dunning.
Frequency holdout	Can fewer sends achieve the same result?	Retention and reactivation.

section 04

Promotion rules

A winner should become the default only when the result matches the goal and does not create a secondary problem. More opens with fewer conversions is not a win for most lifecycle email.

ok Check the primary metric first.
ok Review complaint, unsubscribe, and reply quality before promoting.
ok Keep a record of variant, audience, date, and decision.
ok Retest when the audience or offer changes materially.

reading this as growth and lifecycle teams

Growth work lives or dies on targeting and measurement. What matters is whether behavioral triggers can be built without engineering time, whether the event stream supports attribution past the open, and whether deliverability holds when broadcast volume shares a reputation with product email.

Applied to a/b testing email, that means weighing scheduling, event stream, deliverability, and operating track record ahead of the rest, against behavior-triggered journeys, broadcasts, and winback campaigns measured on revenue.

pillar

A/B Testing Email

What to test first

Sample size limits

Holdouts vs split tests

Promotion rules

reading this as growth and lifecycle teams

related pages

Email marketing for startups

Startup email checklist

Startup Email Examples: From Waitlist to Winback