An email A/B test sends two versions of the same message to two randomly drawn halves of a list, then picks the winner on a single metric. Plan for at least 1,000 contacts and a 95% confidence threshold, with the test duration fixed before sending. One condition comes first: invalid addresses in your list distort the real sample size and every rate calculated from it.
What A/B testing means in email marketing
A/B testing (or split testing) compares a version A, the control, with a version B that differs by one element only. Each version goes to a separate group of recipients, built by random draw. Randomization makes the two groups comparable.
In email, the test usually runs on a sample of the list. The tool measures the results, then sends the winning version to the remaining contacts.
Multivariate testing crosses several elements: 2 subject lines and 2 buttons make 4 versions. At 1,094 recipients per version (calculation detailed below), you need 4,376 contacts to detect a 5-point gap on a 20% open rate.
Choosing the element to test and the metric that judges it
Each element has its own metric. Testing a subject line and judging it on click rate gives no reliable answer, since the subject line acts before the open.
| Element | Example variant | Metric to read | Limit |
|---|---|---|---|
| Subject line | Question vs. number | Open rate | Skewed by Apple Mail Privacy Protection |
| Preheader | With or without preview text | Open rate | Tied to the subject line: change only one of the two |
| Sender name | Brand name vs. person’s name | Open rate, then click rate | Not offered by every tool |
| Call to action | Button vs. text link | Click rate | One main button per version |
| Content | Short vs. long message | CTOR (clicks divided by opens) | Changes several elements at once: avoid |
| Send time | Tuesday 10 am vs. Tuesday 2 pm | Click rate | Same day for both sends, otherwise the day skews the test |
A subject line test on 1,000 recipients per version only detects a gap of at least 5 points: 25.2% opens against 20%. Below that, the result stays inside statistical noise. To find subject line variants to pit against each other, 15 examples of eye-catching email subject lines offer wording that has already proven itself.
Setting up a test in six steps
- Write a numeric hypothesis: “a subject line with the first name will lift the open rate from 20% to 22%”.
- Change one variable only. Two simultaneous changes make the result impossible to interpret, since you cannot tell which one did the work.
- Set the decision metric before sending, such as open rate or click rate. Choosing it afterwards biases the reading.
- Calculate the sample size per version (next section).
- Turn on random split. Never build the groups by hand, for example in alphabetical order.
- Launch the test and wait for the measurement window to close, then send the winning version to the rest of the list.
The metric from step 3 depends on the campaign goal, so the choice of KPIs for your email campaign comes before the test.
Calculating sample size and duration
The HubSpot France blog (February 24, 2026) treats about 1,000 contacts as the relevant threshold: below it, you may need to test 80% to 95% of the list to get a usable result (HubSpot France blog, 2026). That threshold is a minimum. The useful size depends on the minimum detectable effect, meaning the smallest difference you want to spot.
Here are per-version sizes, calculated with a two-sided test on two proportions, a 95% confidence level and 80% power:
- open rate from 20% to 25%: 1,094 recipients;
- open rate from 20% to 22%: 6,510 recipients;
- click rate from 3% to 4%: 5,301 recipients;
- click rate from 3% to 3.6%: 13,914 recipients.
The calculation also reads the other way. With 1,000 recipients per version, the smallest detectable gap starts from a 20% open rate and reaches 25.2%. On a 3% click rate, it reaches 5.5%. With 200 recipients per version, you need a 32.3% open rate against 20% to conclude. A list of 4,000 contacts therefore only detects a large gap: on a small base, test radical changes.
Tools that express the sample as a percentage of the list require a conversion: 10% of 4,000 contacts is 400 recipients, or 200 per version. Check that number before launching.
Duration follows the pace of reactions. Fix it before sending, based on your past campaigns. HubSpot cites an email that generates 70% of its clicks within the first 24 hours and 5% per day afterwards: limiting the test to 24 hours then makes sense.
Reading the results: significance and Apple Mail Privacy Protection
Statistical significance tells you whether the observed gap could come from chance. At 95% confidence, the p-value must stay below 0.05 (HubSpot, 2026). Next, check that both groups received a comparable volume: a smaller group widens the margin of error. Last, set the gap against the business result: a 2% rise in open rate can be significant without changing revenue (HubSpot).
Since September 20, 2021, Apple Mail Privacy Protection preloads an email’s images, tracking pixel included. The open is then recorded without the recipient having read anything. Litmus already measured 49.8% of opens on Apple clients at the end of August 2021 (Litmus, 2021).
A winner picked on opens should therefore be read with caution. Prefer click rate or conversion. To understand what distorts this metric on your own sends, see our guide to the email open rate.
Common mistakes
The costliest one is stopping the test as soon as B pulls ahead. Evan Miller quantified the bias on April 18, 2010: with a 50% conversion rate and a test stopped at the first result significant at 5%, before 150 observations, a test with no real effect shows a false winner in 26.1% of cases (Evan Miller, 2010). Checking the results 10 times during a test turns a displayed 1% threshold into a real 5% risk. With 3 looks, you must display 1.8% to keep 5%. Fix the sample size before sending, and call the result only at the end.
- Testing a subject line on 200 recipients per version and declaring a winner on a 2-open gap: at that volume, only a 32.3% open rate against 20% clears the threshold.
- Comparing two groups of a different nature: long-standing customers against new sign-ups.
- Reusing a result out of context: a subject line that won in B2C on a Tuesday morning will not carry over to a B2B base at the end of the week.
Checking your list before sending the test
The number that counts is the number of addresses that actually receive the message. Take a list of 5,000 contacts with 4% invalid addresses (an example value): 200 hard bounces, or 100 per version. Each group drops to 2,400 real recipients.
When bounces cluster on one side (140 in group A, 60 in group B), click rates are calculated on different bases. The measured gap then depends on the list itself.
Handling your bounce rate therefore starts before the test. Run a sample of your list through CaptainVerify before the next campaign: the number of invalid addresses gives you the real size of your sample. Clean the list and recalculate before launching the test.
