The one-sentence version#
A/B testing sends two or more variants of a message to random samples and measures which performs better on a chosen metric before sending the winner to the rest.
Why you should care#
Opinions about subject lines and layouts are unreliable; tests turn them into evidence and compound small gains over time.
The mental model#
Think of email A/B testing as a contract between you and the mailbox providers receiving your mail. You publish or configure something they can check; they check it on every message; the result feeds their decision about where your message lands. Everything below is about making that check pass consistently.
Your first setup, step by step#
- Choose one variable per test and one primary metric (clicks or revenue, not opens).
- Size samples so results are statistically meaningful; small lists need bigger splits.
- Run long enough to capture late openers, typically 4 to 24 hours.
- Log results so learnings persist beyond one campaign.
Words you will see#
- Mailbox provider: Gmail, Microsoft, Yahoo, Apple, and the corporate gateways that decide where mail lands.
- Authentication: proof that a message is from who it says it is (SPF, DKIM, DMARC).
- Reputation: the provider's running score of your domain and IP.
- Placement: whether a message reaches the inbox, spam, or is rejected.
Common mistakes#
- Testing several changes at once.
- Declaring winners on open rate after Mail Privacy Protection.
Frequently asked questions#
What should I A/B test first in email?
Subject lines, then the primary call to action, then send time. These have the largest measurable effect.