Advanced email A/B testing: edge cases, scale, and monitoring

This guide assumes email A/B testing is already deployed and passing. It covers what breaks at scale and how mature teams operate it.

Edge cases that break a working setup#

• Testing several changes at once.
• Declaring winners on open rate after Mail Privacy Protection.
• Mail forwarded through mailing lists or personal forwarders, which alters headers and content.
• Acquisitions and rebrands that introduce domains nobody audited.
• Vendors silently changing their sending infrastructure.

Operating it as infrastructure#

1. Assign an owner for each sending domain and each vendor relationship.
2. Put DNS records under version control or a change-review process.
3. Alert on authentication pass rate drops and reputation changes, not just outages.
4. Run a quarterly audit against the setup steps below.
5. Document runbooks for the three most common failures.

Reference: the baseline setup#

1. Choose one variable per test and one primary metric (clicks or revenue, not opens).
2. Size samples so results are statistically meaningful; small lists need bigger splits.
3. Run long enough to capture late openers, typically 4 to 24 hours.
4. Log results so learnings persist beyond one campaign.

Frequently asked questions#