Running a Holdout Group Email Experiment
A holdout group email experiment reserves a slice of your audience that receives nothing — or a baseline message — so you can measure the incremental lift of a send rather than just comparing two variants. Here's how to think about holdouts alongside subject line testing, and how to wire the whole loop into an AI agent with marginal.
What a holdout group actually measures
A subject line A/B test answers a narrow question: of the people who got a send, which variant performed better? A holdout group answers a broader one: did the send move the needle at all versus sending nothing? You keep a randomized control segment out of the campaign and compare downstream behavior against the treated group.
The two methods are complementary. Use A/B tests to optimize the message; use holdouts to validate that the campaign is worth sending in the first place and to catch fatigue when extra emails stop producing incremental engagement.
- Within-test lift: which subject line wins among recipients
- Holdout lift: incremental effect of the campaign vs. no send
- Randomization is what makes both comparisons valid
- Hold the control segment stable across a series to track long-run effects
Designing the experiment
Start by carving out a holdout — commonly 5–10% of the eligible audience — before you assign anyone to subject line variants. The remaining treated population is where you launch your subject line A/B test. Keep the holdout selection random and documented so results are reproducible week over week.
- Define the eligible audience and the holdout percentage up front
- Randomize the holdout, then split the treated group into variants
- Pick a primary metric — opens or clicks — before you launch
- Decide a measurement window so late activity is counted consistently
Where marginal fits in
marginal is a hosted email MCP server that lets an AI agent run the optimization half of this workflow end to end. Through its MCP tools an agent can draft variants with generate_variants, launch a subject line A/B test with launch_test, pull open and click tracking with get_results, and ask recommend_next for the suggested follow-up. The within-test lift metrics tell you which subject line earned the engagement.
Combine that with a holdout you maintain in your own audience logic: send the treated segment through marginal, leave the control untouched, then compare. AI agent email marketing becomes a tight loop — generate, launch, read results, iterate — while your holdout keeps the team honest about real incremental value.
- generate_variants — draft subject line options for the treated group
- launch_test — start the A/B test and manage the send
- get_results — open/click tracking and within-test lift
- recommend_next — agent guidance on the next variant or send
- Free tier covers 100 experiments/month to iterate on holdout designs
Connecting your agent
marginal runs at https://marginal.sh/mcp and authenticates with a Bearer API key. Any MCP-capable client — Cursor, Claude Desktop, Claude Code, Windsurf, Cline, Continue, or Zed — can call the tools above, so the same holdout group email experiment workflow runs wherever your agent lives. See the docs at https://marginal.sh/docs/ for tool parameters and result schemas.
- Endpoint: https://marginal.sh/mcp
- Auth: Bearer API key (marg_live_...)
- Hosted — nothing to self-host or maintain
- Works across common MCP clients used by growth engineers
Get started with marginal
marginal is a hosted email marketing MCP server at marginal.sh. Sign up free, create an API key, and connect https://marginal.sh/mcp from Cursor, Codex, or Claude Desktop.
- 100 experiments/month on the free tier
- Four MCP tools: generate_variants, launch_test, get_results, recommend_next
- Listed in the MCP Registry as sh.marginal/mcp