Reading results and picking a winner
How to read an experiment's lift and variant numbers, how EarlyBird picks the winner, and what happens when it's applied.
Each experiment has a results card that compares your variants side by side. Here's how to read it and how the winner is chosen.
Open the results
- For one bidder: open the bidder and click the A/B Tests tab. The current test is at the top, and older ones are under Past experiments.
- For all bidders: click Analytics → Experiments, then click a row.
The card starts with a line such as "Long vs short letter started on … with 60 proposals test size."
Choose a winning metric
You pick the winning metric when you set up the test. It decides which variant wins.
- Reply rate: the share of proposals where the client messaged you. It's the closest to new work, so it's the default and the best choice for most tests.
- View rate: the share of proposals the client opened. Use it for changes clients see before they open your proposal, such as boosting or your first line. You'll also get an answer sooner, since views come before replies.
Read the lift
The two boxes at the top show View rate lift and Reply rate lift, each marked "Best variant vs control". Lift compares the best variant with your control:
- +25.0% means the best variant's rate is a quarter higher than the control's. For example, a 10% reply rate against the control's 8%.
- A negative number, such as -12.5%, means every variant did worse than the control.
- +∞ (5.0%) means the control has no views or replies yet, and the best variant has a 5.0% rate.
- – – means there's nothing to compare yet.
Read the variant table
| Column | What it shows |
|---|---|
| Variant | The letter and name, with a control tag on your baseline and a winner tag once the test is completed |
| Bids | Proposals this variant sent |
| Views | Proposals the client opened |
| Replies | Proposals where the client messaged you |
| View rate | Views divided by bids |
| Reply rate | Replies divided by bids |
Your bidder sends with each variant in turn, A, B, C and back to A, so they get about the same number of bids.
Click a variant, or ⋯ → Preview, to see the settings it sends with: the Bidder algorithm, Cover letter, Answers, Terms settings, Boosting, Advanced settings and Other settings. Use it to check that your variants differ only in what you're testing.
How the winner is picked
When the test completes, the variant with the highest rate on your winning metric wins. If a variant ties with the control, the control wins, because there's no reason to change.
The winner is chosen on the numbers alone, and bigger tests give more reliable results. Aim for at least 10 proposals per variant, and treat a small lift from a small test as a hint, not proof.
Auto-apply the winner
If Automatically apply winning variant settings when experiment completes was on, the winning variant's settings become your bidder's settings when the test completes. Your bidder stays on or off as it was.
If it was off, nothing changes. To use the winner, copy its settings into the bidder's Settings tab yourself. Preview the winning variant to see exactly what to set.
Next steps
Start your next test from the new settings, and change one thing again. See Test ideas: what to test first.