What A/B tests are and when to use them
An A/B test sends different versions of a bidder's proposals to real jobs, then keeps the version that wins more views or replies.
An A/B test, or experiment, lets real clients decide which version of your proposals works best. Instead of guessing whether a short or long cover letter gets more replies, you run both and let the numbers pick.
A/B tests are in beta. They work, and we're still improving them.
How it works
- You pick one bidder. Its job search stays the same for the whole test.
- You set up two to five variants. Each one is a full set of bidder settings: cover letter, algorithm, answers, rates, boosting and profile. One of them is the control, your baseline.
- Your bird sends with each variant in turn as it applies to jobs, so they all get a fair share.
- When the test has sent the number of proposals you chose, EarlyBird waits 72 hours for clients to view and reply.
- The variant with the best Reply rate or View rate, whichever you chose, wins. EarlyBird can switch your bidder to it automatically.
When to use one
Run a test when:
- You're not sure which of two approaches is better, such as Template vs Enhanced AI.
- A bidder gets views but few replies, and you want to try a new cover letter.
- You want to try a different rate without betting the whole bidder on it.
- Your bidder sends a steady stream of proposals. A bidder that sends a few a week takes a long time to finish a test.
Skip it when you're making a change you're sure about, or when the bidder is brand new. Run it as it is for a while first, so the control has a fair baseline.
Where to find A/B tests
- On a bidder: the A/B Tests tab shows that bidder's current and past experiments.
- Across all bidders: Analytics → Experiments lists every experiment you've run.
- On the Bidders page, a bidder running a test shows an A/B tag.
Limits
- One experiment runs on a bidder at a time.
- 2 to 5 variants per experiment, labelled A to E.
- A test size of at least 10 proposals in total.