Go to app

What A/B tests are and when to use them

An A/B test sends different versions of a bidder's proposals to real jobs, then keeps the version that wins more views or replies.

Written by the EarlyBird teamBETA

Last verified Nov 4, 2026

Watch the video walkthrough (opens in a new tab)

An A/B test, or experiment, lets real clients decide which version of your proposals works best. Instead of guessing whether a short or long cover letter gets more replies, you run both and let the numbers pick.

A/B tests are in beta. They work, and we're still improving them.

How it works

  1. You pick one bidder. Its job search stays the same for the whole test.
  2. You set up two to five variants. Each one is a full set of bidder settings: cover letter, algorithm, answers, rates, boosting and profile. One of them is the control, your baseline.
  3. Your bird sends with each variant in turn as it applies to jobs, so they all get a fair share.
  4. When the test has sent the number of proposals you chose, EarlyBird waits 72 hours for clients to view and reply.
  5. The variant with the best Reply rate or View rate, whichever you chose, wins. EarlyBird can switch your bidder to it automatically.

When to use one

Run a test when:

  • You're not sure which of two approaches is better, such as Template vs Enhanced AI.
  • A bidder gets views but few replies, and you want to try a new cover letter.
  • You want to try a different rate without betting the whole bidder on it.
  • Your bidder sends a steady stream of proposals. A bidder that sends a few a week takes a long time to finish a test.

Skip it when you're making a change you're sure about, or when the bidder is brand new. Run it as it is for a while first, so the control has a fair baseline.

Where to find A/B tests

  • On a bidder: the A/B Tests tab shows that bidder's current and past experiments.
  • Across all bidders: Analytics → Experiments lists every experiment you've run.
  • On the Bidders page, a bidder running a test shows an A/B tag.

Limits

  • One experiment runs on a bidder at a time.
  • 2 to 5 variants per experiment, labelled A to E.
  • A test size of at least 10 proposals in total.

Next steps