Go to app

Setting up an A/B test

Create an experiment on a bidder, choose the winning metric and test size, build your variants and start the test.

Written by the EarlyBird teamBETA

Last verified Nov 4, 2026

Watch the video walkthrough (opens in a new tab)

An A/B test runs inside one bidder. The bidder keeps its job search, and you set up two to five versions of how it applies. Setting one up takes about seven minutes.

Before you start: decide the one thing you want to test. See Test ideas: what to test first.

Step 1: Open the experiment editor

On a bidder you already have:

  1. In the sidebar, click Bidders and open the bidder.
  2. Click the Settings tab.
  3. Click Create A/B test, next to Bidder name.

On a new bidder, click Create A/B test on the Settings step, after you save the search.

You can also start from the bidder's A/B Tests tab: click Create your first experiment, then Create A/B test.

Note: A bidder runs one experiment at a time. While one is running, the button reads View experiment instead.

Step 2: Fill in the experiment settings

  1. Experiment name: a name you'll recognize later, such as "Long cover letter vs Short cover letter".
  2. Winning metric: Reply rate or View rate. This decides which variant wins. Reply rate is the default. See Reading results and picking a winner.
  3. Test size (proposals total): how many proposals the whole test sends, across all variants. It starts at 60 and must be at least 10. Aim for at least 10 per variant.
  4. Automatically apply winning variant settings when experiment completes: on by default. Leave it on to have your bidder switch to the winner by itself.

Step 3: Build your variants

The editor starts with two variants:

  • Variant A has your bidder's current settings. It's the control, "the baseline variant that other variants are compared against."
  • Variant B starts from the default bidder settings.

Click a variant to open it. Each one has a Variant name and the full set of bidder settings: algorithm, cover letter, answers, rates, boosting, advanced settings and profile.

To make a clean test:

  1. On Variant A, click the copy icon. An exact copy named "Variant A (copy)" appears below it.
  2. In the copy, change only the setting you're testing, and rename it.
  3. On Variant B, click the bin icon, then Remove, if you don't need it.

You can have up to 5 variants, labelled A to E. Click + Add variant to add one with the default settings.

Important: An experiment needs at least 2 variants. If you remove a variant when only two are left, you'll see "Removing this variant will cancel the A/B test and return to normal bidder mode." Click Remove only if you want to drop the test.

Step 4: Start the test

  1. Check every variant one last time.
  2. Click Save & start experiment.
  3. You'll see "Experiment created and started successfully!"

Variant A's settings are saved as the bidder's settings, and the bidder shows an A/B tag on the Bidders page. If a required field is empty, you'll see "Fill out the required fields" and a message under the field, such as "Experiment name is required" or "Test size must be at least 10".

Important: The test only moves forward while the bidder is turned on and sending proposals. Make sure it's on and inside its working hours.

Next steps

Follow the test on the bidder's A/B Tests tab or in Analytics → Experiments. See Reading results and picking a winner.