Go to app

A/B Testing FAQ

Quick answers about test size, how proposals are split, how long tests take, editing, cancelling and what happens to your bidder afterwards.

Written by the EarlyBird teamBETA

Last verified Nov 4, 2026

Is test size per variant or in total?

In total. A test size of 60 with 3 variants sends about 20 proposals per variant. The minimum is 10, and we suggest at least 10 per variant. See Setting up an A/B test.

How are proposals split between variants?

In turn. Your bird uses Variant A for one proposal, Variant B for the next, and so on, then starts again at A. If a job is skipped, the next job gets the same variant.

Can I change the job search during a test?

The search belongs to the bidder, so all variants share it. Changing it changes the jobs every variant sees, which keeps the test fair but makes before and after hard to compare. It's best to leave the search alone until the test ends.

How long does a test take?

It depends on how fast the bidder sends proposals. Once it has sent the test size, results take up to 72 hours to finalize, and you'll get an email when they're ready. See Experiment statuses and editing rules.

Can I run tests on several bidders at once?

Yes. Each bidder runs one experiment at a time, and different bidders can each run their own. You can have up to 5 bidders turned on.

Can I add a variant after the test has started?

No. Once a test is running you can change variant settings, but not add, copy or remove variants. Cancel it and start a new one instead. See Experiment statuses and editing rules.

Can I pause a test?

No. You can turn the bidder off, which stops it from sending proposals, so the test waits. Or you can cancel the test.

What happens to my bidder after a test?

When the test size is reached, your bidder goes back to its regular settings. When the test completes, the winner's settings are applied if you left auto-apply on. See Reading results and picking a winner.

A test winner replaced my cover letter. How do I get mine back?

Your old settings are kept in the test's control, Variant A. On the bidder's A/B Tests tab, click the test under Past experiments, then click Variant A to preview its cover letter and other settings. Copy what you want into the bidder's Settings tab and save. See Reading results and picking a winner.

What happens if I cancel a test?

It stops straight away, keeps its numbers and moves to Past experiments as Cancelled. Your bidder goes back to its regular settings, and no winner is applied.

Does an A/B test cost extra connects?

No. Each proposal costs the same connects it would without the test. Only a variant with different boosting settings changes what you spend. See What connects are and how EarlyBird spends them.

Can I see which proposals came from a test?

Yes. In Analytics → Dashboard, use the Experiment filter above the Proposals table. Each proposal's details show its Experiment and Variant. See The proposals table and proposal details.