How to test a new AI video against your best-performing ad (2026)

Should the control be the live ad or a fresh copy of it?

Use a copy where the tool allows it. Meta's creative test makes 2 to 7 copies of an existing ad, and you then change the creative on each; leave one copy untouched and it becomes a control that starts on the same terms as the AI videos. Copies start equal.

The live ad carries its own delivery history into any comparison. Whether that history helps or hurts a newcomer has no published platform answer, so design the test so it does not matter. In an A/B test, duplicating the ad set and changing only the creative does that job, and Meta recommends the same budget for both versions "to ensure a fair comparison."

Keep everything else identical. Meta's best practices page says results are more conclusive when the ad sets "are identical except for the variable that you're testing", which for this test means the same audience, placements, offer and landing page.

How many AI video challengers can one test carry?

Fewer than the tool allows. Every extra test ad splits the same budget, so each one collects purchases more slowly, and Kohavi's paper puts a number on the effect of uneven splits: an experiment run at 99%/1% "will have to run about 25 times longer than if it ran at 50%/50%."

Say your campaign spends EUR 1,000 a day and your CPA is EUR 40. At Meta's suggested cap, test ads share about EUR 200 a day. Split across 4 test ads, each gets EUR 50 a day, about 1.25 purchases, or roughly 9 in a 7-day test. With 2 test ads, each collects about 17.

If those counts look thin for a CPA call, test fewer and more different videos, or judge on CTR. TikTok's Split Test guidance asks for test groups whose settings "differ significantly", and a new opening scene or a new reason to buy is a bigger difference than a new colour grade. Our creative testing matrix shows how to tell a new concept from a variant before you spend a cell on it.

Volume still matters across weeks. The IndianBites case study shows 142 AI ads in 11 weeks, with 18 hero concepts each tested in 6 to 10 variant cuts and testing every week.

A platform winner is the better ad inside that platform's delivery

Braun and Schwartz, in a Journal of Marketing paper on divergent delivery, show that ad platforms "deliver different ads to distinct and undetectably optimized mixes of users that vary across ads, even during the test." Their conclusion is that the comparison confounds the effect of the ad content with the effect of algorithmic targeting.

For a buying decision, that is acceptable. You want the ad that sells more in your account, with the delivery you will actually get. What the result cannot do is explain itself: if the AI video won, the cause could be its opening, its offer line or the people the system chose to show it to.

So read a win narrowly. It holds for that platform, that campaign and that period. Before you claim the AI video wins on TikTok as well, run the TikTok test.

Write the hypothesis before the first upload

Meta's best practices ask for a measurable hypothesis and give an example: "My cost per result will be lower when I optimize for landing page views." The same shape works for a creative test. Fill in every bracket:

• Control: [ad name], running since [date], CPA of [amount] over the last 28 days.
• Challenger: [AI video name], which differs from the control in [one thing: the opening scene, the reason to buy or the format].
• Metric and setting: [CPA or CTR] against the control, on [7-day click or your account's standard setting].
• Test: [A/B test, creative test, Split Test or video experiment], [budget] per cell, [7, 14 or 21] days.
• Decision: if the challenger wins, [action]; if it loses, [action]; if the result is inconclusive, [action].

Short is fine. A hypothesis your media buyer can read in ten seconds is one the team will still agree on when the numbers come in.

Scale, rebrief or retest: what to do with the readout

A win needs a manual step. After a Meta creative test, the test ads keep running in the campaign on the original budget, and the test makes no automatic changes, so pause the losers and move budget yourself.

A loss is a brief. Write down what the control has that the challenger lacked, whether that is a faster first second, a clearer product shot or a stronger offer, and make it the first line of the next brief.

An inconclusive result is common. On Google, anything below 95% is directional by Google's own definition. On Meta's creative test there is no confidence level at all, so a narrow lead is a tie; rerun it as an A/B test, or with fewer cells and a bigger creative difference.

If you want an outside team to supply the challengers, the Control Test runs this exact shape: you name your current best ad, 10 variants in 9:16 are built to beat it, they run for 14 days in your own ad account, and one metric agreed up front, CTR or CPA against the control, decides it. Fourteen days will not give every account enough purchases for a confident CPA read, which is why CTR is the other option.

Today, open Ads Manager, sort the last 28 days by cost per purchase, pick the ad you would least like to lose and fill in the hypothesis above for one AI challenger. It takes about 20 minutes.