Incrementality Testing: The Only Way to Know If Your Ads Actually Work

I'm gonna say something controversial: most of your "successful" ad campaigns probably aren't doing what you think they're doing.

You're looking at a 5x ROAS in Facebook Ads Manager, feeling great about your targeting skills, and then you dig into the data and realize half those conversions would've happened anyway. Facebook's just claiming credit for people who were gonna buy from you regardless.

This isn't Facebook being evil (well, not entirely). It's just how attribution works. Platforms get credit for conversions they touched, even if they didn't cause them.

The only way to know for sure if your ads are generating incremental revenue—revenue you wouldn't have gotten otherwise—is incrementality testing.

Let me show you how it works and why it's the most important thing you're probably not doing.

What Is Incrementality (And Why Should You Care)

Incrementality answers one simple question: "Did this ad cause a conversion, or was the person going to convert anyway?"

Here's the classic example: You run a branded search campaign on Google for your own company name. Someone Googles "YourBrand," sees your ad, clicks it, and converts. Google says, "I generated that sale! 10x ROAS! You're welcome!"

But hold on. That person literally searched for your brand by name. They were already looking for you. Were they really not going to click your organic listing right below the ad?

You just paid $2 for a click you would've gotten for free.

This is non-incremental revenue. The ad took credit, but it didn't actually create the sale.

Now contrast that with a cold prospecting campaign targeting people who've never heard of you. Those people definitely wouldn't have converted without the ad. That revenue is 100% incremental.

Most campaigns fall somewhere in between. Your job is to figure out how much of your reported ROAS is incremental vs. just credit-stealing.

Why Attribution Can't Tell You This

Attribution tracks correlation: "This person saw an ad, then they converted."

Incrementality measures causation: "This person converted because they saw the ad."

You can't determine causation from correlation. The person might've converted because:

• They were already planning to buy
• They saw your organic post yesterday
• A friend recommended you
• They need your product seasonally and it's that time of year

Attribution gives them all to the last ad they clicked. Incrementality testing actually measures the causal impact.

The Gold Standard: Holdout Tests

The most reliable incrementality test is stupid simple:

Split your audience into two groups (test and control)

Show ads to the test group, show nothing to the control group

Measure the conversion rate difference between the groups

That difference is your incremental lift

If the test group converts at 5% and the control group converts at 3%, your ads are generating 2% incremental lift. That's real impact.

How to Run a Holdout Test

Let's say you're testing Facebook prospecting campaigns.

Step 1: Define your audience Start with your normal prospecting targeting (lookalikes, interests, whatever you're running). Step 2: Create a randomized split Use Facebook's "Audience Splitting" to randomly assign 90% to the test group and 10% to the control group. The control group sees no ads from you (Facebook calls this a "PSA" - Public Service Announcement - placebo). Step 3: Run the test for 2-4 weeks You need enough time to see normal purchase behavior. If your purchase cycle is 2 weeks, run the test for at least 4 weeks. Step 4: Measure everything Track conversions in both groups. Use your own database/analytics, not platform reporting (platforms can't see control group conversions obviously). Step 5: Calculate incremental lift

Here's the math:

Incremental Lift = (Test Conversion Rate - Control Conversion Rate) / Control Conversion Rate

Incremental Conversions = (Test Conversions) - (Test Group Size * Control Conversion Rate)

True ROAS = (Incremental Conversions * AOV) / Ad Spend

Real Example

I ran this test for an e-commerce client last quarter. Here's what we found:

So Facebook was claiming 4.5x ROAS, but the actual incremental ROAS was 2.25x. Still profitable, but way less magical than the dashboard suggested.

That 0.50% control group conversion rate? That's people converting from organic traffic, word of mouth, repeat purchases, etc. Facebook was claiming credit for all of those too.

Geo Holdout Tests (When You Can't Split Users)

Sometimes you can't split users into test/control. Maybe:

• You're testing TV or radio ads (can't target individuals)
• You're worried about contamination (control group members see ads anyway via shared devices/households)
• The platform doesn't support holdout groups

Solution: Geographic holdouts.

Pick similar markets, run ads in some and not others, compare results.

How to Run a Geo Test

Step 1: Select test and control markets Choose markets that are as similar as possible:

• Similar population size
• Similar demographics
• Similar historical conversion rates
• Similar seasonality

Example: Test in Austin, Dallas, San Antonio. Control in Fort Worth, El Paso.

Step 2: Run ads only in test markets Use geo-targeting to exclude control markets completely. Step 3: Monitor both groups Track conversions in test and control markets using your own data. Step 4: Run long enough to account for noise Geo tests need longer run times (6-8 weeks minimum) because market-level variance is higher than user-level variance. Step 5: Analyze the difference

Incremental Lift = (Test Market Conversion Rate - Control Market Conversion Rate) / Control Market Conversion Rate

Challenges with Geo Tests

Problem 1: Geographic spillover People in control markets might see your ads anyway (streaming services, social media feeds from friends in test markets, etc.). Problem 2: Market differences No two markets are identical. You might see differences due to local events, weather, competition, etc. that have nothing to do with your ads. Problem 3: Sample size You need meaningful population sizes to detect differences. Testing in small towns won't give you statistical confidence.

Despite these issues, geo tests are often your only option for measuring incrementality of broad-reach channels like TV, podcast ads, or display campaigns.

Platform-Based Incrementality Tools

The major ad platforms have built-in incrementality testing now:

Facebook Conversion Lift Facebook will randomly hold out a portion of your audience and measure the difference. It's basically an automated holdout test.

Pros: Easy to set up, Facebook handles the math
Cons: You're trusting Facebook to grade its own homework

Google Ads Experiments Similar concept—Google splits your audience and measures lift.

Pros: Works for search, display, and YouTube
Cons: Requires significant spend to reach statistical significance

TikTok Conversion Lift Same idea as Facebook.

I've used all three. They're convenient, but I always validate them against my own data when possible. The platforms have an incentive to show positive results, so take them with a grain of salt.

Pre/Post Testing (Budget Change Method)

Can't run a proper holdout test? Try this simpler approach:

Method: Dramatically change your ad spend and measure the revenue impact. Step 1: Baseline period Run your normal ad spend for 2-4 weeks. Record total revenue. Step 2: Intervention period Either increase spend by 50%+ or decrease spend by 50%+ for 2-4 weeks. Record revenue again. Step 3: Return to baseline Go back to normal spend for 2-4 weeks. Record revenue. Step 4: Compare Did revenue change proportionally to spend changes? That's your incremental impact.

Example

• Baseline: $10K/week ad spend, $50K/week revenue
• Increased spend: $15K/week ad spend, $58K/week revenue
• Return to baseline: $10K/week ad spend, $51K/week revenue

Analysis: A 50% spend increase drove a 16% revenue increase. That suggests your ads are generating incremental revenue, but with diminishing returns at higher spend.

Limitations

This method doesn't account for:

• Seasonality (revenue might've changed for other reasons)
• External factors (competitors, news events, economic changes)
• Lagged effects (ads take time to convert)

It's better than nothing, but way less reliable than a proper holdout test.

What to Test (And When)

You can't test everything all the time (that'd be expensive and chaotic). Here's what to prioritize:

High-Priority Tests

1. Branded search campaigns These are the most likely to be non-incremental. Test first. 2. Retargeting campaigns People who already visited your site might convert anyway. Measure the incremental lift. 3. Your highest-spend campaigns If you're spending $50K/month on Facebook prospecting, you need to know if it's actually working.

Medium-Priority Tests

4. New channels Before you scale TikTok or Pinterest or whatever, run an incrementality test to validate the platform's attribution claims. 5. Seasonal campaigns Is that Black Friday revenue from your ads or from people who were gonna buy anyway because it's Black Friday?

Lower-Priority Tests

6. Small campaigns If you're spending $1K/month on something, the opportunity cost of testing might exceed the value of the insights. 7. Brand awareness campaigns These are harder to test because the impact is long-term and indirect. Still worth doing eventually, but not urgent.

How AdsMAA Helps with Incrementality

Running incrementality tests manually is a pain. You need to:

• Set up holdout groups
• Track conversions in test and control
• Wait weeks for results
• Do statistical analysis to know if results are significant

AdsMAA automates most of this. We help you:

• Design incrementality tests across platforms
• Monitor test and control groups automatically
• Calculate lift and confidence intervals
• Recommend optimal spend levels based on incremental ROAS

The platform connects to your ad accounts and your conversion data, so you get a unified view of incremental performance across all channels.

Sign up for AdsMAA if you're tired of trusting platform dashboards that overstate results.

Interpreting Incrementality Results

So you ran a test. Now what do the numbers mean?

High Incrementality (70%+ of conversions are incremental)

This is great. Your ads are doing real work. Scale them.

Example: Cold prospecting campaigns, new product launches, awareness campaigns in new markets.

Medium Incrementality (30-70% incremental)

Pretty normal for most campaigns. Some conversions would've happened anyway, but you're also generating new demand.

Example: Lookalike audiences, interest targeting, competitor keyword campaigns.

Low Incrementality (0-30% incremental)

Your ads are mostly claiming credit for conversions that would've happened anyway. Either optimize or cut spend.

Example: Branded search, retargeting site visitors who abandoned cart, email subscriber retargeting.

Negative Incrementality (control group converts more than test group)

This is rare but possible. It means your ads are actively hurting conversions. Maybe they're annoying, mistargeted, or cannibalizing organic traffic.

Action: Turn off the campaign immediately and investigate.

Common Mistakes in Incrementality Testing

I've seen people screw this up in predictable ways:

Mistake 1: Testing for too short a time If your purchase cycle is 2 weeks but you only test for 1 week, you won't capture full impact. Run tests for at least 2x your average purchase cycle. Mistake 2: Not accounting for network effects If control group members see your ads through friends' shares or word of mouth, your test is contaminated. Geo tests help avoid this. Mistake 3: Changing too many things during the test Don't launch new campaigns, change pricing, or run big promotions during an incrementality test. You won't know what caused results to change. Mistake 4: Trusting platform tools blindly Platform-provided lift studies are useful but potentially biased. Validate them with your own data when possible. Mistake 5: Testing everything at once You can't run holdout tests on all campaigns simultaneously (you'd have no ads running). Prioritize high-spend campaigns and test sequentially.

FAQ

Q: How much does incrementality testing cost? You lose some conversions from the control group during the test, so there's an opportunity cost. For a 90/10 split over 4 weeks, you're giving up roughly 10% of conversions for 1 month. If that's $5K in lost revenue, that's the cost of learning the truth. Q: Can I test multiple campaigns at once? Yes, but carefully. Make sure test/control groups don't overlap, or you'll contaminate results. Better to test one major campaign at a time. Q: How do I know if results are statistically significant? Use a significance calculator. You generally want a p-value under 0.05 and a confidence interval that doesn't include zero. Most platforms calculate this for you. Q: What if incrementality is low but platform ROAS looks great? Cut spend or optimize. You're overpaying for conversions you'd get for free. Reallocate budget to higher-incrementality channels.

Wrapping Up

Here's the hard truth: attribution tells you what happened, but incrementality tells you what you caused.

If you're spending $10K+/month on ads, you owe it to yourself to know which campaigns are generating real incremental revenue vs. just stealing credit from organic.

Run a holdout test on your biggest campaigns. You might discover you're crushing it and should scale way up. Or you might discover you're wasting money on campaigns that aren't actually working.

Either way, you'll finally know the truth.

And honestly, that's worth the effort.

Start with your branded search campaigns (easiest to test) and your highest-spend prospecting campaigns (biggest impact). Run a 4-week test, calculate incremental lift, and make decisions based on reality instead of platform dashboards.

Your CFO will thank you. Your investors will thank you. And you'll sleep better knowing your ad dollars are actually doing something.

Now go test some campaigns.