What is Incrementality Testing? A Comprehensive Guide

‍Incrementality testing removes the guesswork that comes with attributing sales and conversions to ad performance. It’s often overlooked by marketers, but it’s a vital testing method that helps marketers understand ad efficiency to inform their budget allocation and creative decisions. 

‍

Imagine this: your payment processing app has just experienced an uptick in downloads and is selling like hot cakes on the app store. Good on you! or is it?

What led to the spike? Is it the recent paid ad campaign you launched over the last few weeks? Or does it just happen to be a holiday season and your target audience decides to download your app as a cheaper alternative? 

That’s what incrementality testing answers. It measures the true lift or additional value that your marketing efforts generate separately from what would have occurred organically. When you test a paid campaign for its incrementality, you can identify the volume of downloads and sales your ads are directly responsible for driving. 

In this guide, we’ll take a deeper look at how incrementality testing works, why it’s vital for mobile app marketing, and how to ensure quality control when conducting it. 

‍

What is Incrementality Testing?

Incrementality testing works with randomized controlled trials (RCTs), where you separate your audience into test and control groups. The test group sees your ads but the control group doesn’t. However, both share similar demographic and past behavioural traits. 

The test will run for a designated period, perhaps, for two weeks. Afterwards, you’ll compare the results. Conversions coming from the test group are the ones that drive actual growth since they represent real users from your audience. 

This approach differs from multi-touch attribution, which relies on estimates of influence across channels. Incrementality is essentially evidence-based. 

‍

Why is Incrementality Testing Vital for Mobile App Marketing?

Without incrementality testing, it’s challenging to determine whether your marketing campaigns are truly driving the value you believe they are. Traditional metrics like cost per install (CPI) and return on ad spend (ROAS) assume all app installs are equal. 

But they’re not. 

Organic baselines, especially for mature apps, can significantly skew attribution. And considering the fact that many apps waste big bucks on UA spend, non-incremental attribution may further contribute to wasted dollars if marketers continue dumping cash on ineffective ads. 

The Privacy Angle

Incrementality testing matters more than ever now because of privacy shifts. For example, Apple’s ATT framework restricts IDFA tracking, turning more than two-thirds of iOS users into “dark” installs. That can mean most of your audience can’t be accounted for with traditional attribution methods. 

Incrementality acts as a flashlight of sorts in these contexts, helping you spot the true efficiency of your ads. For example, your testing of a campaign yielding 500,000 installs might only deliver 100,000 incremental ones, crucial info that can help you prevent overbidding on the wrong channels. 

Ad Fraud

There’s also the threat of ad fraud. Click spamming, fake installs, ad stacking…the list goes on. All of these practices can throw traditional attribution models off, giving you a false sense of ad efficiency. Again, incrementality testing can help you differentiate conversions (and installs) derived from real users versus fraudulent ones. 

How Do You Measure Incrementality?

The formula for measuring incrementality is as follows:

For installs, a test group yielding a 7% install rate and control rate of 4%, lift is 75%. 

You can apply this formula to a variety of metrics including:

• Conversion rates
• Customer lifetime value (CLTV)
• Retention rates
• Revenue per visitor

You then scale it by applying the following formula:

So if you tested 1 million installs, your number of incremental installs at 75% lift would be 750,000 incremental installs. 

Some key metrics to also pay attention in relation to these formulas:

• Absolute incrementality: Raw extra conversions. 
• Relative lift: Percentage gain. 
• iROAS (Incremental ROAS): Revenue from incremental users divided by ad spend. 

Together, measuring these metrics can give you a comprehensive and holistic view of ad performance that traditional attribution can’t. 

‍

What Channels and Uses Cases are Best Measured for Incrementality? 

The channels that best support incrementality testing are those with scale and randomization feasibility. Also, platforms that have built-in tools to support incrementality as well as those with privacy-resilient capabilities are ideal for testing. 

Paid Social Channels

• Meta (Facebook/Instagram): Tools such as their Conversion Lift automates RCTs at the account level, handles cross-device tracking via pixels/CAPI, and calculates incrementality factor (IF) as incremental conversions over reported ones. 
• TikTok/Snapchat: TikTok provides holdouts for viral creatives, while Snapchat uses geo-experiments or chat ads for incremental reach. 

Mobile Ad Platforms

• DSPs: These platforms enable bid-level holdouts, integrating with MMPs for SkadNetwork/privacy-safe iROAS. They include the likes of Youappi (MobuppsX DSP) and Aarki, which allow you to conduct detailed bid-level incrementality experiments. 

There are also various tools that help you measure and test incrementality across these platforms. Many of these are mobile measurement platforms (MMPs), and they include the likes of:

• AppsFlyer: Offers automated RCT holdouts, a unified dashboard for UA/remarketing across Meta, Google, and DSPs. It also measures casual lift. 
• Adjust: Provides cohort analysis, automated workflows for A/B testing, and real-time multi-touch and incrementality attribution. 
• Branch: Offers A/B testing with deep links for incrementality proxies, and integrates SkadNetwork for privacy-safe lift on cross-device user acquisition campaigns. 

Choosing the right tool for incrementality testing depends on your business needs, scale, and the technical setup of the tool. You’ll also want to factor in the ease of access the tool provides, and its integration depth. 

What are the Best Practices for Incrementality Testing?

Incrementality is part-science, part-art, and the key behind successful testing is establishing the right parameters at all stages. That means:

• Designing and setting up your study correctly. 
• Ensuring proper execution and analyzing the right metrics. 
• Integrating the right tools and platforms and iterating the testing process. 

Let’s cover each step. 

Designing/Setup

• Form Your Hypothesis & Goals: Define precisely what you’re testing (i.e., ads, channels), and what a successful outcome looks like. 
• Create Randomized, Comparable Groups: Randomly separate your audiences into a treatment group (those who see your ads) and control group (those who won’t see your ads), to ensure they’re statistically similar and isolated. 
• Choose Sufficient Sample Size & Duration: Use an appropriate sample size and testing duration that allows you to capture full customer cycles and factor in natural variations. In terms of time, common testing periods are 2-4 weeks. 
• Control External Factors: Weigh in seasonal trends and campaigns, local events, and competitor ads that may skew your outcomes. 

Execution/Analysis

• Isolate Exposure: Restrict control groups from seeing ads through additional channels to prevent contamination. 
• Measure Key Metrics: Pay attention to business outcomes like sales, revenue, and iROAS instead of vanity metrics. Emmanuel Okocha, our VP of operations, also stresses an important point here for app companies: “No matter what they have to be persistent with the test.” He makes it clear not to interrupt the test because, “It undermines the data integrity since it breaks the consistency which is the most crucial to the Incrementality test”.
• Use Confidence & Risk: Hone in on expected value and confidence intervals to identify the range of possible outcomes, instead of just finding strict p-values. 
• Automate & Standardize: Create automated pipelines for calculations to eliminate human bias from the study. 

Integration/Iteration

• Combine Methods: Place incrementality tests alongside marketing mix modelling (MMM) and A/B testing for a more comprehensive view. 
• Calibrate & Adapt: Use the results of your experiment to tweak your attribution models and adjust your budget allocation to campaigns that are truly incremental. 
• Continue the Process: Make testing a regular and ongoing part of your marketing strategy to stay agile and adaptable to market changes. 

Workflows for incrementality tests can vary from business to business depending on what precisely you’re testing and the outcomes you’re measuring. Nevertheless, following these steps will ensure that the quality of your testing is high and that the measurements are accurate. 

‍

Incrementality Testing Reduces Waste

If you haven’t already incorporated incrementality testing into your workflow, it’s time to start thinking about it. As we mentioned earlier, user acquisition costs can easily strain your budget, especially if you’re in a high-cost vertical. Couple that with shifts in privacy law that make it harder to track users, and it becomes clear that old-school attribution models have glaring limitations. 

Incrementality testing tells you whether your ads are effective or not, giving you strategic control in terms of how you spend your money and manage your ad campaigns. So don’t let this powerful modality go to waste!

Are you interested in learning how incrementality testing can improve your strategic decision-making for ad campaigns? Contact us so we can show you how to incorporate it into your marketing strategy today!

‍

Frequently Asked Questions

What is an example of incrementality?

An example of an incrementality test is a marketer running analysis on a Meta or Apple Ad to determine whether the platform’s claim of a certain percentage of conversions can truly be attributed to that platform. 

What is the difference between A/B testing and incrementality testing?

Incrementality tests measure whether your ad campaigns are actually driving additional sales beyond what would occur organically. A/B tests compare which version of an ad or landing page performs better.

What is the difference between MMM and incrementality testing?

Marketing mix modelling (MMM) is a big-picture view that shows how various marketing channels contribute to outcomes over time. Incrementality, however, is about demonstrating causality between your ads and outcomes. 

‍