Calibrating MMM with Incrementality Testing

A model of the weather can fit last year's temperatures beautifully and still be wrong about tomorrow, because fitting the past and being right about causation are different achievements entirely. Marketing mix modelling has the same gap built into it, and Google's own documentation for its MMM tool doesn't dress it up: a randomised experiment, not a well-fitting model, is "considered the ideal way to estimate causal effects" (About MMM as a causal-inference methodology, Meridian). A model can explain the past very well and still be wrong about why, which is exactly the gap a real experiment is designed to close.

Calibration is the fix: taking the result of a real, randomised incrementality experiment, a test that actually varies who gets exposed to an ad, rather than just observing who happened to be exposed, and using it to correct or tighten the model's own estimate of a channel's effectiveness. It's the step that turns a plausible-looking model into one with an external, causal anchor.

The experiments themselves take a handful of standard forms

The most common is called a geo holdout test: spend is switched off or reduced in a set of matched regions while it continues as normal elsewhere, and the difference in outcomes between the two is the incremental effect, isolated from everything else moving at the same time. It's the closest thing MMM has to a proper controlled experiment, because it's testing what actually happens when a real region stops seeing the ads, rather than inferring it from a pattern in the data.

On digital platforms, the equivalent is called a conversion lift study: a portion of the audience is deliberately withheld from a campaign and their conversion behaviour compared against the group that saw it, isolating the incremental effect of the exposure itself. A couple of refinements exist for the same purpose, mainly to make sure the comparison is fair: instead of showing the control group nothing at all, they're shown an unrelated message or a different advertiser's ad, so both groups genuinely saw something, and the only real difference between them is whether it was your ad.

An experiment sharpens the model, it doesn't replace it

Calibration isn't a matter of overwriting the model's answer with the experiment's answer. In a Bayesian tool like Meridian, an experiment's result gets folded into the model as strong evidence about that one channel, narrowing the range of answers the model considers plausible around whatever the experiment actually found. A more recent, larger-scale experiment carries more weight than an old one from a fraction of today's spend, which is just common sense applied consistently: a test run two years ago at a much smaller budget shouldn't be treated as equivalent to one run last quarter at full scale.

Where multiple experiments exist for the same channel, the sensible approach is to weigh them together rather than picking a single favourite, giving more recent and more relevant tests more say. The model still does the work of separating dozens of channels and estimating how each one behaves. Calibration just makes sure at least one of those estimates is anchored to something that actually happened, rather than something the model merely infers.

A real experiment is expensive, slow, and doesn't travel well

Calibration's honesty comes at a real cost. A proper geo experiment needs a genuine pre-test period to establish a baseline, then a test period long enough to detect a real effect, which in practice usually stretches to a couple of months from start to finish. It also causes direct, deliberate revenue loss in the regions where spend is suppressed for the sake of the test. That combination means a company can only realistically run a handful of these a year, and a result from six months ago may not hold at today's spend level or in today's season.

Some channels also can't be tested this way at all. A company can suppress its regional TV buy in a handful of markets; it cannot selectively delist itself from Google search results in only those same markets, which puts SEO, PR and a handful of other channels structurally out of reach of geo experimentation. And the practical floor for running a geo test at all, enough distinct regions and enough spend in each to produce a clean read, means the smaller and more regionally concentrated a business is, the harder it is to run a version of the very experiment that would make its model trustworthy. Calibration is the right standard to hold a model to; it isn't a standard every business can meet on its own, which is exactly why matching test design to a company's actual scale is its own piece of judgement, not an afterthought.

If you're relying on your MMM's own fit statistics as proof it's right, that's the exact gap calibration exists to close. Get in touch with Day1data to talk through whether an incrementality test would actually be worth running for your channels.

FAQ

What does "calibration" mean in marketing mix modelling? Using the result of a real, randomised incrementality experiment to correct or tighten an MMM's own estimate of a channel's effectiveness, rather than relying on the model's statistical fit alone.

Why isn't a well-fitting model enough on its own? Because fitting historical data well and correctly identifying cause and effect are different things. A model can fit the past closely while still misattributing why the past looked the way it did, particularly when channels move together or spend tracks seasonal demand (see seasonality).

What's a geo holdout test? An experiment that reduces or removes spend in a set of matched regions while it continues elsewhere, then compares outcomes between the two to isolate the incremental effect of that spend.

What's the difference between a PSA test and a ghost ad? Both withhold the real campaign from a control group; a PSA test shows that group an unrelated public-service message, while a ghost ad shows them a different advertiser's real ad, avoiding the mismatch of comparing "saw an ad" against "saw literally nothing."

How does an experiment actually change a Bayesian model like Meridian? It gets folded in as strong evidence about that one channel, narrowing the model's estimate for it around whatever the experiment actually found, with more recent, larger-scale experiments carrying more weight than old, small ones.

Is calibration expensive? Yes. Geo experiments require real data-science setup, cause deliberate short-term revenue loss in the suppressed regions, and typically run for several months between the pre-test and test windows.

Can every channel be calibrated this way? No. Channels like SEO and PR generally can't be regionally suppressed the way a media buy can, which puts them structurally out of reach of geo-experimentation.

Is calibration realistic for a smaller or regionally concentrated business? It's harder — clean geo tests need enough distinct regions and enough spend in each to produce a reliable read, which is a real constraint for smaller advertisers, and part of why matching the test design to the company's actual scale matters as much as running a test at all.