A better fit isn't always a better model. Don’t chase R-squared.
3 diagnostics which pointed the wrong way, and 1 which didn't

Most marketing mix model reviews start with the fit statistics. R-squared, MAPE, a residual plot. If the numbers look good, the model is trusted. If they improve after a change, the change is kept.
We see the same pattern repeatedly in the Meridian models we build and review. A change improves a standard diagnostic, and the model gets worse at its real job.
The rest of this post walks through a recent piece of work on a client's Meridian model, anonymised for this blog. The figures have been changed to protect the client's confidentiality, but the direction and rough size of each effect are as we found them. Three diagnostics pointed us the wrong way, and one pointed the right way.
How the work unfolded
The work started with a review of the client's production model, where the main concern was that the control variables were not doing enough. Over two days we ran about a dozen variants. Each was saved as a separate, unapproved run, so the signed-off model stayed untouched while we experimented.
We worked through the changes in order, and each variant we kept became the starting point for the next. First, we re-ran the baseline to measure sampling noise. Then we removed an anomaly flag, fixed the timing of the holiday controls, tested a more flexible baseline, and finally ran a holdout test on the candidate model.
First step, measure the noise
Meridian is a Bayesian model fitted by sampling, so two runs of the same specification never give identical answers. Before comparing any variants, we re-ran the unchanged baseline with a new seed.
The two runs agreed closely, so differences between variants could be read as real effects of the change, not sampling luck. It is a cheap step and it is often skipped, but without it you cannot tell a finding from a wobble.
1. R-squared rewarded a flag that memorised the peaks
The model contained an anomaly flag that checked for any week more than three standard deviations above a region's average. These were marked as a suspected data-quality spike and given their own indicator variable. The intent was to stop a handful of contaminated weeks from distorting the media estimates.
The problem was that the biggest genuine weeks of the year also sat more than three standard deviations above average, so the flag caught the main holiday peak.
That had two costs. Firstly, the holiday effect was absorbed by an anomaly flag instead of a holiday control. Secondly, the estimates for any channel that spent heavily around those weeks were distorted, because the model no longer needed to explain the peak.
Removing the flag lowered R-squared from 0.82 to 0.80. Judged on fit alone, we would have kept it. But the extra fit was never real, because the flagged weeks were being matched by construction. Any variable built from the KPI itself will do this as it is the outcome leaking into the inputs.
2. The fit improvement we did trust
With the flag gone, the holiday weeks were visibly under-fitted. The model had holiday dummies, and they fired on the week of the holiday itself. This is the default approach, and for many businesses it is wrong.
Customers often act before the occasion, not during it, e.g. gifts are bought, money is sent and travel is booked ahead. For this client, volume ran 40% above normal in the week before the holiday, and the holiday week itself was flat.
Our first attempt was a single two-week holiday window. It over-predicted the holiday week, because one coefficient had to cover a surge week and a flat week. Splitting the holiday into two terms worked better so we landed on one for the pre-holiday effect and one for the holiday week itself.
The split halved the residuals in the holiday weeks and lifted R-squared to 0.85, without the anomaly flag. A re-run with a new seed reproduced it. We also tested dummies for a few minor calendar events, but they made no difference, so we left them out.
This was a fit improvement too, so why did we trust it and not the anomaly flag? Four reasons:
The mechanism came first. People send, buy and book ahead of an occasion. We could state that before running anything.
It was visible in the raw data. The surge week was there to see without any model.
It fixed the residuals it was meant to fix. The gain came from the holiday weeks, not from somewhere unexplained.
It survived a re-run. The result held on a new seed, so we could be confident it was not sampling noise.
3. The cleanest residuals came from hiding the problem
Even with the holiday fixed, the model's residuals were autocorrelated so errors in one week predicted errors in the next. The Durbin-Watson statistic was 1.2, where 2 means no autocorrelation.
Meridian models the baseline with a time-varying intercept, controlled by a number of knots. More knots give a more flexible baseline, so we tested doubling them from 12 to 24. Durbin-Watson rose to 1.6, the best of any variant.
However, the change also moved about four percentage points of the outcome from media to baseline, and the always-on channels lost the most. Nothing about the marketing had changed. The baseline had simply been given enough freedom to follow anything slow-moving in the data, including the effect of sustained media spend.
Autocorrelated residuals usually mean a driver is missing from the model. Extra knots do not find that driver, instead they give the baseline room to stand in for it and take credit from media to do so. The diagnostic improved because the symptom was treated, and the cost showed up in the attribution, which no fit statistic reports. We did not adopt the change.
4. The holdout showed what in-sample fit could not
By this point the candidate model had an in-sample R-squared of 0.85 and sensible holiday behaviour. The final step was to hold out the most recent eight weeks and ask it to forecast them.
One detail mattered here. We pinned the knots inside the training period, so the baseline could not bend to fit weeks the model was not supposed to have seen. Without that, a holdout test in a model with a flexible baseline is much easier than it looks.
The model under-forecast the holdout period by about 10%, with every week's error in the same direction. That pointed to something structural rather than noise. In other words, something had been lifting the KPI in recent months that the model did not know about.
That agreed with the autocorrelated residuals from the knots test. Our leading suspect is promotional and incentive activity that never made it into the data and no in-sample statistic would have told us this.
The split-holiday model without the anomaly flag became the candidate for sign-off, the extra knots were rejected, and the holdout gave us a clear next step, which is to bring the client's promotional data into the model.
What we check before keeping a change
Fit statistics answer one question: how closely does the model reproduce the history it was given? An MMM exists to answer a different one: what did each channel cause? A change can improve the first while damaging the second.
These are the checks we now run on every proposed change:
Measure the noise first. Re-run the baseline with a new seed, so you know how big a difference must be before it means anything.
Write down the mechanism before the run. A change with a stated reason can be proved wrong whereas a change chosen because it simply improves R-squared cannot.
Ask where each variable comes from. Flags, outlier indicators and anything else derived from the KPI will improve fit by construction.
Look at where the credit moved. Compare channel contributions and the baseline before and after. A fit gain that quietly shifts attribution needs a causal explanation.
Hold out recent weeks. Fix the baseline's flexibility inside the training period, then check both the size and the direction of the errors.
Keep experiments away from production. We run variants through an override on the fit, saved as separate unapproved runs, so the signed-off model is never touched until a variant has earned it.
The takeaway
None of this makes R-squared, Durbin-Watson or MAPE useless. They are good at telling you something is wrong, but they are poor at telling you whether a fix is right. Technically a model can score better on all of them while becoming worse at its actual job.
If your MMM review is a single slide of fit statistics, ask two more questions. What moved when the model last changed? And how does it perform on weeks it has never seen?
Blog post written by George Gascoine, Marketing Scientist

