Measure causal impact and report analytics to decision makers

Not every change can be A/B tested. A national TV campaign, a price change in one region, a new store format or a policy change affects everyone at once, or one region at a time. Quasi-experimental methods estimate the causal effect by constructing a credible counterfactual: what would have happened without the change?

Interrupted time series

With a long series before and after an intervention, interrupted time series (ITS) analysis fits the pre-intervention trend and seasonality, and tests whether the level and the slope changed at the intervention date. The counterfactual is the continuation of the earlier pattern.

Before the intervention in July 2024, the outcome grew by 3.37 per period. At the intervention, the level jumped by 44.40 (95% CI [31.55, 57.24], p < .001), and the slope increased by 2.245 per period (p < .001). Compared with the counterfactual, the outcome was on average 65.72 higher per period after the intervention, a cumulative gain of 1,183 (9.8%; 95% CI [908, 1,458]). Newey-West standard errors account for autocorrelation, and seasonal terms remove the seasonal pattern.

The weakness of ITS: the estimate is causal only if nothing else changed at the same time. A competitor leaving the market in July 2024 would be attributed to your campaign. Adding a control series that was not exposed to the intervention strengthens the design, which leads to the next two methods.

Synthetic control

When one unit (a region, a store, a country) receives the intervention and several others do not, the synthetic control method (Abadie et al., 2010) builds a weighted combination of untreated units that closely tracks the treated unit before the intervention. After the intervention, the gap between the treated unit and its synthetic twin is the effect.

In the example, the North region launched a campaign in 2018. The synthetic North is built from East (weight 0.33), Lakes (0.24), Valley (0.18), Coast (0.14) and South (0.11), and tracks North closely before 2018 (RMSPE 1.17, 0.9% of the mean). After 2018, North is on average 11.36 above its synthetic control (7.5%). The placebo test repeats the analysis pretending each untreated region had been treated: North’s post/pre error ratio ranks first of 16 regions, giving a placebo p-value of .063 (1 in 16). The effect stands out, although with only 16 units the smallest possible p-value is .063.

Difference-in-differences

With a treated group and a control group observed before and after, difference-in-differences (DiD) compares the change in the treated group with the change in the control group. The key assumption is parallel trends: without the intervention, both groups would have changed by the same amount. Check it by plotting the pre-intervention trends. The difference-in-differences tool on this site estimates the effect with a regression and robust standard errors.

Choosing a method

Presenting analytics to decision makers

The best analysis is useless if it does not change a decision. Some principles that work:

• Start with the decision and the answer: “Shift 20% of the social budget to search: expected +9% media-driven sales.”
• Quantify in money: incremental revenue, profit, CLV, cost per incremental customer.
• Show uncertainty: ranges and confidence intervals, and what would change the conclusion.
• Separate correlation from causation: say which results come from experiments, which from quasi-experiments and which from observational models.
• Propose the next test: every analysis should end with an experiment that would confirm or refute it.
• Build a measurement roadmap: dashboards for metrics, experiments for decisions, models for planning.

You have completed the course. The final assessment covers all nine lessons; pass it to download your certificate.