Product decisions · 5 min read · Sep 20, 2026
A launch is not an experiment just because the chart moved
A team launches a new onboarding flow on Monday.
By Friday, completion has increased from 42% to 48%.
The launch deck practically writes itself:
The new experience increased completion by six percentage points.
Maybe it did.
But perhaps traffic changed. A marketing campaign brought in people with stronger intent. A payment bug was fixed on Tuesday. Fewer mobile users visited that week. A holiday changed the usual pattern. Or completion had already begun improving before the launch.
The chart moved after the product changed. That tells us when two things happened. It does not prove that one caused the other.
“Before versus after” answers a smaller question
A simple comparison can tell us: Was the metric different after launch?
It cannot reliably tell us: Would the metric have changed if we had not launched?
That second question requires a counterfactual—a credible estimate of what would have happened without the change.
In a randomized experiment, the control group provides that estimate. Because users are assigned between control and treatment, seasonality, marketing activity and many other outside influences should affect both groups rather than only one.
That is why controlled experiments are used to establish causal impact, not merely to observe correlation. Microsoft’s experimentation research describes randomization as the mechanism that distributes external factors across control and treatment, making the product change the meaningful difference between them. Read the research.
But not every launch can be cleanly randomized.
Sometimes the feature affects an entire marketplace. Sometimes users influence one another. Sometimes the infrastructure cannot serve two versions safely. Sometimes a legal or reliability change has to reach everyone.
The answer is not to give up on measurement. It is to be precise about what the evidence can support.
The evidence ladder
A stronger claim needs a stronger comparison.
Do not climb higher in the conclusion than the measurement design allows.
Decide what kind of question you are answering
There are three different questions teams often collapse into one:
| Question | Evidence needed |
|---|---|
| Did the launch work technically? | Errors, latency, missing events, support issues |
| Did behaviour change after launch? | Pre/post trends and segmented monitoring |
| Did the launch cause the change? | Experiment or credible comparison group |
All three are useful. Only the third supports a causal claim.
A launch can work technically while making no difference to behaviour. Behaviour can improve after launch even when the feature caused none of the improvement.
Measurement becomes clearer when the team chooses the question before looking at the result.
Build the comparison before the launch
The weakest time to decide how success will be measured is after the chart moves.
At that point, it becomes very easy to choose the date range, segment or metric that tells the best story.
Before launch, write a short evidence plan:
- Decision: What will we do differently if the result is positive, neutral or harmful?
- Primary outcome: Which behaviour is the feature expected to change?
- Mechanism: Why should the feature change that behaviour?
- Guardrails: What must not become worse?
- Unit: Are we assigning users, accounts, locations, campaigns or time periods?
- Comparison: What represents the world without the launch?
- Minimum useful effect: How much change would justify the cost or risk?
- Observation window: How long does the behaviour need to appear?
- Known interference: What else is launching or changing at the same time?
The mechanism is especially important.
If a feature is supposed to improve activation by reducing confusion, we should expect an earlier step—fewer repeated attempts, less time stuck or fewer exits—to move before long-term retention does.
If the final metric improves but none of the expected intermediate behaviour changes, we should be cautious about crediting the feature.
When an A/B test is not possible
A few alternatives can make a launch evaluation stronger than a basic before-and-after chart:
Phased rollout: Release to comparable groups at different times. Groups that have not received the feature yet can provide a temporary comparison.
Matched comparison: Find accounts, regions or products with similar historical behaviour that did not receive the change.
Interrupted time series: Use a longer history to determine whether the post-launch movement is meaningfully different from the existing trend.
Switchback design: Alternate treatment and control across time windows when the intervention operates at a market or system level. This requires care because effects can spill into later windows.
None of these becomes trustworthy merely because it has a technical name. A matched group can differ in an important hidden way. A phased rollout may send the feature to the most promising customers first. A time-series result may be distorted by another event occurring on the same date.
The limitations belong next to the conclusion.
Use AI as a critic, not a causal calculator
Before analysing a launch, give an AI assistant the evidence plan and ask:
I am evaluating whether this product launch caused a change in the primary metric.
Separate: facts directly observed; causal claims the design can support; plausible alternative explanations; data-quality checks required; segments that could reveal a composition change; and assumptions that cannot be validated from the supplied data.
Do not treat “after launch” as “caused by launch.” Suggest the narrowest defensible conclusion.
AI can help identify missing comparisons and confounders. It cannot repair an evaluation whose control group was never created or whose instrumentation began after launch.
Look for composition before celebrating movement
Whenever an aggregate metric changes, check whether the underlying population changed too.
Completion might rise because more returning users arrived, one high-performing country represented more traffic, desktop traffic increased, a low-performing acquisition channel spent less, eligibility rules excluded difficult cases or missing events reduced the denominator.
Recalculate the metric within stable segments. Compare volumes and distributions, not only rates. Confirm that the numerator and denominator still represent the same behaviours they represented before launch.
Sometimes the product improved. Sometimes the mix changed.
Both are insights—but they lead to different decisions.
Try this
Take one recent launch described as successful.
Write two sentences:
What we observed:
After launch, the metric changed from ___ to ___.
What we can conclude:
Because we used ___ as the comparison, we can reasonably attribute ___ to the launch, assuming ___.
If the second sentence is difficult to complete, that does not make the analysis worthless. It means the claim should become smaller.
“The metric improved after launch” may be honest.
“The launch increased the metric” must be earned.