Blog · Board and management reporting
Compare test and holdout conversions and revenue per eligible customer from assigned experiment groups. Work through the lift calculation, reconcile the totals, and keep the assumptions behind causal interpretation visible.
Incrementality analysis compares outcomes in an assigned treatment group with outcomes in a holdout to estimate what the intervention added. Start with conversion and revenue per eligible customer, since unequal group sizes make raw totals misleading. Keep the assignment and measurement assumptions beside the result.
Start with groups already assigned for an approved experiment. For example, a US distributor tests a renewal reminder against its usual process. The treatment group receives the reminder; the holdout remains on the usual process during the same outcome window.
Obtain a customer-level assignment export and an outcome export for that window. Keep customer ID, experiment ID, assigned group, assignment date, eligibility flag, conversion flag, and net revenue in USD. Aggregate multiple orders to the customer before combining outcomes with assignments, so order counts do not multiply eligible customers.
Random assignment matters because systematic differences between groups can explain their outcomes. The NIST guide to completely randomized designs describes random assignment of treatment levels to experimental units. This analysis preserves the assignment already made; a workbook summary cannot establish that the original assignment was valid.
Document the conversion definition and revenue cutoff. Retain assigned eligible customers with no purchase as nonconverters with zero revenue when the complete outcome records establish that result. An unmatched customer with unavailable tracking is an unresolved observation, not an automatic zero.
Prepare an analysis sheet with one row per assigned eligible customer. Put Customer ID in A, Assigned group in B, Converted in C, and Net revenue in D. Converted is a numeric flag: 1 for a qualifying conversion and 0 otherwise.
The following example contains 15,000 eligible customers across the two groups. The summary shows the inputs, not a comparison of differently filtered customer lists.
| Assigned group | Eligible customers | Converted customers | Net revenue (USD) |
|---|---|---|---|
| Treatment | 10,000 | 600 | 72,000 |
| Holdout | 5,000 | 250 | 27,500 |
| Total | 15,000 | 850 | 99,500 |
Put the group names in F2:F3. In Excel, use these formulas for eligible customers in G, conversions in H, and revenue in I, filling down:
=COUNTIFS($B$2:$B$15001,F2)
=SUMIFS($C$2:$C$15001,$B$2:$B$15001,F2)
=SUMIFS($D$2:$D$15001,$B$2:$B$15001,F2)
Microsoft's COUNTIFS documentation describes counting rows that meet specified criteria. These formulas assume the sheet has already passed uniqueness, eligibility, and missing-outcome checks. They do not perform those checks themselves.
Conversion rate = Converted customers / Eligible customers
Absolute conversion lift = Treatment conversion rate - Holdout conversion rate
Relative conversion lift = Absolute conversion lift / Holdout conversion rate
In J2 calculate conversion rate with =H2/G2; in K2 calculate revenue per eligible customer with =I2/G2. Fill both down to the holdout row.
| Measure | Treatment | Holdout | Treatment minus holdout |
|---|---|---|---|
| Conversion rate | 600 / 10,000 = 6.0% | 250 / 5,000 = 5.0% | 1.0 percentage point |
| Revenue per eligible customer (USD) | 72,000 / 10,000 = 7.20 | 27,500 / 5,000 = 5.50 | 1.70 |
Percentages are displayed to one decimal place; dollar averages use two. Calculate with unrounded values. Relative conversion lift is (6.0% - 5.0%) / 5.0% = 20.0%. It describes the increase relative to the holdout rate, not the percentage of treatment customers who converted.
If the holdout conversion rate is zero, relative lift is undefined. Report the absolute difference and group counts rather than forcing a percentage.
Estimate treatment outcomes at the holdout rate, then subtract that baseline from the observed treatment outcomes:
| Step | Calculation | Result |
|---|---|---|
| Baseline treatment conversions | 10,000 × 5.0% | 500 |
| Estimated incremental conversions | 600 - 500 | 100 |
| Baseline treatment revenue (USD) | 10,000 × 5.50 | 55,000 |
| Estimated incremental revenue (USD) | 72,000 - 55,000 | 17,000 |
Using the summary cells, the corresponding formulas are:
=H2-G2*J3
=I2-G2*K3
The revenue identity proves the arithmetic: baseline revenue of $55,000 plus the estimated increment of $17,000 equals observed treatment revenue of $72,000. Also reconcile both groups' customer counts to the assignment file and their combined $99,500 to the filtered outcome export.
The $17,000 is a point estimate under the stated assumptions. It is not all treatment revenue, a profit figure, or an automatic rollout forecast. A return on investment calculation additionally needs the appropriate costs and benefits.
Have the experiment owner review the assigned split, baseline group characteristics, missing outcomes, and customers exposed to both conditions. The reviewer should also assess uncertainty using a method appropriate to the assignment unit and outcome distribution. NIST describes comparisons of two independent proportions; a conversion-rate comparison and a revenue comparison require different attention to their outcome data.
This table contains no significance test or confidence interval. A positive point estimate alone is insufficient to establish a reliable business effect. If assignment occurred by store or territory, treating individual customers as independent observations would also misstate uncertainty.
For channel credit, see lead attribution. For comparing changing revenues and costs in a business proposal, see the separate incremental analysis example.
Covirage is not affiliated with Microsoft.
Bring a sample of your own assigned-group and outcome files to Covirage. Covirage maps the columns, and its tools compute and check the supplied group summaries; the external AI model explains the differences with every figure traced to its rows, while the experiment owner remains responsible for causal assumptions. Create a free Covirage account and bring a sample of your own data.
See Covirage for self-service analytics for the supporting file-based workflow.
Attribution assigns credit for observed outcomes to channels or interactions. Incrementality asks how outcomes differ from a credible untreated baseline. A channel can receive attribution for purchases that would have happened anyway, so attributed revenue and estimated incremental revenue answer different questions.
Yes, but compare rates or amounts per eligible customer rather than raw totals. The holdout still needs enough information for an uncertainty assessment. Preserve the approved allocation and assignment records, and investigate unexplained deviations before interpreting the estimated lift.
For an analysis based on assigned groups, retain eligible customers under their original assignment even when delivery fails or they do not engage. Excluding them after assignment can select a different population. Use the experiment's approved eligibility and outcome rules consistently in both groups.
No. A positive difference is a descriptive result until assignment quality, consistent measurement, interference, and statistical uncertainty have been assessed. Random assignment supports causal interpretation, but missing outcomes, spillovers, or departures from the original design can still undermine the comparison.