Analyze exported marketing experiment results using conversions, revenue, and contribution per assigned account. Follow a worked spreadsheet example, check uncertainty, and identify the conditions needed for a causal conclusion.
AB test analysis for marketing compares conversions and financial outcomes between assigned groups over the same observation window. Start with every assigned account, including those that never click or buy. Then check whether the conversion result and the contribution result support the same business decision.
Write down the treatment, assignment unit, primary outcome, observation window, planned sample, stopping rule, and decision threshold before launch. A US supplier testing two email offers might assign accounts rather than contacts, so colleagues at the same customer do not receive conflicting offers.
NIST's guide to completely randomized designs describes random assignment of treatment levels to experimental units. For this business example, assignment occurs at the account level and outcomes are summarized at that same level. Repeated orders are combined before measuring whether an account converted.
Separate the primary business outcome from supporting measures. The KPI, metric, and measure distinction helps prevent choosing a favorable metric after seeing the results.
Start with the experiment assignment export: account key, assigned group, assignment date, and eligibility status. Match it to the order export, returns, fulfillment costs, and treatment-specific marketing costs. Keep assigned accounts without purchases in the population with zero recorded revenue after verifying outcome coverage.
Use the same purchase cutoff and return allowance for both groups. Label late or unavailable outcomes as incomplete rather than silently setting them to zero. The marketing cohort analysis guide covers aligning observation windows across groups.
For the example, both groups have complete outcomes. Each conversion means an account with a qualifying purchase. The invented figures below illustrate arithmetic, not expected experiment performance.
Put these columns in A through G, with headers in row 1 and groups in rows 2 and 3. Revenue is net of the recorded discounts and returns; variable costs cover fulfillment of those orders.
| Group | Assigned accounts | Converted accounts | Conversion rate | Net revenue (USD) | Variable costs (USD) | Marketing costs (USD) |
|---|---|---|---|---|---|---|
| A: control offer | 10,000 | 400 | 4.0% | 40,000 | 24,000 | 2,000 |
| B: revised offer | 10,000 | 460 | 4.6% | 43,700 | 28,405 | 2,500 |
| Total | 20,000 | 860 | 4.3% | 83,700 | 52,405 | 4,500 |
In your spreadsheet, calculate conversion rate in D and add contribution in H and contribution per assigned account in I. Enter these formulas in D2, H2, and I2, respectively:
=C2/B2
=E2-F2-G2
=H2/B2
Fill down to row 3. Calculate the absolute conversion difference and relative lift with these formulas, respectively:
=D3-D2
=(D3-D2)/D2
| Result | A | B | B minus A |
|---|---|---|---|
| Conversion rate | 4.0% | 4.6% | 0.6 percentage points |
| Revenue per converted account (USD) | 100.00 | 95.00 | -5.00 |
| Contribution after listed costs (USD) | 14,000 | 12,795 | -1,205 |
| Contribution per assigned account (USD) | 1.4000 | 1.2795 | -0.1205 |
Relative conversion lift is (4.6% - 4.0%) / 4.0% = 15.0%. Yet contribution falls by $1,205 because the revised offer produces less revenue per converted account and incurs higher fulfillment and marketing costs. Contribution here excludes corporate overhead; keep that scope visible.
Rates and lift are displayed to one decimal, revenue per buyer to two decimals, and contribution per assigned account to four decimals. Calculate with unrounded values. The conversion lift alone cannot justify adopting the revised offer.
For independent randomized accounts with a binary conversion outcome, a large-sample approximation can describe uncertainty around the rate difference. NIST's comparison of two proportions explains proportion comparisons and the distinction between large-sample and exact methods. The interval below uses the unpooled standard error for a difference, rather than the pooled null-hypothesis test statistic.
With the worksheet above, enter these formulas consecutively in K2 through K6:
=D3-D2
=SQRT(D2*(1-D2)/B2+D3*(1-D3)/B3)
=NORM.S.INV(0.975)
=K2-K4*K3
=K2+K4*K3
This uses a prespecified two-sided 95% confidence level. The standard error is approximately 0.002869 and the critical value is approximately 1.960. The resulting approximate interval runs from 0.038 to 1.162 percentage points, rounded to three decimals.
That interval concerns conversion, not contribution. Financial outcomes need their own uncertainty assessment, using outcomes at the assignment unit and a method appropriate for skewed spending. A randomized comparison also needs intact assignment, complete measurement, and no material interference between accounts before supporting a causal interpretation.
The assignment counts must tie to the eligible assigned population: 10,000 + 10,000 = 20,000. Converted accounts must be unique within their assigned group, and 400 + 460 = 860 must match the matched conversion list.
The financial identity is 83,700 - 52,405 - 4,500 = $26,795. It also equals $14,000 + $12,795, the sum of the group contributions. Check the revenue and cost totals against the experiment-scoped finance extracts, not the company's entire reporting period.
Use marketing funnel analysis to investigate where behavior changed after the primary comparison is complete. Record exploratory findings separately from the planned decision.
For formal inference beyond this simple setup, use the statistical analysis in marketing guide and have the experiment owner review the design assumptions.
Bring your assignment, outcome, and cost files; Covirage maps the columns, and its tools compute and check the group summaries. The external AI model explains the supplied results with every figure traced to its rows. Create a free Covirage account and bring a sample of your own data.
See sales insights for the supporting business context.
Use the group assignment population for the primary comparison. Clicking can be affected by the treatment, so restricting the analysis to clickers can select different kinds of accounts in each group and undermine the randomized comparison.
Percentage points subtract one conversion rate from another. Relative lift divides that difference by the control rate. Report both with their labels and the group counts, because a relative change alone hides the starting conversion rate.
Aggregate totals show the observed financial difference. To assess uncertainty around contribution per account, retain account-level outcomes or appropriate variance summaries and use a method suited to the assignment design and the distribution of financial outcomes.
Bring the assignment and outcome records from an experiment conducted elsewhere. Covirage's tools compute and check supplied figures, and the external AI model explains them. Treat experimental design and any formal statistical inference as separate responsibilities.