Evaluate supplied churn scores against subsequent customer losses. This worked example separates expected losses, observed churn, outreach capacity, and evidence that a retention action helps.
Churn rate prediction estimates future customer departures over a stated period, but a business team needs evidence that the supplied scores help a decision. Compare scores exported before the period with the customers who actually leave afterward. Keep prediction quality separate from the effectiveness of any retention offer.
Ask the score supplier to state the customer population, scoring date, outcome window, and definition of departure. A ranking from low to high risk is different from a probability of leaving. Do not add arbitrary scores together to estimate total losses.
The churn glossary defines the customer measure, while renewal, retention, and churn distinguishes a beginning customer base from contracts eligible for renewal. A score for failed renewal cannot be evaluated against a different definition based on inactivity.
This article evaluates exported predictions already supplied to the business. It does not describe producing individual probabilities or training a prediction system. The practical question is whether the supplied file identifies a useful group for further investigation.
Keep a dated export containing account ID, score date, prediction horizon, probability if supplied, and customer segment. Join it to a later customer-status export using the same account ID. Preserve the earlier file so later cancellations cannot change the recorded prediction.
For an illustrative US subscription service, all accounts were active on April 1, 2026. Their supplied probabilities refer to cancellation by April 30, 2026. The business waits until the outcome period is complete and confirms that every scored account has an observed status.
Record any retention outreach during the period. If the highest-risk accounts receive special help, their subsequent churn reflects both initial risk and that intervention. Report the contact policy with the evaluation rather than treating the outcomes as untouched evidence of prediction quality.
The export contains the following probability groups. Every account within an illustrative group has the displayed probability. A real export would sum the individual probabilities within each group instead.
| Supplied risk band | Starting customers | Supplied probability | Expected losses | Observed losses | Observed churn |
|---|---|---|---|---|---|
| High | 100 | 40.0% | 40 | 35 | 35.0% |
| Medium | 200 | 15.0% | 30 | 30 | 15.0% |
| Low | 700 | 3.0% | 21 | 35 | 5.0% |
| Total | 1,000 | 91 | 100 | 10.0% |
Expected losses = Sum of the supplied customer probabilities
Observed churn = Observed losses / Starting eligible customers
Expected losses are 100 × 40% + 200 × 15% + 700 × 3% = 91. Observed losses are 35 + 30 + 35 = 100. The expected aggregate rate is 91 / 1,000 = 9.1%, compared with observed churn of 10.0%.
The total understates losses by nine customers, or 0.9 percentage points. More importantly, the high group overstates its observed rate while the low group understates it. A near total does not establish that the probability levels are reliable within groups.
Check uncertainty before interpreting these gaps as persistent errors. NIST's guidance on proportion intervals explains why small groups and few observed events need care, including methods that avoid unreliable normal approximations. Repeat the evaluation on later completed cohorts.
Suppose the success team can contact the 100 high-band customers. Of those, 35 subsequently leave and 65 stay. The remaining 900 customers include 65 departures and 835 retained accounts.
| Evaluation measure | Calculation | Result |
|---|---|---|
| Loss rate in contact list, or precision | 35 / 100 | 35.0% |
| Share of all losses in contact list, or recall | 35 / 100 | 35.0% |
| Cohort baseline loss rate | 100 / 1,000 | 10.0% |
| Concentration relative to baseline, or lift | 35.0% / 10.0% | 3.50 times |
The equal precision and recall here are a coincidence: the list size and total departures both happen to be 100. Google's classification evaluation reference defines these measures with different denominators. Covirage is not affiliated with Google.
If capacity expands to the high and medium groups, the list contains 300 accounts and 65 departures. Precision becomes 65 / 300 = 21.7%; recall becomes 65 / 100 = 65.0%; lift becomes 21.7% divided by 10.0%, using unrounded values, or 2.17 times. Percentages are rounded to one decimal place and lift to two decimals.
The wider list captures more departures but requires more contacts per observed loss. Choose a list size using capacity and the economics of an action, not whichever percentage looks strongest.
Labeling every customer as retained would be correct for 900 / 1,000 = 90.0% of accounts in this cohort. Treating only the high band as predicted departures would be correct for 35 + 835 = 870 accounts, or 87.0%. Yet that high band still concentrates departures at 3.50 times the cohort rate.
This is why overall accuracy alone is a poor decision rule when most customers stay. Also compare results with a simple business rule, such as a list based on an approaching renewal and an unresolved service issue, using the same list size and outcome window. Retain the previous completed cohort's rate as a historical baseline, checking that eligibility and customer mix are comparable. Do not invent a benchmark from an unrelated industry.
Churn rate analysis helps check whether segment mix explains changing performance. Once a list is useful, reducing churn requires a separate intervention test rather than assuming every predicted loss can be prevented.
Bring your dated score export and subsequent customer outcomes as files or a Google Sheet. Covirage maps the columns; its tools compute and check the outcome comparisons, and the external AI model explains the results with every figure traced to its rows. Create a free Covirage account and bring a sample of your own export.
See sales insights for the supporting business context.
Only if its supplier defines it that way. A score can be a ranking without representing a customer-level probability. Confirm the scale, prediction horizon, eligible population, and event definition before adding scores together or interpreting a score as the expected percentage of customers who leave.
Freeze the scores before the outcome period, then compare them with completed customer outcomes. Check whether higher-scored groups lose more customers, whether stated probabilities match observed rates, and whether the outreach list captures enough losses to justify the available staff capacity.
No. A risk score can identify likely departures without identifying customers whose decisions an intervention can change. Evaluate the retention action separately using comparable groups, equal observation windows, and contribution after the cost of outreach, concessions, and additional service.