Propensity models in marketing: how to evaluate scores

Propensity models in marketing produce scores intended to rank who is more likely to take a defined action, such as buying within a specified period. To evaluate supplied scores, compare score bands with observed conversion and contribution using the same eligible population and outcome window.

A ranking can help allocate limited contact capacity without providing accurate probabilities. A good ranking also does not establish that contacting high-scoring prospects causes additional sales. Keep those questions separate when deciding how to use a scored list.

Ask what the supplied score represents

Before opening a campaign report, get the score definition from whoever produced it. Record the action being predicted, the prediction window, the score date, the eligible population, and whether larger values mean greater likelihood.

Confirm whether the score is a probability or only a rank. A value displayed as a decimal is not automatically a reliable probability. If the supplier describes the score as a probability estimate, compare its stated expectation with observed outcomes as well as checking the order of the bands.

The business owners have distinct jobs. Marketing operations validates the scored audience and contact history. Sales operations validates purchases and prospect-to-customer matches. Finance validates net sales, fulfillment costs, and contact expenses. The supplier owns questions about how the scores were produced.

This article evaluates an existing score export. It does not describe training, deploying, or serving propensity models.

Prepare an evaluation file with outcomes that came later

Use a scored prospect export containing prospect ID, score date, score definition, and supplied score. Match later purchases to the same ID, with purchase date, net sales, and defined variable costs. Add the contact date and cost from the campaign records.

Keep one row per eligible prospect in the evaluation population, including prospects who did not buy. For this example, conversion means at least one completed purchase during the agreed follow-up window. Multiple purchases can add sales and contribution, but the prospect still counts once as converted.

Use a completed historical group that was not used to fit or choose the supplied scoring approach, and retain scores recorded before the purchase window began. Do not combine recently scored prospects with prospects whose full follow-up window has finished. Otherwise, differences in observation time can look like differences in score quality.

For checks on the underlying sales file, see asking questions of a sales workbook. The calculation below starts after IDs, dates, and costs have been reconciled.

Work the ranking from four score bands

A US office-equipment supplier evaluates a completed campaign in which every prospect received the same type of contact. The supplier's export describes scores as probability estimates. Each band contains the prospects whose recorded scores fall in its range; no prospect appears twice.

Observed conversion rate = Converted prospects / Eligible prospects

Lift = Band conversion rate / Overall conversion rate

The overall conversion rate is (50 + 30 + 15 + 5) / (100 + 100 + 100 + 100) = 25.0%. The highest band's lift is 50.0% / 25.0% = 2.00. Conversion declines as scores decline, so the ranking separates higher- and lower-converting groups in this historical population.

Calculate rates from counts, retaining full precision before display. Percentages here are rounded to one decimal place and lift to two decimals. Check that band counts sum to the eligible prospect total and that converted plus nonconverted prospects equal that same total.

Check probability estimates separately

The highest band's average supplied probability is 0.60, implying 100 × 0.60 = 60 expected conversions. It delivered 50. A score can therefore order prospects usefully while overstating the observed purchase frequency.

Expected conversions total 60 + 40 + 20 + 10 = 130, against 100 observed. That discrepancy is a reason to ask the supplier about probability accuracy, population changes, and uncertainty. It is not enough evidence to diagnose a technical cause. Do not put the supplied probabilities straight into a revenue budget without evaluating their behavior across comparable completed periods.

Translate ranking into a capacity decision

Suppose the next campaign has capacity for a limited number of contacts. Compare historical results above each available cutoff. Changing a cutoff changes who is selected; classification threshold guidance describes the resulting trade-off between selecting more cases and excluding more cases.

For the highest two bands, selected conversion is (50 + 30) / 200 = 40.0%, while capture is 80 / 100 = 80.0%. These use the same numerator with different denominators. They correspond to the distinction between precision and recall: success among selected cases versus coverage of actual successes.

This table describes historical allocation. It does not promise the same conversions from the next list. Link the cutoff to the available contact capacity, and use a comparable completed group to monitor what happens afterward.

Test whether the selected business pays

Higher conversion does not necessarily mean higher contribution. In this example, use actual net sales and fulfillment costs, then subtract the contact expense assigned to each band.

The check is 50,000 − 30,000 − 4,000 = $16,000. The highest two bands account for 9,000 + 5,000 = $14,000 after contact costs. Adding the lowest band contributes nothing further under these cost assumptions. The result remains an observed contribution allocation, not incremental campaign profit established by an experiment.

Keep marketing attribution separate from score evaluation: assigning sales credit is another question. Use customer acquisition cost when the decision concerns the fully defined cost of acquiring customers.

Where score evaluation goes wrong

• Only converted prospects are exported. The denominator disappears and conversion rates become meaningless. Retain every eligible scored prospect.
• Scores use information from after the outcome. The comparison becomes hindsight. Preserve the score snapshot and date from before the purchase window.
• Incomplete windows are mixed with completed ones. Newer prospects have had less time to convert. Compare equally mature groups.
• Ranking is treated as probability accuracy. Check both ordering and expected-versus-observed counts when probabilities are supplied.
• Observed purchases are called incremental sales. Some selected customers would have purchased without contact. A causal claim needs an appropriate experimental comparison.

Evaluate your supplied score export in Covirage

Bring a sample of your own supplied scores, matched purchase outcomes, and campaign costs. Covirage maps the columns; its tools compute and check score-band conversion and contribution, while the external AI model explains the results with every figure traced to its rows. Create a free Covirage account in about a minute, then set up your business and bring your own files.

See sales insights for the supporting business context.