How to choose analytics software for compliance: questions, data and traps

Most buying decisions for analytics start from a feature list. For compliance teams the better start is the questions that come back every month, the files already on hand, and the traps that make a tool look right in a demonstration and wrong in the first board meeting.

Start from the questions

Any tool you consider should answer these from your data, not from a sample. Ask to see it.

The data you already hold

• Obligations register
• Control library
• Regulatory change log
• Testing schedule
• Test results
• Issues log
• Learning system
• Role matrix

If a vendor needs a warehouse built before it can read these, count that in the cost and the time.

Ten questions to ask any vendor

1. What does it need in place before the first answer? A warehouse, a data model, a modelling language, a partner? Ask for the list and the typical weeks.
2. Who does the setup, and who maintains it? Your team, the vendor, or a partner, and what that costs after year one.
3. Does the AI calculate figures, or choose from computed ones? A language model that writes queries or code can produce a plausible wrong number. Ask what it is allowed to do.
4. Does every total reconcile to a control figure? Ask to see a bridge that does not sum and what the product does about it.
5. Can every figure be opened to its rows? An answer nobody can check becomes a debate in the meeting.
6. What does it cost in the first year, all in? Licences, consumption, implementation, modelling and training, not only the seat price.
7. How does data arrive, and who holds credentials? A file your systems already export, a scheduled drop, or a live connection with the vendor holding keys.
8. What happens to the data, and where is it stored? Residency, retention, deletion, and whether names can be replaced with identifiers.
9. Can we see it on our own data before we sign? A demonstration on a sample dataset tells you little about your own.
10. What does the tool do when it cannot answer? It should say so. A confident guess does more harm than no answer.

Checks specific to compliance

Ask whether the tool enforces these, and what it does when they fail:

• Coverage: Obligations = mapped, owned and tested + mapped with a gap + unmapped
• Testing: Tests due = completed on time + completed late + overdue
• Issues: Opening issues + raised − closed = closing issues
• Training: Required = completed in date + expired + never completed

The traps

Obligations with no control mapped. Reporting starts from the control library, so an obligation that never got a control never appears.

Incident reporting against activity. Raw counts make the busiest, most diligent unit look worst.

Repeat failures. A control that fails, is remediated, and fails again counts as one failure each year.

Measures to leave out

Number of controls. More controls is not more compliance.

Training completion as one firm-wide percentage. It hides the unit at 60 percent and ignores what each role requires.

Incidents reported, as a raw count. Always against activity.

A scorecard

Where Covirage fits

Covirage reads the exports above, answers the questions with figures our tools compute and check, and is set up for you within a week. See analytics software for compliance compared, AI analytics for compliance and Covirage for Compliance.

For the measures in full, with formulas and exports, read KPIs for compliance teams.