A forecast does not save inventory. A churn score does not retain a customer. A failure probability does not schedule maintenance.
Someone has to change a purchase, call, route, price, inspection, or staffing decision because of the prediction. If no named person or system will act differently, the model has no operating job.
Predictive analytics consulting should therefore be priced around a decision, not a model. The first deliverable is a decision receipt. It records what happens today, what information arrives too late, which choices are actually available, and what each kind of error costs.
Our machine learning consulting guide covers custom ML fit and broad engagement economics. This guide starts one level closer to use: a buyer already has a forecast or scoring question and needs to determine whether it deserves a production system.
Write the decision receipt
Complete this sentence without mentioning an algorithm:
Every [cadence], [owner] chooses [action] for [unit] using [current evidence], before [deadline].
“Predict demand” is not complete. “Each Thursday, the replenishment manager sets next week’s case order for each store before the distributor cutoff, using recent sales, on-hand estimates, planned promotions, and local judgment” is testable.
Add the fields that make the decision purchasable:
| Receipt field | Question |
|---|---|
| Decision owner | Who is accountable for the action? |
| Unit and cadence | One customer, store, machine, or invoice, and how often? |
| Available actions | What can the owner actually change? |
| Current baseline | What rule, forecast, or judgment is used now? |
| Decision deadline | When does new information become useless? |
| False positive cost | What happens when the system acts unnecessarily? |
| False negative cost | What happens when it misses the event? |
| Capacity constraint | How many cases can people or systems act on? |
| Outcome delay | When will the true result be known? |
This receipt is a proposed gmware method. It keeps the project honest when the easiest metric is not the one that matters to the operation.
Make the baseline hard to beat
A model needs an opponent. Use the actual current method, not a deliberately weak straw man.
For demand, that might be last year’s same-week value adjusted by a known promotion. For churn review, it might be the account team’s existing risk flag. For maintenance, it might be an age and usage threshold. Record the baseline’s inputs, timing, and exceptions.
A simple baseline can be useful because it is cheap, legible, and stable. If a complex model improves an abstract score but cannot change the decision before its deadline, the baseline still wins.
Require the consultant to preserve baseline results in the evaluation packet. “Better than our previous model” is meaningless if nobody can reproduce the previous result.
Test on the future, not a shuffled echo
Many business datasets have time in them. Training on randomly mixed past and future rows can leak later conditions into an earlier prediction.
Use a holdout period that occurs after the training period when that matches production. Freeze the target definition before examining results. Check whether fields were created after the decision deadline. A “days overdue” value calculated at collection time, for example, can contain information the operator would not have had when acting.
The consultant should list every leakage check and excluded field. It should also explain whether seasonality, policy changes, acquisitions, product launches, or missing-history periods make the holdout unrepresentative.
NIST’s AI RMF says measurement should include testing, performance assessment, uncertainty, benchmarks, and documented results. The framework is voluntary. It supplies a useful risk-management vocabulary, not a certification or a promise that one test proves trustworthiness.
Score the consequence of being wrong
An average can hide the error that determines whether the system is useful.
Suppose a team can call only 100 at-risk customers each week. Ranking quality near that operational cutoff may matter more than a metric across every customer. If a false fraud alert freezes a legitimate payment, the review burden and customer harm belong in the score. If a missed equipment failure can halt a line, the cost is different again.
Build a consequence table before selecting the metric:
| Model outcome | Operational consequence | Evidence |
|---|---|---|
| Correct action | Avoided loss or improved service | Observed later outcome |
| Unnecessary action | Labor, delay, discount, inspection, or customer friction | Action record and cost |
| Missed event | Loss, outage, churn, waste, or escalation | Outcome record |
| Correct inaction | Capacity preserved | No-action record |
Then test several thresholds. The useful threshold respects available capacity and stated risk tolerance. It is not automatically 0.5.
Run in shadow before granting action
Shadow mode produces predictions on the production cadence without allowing them to trigger the real action. Operators can compare the result with their current method, record disagreements, and find missing context.
The shadow log should include prediction time, input version, model version, score, proposed action, operator decision, reason for override, and eventual outcome. Do not ask operators to annotate every case forever. Sample enough to learn where the system’s view and the business process diverge.
Shadow mode is also where timing failures surface. A forecast delivered after the order cutoff may score well and still be useless. A risk alert without the evidence an account manager needs will be ignored. Those are system failures, not communication nits.
Set the promotion rule before the run begins. Include performance against baseline, action capacity, data freshness, override review, and unresolved harmful cases. A project should be allowed to stop.
Contract for drift and response
Production conditions move. Customer mix changes. A new pricing rule alters behavior. Sensors are replaced. Operators adapt to the score.
The NIST AI RMF Playbook recommends comparing production behavior with pre-deployment measures and monitoring distribution differences, error response, and feedback. Turn that general advice into a specific operating agreement:
- which input, output, outcome, and business measures are watched;
- who reviews them and how often;
- which threshold opens an investigation;
- what can pause or roll back use;
- how overrides and complaints are sampled;
- when retraining is considered;
- who approves a new model version;
- when the system is retired.
Drift is broader than a statistical alert. A model can remain statistically stable while the action becomes unavailable, too costly, or prohibited by a new policy. Monitor the decision receipt too.
Separate the builder from acceptance
The consultant who builds the model can prepare the evidence. The buyer should own acceptance.
Name a decision owner, data owner, technical operator, and independent reviewer. The reviewer need not rebuild the model. That person needs access to the frozen target, data lineage, leakage checks, baseline, holdout results, error slices, shadow log, and unresolved limitations.
NIST describes independent review as one way to reduce internal bias and conflicts in testing. Use it in proportion to consequence. A forecast that helps arrange a marketing test and a score that affects a person’s access to a service do not need the same review depth.
Compare firms with one fixed packet
Give each shortlisted consultancy the same decision receipt, a data dictionary, a representative sample, and the current baseline. Ask for a written response to the same questions:
- What would make you recommend no model?
- Which fields may leak the target?
- How will you construct the time-based holdout?
- Which errors matter to the decision owner?
- How does the score enter the workflow before the deadline?
- What is monitored after launch?
- What will the internal team own at exit?
Score the specificity of the answer. A long list of algorithms is not evidence of fit. A credible provider may narrow the target, propose better data collection, keep the baseline, or decline the build.
A useful first engagement
Bring the decision owner, the current spreadsheet or rule, ten awkward historical cases, and a data owner. In one working session, the team can draft the decision receipt and identify the most dangerous leakage and outcome gaps.
The next step might be a data audit, a baseline replay, a shadow test, or no model. That last answer can be the best return on a short consulting engagement. It prevents a prediction with nowhere to go.