Getting models out of notebooks and keeping them honest

What this actually solves

The hard part of machine learning turned out not to be the modelling. A capable data scientist can build a genuinely useful churn or demand or fraud model in a couple of weeks. Keeping three of them running correctly for three years, through staff changes and platform migrations and a business that keeps changing shape underneath them, is the part nobody budgeted for.

The state this usually reaches looks like this. Models live in notebooks on a shared server, triggered by a cron entry someone added a year ago. Predictions land in a database table. The code is in version control but the models are not, so nobody can say with certainty which version is serving or what data it was trained on. Accuracy gets checked when someone remembers. And the whole arrangement depends on one person who knows how it fits together, which means it is fragile in a way that never appears on a status page.

The failure that makes this urgent is a silent one. A model does not crash when it becomes wrong. It carries on responding in milliseconds, returning perfectly well-formed predictions, and every dashboard stays green while the decisions those predictions drive get steadily worse. Weeks pass before someone notices the business outcome, and by then the cause is weeks behind. What is being bought here is not deployment. Deployment is the easy half. It is the ability to notice, to know which version did it, and to get back to the previous one quickly.

How the work runs

1. Assess what actually runs, and where. Every model in production or claiming to be, how it is triggered, what data it consumes, who owns it, and how anyone would know if it broke. This is done by looking rather than by asking, because the documented picture and the real one rarely match. Output is a maturity placement and a phased roadmap with the worst pain first. One to two weeks.
2. Put a registry in first. Every model version recorded with its training data, its parameters, its validation scores and its deployment status, through MLflow or Weights and Biases. This comes before serving because until you can answer which version is live, nothing else you build can be trusted. One to two weeks.
3. Containerise and serve, with a route back. Models packaged in Docker and served behind a versioned API using BentoML, KServe or your cloud's managed endpoints, with the rollback path built and — this is the part usually skipped — actually tested by performing a rollback before anything depends on it. Two to four weeks.
4. Instrument the predictions. Every prediction logged with its inputs, its output, the model version that produced it and the latency, into a store you can query. Prometheus and Grafana for the operational view, joined to the model view so an incident can be traced from a business symptom back to a specific version. Two to three weeks.
5. Add drift detection and set thresholds that mean something. Input distributions and prediction distributions compared against the training baseline through Evidently AI, with alert thresholds tuned so they fire on real movement rather than on ordinary weekly variation. An alert nobody believes is worse than no alert. One to two weeks.
6. Automate retraining behind a validation gate. The pipeline in Airflow or Dagster, and a gate that refuses to promote a newly trained model unless it beats the incumbent on a held-out set. Retraining without that gate is an automated route to shipping a worse model. Two to four weeks.
7. Document and hand over. Runbooks for the failures that will actually occur, the retraining trigger, the rollback procedure, and a working session with the team who will own it. About a week.

What moves the price

The range is $3,000–$30,000 over 1–4 months, with a standalone assessment sitting at the bottom of it. Three things drive the rest.

How many models and how uniform they are. Three models built by the same person in the same framework standardise easily. Three built over four years in different frameworks with different dependency requirements need their differences resolved before any shared platform can hold them.

The platform, and whether the data pipeline underneath is sound. A team already on one cloud with clean, scheduled data feeds is a contained project. A team whose training data is assembled semi-manually each time, or who are split across environments, is carrying a data engineering project inside the MLOps one, and it has to be sequenced first.

Compliance and where it has to run. Regulated environments add auditability requirements — reproducing exactly which model version produced a specific historical decision, and proving it. On-premise or air-gapped deployment removes the managed services that would otherwise absorb a large part of the work.

What usually goes wrong

Infrastructure monitoring gets mistaken for model monitoring. The dashboards are comprehensive, the alerts are wired to a pager, uptime is excellent and response times are low. None of it watches the thing that matters. A model whose input population has shifted returns confident predictions at full speed, and every operational signal reports health while the predictions decay. The distinction has to be built in deliberately, because nothing about the infrastructure stack suggests it is missing.

Retraining is automated before there is a gate to stop it. Automatic retraining sounds like maturity and is dangerous without validation. If the drift that triggered the retrain came from a broken upstream feed or a contaminated period of data, the newly trained model is worse than the one it replaces, and the automation promotes it without a human in the path. The gate is simple — the challenger must beat the incumbent on held-out data or it does not ship — and it is what makes the automation safe rather than merely fast.

The deployment path is automated and the way back is not. Pushing a new model version becomes a single command, while rolling back is a manual sequence written in a document that has never been executed. The first bad promotion then turns into a slow, improvised afternoon under pressure, usually with the person who wrote the document unavailable. The rollback is built at the same time as the deployment and rehearsed once while nothing is wrong, which is the only time it can be rehearsed calmly.