Most ML models never make it to production. Of those that do, most fail silently — drift, stale features, broken dependencies, broken retraining pipelines. We've shipped 30+ production models over the last three years, and we've made every mistake there is to make. This article is the playbook we built to stop making them.
MLOps is not DevOps with extra steps. It's a different discipline because models fail differently. Code either works or it doesn't. Models degrade. The infrastructure has to assume degradation is a normal operating mode, not an exception.
The five pieces of the MLOps stack
Every production ML system we ship has these five pieces. None is optional.
- Feature store — versioned features with point-in-time correctness for training and serving
- Model registry — every model versioned with training data hash, code hash, metrics, and approver
- Inference API — REST + gRPC with latency budgets and per-model autoscaling
- Drift monitoring — input distribution + prediction distribution + ground-truth-vs-prediction
- Retraining pipeline — scheduled, with human approval for production rollout
Models are versioned like code
Every model artifact has a SHA. The SHA is composed of: training code commit, training data hash, hyperparameters, evaluation metrics, approver. The model doesn't ship to production unless all five are recorded and the approver has signed off. This sounds bureaucratic. It's not — it's automated and takes 4 minutes from training run to production deploy.
“If you can't tell me which version of which model served which prediction three weeks ago, you don't have a production ML system. You have a notebook with a server in front of it.”
Drift is the silent killer
The single most common cause of production ML failure we see is undetected drift. The model was correct at training time. By month 6, the input distribution has shifted 18%. The model is confidently wrong. No one notices because the application code still works and the API still returns 200.
Our drift monitoring tracks three distributions: input features, prediction distribution, and (where ground truth becomes available) actual-vs-predicted. Any of the three breaching a threshold pages the on-call engineer. The threshold is set per-model based on training-time variance.
The bottom line
Production ML is a discipline, not a notebook. The five-piece stack above has shipped 30+ models for us across fintech, healthcare, and retail. The models that fail are the ones that skipped one of the pieces. The ones that succeed all have all five.
Author
Vikram Iyer
Head of AI & Data, Globantis Labs
Vikram writes about engineering at Globantis Labs. Reach out if you'd like to discuss this article or your own project.
