What is model monitoring?
Continuous surveillance of models in production: performance, data drift and answer quality, with alarms before the damage.
Model monitoring is the continuous surveillance of a model after release: how performance is doing, whether incoming data still resembles the training data, whether answers stay within expected quality. It is needed because models do not break like software: they degrade silently, with no exceptions in the logs, as the world drifts away from the data they learned on. That is why model monitoring is not a task separate from MLOps, but its natural extension after release: the same versioning and traceability principles governing training keep applying for as long as the model stays in production. A model that stops being observed effectively also stops being managed. It should be kept conceptually separate from pre-production testing: verifying a model performs well before release says little about how it will behave six months from now, once the data it receives has inevitably drifted from what it was validated on in the lab.
See also: MLOps · MLOps is the set of practices that takes machine learning models to production and keeps them there: versioning, deployment, monitoring and retraining.
What gets monitored
Three families of signals. Data drift: the input distribution changes (new products, new customers, an updated price list) and the model starts receiving questions it has never seen. Model drift (or concept drift): the relationship between inputs and reality changes, a case not to confuse with data leakage, where the problem is not that data changes after release but that the model was already too optimistic in the lab because of a flaw in how the training set was built, and yesterday's predictions no longer hold. And business performance: the metric that truly matters, measured when ground truth arrives (the demand forecast against actual sales). For LLM systems add answer quality, refusal rates, costs and latency: there monitoring merges with continuous evals and chain observability.
How it is done in practice
The foundations are unexotic: thresholds on input distributions, a sample of predictions verified against reality as soon as it is available, dashboards with an owner mandated to act, and retraining (or rollback) as a defined procedure, not creative firefighting. The rule separating mature teams: every production model has an owner, a health metric and a threshold beyond which something happens. A model without monitoring is not in production: it is on parole without a parole officer.
Related terms
- MLOps · MLOps is the set of practices that takes machine learning models to production and keeps them there: versioning, deployment, monitoring and retraining.
- LLMOps · MLOps applied to language models: evaluating, monitoring and controlling the cost and quality of LLMs, RAG and agents in production.
- Data observability · Continuous monitoring of data freshness, volume, schema and distribution to catch problems before they reach dashboards.
- Eval alerting · The operating threshold on an eval score: when a quality drop wakes a person up, and with what severity.
- Model collapse · Progressive quality decay of an AI model trained recursively on data generated by other models, not people.