MLOps: Reproducible Pipelines, Serving, Deployment and Monitoring

You trained a model. The validation score looks great, and the notebook runs perfectly on your laptop. Then someone asks you to put it in front of real users, and suddenly nothing is simple. Today we learn how to ship a model and keep it alive.

• Great score is not a product
• Production is a different problem

Ordinary software fails loudly. A bug throws an error and a page turns red. A machine learning system fails quietly. It keeps returning numbers, but the numbers slowly become wrong because the world changed. MLOps is the set of habits that make those silent failures visible.

• Wrong numbers, no errors
• MLOps makes failure visible

In lesson 62 we studied uncertainty, shift, fairness and robustness. Now we turn those worries into engineering. We will version everything, track experiments, save models safely, test them, serve them, monitor them, and finally count the cost and the security risks.

• Version, track, save, test
• Serve, monitor, secure

A result is reproducible if someone else, or you in six months, can rerun it and get the same answer. That needs four things to be pinned down: the code, the data, the configuration, and the environment. Miss any one and the result becomes folklore.

• Code, data, config, environment
• Miss one and it breaks

Git records every change to your code as a commit with a unique hash. Given a commit hash, you can recover the exact code that produced a model. A common mistake is training from uncommitted changes, so the model points to a commit that does not match what really ran.

• Hash identifies exact code
• Never train on dirty working tree

Git is bad at big files, so data is versioned with a tool like DVC. The large dataset lives in storage, while git tracks a tiny pointer file holding a content hash. Check out an old commit, pull the matching data, and you have the exact training set from that day.

• Git stores the pointer
• Storage holds the big data

Experiment tracking tools, like MLflow, log each training run. For every run you record parameters, metrics, the git commit, the data hash, and the saved model. Suppose you try three learning rates and get validation accuracy of 0.91, 0.94 and 0.89. The log tells you which run won and exactly how to repeat it.

• Params, metrics, commit, data hash
• Compare runs, repeat the winner

Randomness is the next enemy. Set the random seed for shuffling, splits, and initialization, and record it with the run. Also freeze library versions in a lock file, because a minor update to a library can change results. Even then, some GPU operations are not perfectly deterministic, so expect small differences.

• Record the random seed
• Pin library versions

A trained model is just numbers and structure in memory. Serialization writes it to a file so another process can load it later. The key question is what format you choose, because the format decides portability, speed and safety.

• Memory to file and back
• Format decides safety and portability

For scikit-learn models, joblib is the convenient choice. It handles large numpy arrays efficiently. Under the hood it builds on pickle, Python's general object format. Both only work reliably when you load with the same library versions that saved the file.

• Joblib suits scikit-learn
• Same versions on load

Here is the serious danger. Loading a pickle file does not just read data, it can execute arbitrary code embedded in the file. If you load a pickle from an untrusted source, an attacker can run commands on your machine. Never unpickle files you did not create, and treat downloaded model files with the same suspicion as downloaded programs.

• Loading can execute code
• Never load untrusted pickles

ONNX is an open format that stores the model as a graph of operations, with no arbitrary code. You can train in one framework and run it in a fast runtime elsewhere. Safer weight-only formats exist for deep learning too. The trade-off is that exotic custom operations may not convert, so always check the converted model gives the same predictions.

• Graph of operations, no code
• Verify outputs match

Normal software tests check code. ML systems have three moving parts, code, data and model behavior, so we test all three. A pipeline can have perfectly correct code and still produce garbage because the data changed or the model learned something strange.

• Code tests
• Data tests
• Model behavior tests

Data tests check the contract of your inputs. Does the age column exist, is it numeric, is it between 0 and 120, and is the share of missing values below five percent? Run these checks before training and before serving. A single bad batch with ages stored in months instead of years can wreck predictions without any crash.

• Check type, range, missing rate
• Run before training and serving

Unit test your feature functions with tiny hand-made inputs. If a function scales income, feed it three numbers and check the output by hand. Also test that the pipeline applies the same transform at training and serving time. A mismatch between the two, called training serving skew, is one of the most common production bugs.

• Small hand-checked inputs
• Share one feature function

Behavior tests ask whether the model acts sensibly. Check that accuracy beats a simple baseline, that predictions on a fixed golden set do not change unexpectedly, and that small harmless changes to an input do not flip the answer. If a house gets one square foot larger and the predicted price falls by half, something is wrong. Also check metrics on important subgroups, as we discussed in lesson 62.

• Beat a baseline
• Golden set and invariance checks

A reproducible pipeline is a chain of named steps: ingest, validate, split, featurize, train, evaluate, package. Each step reads declared inputs and writes declared outputs. Tools like DVC pipelines or workflow orchestrators rerun only the steps whose inputs changed, which saves time and keeps results consistent.

• Declared inputs and outputs
• Rerun only what changed

Recall lesson 35: fit scalers and encoders on training data only. The safest way is to bundle preprocessing and the model into a single pipeline object, then save and serve that one object. If you save the model alone, the server will have to rebuild preprocessing by hand, and mistakes creep in.

• Bundle preprocessing with model
• Save and serve a single artifact

Continuous integration runs your tests automatically on every change. A typical gate: unit tests pass, data checks pass, and the new model scores at least as well as the current one on a fixed evaluation set. If any gate fails, the change does not ship. This turns good intentions into a rule.

• Automatic tests on each change
• Metric gate protects quality

A model registry is a catalogue of trained models with versions and stages such as staging and production. Each entry links back to the run, code commit and data version. When something goes wrong, you can roll back to the previous version in minutes instead of retraining from scratch.

• Versions and stages
• Fast rollback

There are two main ways to serve predictions. Batch inference scores many rows on a schedule, such as nightly churn scores written to a table. Online inference answers one request at a time, in milliseconds, such as a fraud check during payment. Batch is cheaper and simpler, so choose online only when freshness truly matters.

• Batch: scheduled, cheap
• Online: instant, per request

FastAPI lets you wrap a model in a web endpoint with very little code. You load the saved pipeline once at startup, define a function for the route slash predict, and return the prediction as JSON. Loading once matters. If you reload the model on every request, latency explodes.

• Load model once at startup
• Return JSON prediction

Never trust the input. With a Pydantic schema, you declare that area must be a number greater than zero and bedrooms must be an integer from zero to twenty. If a request sends the text banana, the server rejects it with a clear error before the model sees it. Validation prevents crashes and blocks many abuse attempts.

• Schema declares allowed values
• Bad input rejected early

Docker packages your app with its exact environment into an image. A container is a running instance of that image. You write a Dockerfile: start from a base Python image, copy a requirements file, install packages, copy your code and model, then set the start command. The same image runs identically on your laptop and on a server.

• Image bundles code and environment
• Containers run it anywhere

Latency is how long one request takes. Do not report only the average. Suppose ten requests take 20 milliseconds, and one takes 400. The mean is about 54 milliseconds, but a user sometimes waits 400. So we report percentiles, like the median, the ninety-fifth and the ninety-ninth, because the slow tail is what people feel.

• Mean hides the tail
• Report p50, p95, p99

Throughput is requests handled per second. If one request takes 20 milliseconds on one worker, that worker handles about 50 requests per second. Four workers give roughly 200. Grouping several requests into a mini batch can raise throughput further, especially on a GPU, but it adds waiting time, so latency and throughput pull against each other.

• 1 / 0.02 s = 50 per second
• Batching trades latency for throughput

Once deployed, a model meets data it was never trained on. Monitoring watches three layers: system health like errors and latency, data quality and drift in the inputs, and model quality when true labels eventually arrive. Labels are often delayed by days or weeks, so input monitoring is your early warning.

• System, data, model quality
• Labels arrive late

Data drift means the input distribution changes, for example customers get younger. Concept drift means the relationship between inputs and the label changes, for example the same transaction pattern stops meaning fraud. Data drift may not hurt accuracy, while concept drift always does. We detect the first from inputs alone, but the second usually needs labels.

• Data drift: inputs change
• Concept drift: rules change

The population stability index compares a feature's training distribution with live data. Split the feature into bins. For each bin, take the training share e, and the live share a, then add up a minus e, times the natural log of a over e. Say a bin goes from 0.30 to 0.20. The term is minus 0.10 times log of 0.667, which is about 0.041. Common rules of thumb call below 0.1 stable and above 0.25 a major shift.

• Sum of (a - e) times ln(a / e)
• Under 0.1 stable, over 0.25 shifted

From lesson 39 we know hypothesis tests. The two sample Kolmogorov Smirnov test compares two empirical cumulative distribution curves and takes the largest vertical gap between them, called D. A big D with a tiny p value says the samples likely come from different distributions. Beware: with millions of rows, trivial shifts become significant, so also look at the size of D itself.

• D is the largest CDF gap
• Large samples flag tiny shifts

When should you retrain? You can retrain on a schedule, such as weekly, or on triggers: drift above a threshold, a drop in measured accuracy, or a business event. A common mistake is to retrain automatically and deploy without checks, which can ship a worse model. Retrained models should pass the same gates as any new model.

• Schedule or trigger
• Retrained models pass the gates

A shadow deployment runs the new model beside the old one on real traffic, but only the old model's answers reach users. You log both outputs and compare them. This tests latency, errors and prediction differences with zero user risk, and it is a great first step.

• New model sees real traffic
• Users never see its output

A canary release sends a small share of live traffic, say five percent, to the new model. If errors, latency and business metrics hold, raise it to twenty five, then fifty, then all. If anything looks wrong, route everything back to the old version. A small blast radius turns a disaster into a minor incident.

• Start with 5 percent traffic
• Roll back on any alarm

Bigger models are not always better business. Suppose a model scores 0.92 and costs 50 milliseconds and a GPU, while a smaller one scores 0.91 and costs 5 milliseconds on a cheap CPU. One point of accuracy may not justify ten times the latency and a much larger bill. Techniques like quantization, distillation, caching and batch scoring cut cost, so always measure the trade-off.

• Accuracy versus latency and cost
• Quantize, distill, cache

Training data often holds personally identifiable information, like names, emails and addresses. Collect only what you need, remove or hash identifiers, and keep logs free of raw personal data. Models can also memorize rare training records and leak them. Follow the privacy laws and company rules that apply to you, and involve the experts.

• Minimize and remove identifiers
• Keep personal data out of logs

An attacker with unlimited access to your prediction endpoint can query it thousands of times and train a copy. This is called model extraction. Defenses include authentication, rate limits, monitoring unusual query patterns, and returning less detail, like labels rather than full probabilities. Add input validation and keep dependencies updated as well.

• Many queries can clone a model
• Authenticate and rate limit

Let us put it all together. Data sources feed validated pipelines. Training runs are tracked, tested and stored in a registry. A CI gate promotes a model to a container behind an API, deployed through shadow and canary stages. Logs flow into monitoring for latency, drift and quality, and alarms can trigger retraining, closing the loop.

• A loop, not a line
• Monitoring feeds retraining

The mistakes repeat. Training from uncommitted code, loading untrusted pickles, reloading the model on every request, reporting only average latency, skipping data checks, and deploying retrained models without gates. Each has a simple fix, and each fix is a habit you can start today.

• Pin, test, validate, gate
• Measure the tail, not the mean

To recap: version code, data and config, track every run, save models in safe formats, test data, code and behavior, serve with validation and containers, measure latency and throughput, monitor drift with PSI and the KS test, release with shadow and canary, and respect cost and security. Next, in lesson 64, we start the capstones with house prices and customer churn.

• Version, test, serve, monitor
• Next: capstone projects