Reinforcement Learning, Causal Inference and MLOps: The ML Capstone

Every model in this course so far learned from a fixed table of data. But what if the data depends on what the model does? And what happens after you train a model, when the real world keeps changing underneath it? This final episode answers both questions.

• Learning by acting
• Advanced ideas
• Shipping to production

Last time we studied transformers and modern language models. Today we cover reinforcement learning, a set of advanced tools for uncertainty, causality and fairness, and then the practical craft of MLOps. We finish with capstone projects and a study roadmap you can follow on your own.

• Reinforcement learning
• Advanced topics
• MLOps and capstones

In reinforcement learning there is no teacher giving correct answers. An agent observes a state, takes an action, and the environment replies with a reward and a new state. Think of training a dog: you never show it the right move, you just reward good outcomes and let it figure out the rest.

• Agent and environment
• State, action, reward
• No labels given

We formalize this as a Markov decision process, made of states, actions, transition probabilities, a reward function and a discount factor called gamma. The Markov property says the next state depends only on the current state and action, not on the whole history. That one assumption makes the mathematics tractable.

• States S, actions A
• Transition probabilities P
• Reward R, discount gamma

The agent does not want one big reward, it wants the largest total reward over time. We call this the return: the first reward, plus gamma times the second, plus gamma squared times the third, and so on. With gamma equal to zero point nine and rewards of one, one, and ten, the return is one plus zero point nine plus eight point one, which is ten.

• G = r1 + gamma r2 + gamma^2 r3 ...
• Gamma near 0: short-sighted
• Gamma near 1: far-sighted

A policy is the agent's strategy: a rule that picks an action in each state. The state value is the expected return if you start in that state and follow the policy. The action value, called Q, is the expected return if you take a specific action first and then follow the policy.

• Policy pi: state to action
• V(s): expected return
• Q(s,a): value of an action

Here is the key insight of the field. The value of a state equals the immediate reward plus the discounted value of where you land next. That turns an infinite-horizon problem into a one-step relationship that refers to itself, just like a recursive function.

• Value = reward + future value
• Recursive structure
• Break long horizon into one step

For the best possible behavior we use the Bellman optimality equation. The optimal Q of a state and action equals the expected reward plus gamma times the maximum optimal Q over actions in the next state. Once you know Q star, the optimal policy is simple: in every state, pick the action with the highest Q value.

• Q*(s,a) = r + gamma max Q*(s',a')
• Max over next actions
• Optimal policy: pick argmax

Let us try a tiny example. Two states in a line: from state A you can move to state B for reward zero, and from B you reach the goal for reward ten. Start with all values at zero. After one sweep B is worth ten, after the next sweep A is worth gamma times ten, which is nine with gamma zero point nine. The reward information propagates backwards one step at a time.

• Start all values at 0
• Sweep and update
• Values converge

Value iteration needs to know the transition probabilities, but in the real world we usually do not. Q-learning learns from raw experience instead. After each step, it nudges the current estimate toward a better target built from the observed reward and the best next value.

• Unknown transitions
• Learn from experience
• Temporal difference update

The update rule reads: new Q equals old Q plus alpha times the difference between the target and the old Q. The target is the reward plus gamma times the best next Q. That difference is the temporal difference error, and it plays the same role as the gradient error signal in earlier episodes.

• Q <- Q + alpha (target - Q)
• Target = r + gamma max Q(s',a')
• TD error = target - Q

Suppose the current Q is two, you receive reward one, and the best next Q is five. With gamma zero point nine, the target is one plus four point five, which is five point five. The error is five point five minus two, which is three point five. With alpha equal to zero point one, the new Q is two plus zero point three five, which is two point three five.

• Q = 2, r = 1
• max Q(s') = 5
• alpha 0.1, gamma 0.9

If the agent always picks its current best action, it may never discover something better. If it always acts randomly, it never uses what it learned. Epsilon-greedy balances the two: with probability epsilon pick a random action, otherwise pick the best one. Usually epsilon starts high and decays over training.

• Exploit: best known action
• Explore: try something new
• Epsilon-greedy

Beginners often hit the same traps. A badly designed reward can be exploited: an agent rewarded for speed might find a glitch rather than finish the task. Rewards that arrive rarely make learning very slow. And decaying epsilon too fast freezes the agent into a poor habit before it has seen enough.

• Reward hacking
• Sparse rewards
• Too little exploration

A table of Q values works for small grids, but not for images or continuous states, where the number of states is astronomical. So we replace the table with a neural network that takes a state and outputs a Q value for each action. This is the idea behind the deep Q network, or DQN.

• Tables explode in size
• Network approximates Q
• Generalize across states

Training a network on its own moving targets is unstable. DQN uses two tricks. A replay buffer stores past experiences and samples random mini-batches, which breaks the correlation between consecutive steps. A separate target network, updated only occasionally, keeps the regression target from shifting every step.

• Replay buffer
• Target network
• Decorrelate the data

Instead of learning values and deriving a policy, policy gradient methods learn the policy itself, a network that outputs action probabilities. The core idea is simple: if an action led to a high return, increase its probability; if it led to a low return, decrease it. The update is proportional to the gradient of the log probability of the action, scaled by the return.

• Learn the policy directly
• Raise probability of good actions
• Works for continuous actions

Policy gradients are noisy, because a single episode's return is a rough signal. Subtracting a baseline, such as a value estimate, reduces the variance without changing the expected direction. Actor-critic methods combine both worlds: an actor network chooses actions while a critic network estimates values to judge them. Modern algorithms such as PPO build on this idea.

• Returns are noisy
• Subtract a baseline
• Actor picks, critic judges

Now to advanced topics. Maximum a posteriori estimation, or MAP, picks the parameter that maximizes the posterior: likelihood times prior. Taking the logarithm turns this into the log likelihood plus the log prior. With a Gaussian prior on the weights, the log prior becomes a squared penalty, so MAP is exactly ridge regression. Regularization is a belief about the weights.

• MLE plus a prior
• Posterior is proportional to likelihood times prior
• Prior acts as regularization

A Gaussian process goes further by placing a distribution over entire functions. Given a few data points, it predicts a mean curve and an uncertainty band that is narrow near observations and wide far from them. The kernel decides how smooth the functions are, and how far information travels between nearby inputs.

• Distribution over functions
• Mean and uncertainty band
• Kernel sets smoothness

That uncertainty is useful. Bayesian optimization uses it to choose the next experiment, trading off trying promising settings against exploring unknown ones, which is the same exploration idea from reinforcement learning. The main limitation is cost: exact Gaussian processes scale with the cube of the number of data points, so they shine on small datasets.

• Bayesian optimization
• Cost grows cubically
• Best for small data

Conformal prediction wraps any model with honest uncertainty. Hold out a calibration set and record the absolute errors. If you want ninety percent coverage, take the ninetieth percentile of those errors, say four units, and report each new prediction plus or minus four. Under the assumption that new data is exchangeable with calibration data, the true value falls inside the interval about ninety percent of the time.

• Calibration set
• Prediction sets with coverage
• No distribution assumptions on model

Prediction asks what will happen. Causal inference asks what would happen if we intervened. Ice cream sales and sunburn rise together, but ice cream does not cause sunburn; hot sunny weather causes both. That hidden third variable is called a confounder, and ignoring it leads to confident but wrong decisions.

• Confounders
• Ice cream and sunburn
• Prediction versus intervention

The gold standard is the randomized experiment. By assigning treatment at random, we break the link between the treatment and any confounder. If the treated group averages a conversion rate of twelve percent and the control group averages ten percent, the estimated average treatment effect is two percentage points.

• Random assignment
• Treatment versus control
• Average treatment effect

When experiments are impossible, we use observational data and adjust for confounders by comparing like with like, by stratifying, matching, or including them in a regression. This only works if we measured every important confounder, an assumption that no amount of data can verify. The common mistake is to add every available variable, including ones affected by the treatment, which can create bias instead of removing it.

• Observational data
• Stratify or regress
• Needs no hidden confounders

Every guarantee we met in this course assumed the test data looks like the training data. In practice it often does not. Covariate shift means the inputs change, label shift means the class proportions change, and concept drift means the relationship between inputs and labels itself changes. A fraud model trained last year may struggle against new tactics today.

• Covariate shift
• Label shift
• Concept drift

Anomaly detection looks for the rare points that do not fit. The simplest version models normal behavior and flags anything far from it, for example a value more than three standard deviations from the mean. Methods such as isolation forests and autoencoders extend this idea to many dimensions. Because anomalies are rare and labels scarce, evaluate carefully and expect many false alarms.

• Model normal behavior
• Score distance from normal
• Isolation forest, autoencoder

A model can have high overall accuracy and still treat groups very differently. We can compare error rates, selection rates, or calibration across groups. Importantly, several common fairness definitions cannot all be satisfied at once when base rates differ between groups. So fairness is not one number to optimize; it requires choosing what matters for the application, and involving the people affected.

• Groups can see different error rates
• Metrics can conflict
• Context and humans matter

Models can also be fragile. Adding a carefully chosen, nearly invisible perturbation to an image can flip a classifier's prediction with high confidence. Robustness means testing beyond clean data: noisy inputs, unusual cases, and deliberate attacks. Training with augmented or adversarial examples helps, but no model is robust to everything.

• Tiny perturbations fool models
• Adversarial examples
• Test under stress

A model in a notebook is not a product. In production, someone must reproduce the training, serve predictions reliably, and notice when quality decays. MLOps is the set of practices that treats a machine learning system as software plus data plus models, all of which change over time.

• Notebook is not a product
• Models decay
• Reproducibility

If you cannot recreate a model, you cannot debug it. Version your code in git, your datasets with snapshots or content hashes, and your configuration and random seeds. Experiment trackers record each run's parameters, metrics and artifacts. A model registry then stores approved versions with the exact inputs that produced them.

• Code in git
• Data snapshots and hashes
• Config and random seeds

A classic production bug is training-serving skew: the preprocessing used at training differs from the one used at prediction time. Packaging the scaler, encoder and model into one pipeline object avoids this, and also prevents leakage, as we saw in earlier episodes. Add automated tests for data schema, value ranges and minimum model quality before anything is deployed.

• Same preprocessing at serve time
• Pipeline objects
• Tests for data and model

The most common way to deploy is behind an API. A small web service loads the saved model once at startup, validates the incoming request, runs the pipeline, and returns the prediction as JSON. With a framework like FastAPI this is a few dozen lines, and the service can be packaged in a container so it runs identically anywhere.

• Load model once
• Validate input
• Return JSON prediction

Choose online serving when you need an answer in milliseconds, and batch scoring when predictions can be computed overnight. Measure latency and throughput, and keep a budget. Roll out new models gradually, sending perhaps five percent of traffic to the new version first, a canary release, and keep the ability to roll back instantly if metrics get worse.

• Online versus batch
• Latency budgets
• Canary and rollback

Once deployed, monitor three things: system health, such as errors and latency; data quality and drift in the inputs; and model performance. Labels often arrive late, so input and prediction distributions act as early warnings. Compare the live distribution of each feature to a training reference using a statistical test or a distance measure, and alert when it crosses a threshold.

• Input drift
• Prediction drift
• Performance when labels arrive

Here is a simple drift check. Split a feature into bins using the training data. Suppose bin one held forty percent of training data but now holds twenty percent, while bin two grew from sixty to eighty percent. Big shifts in bin proportions produce a high drift score, such as a population stability index. A common practice is to treat small scores as stable, and investigate large ones, but thresholds should be tuned to your application.

• Population stability idea
• Bin the feature
• Compare proportions

When drift appears, you can retrain on a schedule, or trigger retraining when monitors fire. Either way, treat the new model like any release: evaluate it on fresh held-out data, compare it with the current model, and only promote it if it is truly better. Retraining blindly on bad data can make things worse.

• Scheduled retraining
• Triggered by drift
• Validate before promoting

Reading and watching only take you so far. Capstone projects force you to face messy data, ambiguous goals and trade-offs, which is where real skill grows. Each project below should include a clear question, a baseline, a careful evaluation, and a short write-up of what failed.

• Projects prove skill
• End-to-end thinking
• Portfolio and confidence

Start with tabular projects. Predict house prices with regularized regression and gradient boosting, and explain the result with SHAP. Build a churn classifier and choose the threshold using business costs. Then build a credit risk model, add calibration and a fairness audit across groups, and write down the limitations.

• House price regression
• Customer churn classifier
• Credit risk with fairness check

Next, train a convolutional network on an image dataset and test it under noise and shift. Fine-tune a pretrained transformer for text classification and compare it with a simple bag of words baseline. Build a demand forecasting model with proper time-based validation, and give prediction intervals using conformal methods.

• CNN image classifier
• Fine-tune a transformer
• Demand forecasting

For reinforcement learning, code Q-learning from scratch on a grid world, then train a DQN on a simple control task. Finally, take your best model from any project and deploy it: wrap it in an API, containerize it, log predictions, and build a drift dashboard. A model that someone else can call and trust is the true capstone.

• Q-learning grid world
• DQN on a simple game
• Deploy with API and monitoring

Here is how to continue. First, make sure the foundations are solid: redo derivations for linear regression, logistic regression and backpropagation by hand. Second, implement algorithms from scratch, then compare with library versions. Third, choose a direction and go deep, such as deep learning research, applied tabular modeling, reinforcement learning, causal inference or ML engineering.

• Solidify foundations
• Build from scratch
• Specialize deeply

Habits matter more than any single resource. Read papers slowly and try to reproduce a result from each one. Keep a notebook of mistakes, because debugging is most of the job. And explain concepts to others, since teaching reveals exactly where your understanding is thin.

• Read papers slowly
• Reproduce results
• Teach what you learn

Let us recap. Reinforcement learning maximizes return in a Markov decision process, with Bellman equations linking values across time. Q-learning learns values from experience, DQN scales it with networks, and policy gradients learn the policy directly. Among advanced tools, MAP is regularization as a prior, Gaussian processes and conformal prediction quantify uncertainty, and causal inference separates intervention from correlation.

• MDP, return, Bellman
• Q-learning, DQN, policy gradient
• MAP, GP, conformal, causal

Real systems face distribution shift, fairness concerns and fragile inputs, so we test for them. MLOps means versioning code, data and configs, serving through an API, monitoring drift, and retraining safely. Together these turn a clever model into a dependable one.

• Watch shift and fairness
• Version everything
• Serve, monitor, retrain

You started this series with the question, what is machine learning, and you now understand the mathematics, the algorithms and the systems behind it. Pick one capstone and start building today. Next, we will explore new deep dives and projects beyond this course. Thank you for learning with me.

• 30 episodes complete
• Subscribe for more
• Build something today