Time Series and Forecasting: ARIMA, Smoothing and Leakage Explained

Imagine someone hands you a table of daily sales and asks you to predict next week. If you shuffle the rows first, you destroy the one thing that matters: the order. Time series data has memory, and today is tied to yesterday. In this video we learn how to read that memory and use it to forecast.

• Order matters
• Today depends on yesterday

Last episode we learned how to evaluate and select models without fooling ourselves. This time that lesson gets harder, because time breaks the usual rules of random splitting. We will cover decomposition, lags, autocorrelation, stationarity, the ARIMA family, exponential smoothing, honest validation, error metrics, and the leakage traps that catch even experienced people.

• Last time: model evaluation
• Now: data with a clock

A time series is a list of values recorded in order, usually at equal spacing, like monthly sales or hourly temperature. We write the value at time t as y sub t, so y sub one is the first observation, y sub two the second, and so on. Forecasting means estimating y sub t plus h, where h is how many steps ahead we look. The key assumption we will test is that the past carries information about the future.

• Values indexed by time
• Equal spacing is typical
• Notation: y sub t

Most series can be pulled apart into three ingredients. The trend is the slow, long-term direction, like a business growing year after year. Seasonality is a pattern that repeats on a fixed calendar, like a spike every December. The residual is whatever is left over, the noise and surprises that neither of the first two can explain.

• Trend
• Seasonality
• Residual

The additive model says y sub t equals trend plus seasonal plus residual. Suppose in some month the trend is one hundred, the December seasonal effect is plus twenty, and the residual is minus three. Then the observed value is one hundred plus twenty minus three, which is one hundred seventeen. Additive fits best when the seasonal swing stays about the same size as the level grows.

• y = T + S + R
• Seasonal swing stays constant

Sometimes the seasonal swings grow as the series grows. Then a multiplicative model fits better: y sub t equals trend times seasonal times residual. Here the seasonal factor is a ratio, so one point one means ten percent above normal. If the trend is two hundred and the factor is one point one, the seasonal value is two hundred twenty. Taking logarithms turns products into sums, so a multiplicative series becomes additive.

• y = T × S × R
• Swings grow with level

Classical decomposition works in steps. First estimate the trend by smoothing the series with a moving average. Subtract that trend, which leaves a wavy series with no long-term direction. Then average all the Januaries together, all the Februaries together, and so on, to get the seasonal pattern. Whatever remains after removing both pieces is the residual, and it should look like patternless noise.

• Estimate trend by averaging
• Remove it, average by season
• Residual is the rest

The residual is your report card. If it looks like random scatter around zero, the decomposition captured the structure. If you still see waves or drift in it, something was missed, perhaps a second seasonality or a change in trend. Later, this same idea will return when we check whether our forecasting model is any good.

• Residual should look random
• Patterns mean something missed

To feed a time series to an ordinary model, we turn the past into columns. A lag feature is just the series shifted down. Lag one holds yesterday's value, lag seven holds the value from the same weekday last week. The first few rows have no lag available and are empty, so we drop them. Now each row says: here is what happened before, and here is what we want to predict.

• Lag 1 = yesterday
• Lag 7 = same weekday last week
• Shift the column

A moving average smooths noise by averaging the most recent k values. Take the series ten, twelve, fourteen, thirteen, fifteen with a window of three. The first average is ten plus twelve plus fourteen, divided by three, which is twelve. Slide forward and you get thirteen, then fourteen. The bigger the window, the smoother the line, but the more it lags behind real changes.

• Average the last k points
• Smooths noise
• Example: window of 3

Rolling statistics such as the rolling mean and rolling standard deviation make great features. But there is a catch. When predicting day t, the window must end at day t minus one, never include day t. A safe habit is to shift the series by one step first, and then roll. We will see in the leakage section why this single detail matters so much.

• Rolling mean, rolling std
• Use only past values
• Shift before rolling

Autocorrelation asks a simple question: how strongly is the series related to a shifted copy of itself? Take each value and pair it with the value k steps earlier, then measure how well the pairs line up. If high values tend to follow high values, the lag one autocorrelation is positive. If the series jumps up and down alternately, it is negative.

• Compare series with its lag
• Lag k autocorrelation

Here is the formula. R sub k equals the sum of y sub t minus the mean, times y sub t plus k minus the mean, divided by the sum of y sub t minus the mean, squared. The top measures how deviations from the average move together at lag k. The bottom is the total variation, which scales the answer into the range from minus one to plus one. Lag zero always gives exactly one.

• r_k = Σ(y_t − ȳ)(y_{t+k} − ȳ) / Σ(y_t − ȳ)²
• Between −1 and 1

Let us compute it for the series one, two, three, four, five. The mean is three, so the deviations are minus two, minus one, zero, one, two. The denominator is four plus one plus zero plus one plus four, which is ten. For lag one, multiply neighboring deviations: two, zero, zero, and two, which sum to four. So r one is four over ten, or point four.

• Series 1,2,3,4,5
• Mean 3
• r1 = 4 / 10 = 0.4

We plot autocorrelation at every lag as bars, called the ACF plot. A shaded band around zero shows how large a bar must be before it is likely more than chance. A slow decay means a trend. Repeated bumps, say at lags twelve and twenty-four in monthly data, reveal yearly seasonality. This plot is the main tool for choosing the models later in this video.

• Bars for each lag
• Blue band = noise level
• Spikes at seasonal lags

Autocorrelation at lag two can be large simply because lag two influences lag one, which influences today. The partial autocorrelation removes those middle links and measures only the direct effect of lag k. It is like asking how much the day before yesterday matters once yesterday is already known. Together, the ACF and PACF give fingerprints of different models.

• Direct effect of lag k
• Removes indirect paths

A series is stationary when its statistical behavior does not change over time: the mean stays constant, the variance stays constant, and the autocorrelation depends only on the lag. A flat, wobbling series qualifies. A series with a trend or a growing swing does not. Stationarity matters because a model can only learn patterns that hold steady, and the classical models we study next assume it.

• Constant mean
• Constant variance
• Stable autocorrelation

Non-stationary data causes real trouble. Two unrelated series that both trend upward will show a very high correlation purely because both climb with time. That is a spurious relationship. A model trained on a rising period also learns levels that may never appear again. Making the series stationary removes this illusion and gives patterns that can truly be reused.

• Statistics drift
• Spurious relationships
• Models extrapolate badly

The main tool is differencing: replace each value with the change from the previous one. Take one hundred, one hundred three, one hundred seven, one hundred twelve. The differences are three, four, five. The trend is mostly gone, and the series now describes growth instead of level. Difference once more and you get one, one, a flat line. For seasonal data, subtract the value from one season ago, for instance twelve months back.

• Δy_t = y_t − y_{t−1}
• 100,103,107,112 → 3,4,5
• Second difference → 1,1

How do you check? Look at the plot, then the ACF: a very slow decay signals non-stationarity. Formal tests exist, such as the augmented Dickey-Fuller test, which tests for a unit root. A common mistake is differencing too much. Over-differencing adds artificial noise and creates a strong negative autocorrelation at lag one. Difference only until the series looks stable, usually once, rarely twice.

• Plot and ACF first
• Unit root tests exist
• Do not over-difference

An autoregressive model predicts the series from its own past. For order one: y sub t equals c plus phi times y sub t minus one plus an error epsilon. Here c is a constant, phi is the memory strength, and epsilon is fresh random noise. Let c be two and phi be point eight. If yesterday was fifteen, the forecast is two plus point eight times fifteen, which is fourteen. The series is pulled back toward its long-run mean, c over one minus phi, which is ten.

• y_t = c + φ y_{t−1} + ε_t
• c=2, φ=0.8
• Previous 15 → forecast 14

AR of order p simply uses p past values, each with its own weight phi. Fitting this is a linear regression on lag features, so it connects directly to what you learned about linear regression. The PACF is the clue: if it has strong spikes up to lag p and then cuts off, an AR of order p is likely. For AR one, phi must stay between minus one and one, or the series explodes instead of settling.

• Use p past values
• PACF cuts off after lag p
• Need |φ| < 1 for AR(1)

Do not confuse this with the moving average smoother. The MA model says the series is a baseline plus the current shock plus a fraction of the previous shock. In symbols, y sub t equals mu plus epsilon sub t plus theta times epsilon sub t minus one. If mu is fifty, theta is point five, and last period's surprise was four, the forecast is fifty plus two, which is fifty-two. The effect of a shock lasts only q steps for an MA of order q.

• y_t = μ + ε_t + θ ε_{t−1}
• Memory of past shocks
• μ=50, θ=0.5, ε=4 → 52

Here is the practical cheat sheet. For an AR model, the ACF decays gradually while the PACF cuts off after lag p. For an MA model, it is the opposite: the ACF cuts off after lag q while the PACF decays. Real data is messier than the textbook, so treat this as a heuristic and confirm by comparing candidate models on validation data.

• AR: ACF decays, PACF cuts off
• MA: ACF cuts off, PACF decays
• Rule of thumb only

ARMA combines both ideas. The next value depends on a weighted sum of p past values and a weighted sum of q past shocks, plus new noise. Mixing the two often describes a series with fewer parameters than a long pure AR or pure MA model would need. But ARMA still requires a stationary series, which brings us back to differencing.

• AR part: past values
• MA part: past shocks
• Fewer parameters overall

ARIMA stands for autoregressive integrated moving average. It is a three-step pipeline with three numbers. D says how many times to difference the series. P is the number of autoregressive terms and q the number of moving average terms, both applied to the differenced series. At the end the differences are added back, the integration step, to return forecasts on the original scale.

• p: AR order
• d: times differenced
• q: MA order

A worked example. Let the last observation be one hundred twenty, and the last change w be four. Take phi equal to point five, theta equal to point three, no constant, and a last error of two. The forecast change is point five times four, plus point three times two, which is two plus point six, or two point six. Add it back to the last level: one hundred twenty plus two point six gives one hundred twenty-two point six.

• w_t = y_t − y_{t−1}
• w_t = φ w_{t−1} + θ ε_{t−1}
• Forecast 122.6

In practice, a library estimates phi and theta by maximum likelihood, the idea from episode ten. You try a few orders and compare them with a criterion such as AIC, which rewards fit but penalizes extra parameters. Finally check the residuals: their ACF should show no significant bars. If structure remains, the model missed something. Seasonal ARIMA, called SARIMA, adds the same ideas at the seasonal lag.

• Fit by maximum likelihood
• Compare models with AIC
• Residuals should be white noise

Exponential smoothing takes a different route: keep a running estimate called the level, and nudge it toward each new observation. The new level equals alpha times the new value plus one minus alpha times the old level. Alpha, between zero and one, controls how fast you adapt. Unrolled, this is a weighted average of all past values in which weights shrink geometrically, so the recent past counts most.

• ℓ_t = α y_t + (1−α) ℓ_{t−1}
• Weighted average of past
• Recent matters more

Let alpha be point three, the old level fifty, and the new observation sixty. The new level is point three times sixty, which is eighteen, plus point seven times fifty, which is thirty-five. That totals fifty-three, so the level moved only partway toward sixty. The weights on recent observations are point three, then point three times point seven, which is point two one, then point one four seven, and so on.

• α = 0.3, old level 50, y = 60
• New level = 53
• Weights 0.3, 0.21, 0.147

A large alpha reacts quickly but chases noise. A small alpha is smooth but slow to notice a real shift in level. You choose alpha by minimizing squared forecast error on past data, which the software does for you. Note that simple exponential smoothing forecasts a flat line, because it has no notion of trend or season.

• High α: fast, noisy
• Low α: slow, smooth
• Fit by minimizing error

Holt's method adds a second equation that tracks the trend, so forecasts continue along a slope. The Holt-Winters method adds a third that tracks the seasonal pattern, in additive or multiplicative form. Each component is updated with its own smoothing parameter. These methods are simple, fast, and often surprisingly hard to beat, and they are a good baseline for any forecasting project.

• Add trend: Holt
• Add seasonality: Holt-Winters
• Level + trend + season

Now to evaluation. With a random train test split, a test point from Tuesday can have training neighbors from Monday and Wednesday. The model can interpolate between them, which is not forecasting. In reality you only know the past. So the split must respect time: train on earlier data, test on later data.

• Random split leaks the future
• Train on past, test on future

One split gives a noisy estimate, so we repeat it. In walk-forward validation, fold one trains on the first stretch and tests on the next block. Fold two extends the training window and tests on the block after that, and so on. You can use an expanding window or a fixed rolling window. Averaging the errors across folds gives a more trustworthy score. Scikit-learn offers TimeSeriesSplit for this.

• Expanding or rolling window
• Several folds
• Average the errors

Two refinements. If your features use long windows or the target spans several steps, insert a gap between train and test so no overlapping information crosses the boundary. And keep a final holdout period at the very end that you touch only once, after all tuning. When you are done, retrain on all available data before producing real forecasts.

• Add a gap if features look ahead
• Keep a final untouched test
• Retrain on all data to deploy

The forecast error is the actual value minus the forecast. Say actual sales are one hundred, one hundred twenty, and eighty, and the forecasts are one hundred ten, one hundred fifteen, and ninety. The absolute errors are ten, five, and ten. Their average, the MAE, is twenty-five over three, about eight point three three. It is easy to read because it is in the same units as the data, here units sold.

• e_t = y_t − ŷ_t
• MAE = mean |e_t|
• Same units as data

RMSE squares the errors first, then averages, then takes the root. For our example that is the square root of seventy-five, about eight point six six, and it punishes large misses more. MAPE divides each error by the actual value. Ten percent, four point one seven percent, and twelve point five percent average to about eight point nine percent. Percentages feel intuitive and let you compare across series of different sizes.

• RMSE punishes big misses
• MAPE = mean |e_t / y_t| × 100
• Here: about 8.9%

MAPE has well-known problems. If an actual value is zero, you divide by zero. If an actual value is tiny, say one, a forecast of eleven gives a one thousand percent error and dominates the average. It is also asymmetric: under-forecasting can never exceed one hundred percent, while over-forecasting has no limit, so MAPE quietly favors low forecasts. For intermittent demand, avoid it.

• Undefined when actual = 0
• Asymmetric: favors low forecasts
• Explodes near zero

A good error number needs context. Always build a baseline first. The naive forecast repeats the last value, and the seasonal naive repeats the value from the same point last season. The mean absolute scaled error, MASE, divides your MAE by the naive method's MAE. A value below one means you beat the baseline. If a complex model cannot beat seasonal naive, it is not ready.

• Naive forecast: repeat last value
• Seasonal naive: repeat last season
• MASE < 1 beats naive

The machine learning approach turns forecasting into ordinary supervised learning. Each row has features such as lags, rolling means, the day of the week, the month, and holidays, and the target is the next value. Then you can use linear regression, random forests, or gradient boosting. Trees are handy because they capture nonlinear effects and interactions among many extra inputs, such as price or weather.

• Features: lags, rolling stats, calendar
• Target: next value
• Any regressor works

To forecast several steps ahead there are two strategies. The recursive approach predicts one step, feeds that prediction back in as a lag, and repeats. It is simple, but errors can compound. The direct approach trains a separate model for each horizon, avoiding compounding at the cost of more models. ARIMA does the recursive style naturally, which is why its long forecasts converge toward a mean.

• Recursive: feed predictions back
• Direct: one model per horizon
• Errors can compound

So which should you choose? Classical models shine with a single series, limited data, and a need for interpretable parameters and prediction intervals. Machine learning shines when you have many related series, rich external features, and nonlinear effects. Neither wins by default. Run both under the same walk-forward validation, against the same naive baseline, and let the evidence decide.

• Few points, one series: classical
• Many series, many features: ML
• Always compare to baseline

Leakage means your model saw information that would not exist at prediction time. The result is a beautiful validation score and a disappointing production result. In time series it is especially easy to cause, because the future sits right there in the same table. Here are the most common traps.

• Training uses info from after prediction time
• Scores look great
• Production fails

Trap one is random splitting, which we have covered. Trap two is a centered moving average, which averages the points before and after day t. That is fine for describing history, but when used as a feature it quietly contains tomorrow. Trap three is a rolling feature whose window includes the current target value. Always use trailing windows, shifted so they end one step before the prediction.

• Shuffled splits
• Centered moving average uses t+1
• Window includes current target

More traps hide in preprocessing. If you fit a scaler or fill missing values using the entire series, test statistics leak into training. Fit them on the training window only, ideally inside a pipeline. Also watch for features that are not known in advance: yesterday's actual temperature is fine, but tomorrow's actual temperature is not. Use the weather forecast instead, exactly as it would be available at the time.

• Fit scaler on training only
• Lag 0 and future-known covariates
• Imputing with future values

Here is a simple checklist. Split by time first. Fit all preprocessing on training data only. Use trailing windows only. And for every feature, ask whether you would truly know it at the moment you make the forecast. If your forecast of tomorrow's sales is nearly perfect, do not celebrate. Go hunting for leakage.

• Ask: known at prediction time?
• Split by time, then preprocess
• Be suspicious of great scores

Let us recap. A series splits into trend, seasonality, and residual. Lags, moving averages, and autocorrelation reveal its memory. Differencing makes it stationary, and AR, MA, and ARIMA models describe it with a few parameters. Exponential smoothing offers a simple alternative. Validate with walk-forward splits, measure with MAE or MASE while respecting the caveats of MAPE, and guard against future leakage.

• Decompose, lag, autocorrelate
• Stationarity, differencing, ARIMA
• Smoothing, validation, errors, leakage

In the next episode, we build a neural network from scratch and derive backpropagation step by step, using the chain rule you already know.

• Episode 26: Neural networks from scratch
• Backpropagation
• Subscribe