Scaling Laws Explained: Why Bigger Models Keep Getting Better

Imagine training a model for months, spending millions of dollars, and knowing roughly how good it will be before you press start. That sounds like fortune telling, but it is close to what happens in modern AI. The reason is a set of empirical regularities called scaling laws, and they quietly shaped almost every model you use.

• Loss falls smoothly
• Before training even starts

In the previous lesson we saw how GPT is trained by predicting the next token. Today we ask a different question: what happens when you make that recipe bigger? We will define the key quantities, derive a famous compute formula, estimate GPU-days for a real-sized model, and ask how big a model should be for a given budget.

• Power laws and compute
• Chinchilla and over-training
• Limits of the laws

Four symbols will carry the whole lesson. N is the number of parameters in the model. D is the number of tokens it trains on. C is the total training compute, counted in floating point operations, or FLOPs. And L is the loss on held-out text, the cross-entropy we met in the ML course. Scaling laws relate these four.

• N: parameters
• D: training tokens
• C: compute in FLOPs
• L: loss

Here is the central form. Loss as a function of parameters looks like L of N equals N sub c divided by N, all raised to the power alpha. N sub c is just a fitted constant, and alpha is a small positive exponent, well under one. Bigger N means smaller loss, but with diminishing returns: each step down costs more than the last.

• L(N) = (N_c / N)^α
• Small α, steady gains

Why do researchers love power laws? Take the logarithm of both sides. Log L equals alpha log N sub c, minus alpha log N. That is a straight line, with slope minus alpha. So on a log-log plot, a power law is a ruler-straight line, and you can extrapolate it by eye. That is the secret to predicting a big run from small ones.

• Take logs of both sides
• Slope equals minus α

Let us make that concrete with an illustrative exponent of zero point zero eight. Multiply parameters by ten, and loss gets multiplied by ten to the minus zero point zero eight, about zero point eight three. That is roughly a seventeen percent drop. Merely doubling parameters multiplies loss by about zero point nine five, a drop near five percent. Small exponents mean slow, steady, reliable progress.

• α = 0.08 example
• 10x params: loss times 0.83
• 2x params: about 5% lower

The same pattern shows up along three axes: parameters, data, and compute. One common way to combine them says loss equals an irreducible floor E, plus a term that shrinks with parameters, plus a term that shrinks with data. The floor reflects the natural randomness of text; even a perfect model cannot predict every next word.

• Loss vs parameters
• Loss vs data
• Loss vs compute

Notice the two separate terms. If you make the model enormous but feed it very little data, the data term dominates and you stop improving. If you pour huge data into a tiny model, the parameter term dominates. Progress needs both to grow, which is exactly why the question of how to split a budget becomes so important.

• Big model, tiny data: stuck
• Big data, tiny model: stuck

Compute is the currency of training. A FLOP is one floating point operation, one addition or one multiplication. Total compute C is the count of all such operations in the run, and it translates directly into GPU time, electricity, and money. So we want a simple rule linking C to the things we choose: N and D.

• Compute = money and time
• FLOPs count the arithmetic

Here is the derivation sketch. During a forward pass, almost every parameter takes part in one multiply and one add for each token: a weight times an activation, accumulated into a sum. That is two FLOPs per parameter, so the forward pass costs about two N FLOPs per token. We ignore smaller terms like attention over long contexts.

• Each weight: multiply and add
• Forward costs about 2N FLOPs

Backpropagation, from the ML course, needs two matrix multiplications per layer. One computes the gradient with respect to the layer's inputs, to pass the error backward. The other computes the gradient with respect to the weights, for the update. Each costs about as much as the forward pass, so the backward pass is roughly four N FLOPs per token.

• Gradient wrt activations: 2N
• Gradient wrt weights: 2N

Add them up: two N forward plus four N backward gives six N FLOPs per token. Multiply by the D tokens you train on, and you get the famous rule, C approximately equals six times N times D. It is an approximation, not an exact count, but it is accurate enough to plan a training run on the back of an envelope.

• 2N + 4N = 6N per token
• Times D tokens
• C ≈ 6ND

A few honest caveats. Attention adds extra work that grows with context length. Real GPUs never hit their peak speed, because memory movement and communication get in the way. And tricks like activation recomputation add extra forward passes. Six N D counts the useful math; the hardware efficiency we handle separately in the next example.

• Attention adds extra cost
• Real hardware wastes cycles
• Recompute raises the bill

Time for a worked example. Take a hypothetical seven billion parameter model trained on two trillion tokens. So N is seven times ten to the ninth, and D is two times ten to the twelfth. This is not a specific real model; it is just a plausible size for us to practice the arithmetic with.

• N = 7 × 10^9
• D = 2 × 10^12
• Hypothetical model

Step one, total FLOPs. Six times seven billion is forty-two billion, or four point two times ten to the tenth FLOPs per token. Multiply by two trillion tokens and we get eight point four times ten to the twenty-second FLOPs. That is a staggering number, but it is just one multiplication.

• 6 × 7e9 × 2e12
• = 8.4 × 10^22 FLOPs

Step two, state our assumptions clearly. Suppose each GPU has a peak of about three hundred twelve trillion operations per second in low precision, similar to an A100 class chip. Real training rarely exceeds a fraction of peak, so assume forty percent utilization. Effective speed is then about one point two five times ten to the fourteenth FLOPs per second per GPU.

• Assume 312 TFLOP/s peak
• Assume 40% utilization
• About 1.25 × 10^14 FLOP/s

Step three, divide. Eight point four times ten to the twenty-second, over one point two five times ten to the fourteenth, gives about six hundred seventy million GPU-seconds. Divide by thirty-six hundred for roughly one hundred eighty-seven thousand GPU-hours, and by twenty-four for about seven thousand eight hundred GPU-days.

• 8.4e22 / 1.25e14 = 6.7e8 s
• About 187,000 GPU-hours
• About 7,800 GPU-days

Nobody trains on one GPU. Spread the work across one thousand twenty-four GPUs, and seven thousand eight hundred GPU-days becomes about seven and a half days of wall-clock time, if scaling were perfect. In practice there are failures, restarts, and communication overheads, so treat it as a lower bound and a planning estimate, not a promise.

• 1,024 GPUs in parallel
• About 7.6 days wall-clock
• Assumes perfect scaling

Suppose you have a fixed compute budget. Because C is about six N D, you can trade model size against data: a bigger model trained on fewer tokens, or a smaller model on more tokens, for the same cost. Draw the rectangle with area C. The question is which shape gives the lowest loss.

• Fixed C
• Choose N and D
• Big model or more data?

The first large study, from Kaplan and colleagues in 2020, concluded that with more compute, most of the extra budget should go into bigger models, with data growing much more slowly. Many labs followed that advice, producing very large models trained on relatively few tokens by today's standards.

• Early scaling-law study, 2020
• Suggested growing N faster
• Many models under-trained

Then came Chinchilla, in 2022. DeepMind retrained the question carefully across many sizes and found that parameters and tokens should grow together. Their seventy billion parameter model, trained on about one point four trillion tokens, outperformed the much larger two hundred eighty billion parameter Gopher, which saw only around three hundred billion tokens, using a similar compute budget.

• 2022 DeepMind result
• 70B model on 1.4T tokens
• Beat a 280B model on 300B tokens

A handy summary is that the compute-optimal number of tokens is roughly twenty times the number of parameters. Check it: twenty times seventy billion is one point four trillion, matching Chinchilla. Combine it with six N D and you can solve for the best N and D for any budget C. Remember, this is a rule of thumb, not a law of nature.

• D ≈ 20 N
• Rule of thumb only
• 70B → 1.4T tokens

Apply it to our seven billion model. Twenty times seven billion is one hundred forty billion tokens, costing six times seven billion times one hundred forty billion, about five point nine times ten to the twenty-first FLOPs. Our earlier two trillion token run used roughly fourteen times more compute, and about two hundred eighty-six tokens per parameter. So why would anyone do that?

• 20 × 7B = 140B tokens
• C = 6 × 7e9 × 1.4e11
• ≈ 5.9 × 10^21 FLOPs

Chinchilla optimizes for one thing: the lowest loss for a given training budget. But a deployed model is trained once and then served for months, to enormous numbers of users. Every request costs compute proportional to N. Over its lifetime, serving can dwarf the training bill, so a smaller model has a big advantage.

• Train once
• Serve billions of requests
• Inference cost scales with N

So teams deliberately over-train: pick a smaller model and push far past twenty tokens per parameter, spending extra training compute to reach quality that a bigger model would have given. Recent open models of a few billion to tens of billions of parameters have been trained on well over ten trillion tokens, which is hundreds of tokens per parameter.

• Smaller N, much more D
• Same quality target
• Cheaper every request

Treat the twenty to one rule with care. The best ratio depends on data quality, the architecture, and how the curves were fitted. And going beyond it is not wasted: loss keeps improving past twenty tokens per parameter, just with diminishing returns. The right ratio depends on how much inference you expect to do.

• Depends on data quality
• Depends on architecture and fit
• Returns diminish, but continue

Loss falls smoothly, yet some abilities seem to appear suddenly as models grow, which people call emergence. Later work argued that many of these jumps depend on the metric: an all-or-nothing score like exact match can hide steady underlying progress. The debate is unsettled, so be cautious when someone promises that a specific skill will appear at a specific size.

• Loss: smooth curve
• Some tasks: sudden jumps
• Debate: real or metric artifact?

Scaling data runs into a limit: the supply of high-quality human text is finite. Repeating the same data for several passes helps, but with diminishing value. Labs are exploring filtering, synthetic data, and other modalities. How far these routes can go is an open research question, so we should not claim a firm answer.

• High-quality text is finite
• Repeating data helps less
• Synthetic data: promising, unproven

There is also a newer axis. Instead of only spending compute during training, a model can spend more at answer time, by generating longer reasoning before it replies, or by sampling several attempts. Reasoning models exploit this, and results suggest accuracy on hard problems can rise with thinking budget. We will study that in the chain-of-thought lesson.

• Spend compute when answering
• Think longer, answer better
• A new scaling axis

Now let us code it. Everything here is synthetic, generated by us, not real measurements. We create twelve model sizes spaced logarithmically between a million and ten billion with numpy, then build losses from a known power law with alpha equal to zero point zero eight, and add a little multiplicative noise.

• Clearly synthetic
• Known α and noise
• No real benchmark numbers

To recover the law, take logs of both columns and call numpy polyfit with degree one. The slope is minus alpha, and the intercept equals alpha times log N sub c, so N sub c is the exponential of intercept divided by alpha. Run it, and the fitted alpha lands very close to the true zero point zero eight we planted.

• np.polyfit on logs
• Slope gives −α
• Intercept gives N_c

The payoff is extrapolation: evaluate the fitted line at ten times the largest size and you have a prediction. The honest practice is to hold out your largest runs and check the prediction against them. Common mistakes are fitting on a narrow range, ignoring an irreducible loss floor that bends the curve, and trusting the line far beyond your data.

• Predict a 10x larger model
• Compare with held-out point
• Watch for the loss floor

Now the limits. Scaling laws are empirical: they are curves fitted to experiments, not theorems. They may bend or break outside the tested range. They predict loss, not usefulness, safety, or truthfulness. Different data and architectures shift the constants. Use them as strong planning tools, but keep checking with small experiments.

• Empirical fits
• No guarantee beyond data
• Loss is not everything

Let us recap. Loss falls as a power law in parameters, data, and compute, appearing as straight lines on log-log plots. Training compute is about six N D, from two N forward and four N backward per token. Chinchilla suggests about twenty tokens per parameter for training efficiency, while real models over-train to make serving cheaper. Data limits and test-time compute open new questions.

• L falls as a power law
• C ≈ 6ND
• ~20 tokens per parameter, then over-train

A trained model gives us probabilities for the next token, but how do we turn them into text? In the next lesson we cover sampling: greedy decoding, temperature, top-k and top-p, with real numbers.

• Greedy, temperature, top-k, top-p
• Subscribe for the series