Hyperparameter Search and Bayesian Optimization Explained With Numbers

You have a model, a dataset, and a dozen dials to set. Learning rate, tree depth, regularization strength. Every setting costs minutes or hours to test. Most people twiddle them by hand and hope. Today you will learn how to search smartly, how to avoid fooling yourself, and how a method called Bayesian optimization picks the next experiment.

• Every model has dials
• Each trial costs time
• Search needs a strategy

In lesson 44 we fit Gaussian mixtures with expectation maximization, and we had to choose the number of components by hand. That choice is exactly what this lesson is about. We will cover search spaces, grid and random search, nested cross-validation, early stopping of bad trials, Bayesian optimization, and then code it all in Python.

• Lesson 44: EM for mixtures
• Lesson 45: tuning strategy
• Goal: best model per budget

Parameters are numbers the training algorithm learns, like the weights in linear regression from lesson 14. Hyperparameters are choices you make before training starts, like the regularization strength from lesson 17 or the depth of a tree from lesson 19. Training cannot tune them by gradient descent on the training loss, because a more flexible setting would always look better there. So we judge them on validation data.

• Parameters: learned from data
• Hyperparameters: set before training
• Chosen by validation score

A search space lists each hyperparameter with its type and range. Tree depth is an integer between two and twenty. Regularization strength is a positive float. The kernel is a category: linear or radial. A good space is wide enough to contain a good answer but narrow enough that you do not waste trials. Past experience and defaults are your best guide.

• Type: integer, float, category
• Range: lower and upper bound
• Prior knowledge narrows it

Suppose the learning rate may range from zero point zero zero zero one to zero point one. If you sample uniformly, ninety percent of draws land above zero point zero one, and the small values almost never appear. Sample uniformly in the logarithm instead, and each of the three decades gets about one third of the trials. Use log scales for learning rates, regularization strengths, and anything that spans orders of magnitude.

• Learning rate: 0.0001 to 0.1
• Uniform sampling wastes trials
• Log-uniform covers each decade

Grid search picks a few values per hyperparameter and tries every combination. It is simple and reproducible. But the cost multiplies. Three values for each of four hyperparameters gives three to the fourth power, eighty-one runs. Add a fifth dial and you need two hundred and forty-three. The grid also forces you to choose values in advance, even though you do not know where the good region is.

• Try every combination
• Cost multiplies per dimension
• 3 values, 4 dials = 81 runs

Random search draws each hyperparameter independently from its distribution, using the log scale where appropriate. You decide the number of trials, whether that is ten or a thousand, and you can stop at any time. Each trial tries a fresh value of every dial. That detail turns out to matter enormously, as the next example shows.

• Sample each dial independently
• Every trial tries new values
• Budget is your choice

Imagine two hyperparameters, but only the learning rate really affects the score. A three by three grid uses nine trials, yet only three distinct learning rates, because each is repeated three times. Nine random trials give nine distinct learning rates. The effective dimension of the problem is lower than the number of dials, and random search exploits that automatically. Research by Bergstra and Bengio made this point forcefully.

• Often only a few dials matter
• Grid repeats important values
• Random explores more values

Three mistakes show up again and again. First, tuning on the test set, which destroys its meaning as an honest final check. Second, a range so narrow that the best value sits on the boundary, which is a sign you should widen it. Third, sampling a learning rate linearly. If your best trial lies at the edge of the space, the search is not finished.

• Tuning on the test set
• Too-narrow ranges
• Ignoring log scale

Here is a subtle trap. Suppose you try fifty random models whose true accuracy is exactly fifty percent, pure noise. Each gets a validation score that wobbles around fifty. The best of the fifty might show fifty-eight percent. That number is not an honest estimate of the winner. Choosing the maximum and reporting the maximum are two uses of the same data, and the report is biased upward.

• Best of many trials is lucky
• Score is biased upward
• Selection uses the same data

In lesson 24 we used cross-validation to estimate performance. Nested cross-validation adds a second loop. The outer loop splits the data into, say, five folds. For each outer training set, an inner cross-validation runs the whole search and picks the best hyperparameters. That winner is then scored once on the outer test fold, which the search never saw. The five outer scores give an honest estimate of the whole tuning procedure.

• Outer loop: honest evaluation
• Inner loop: hyperparameter search
• Fresh search in every outer fold

Count the work. Five outer folds, three inner folds, and twenty configurations gives five times three times twenty, which is three hundred model fits, plus five refits of each winner. That is expensive, so nested cross-validation is used to estimate how well a tuning procedure generalizes, not for every project. For the final model, rerun the search on all the training data and report the nested score as your expected performance.

• 5 outer x 3 inner x 20 configs
• 300 inner fits + 5 refits
• Estimate the procedure, not one model

Many hyperparameter settings are clearly bad after only a little training. Their loss curves sit high from the start. Why wait for the full run? The idea of early stopping applied to search is to give lots of configurations a tiny budget, throw away the worst, and give survivors more. Budget can mean epochs, number of trees, or a subsample of the data.

• Bad configs reveal themselves early
• Spend little on many
• Spend much on few

Start with twenty-seven configurations, each trained for one epoch. Keep the best third, nine of them, and train for three epochs. Keep three, train for nine. Keep one, train for twenty-seven. Every round costs twenty-seven epochs, so the total is one hundred and eight. Training all twenty-seven for the full twenty-seven epochs would cost seven hundred and twenty-nine, more than six times as much.

• 27 configs at 1 epoch
• Keep top third each round
• Total 108 vs 729 epochs

Successive halving has a weakness. Some configurations start slowly, like a small learning rate, and look bad early but win in the end. Hyperband hedges by running several brackets of halving. One bracket starts with many configurations and tiny budgets, another with fewer configurations and larger budgets. This way, a slow starter gets a fair chance somewhere, without a lot of extra cost.

• Early scores can mislead
• Slow starters get cut
• Run several brackets

Random search never learns. Trial fifty is chosen as blindly as trial one. Yet every finished trial tells you something about how the score depends on the hyperparameters. Bayesian optimization uses that history. It builds a cheap statistical model of the expensive score function, then uses the model to decide where to run the next trial. It shines when each trial is costly.

• Random search ignores history
• Each trial reveals information
• Model the score surface

That cheap model is called a surrogate. The classic choice is a Gaussian process. Informally, it fits a smooth curve through the observed points and gives, at every candidate setting, a predicted mean and an uncertainty. Near observed points the uncertainty is small. Far from them it is large. We treat the mathematics as a black box here; lesson 61 derives it properly.

• Predicts a mean at every point
• Predicts uncertainty too
• Gaussian process, stated informally

Where should we try next? We could exploit, picking the spot with the best predicted mean. Or we could explore, picking a spot where we know little, since a surprise might hide there. Pure exploitation gets stuck near what we already know. Pure exploration wastes trials. An acquisition function scores each candidate by combining both, and we run the trial at its maximum.

• Exploit: high predicted mean
• Explore: high uncertainty
• Acquisition function balances both

The most popular acquisition function is expected improvement. Let f best be the best score so far, mu the predicted mean at a candidate, and sigma its uncertainty. Define z as mu minus f best, divided by sigma. Then expected improvement equals mu minus f best, times the normal cumulative distribution of z, plus sigma times the normal density of z. In words, it is the average amount by which we beat the best, counting zero when we do not.

• Best so far: f best
• z = (mu - f best) / sigma
• EI = (mu - f best)Phi(z) + sigma phi(z)

Let us maximize accuracy. The best so far is zero point eight zero. Candidate A has predicted mean zero point eight one and uncertainty zero point zero two. Then z is zero point zero one over zero point zero two, which is zero point five. The cumulative distribution at zero point five is about zero point six nine two, and the density is about zero point three five two. So expected improvement is zero point zero one times zero point six nine two, plus zero point zero two times zero point three five two, about zero point zero one four.

• f best = 0.80 accuracy
• A: mean 0.81, sigma 0.02
• EI is about 0.014

Candidate B has a worse predicted mean, zero point seven eight, but a large uncertainty of zero point zero eight. Now z is minus zero point zero two over zero point zero eight, which is minus zero point two five. The cumulative distribution is about zero point four zero one and the density is about zero point three eight seven. Expected improvement is minus zero point zero two times zero point four zero one, plus zero point zero eight times zero point three eight seven, about zero point zero two three.

• B: mean 0.78, sigma 0.08
• z = -0.25
• EI is about 0.023

Candidate B wins, zero point zero two three against zero point zero one four, even though its predicted mean is lower. The reason is the long tail. B has a real chance of landing far above the best, while A can only nudge past it. That is the explore and exploit trade in a single number. As trials accumulate, uncertainty shrinks and the choice shifts toward exploiting the best region.

• B beats A: 0.023 vs 0.014
• Worse mean, bigger upside
• Next trial goes to B

The full loop is short. Run a few random starting trials. Fit the surrogate to all results so far. Find the candidate that maximizes expected improvement. Evaluate the real model there, add the result to the history, and repeat until the budget runs out. Each pass is cheap compared with training a real model, so the extra thinking pays for itself.

• Fit surrogate to history
• Maximize acquisition
• Run trial, add result, repeat

Time to code. For random search, we write a loop. Each iteration draws the learning rate as ten raised to a uniform number between minus four and minus one, calls an objective function that returns the validation score, and stores the pair. After the trials finish, we take the maximum. Notice the exponent trick: uniform in the exponent means log-uniform in the value.

• Sample log-uniform learning rate
• Score each trial
• Keep the best

Now the one-dimensional Bayesian optimizer. We evaluate three random starting points. Then, in a loop, we fit a Gaussian process from scikit-learn to the points so far, predict the mean and standard deviation on a fine grid of candidates, compute expected improvement with the formula we just used, and pick the grid point with the highest value. We evaluate the true objective there and append it to the data.

• Start with 3 random points
• Fit Gaussian process surrogate
• Pick max expected improvement

Run it on a bumpy one-dimensional function and watch. The first picks land in unexplored gaps, where uncertainty is high. As the surrogate sharpens, the picks cluster around the peak. On a toy problem like this, ten to fifteen Bayesian trials typically match what random search needs many more trials to find. Gains are less dramatic with many dimensions or cheap trials, so treat it as a tool rather than a guarantee.

• Early picks explore gaps
• Later picks cluster at the peak
• Fewer trials than random

In practice you rarely write your own optimizer. Optuna is a popular library. You define an objective function that receives a trial object. Inside it you ask the trial to suggest values: a float with a log flag for the learning rate, an integer for depth. You build the model, compute cross-validated score, and return it. Then you create a study, set the direction to maximize, and call optimize with a number of trials.

• Define an objective function
• Suggest values from a trial
• study.optimize runs the loop

Optuna's default sampler, called TPE, is a Bayesian method that models good and bad regions of the space rather than a Gaussian process, and it handles integers and categories easily. Its pruners implement the early stopping ideas from earlier, including successive halving and Hyperband. Your training loop reports intermediate scores, and the pruner cancels trials that fall behind.

• Default sampler is TPE
• Pruner stops weak trials
• Halving idea built in

Say we tune a gradient boosting classifier on a tabular dataset. The search space covers learning rate on a log scale, depth, number of trees, and subsample fraction. Defaults give about eighty-six percent in cross-validation. After fifty trials, the best setting reaches about eighty-nine. Illustrative numbers, but typical in size: tuning usually gives a modest, real gain, not a miracle. Check the study's importance plot to see which dials mattered.

• Tune learning rate, depth, trees
• Default accuracy 0.86
• Tuned accuracy 0.89 on holdout

Search itself can overfit. Each trial is a peek at the validation set, and after hundreds of trials the best score is partly luck, exactly as in the fifty-noise-models example. Protect yourself with a final test set used once, with cross-validation instead of a single split, and by preferring simpler settings when scores are within noise. If the top ten configurations differ by less than the standard error, any of them is fine.

• Many trials overfit validation
• Use a final untouched test set
• Prefer simpler settings on ties

Plan your budget. Start with a wide, coarse search on a small data subsample or few epochs. Narrow the ranges around the promising region and search again. Watch the running-best curve, and when it flattens, stop. Remember that data quality and features from lessons 17 and 34 usually beat tuning, so do not spend a week squeezing a tenth of a percent out of dials.

• Start coarse, then refine
• Stop at diminishing returns
• Cap by time, not trials

A tuning result you cannot reproduce is a rumor. Fix random seeds for the sampler, the splits, and the model. Log every trial with its parameters, scores, code version, and data version. Save the search space along with the study, so a colleague can see not only the winner but what was never tried. Lesson 63 on MLOps builds on exactly this habit.

• Fix random seeds
• Log every trial and version
• Save the study and the space

Let us recap. Hyperparameters are set before training and judged on validation data. Use log scales for dials spanning orders of magnitude. Random search beats grid search because it tries more distinct values of the dials that matter. Nested cross-validation gives an honest estimate of the tuning procedure. Successive halving and Hyperband save compute by stopping bad trials early. Bayesian optimization uses a surrogate and expected improvement to balance exploring and exploiting.

• Log scales, random beats grid
• Nested CV for honest scores
• Halving, Hyperband, Bayesian opt

Tuning is a habit, not a trick: plan the space, protect the test set, and spend your budget wisely. In lesson 46 we return to time series and derive ARIMA and exponential smoothing with real numbers.

• Lesson 46 next
• Time-series models by hand
• Subscribe for the full course