Backtest Metrics: Welford Sharpe, Sortino and Drawdown in C++

Four equity values. One hundred, one hundred and one, one hundred and a half, and one hundred and two. Feed them to a Sharpe calculation and you get a number above twelve. That would be a miracle, if it meant anything. Today we see how Kestrel computes its backtest metrics, and why this number should make you suspicious.

• Four equity points
• Sharpe above 12
• Why be suspicious?

In the previous episode we built an implied volatility surface from option prices. This episode sits at the other end of the pipeline. A backtest has finished running, and we have a long record of equity and fills. How do we boil that down to a handful of honest numbers? That is the job of BacktestMetrics.

• Episode 14
• End of the pipeline
• BacktestMetrics

A reminder before we begin. This series is educational. Nothing here is financial advice, and no number in this episode implies that any strategy works. Every example is illustrative. Our goal is narrower and more useful: to compute risk measures correctly, and to avoid the classic mistakes.

• Educational only
• Numbers are illustrative
• Goal: compute correctly

BacktestMetrics listens through two inputs. The equity observer is called once per simulated second with the current account value. The fill observer is called on every fill. Equity gives us the shape of the curve. Fills give us trades, turnover and costs. Everything else is derived from these two streams.

• Equity observer: once per sim second
• Fill observer: every fill
• Everything derives from both

Why not compute returns directly on those per second values? Because a one second return and a one day return are different animals, and a strategy that trades rarely produces long flat stretches. We first resample equity onto a fixed grid, so every return covers exactly the same amount of time.

• Returns need equal time spans
• Fix the grid first

The grid starts at the beginning of the run. For each grid point we take the last equity value at or before that point, a forward fill. Never a value from after it. That rule matters: looking ahead to the next sample would quietly leak the future into the past, the same sin the simulated venue avoids.

• Grid starts at run start
• Last value at or before the point
• Never look ahead

The step is chosen automatically from the span of the run. A span of at least sixty days uses a one day step. At least two days uses one hour. At least two hours uses one minute. Anything shorter uses one second. A long run gets coarse, stable returns; a short run keeps enough points to say something.

• 60 days or more: 1 day
• 2 days or more: 1 hour
• 2 hours or more: 1 minute
• Otherwise: 1 second

Sharpe is also reported at three fixed steps: sixty seconds, thirty six hundred seconds, and eighty six thousand four hundred seconds. This is the sharpeByStepSec table. It lets you compare two strategies at the same sampling interval, instead of comparing numbers that were quietly computed on different grids.

• sharpeByStepSec
• 60 s, 3600 s, 86400 s
• Same grid, fair comparison

On the grid, each period gives an excess return x sub i. It is the equity at the end of the period divided by equity at the start, minus one, minus the risk free rate divided by P. Here P is periods per year: three hundred sixty five point two five days, in seconds, divided by the step in seconds.

• x = E_i / E_(i-1) - 1 - rf / P
• P = 365.25 * 86400 / step

Notice that annualising uses calendar time, three hundred sixty five point two five days a year, not trading days. A one day step therefore gives about three hundred sixty five periods per year, even though markets are closed on weekends. Flat weekend returns are simply part of the series. That is a design choice worth knowing when you compare against other tools.

• Calendar year: 365.25 days
• Weekends count as periods
• Know it when comparing tools

To get a standard deviation the naive way, you store every return, compute the mean, then loop again for squared deviations. That needs memory proportional to the run, and two passes. The one-pass shortcut, sum of squares minus square of the sum, is worse: it subtracts two huge nearly equal numbers and loses precision to cancellation.

• Two passes, stored array
• Sum of squares trick cancels badly

Welford's algorithm updates the mean and a running sum of squared deviations, called m2, one observation at a time. It never stores the history, and it never subtracts huge numbers. Each new value nudges the mean by its surprise divided by n, and m2 grows by a product of two deviations, one measured before the update and one after.

• Running mean
• Running m2
• One pass, stable

Here is the heart of it, verbatim from Kestrel. Emit takes the next equity value. If the stream has stopped, it returns. If the value is not strictly positive, it stops the stream and flags it. Otherwise it forms the excess return, updates the mean and m2, and accumulates the squared downside.

• Guard, return, Welford, downside

Look closely at the m2 line. The delta was computed against the old mean. The factor x minus mean is computed after the mean has moved, so it is the new deviation. Their product is exactly the increment to the sum of squared deviations. Swap them for two old deltas, and the variance comes out wrong.

• delta: against the OLD mean
• x - mean: against the NEW mean
• Product updates m2 exactly

Now the guard. It is written as not, value greater than zero, rather than value less than or equal to zero. Why? Because every comparison with a NaN is false. A less-than-or-equal test would let NaN slip through and poison the mean forever. The negated form catches zero, negatives and NaN in one condition.

• !(value > 0.0)
• NaN > 0 is false
• One test catches 0, negative, NaN

If equity reaches zero or below, a return is meaningless: you cannot divide by a ruined account. So the stream stops for good, and the flag equityWentNonPositive is set. Metrics computed so far stay valid, and the result tells you honestly that the account was wiped out, instead of printing a clean looking ratio.

• Return stream stops
• equityWentNonPositive = true
• Honest flag, no fake ratio

Now the worked example, on a daily step with a risk free rate of zero. Equity goes one hundred, one hundred one, one hundred point five, one hundred two. The three returns are plus one percent, minus zero point four nine five percent, and plus one point four nine two five percent. As decimals: point zero one, minus point zero zero four nine five, and point zero one four nine two five.

• x1 = 0.010000
• x2 = -0.004950
• x3 = 0.014925

Run Welford on those. After the first return, the mean is point zero one and m2 is zero. After the second, the mean drops to about point zero zero two five and m2 becomes about one point one eight times ten to the minus four. After the third, the mean settles at point zero zero six six five eight, and m2 reaches about two point one four times ten to the minus four.

• n=1: mean 0.0100, m2 0
• n=2: mean 0.00252, m2 0.000112
• n=3: mean 0.006658, m2 0.000214

Sample variance divides m2 by n minus one, which is two, giving a standard deviation of point zero one zero three five. Sharpe is the mean over that, times the square root of three hundred sixty five point two five, about nineteen point one one. That is twelve point two nine. For Sortino, the only downside is the minus point zero zero four nine five. Its square over three, rooted, is about point zero zero two eight six, giving forty four point five two.

• sd = 0.0103507, Sharpe = 12.29
• Downside rms = 0.002858
• Sortino = 44.52

Annualised volatility is the standard deviation times root P, which is nineteen point seven eight percent. Now the warning. Three returns is nothing. With a handful of samples, a tiny difference in one return swings these ratios wildly, and annualising multiplies the noise by nineteen. That is why sweeps require a minimum number of trades before they trust a score.

• Volatility = 19.78%
• 3 samples prove nothing
• Sweeps require minimum trades

Here are the definitions, simplified. Sharpe is the mean over the sample standard deviation, using n minus one, times root P. Sortino uses a different denominator: the square root of the sum of squared negative returns, divided by n, again times root P. Volatility is the standard deviation times root P. The downside deviation divides by n, and only penalises losses.

• Sharpe = mean / sd * sqrt(P)
• Sortino = mean / downside rms * sqrt(P)
• Volatility = sd * sqrt(P)

Some of these are undefined. With fewer than two returns there is no sample standard deviation. With zero deviation, Sharpe divides by zero. In those cases Kestrel reports null. It never emits NaN, because NaN is not valid JSON, and a single one can break the portal's response or a browser parser downstream.

• n < 2: null
• Zero sd: null
• NaN is not valid JSON

Growth is simpler. Total return is final equity over starting equity, minus one. Years is the span divided by three hundred sixty five point two five days. CAGR is final over start, raised to one over years, minus one. For instance, a doubling over two years gives a root two, so about forty one percent a year. A made up number, purely for arithmetic.

• totalReturn = final/start - 1
• years = span / 365.25 d
• CAGR = (final/start)^(1/years) - 1

Drawdown needs a memory of the best equity seen so far, the running peak. At each grid point the drawdown is equity divided by that peak, minus one. It is zero at a new high and negative below it. Maximum drawdown is the largest value of minus drawdown: the deepest fall from any previous high.

• Running peak
• dd = E / peak - 1
• maxDrawdown = max(-dd)

Kestrel does not only keep the depth. It records three timestamps: the peak where the worst drop began, the trough at its bottom, and the recovery, when equity regained the old peak. From these comes mddDurationSec for the worst drawdown, and longestDrawdownSec, the longest time spent under any previous high. These two are often different episodes.

• peak, trough, recovery timestamps
• mddDurationSec
• longestDrawdownSec

A quick example with invented numbers. Equity climbs to one hundred and twenty, falls to ninety, then climbs back to one hundred and twenty five. Drawdown at the bottom is ninety over one hundred twenty, minus one: minus twenty five percent. Recovery happens only when equity first returns to one hundred twenty, not when it merely stops falling.

• Peak 120, trough 90
• dd = 90/120 - 1 = -25%
• Recovery at 120, not before

Equity says how much, but not which trades did it. For trade statistics, the fill observer keeps separate average cost books, one for each pair of strategy and instrument. Two strategies trading the same contract never net against each other here. Each book follows its own position through time.

• Fill observer feeds books
• One book per (strategy, instrument)
• No cross-netting

A trade closes when its position returns to zero, or flips sign. On a flip, say from long ten to short five, fifteen units trade. Ten close the old trade and five open a new one. The commission is split by the closing quantity over the absolute fill quantity, so only the closing share is charged to the finished trade. Profit and loss includes commissions.

• Closes at zero or on a flip
• Flip: commission split closing/|q|
• P&L includes commissions

From closed trades come the familiar numbers. Win rate is the share with positive profit and loss. Profit factor is the sum of wins over the absolute sum of losses. If there are no losers, that is a division by zero, so the value is null. Average, best and worst trade are also reported. Remember, a high win rate says little without the sizes.

• winRate
• profitFactor = sumWin / |sumLoss|
• Null with no losers

Three more measures describe how the capital was used. Exposure is the fraction of time with gross position value above zero. Average gross leverage is gross exposure over equity, weighted by time. Turnover is traded notional, excluding settlement fills, divided by time weighted mean equity, then annualised. Settlement is not a decision to trade, so it must not inflate turnover.

• Exposure: time with gross > 0
• avgGrossLeverage: time-weighted
• Turnover excludes settlement fills

A year of one minute equity is hundreds of thousands of points. The portal sends it to a browser chart, which cannot draw all of them, and simple downsampling, keeping every hundredth point, can erase the worst spike of the year. A drawdown chart that hides the drawdown is worse than none.

• Hundreds of thousands of points
• Naive downsampling hides spikes

SeriesDecimator fixes the number of buckets. Each bucket keeps four points: the first, the minimum, the maximum and the last. A year of one minute equity becomes about two thousand points, yet every peak and trough survives. The spike lands in some bucket's minimum and so stays on the chart.

• Fixed number of buckets
• Keep first, min, max, last
• About 2000 points, spikes kept

Now the classic mistakes. First: annualising returns sampled irregularly. If you take a return only when a trade happens, each return covers a different duration, and multiplying by root P assumes they are all equal. Kestrel avoids it by resampling onto a fixed grid before any return is computed.

• Irregular returns, wrong root P
• Fix: fixed grid first

Second mistake: comparing strategies on different steps. A Sharpe from one minute returns and one from daily returns differ even for identical equity, because serial correlation and noise change with the step. Sweeps therefore force a step of eighty six thousand four hundred seconds, so every candidate is scored on the same daily grid.

• Different steps, different Sharpe
• Sweeps force 86400 s

Third, and we have seen it: NaN in JSON, which Kestrel prevents by returning null. Fourth, trusting tiny samples, as our twelve point two nine showed. Fifth, ignoring commissions in trade statistics, which is why trade profit and loss includes them. None of these are exotic. They are what happens by default when metrics are written quickly.

• NaN in JSON: use null
• Tiny samples mislead
• Include commissions

Let us recap. Equity is sampled on a fixed grid with forward fill and no lookahead. Excess returns feed a Welford accumulator: one pass, stable, and careful about NaN. Sharpe, Sortino and volatility are annualised with root P, and undefined values are null. Drawdown tracks a running peak, with timestamps. Trades, exposure and turnover come from the fills, and the decimator keeps spikes.

• Fixed grid, forward fill
• Welford: one pass, stable
• Null instead of NaN
• Drawdown, trades, decimator

Metrics are only useful when you can run many backtests and compare them. In the next episode we look at the portal: backtest jobs, parameter sweeps, and walk-forward testing, and how they use the numbers we just built.

• Next: portal backtest jobs
• Sweeps and walk-forward
• Subscribe