Kestrel Backtest Jobs, Parameter Sweeps and Honest Walk-Forward

Run a thousand parameter combinations, pick the best one, and you have almost certainly found noise. The hard part of research is not running backtests. It is running them in a way that cannot fool you. Today we follow a backtest through Kestrel's research service, and then through a sweep tool designed to stay honest.

• Best of many is often noise
• Honest research needs structure

In episode fourteen we built the metrics: Sharpe, Sortino and drawdown, computed with Welford updates. We will not repeat those formulas here. This episode is about the machinery around the engine: the portal that queues jobs, and the sweep tool that scores them without lying to you.

• Episode 14 gave us metrics
• Now: jobs, sweeps, walk-forward

First the portal: its config, its submit endpoint, its job manager and the way a result is saved. Then the offline sweep tool, with its proof that nothing leaves the machine. Finally walk-forward folds and the score function, with a worked example using concrete numbers. Everything here is educational, and every number is illustrative.

• Portal and job pipeline
• Offline sweep tool
• Walk-forward and scoring

kestrel_portal loads config slash portal dot json. It listens on 127.0.0.1 port 8093, loopback only. Limits are explicit: two backtest workers, one download worker, at most 32 queued jobs, 200 retained results, a 366 day maximum backtest span, and 3600 seconds per job.

• Listener 127.0.0.1:8093
• 2 backtest workers, 1 download
• maxQueued 32, maxRetained 200

Result size is bounded too. Curves and price series are decimated to 2000 points, and at most 20000 fills are stored. The defaults for a job are capital of 1,000,000 rupees, the india cost model, slippage of 2, 1 and 5 basis points, and pricing with a rate of 6.5 percent and volatility of 20 percent. A request only needs to state what differs.

• Curves capped at 2000 points
• maxStoredFills 20000
• Defaults: capital 1,000,000

The portal speaks JSON only. A request for the root path gets a plain 404, because there is no web page here; the browser gateway from the series overview lives elsewhere. Every handler sits behind a RequestGuard, which we cover in episode eighteen. That keeps the research API from being poked by anything that is not meant to talk to it.

• JSON only; GET / is 404
• Every handler behind RequestGuard
• Details in episode 18

On startup the portal registers itself with the role research. That role matters for safety: the research process is not the one that holds broker sessions or places orders. It reads history from the local store and runs simulated venues, which is exactly what a backtest needs.

• Registers with role research
• Reads the local history store
• No broker connection needed

A backtest starts with a POST to slash api slash backtests. The body is a spec: strategy, symbols, dates and optional overrides. Before anything runs, the portal pushes it through a chain of four stages, and any stage can reject it with a clear error. Cheap checks come first, so a bad request fails in milliseconds rather than after an hour.

• Four checks, then submit
• Cheap checks first
• Fail fast with clear errors

parseBacktestSpec reads the spec. The venue, risk and pricing sections are merge-patched over the portal defaults. Merge patch means a key you send replaces the default for that key, and keys you omit keep their default values. So sending only a slippage change leaves capital and costs alone. If the strategy uses derived instruments, such as options, the spec must name a stored underlying.

• Request merge-patched OVER defaults
• Omitted keys keep defaults
• Derived needs a stored underlying

Next, buildEngineConfigJson turns the spec into a real engine config. Instruments come from the provider's instrumentTemplate. There is exactly one store source, the venue is a simulated venue, and runSeconds is set to maxJobSeconds so a runaway job has a ceiling. This is the same three-file run config idea from earlier episodes, assembled in memory.

• Instruments from instrumentTemplate
• One store source, simulated venue
• runSeconds = maxJobSeconds

validateAgainstRegistries does something stricter than checking names. It actually constructs each alpha from its parameters. If a parameter is missing or out of range, the constructor complains now, at submit time, instead of inside a worker minutes later. Then a coverage check confirms the store really holds bars for the requested symbols and dates.

• Constructs each alpha for real
• Errors surface at submit time
• Coverage check against the store

Then JobManager dot submit decides the answer. A 202 means accepted, and you get an id such as bt, dash, 20261026, dash, 101500, dash, 3fa2: a date, a time and a short suffix. A 429 means the queue is full at 32. A 409 means the request has conflicting keys. Each code tells the caller exactly what to do next.

• 202 accepted with id
• 429 queue full
• 409 conflicting keys

JobManager keeps one worker pool per job kind. Backtests get two workers, downloads get one. Each kind has its own queue guarded by a condition variable, so idle workers sleep instead of spinning, and a slow download can never starve a backtest.

• One pool per job kind
• Condvar queues, no spinning
• Downloads cannot starve backtests

A worker waits on its condition variable with a predicate: wake when the queue is non-empty or the manager is shutting down. Submit pushes a job and notifies one worker. The predicate guards against spurious wakeups, and the shutdown flag lets every worker drain out cleanly when the portal stops.

• Wait on predicate
• Notify one worker per job
• Shutdown flag drains workers

Each running job receives a JobContext. It offers progress, taking a fraction and a message, and cancelled, which the job can poll. A job can also install a stop handler. When you cancel, the manager flips the flag and calls that handler, which for a backtest calls engine dot requestStop.

• progress(fraction, msg)
• cancelled() polled by the job
• Cancel calls engine.requestStop()

Notice how the limits interlock. Two workers cap CPU use, a queue of 32 caps memory held by waiting specs, and 200 retained results cap disk. Each limit turns an unbounded problem into a bounded one, and each has a visible failure, like the 429, rather than a slow degradation.

• Workers cap CPU
• Queue cap bounds memory
• Retention cap bounds disk

When a worker picks up a job it calls runBacktest. This builds an Engine from the config, attaches BacktestMetrics from episode fourteen, a FillRecorder that remembers fills up to the storage cap, and a ProgressReporter that feeds the JobContext. The engine then runs exactly as it would in any other mode.

• Engine + BacktestMetrics
• FillRecorder + ProgressReporter
• Same engine as every mode

The result is a JSON document. It holds the metrics, a decimated equity curve and drawdown curve, price series, the stored fills, any dataWarnings, and a partial flag. The partial flag is honest bookkeeping: if the job was stopped or cut short, the result says so instead of passing itself off as complete.

• Metrics, equity, drawdown, prices
• Fills and dataWarnings
• partial flag for cut-short runs

The result file is saved atomically. The portal writes to a temporary file, then renames it onto the final name. A rename within one directory is atomic on POSIX filesystems, so a reader sees either no result or the whole result, never half a file. It is the same idea as the history store from episode four.

• Write to temp file
• Rename onto final name
• Readers never see half a file

Each job also gets its own log. The logging sink is thread-scoped: the worker installs it for its own thread, so two jobs running at the same time write to separate files without mixing lines. When something looks wrong in a result, you open that job's log and read exactly what the engine said.

• Thread-scoped log sink
• Two jobs, two separate logs
• Debug a result by reading its log

One backtest answers one question. A parameter sweep asks it many times. For that, Kestrel ships tools slash sweep slash sweep dot py, written with only the Python standard library. It drives the portal over HTTP, so a sweep uses exactly the same job pipeline we just saw.

• tools/sweep/sweep.py
• Python standard library only
• Drives the same portal pipeline

The sweep does not use your running portal. It starts its own on 127.0.0.1 port 8097. It blanks every environment variable starting with FYERS underscore, so no credential can leak in. And it points KESTREL underscore DATA underscore ROOT at a fresh directory containing only the public symbol masters and no token.

• Own portal on 127.0.0.1:8097
• Every FYERS_* variable blanked
• Fresh data root, no token

Finally it sets the HTTP proxy variables to 127.0.0.1 port 9. Nothing listens there, so any outbound call a library tries to make fails at once. Research is not merely supposed to be offline. It is provably offline, because the first stray network call would crash loudly instead of quietly fetching data.

• HTTP proxies set to 127.0.0.1:9
• Any outbound call fails at once
• Offline is enforced, not hoped

This matters beyond security. If a backtest could fetch fresh data, two runs a week apart might differ for reasons you cannot see. With a fixed local store and no network, the same spec gives the same numbers. Episode sixteen builds on exactly this property.

• Same store, same spec
• Same numbers every time
• Foundation for episode 16

A sweep is described by one spec. It names the strategy, provider and symbols, a baseParams object, and a grid. It has start and end dates, where end is exclusive, plus costs, a walkForward block, a score block, topN, and metricsResampleSec, which defaults to 86400, one day.

• baseParams plus grid
• end is exclusive
• metricsResampleSec 86400

The grid is a cartesian product. If fast takes three values and slow takes four, that is twelve candidates. Constraints, written as a left side, an operator and a right side, prune nonsense. For instance, requiring fast to be less than slow removes every pair where the order is backwards before any job is submitted.

• Grid is a cartesian product
• 3 x 4 gives 12 candidates
• Constraints prune before running

Costs are part of the spec, so every candidate pays the same fees and slippage. The metrics resample step is part of it too, and that detail matters later. A Sharpe ratio computed from daily returns and one from hourly returns are measured on different scales, so a sweep fixes one step for all candidates.

• Same costs for every candidate
• One resample step for all
• Sharpe depends on the step

Walk-forward splits time into folds. In each fold you train on an earlier window and test on the window right after it. You pick parameters using only the training window, then look at how those parameters did on the test window, which they never saw.

• Train on the past
• Test on the window right after
• Parameters chosen on train only

Take a twelve month train, a three month test, stepping three months across two years. Fold one trains on months zero to twelve and tests on twelve to fifteen. Fold two trains on three to fifteen and tests on fifteen to eighteen. Fold three is six to eighteen, then eighteen to twenty one. Fold four is nine to twenty one, then twenty one to twenty four.

• Train 12, test 3, step 3
• Four folds over 24 months
• Test windows tile months 12 to 24

With anchored set to true, training always starts from day one and simply grows. The step defaults to the test length. If the final test window runs past the data, the fold is kept only when the truncated test covers at least minLastTestFraction of a full one. That stops a two day test from counting as a fold.

• Anchored trains from day one
• Step defaults to test length
• Short last test needs minLastTestFraction

For each fold, the best training score picks the parameters. The report then gives those parameters' test metrics and their rank on the test window. The out-of-sample figure compounds the chosen test returns end to end. Outputs are results dot csv, walkforward dot csv, top underscore full dot csv and summary dot json.

• Train score picks the params
• Report shows their test metrics and rank
• outOfSample compounds test returns

Which parameters are best? That needs a number, and sweep dot py defines it in score underscore of. This is simplified Python. A failed run, zero trades, or a missing Sharpe returns None, meaning not eligible. The profit factor gate does the same. Then the trade count scales the Sharpe through a factor f.

• Gates return None
• f = min(1, trades / minTrades)
• Positive scaled down, negative up

Look at the last line. With few trades, f is below one. A positive Sharpe is multiplied by f, so it shrinks. A negative Sharpe is divided by f, so it grows more negative. Thin evidence can never help a candidate, whichever direction it points.

• Positive: multiply by f
• Negative: divide by f
• Few trades never help

Let minTrades be 20. Candidate A has a Sharpe of 1.2 from only 10 trades. So f is 0.5, and the score is 0.6. Candidate B has a Sharpe of 0.9 from 40 trades. Here f is capped at 1, so the score stays 0.9, and B wins. Candidate C has a Sharpe of minus 0.4 from 10 trades. Dividing by 0.5 gives minus 0.8. These numbers are illustrative.

• A: 1.2, 10 trades gives 0.6
• B: 0.9, 40 trades gives 0.9, wins
• C: -0.4, 10 trades gives -0.8

The second gate handles profit factor, the ratio of gross profit to gross loss. If it is undefined but there were losing trades, the candidate is rejected. If it falls below minProfitFactor, which defaults to zero, it is rejected too. The default of zero rejects nothing by itself, but you can raise it. The defaults here are minTrades of 20 and minProfitFactor of 0.

• Undefined PF with losers: rejected
• Below minProfitFactor: rejected
• Defaults: minTrades 20, PF 0

The first pitfall is choosing parameters on test data. If you look at test results, then adjust the grid and rerun, the test window quietly becomes training data. Walk-forward protects you only while the choice uses training scores alone. The report shows test metrics for the chosen parameters, and it should be read, not optimised against.

• Select on train only
• Looking at test then tuning leaks
• Read test results, do not optimise them

The second pitfall is comparing Sharpe ratios computed at different resample steps. A daily Sharpe and an hourly Sharpe are not the same unit. That is why the spec carries one metricsResampleSec for every candidate. Compare numbers only when they were produced with the same step.

• Different steps, different units
• One metricsResampleSec per sweep
• Compare like with like

The third pitfall is trusting a high Sharpe from few trades, which the score function penalises. There is also a quieter limit. Walk-forward here has no separate warm-up, so each test window starts cold, with indicators and positions empty. Early test bars may behave differently than they would in a continuous run, and the report cannot hide that.

• Few trades make fragile Sharpe
• No separate warm-up
• Each test window starts cold

To recap: kestrel_portal bounds everything with explicit limits and validates a spec by really constructing the alphas. JobManager runs worker pools with condition variable queues, progress and cancellation, and results are saved atomically. The sweep tool runs its own offline portal, then walk-forward folds choose on training and report on test, scored so that few trades never help.

• Bounded, validated job pipeline
• Atomic results, per-job logs
• Offline sweeps, honest folds

This is educational material, not financial advice, and none of these numbers claim that any strategy works. Next time, episode sixteen: proving determinism with golden tests, and the run archive.

• Next: golden tests and run archive
• Educational only, not advice
• Subscribe for episode 16