Golden Tests, Fakes and the Run Archive: Proving a Trading Engine Unchanged

Imagine you rewrite the heart of a trading engine. Thousands of lines move around. Every test passes. But how do you know the engine still makes exactly the same decisions, down to the last paisa? Today we look at how Kestrel answers that question with a single hash.

• Refactor the engine
• Prove nothing changed

This is episode sixteen. In the previous episode we saw portal backtest jobs, sweeps and walk-forward. Now we turn to trust. We will look at the test layout, how everything runs offline, the golden tests, a few unit tests worth naming, and finally the run archive that records paper and live sessions without stalling trading.

• Layout and offline
• Golden tests
• Run archive

The idea is simple. A backtest is deterministic, so its whole output is a function of its inputs. Turn that output into text, hash the text, and compare the hash to one stored earlier. If the hash matches, behaviour is identical. If not, something changed, and you want to know exactly what.

• Deterministic run
• Text, then hash
• Compare with stored

Kestrel uses doctest. Each module has its own folder under tests, and every cpp file in that folder is compiled together with tests slash test underscore main dot cpp. That is the only file that defines DOCTEST underscore CONFIG underscore IMPLEMENT underscore WITH underscore MAIN. The result is one executable per module, named kestrel underscore tests underscore the module name.

• tests/<module>/*.cpp
• One main file only
• kestrel_tests_<module>

Why define the doctest main in exactly one file? Define it twice and the linker complains about duplicate symbols. Define it in none and there is no entry point. Keeping it in a single shared file means every module executable gets an identical runner, and adding a new test file never requires touching build plumbing.

• Two mains: link error
• One shared main: clean

Every executable is registered with ctest, so one command runs the whole suite. Fixture files, such as small bar files or symbol masters, are found through the KESTREL underscore TEST underscore DATA underscore DIR definition, rather than through the current directory. The Python tests for the browser gateway also run under ctest, so there is one green or red answer.

• ctest runs everything
• KESTREL_TEST_DATA_DIR
• UI Python tests too

The first pitfall in trading tests is touching the network. A test that calls a real broker is slow, flaky, and depends on a login that expires every day. In Kestrel the FYERS side is reached through injected fake transports, and symbol masters come from fixture files. A test run needs no internet and no credentials.

• Injected fake transports
• Fixture symbol masters
• No credentials needed

Tests also protect themselves from the developer's machine. Every FYERS environment variable is blanked with setenv of the key, an empty string, and the overwrite flag set to one. The detail matters: an empty value is not overwritten by the dot env loader, so a real token sitting in a file can never leak into a test. The data root is also a fresh temporary directory.

• setenv(KEY, "", 1)
• Empty is not overwritten
• Temp KESTREL_DATA_ROOT

A temporary data root prevents two bugs. Tests cannot read your real history by accident, so results do not depend on what you downloaded. And tests cannot write into your real archive, so a run of the suite never pollutes paper run folders. Every test starts from a clean world that it builds itself.

• No accidental real data
• No archive pollution

Shared helpers live in tests slash support. There is a VectorSource, an ImmediateVenue, a RecordingListener, a CoreEnv, a fake FYERS provider, order-service fixtures, and golden dot h p p. Think of them as a small workshop: they let a test assemble a real engine around fake edges.

• VectorSource, ImmediateVenue
• RecordingListener, CoreEnv
• golden.hpp

VectorSource is a bar source of type vector. You can hand it explicit rows, or ask for a seeded random walk. The random walk uses std mt19937 sixty-four, seeded with the seed plus the instrument id. Each instrument gets its own stream, so adding a third symbol never changes the prices of the first two.

• Explicit rows or random walk
• mt19937_64
• seed + instrument id

This is the second pitfall: non-deterministic seeds. If a test seeds from the clock or from random device, a failure cannot be reproduced and a golden hash can never be stable. Fixed seeds turn randomness into a reproducible fixture. The data looks random, but it is the same data every single time.

• Clock seed: unreproducible
• Fixed seed: same data always

ImmediateVenue fills orders at once under simple rules, which lets tests of the order layer ignore fill modelling. RecordingListener sits on the event bus and records everything it hears, so a test can assert on the exact sequence of reports. CoreEnv wires the pieces into a ready engine so each test stays short.

• ImmediateVenue fills at once
• RecordingListener captures events
• CoreEnv wires it up

There are four golden tests. G one runs an sma and a z-score alpha on two random-walk equities, five thousand bars each. G two runs a delta-hedged derived option and a future that both expire inside the data. G three covers position-limit clipping and limit-order square-off. G four is paper trading with vector rows, a carrying alpha, and end-of-day files.

• G1 sma + zscore, 5000 bars
• G2 derived expiry, hedged
• G3 limits and square-off
• G4 paper rows, EOD files

Each scenario was chosen to cover a different risk. G one exercises the merge and the alpha framework at volume. G two crosses expiry settlement and derived bars, which touch many components. G three hits risk clipping and forced exits. G four covers the paper path and its files. Between them, most of the engine's moving parts leave a footprint.

• Merge and alphas
• Expiry and derived bars
• Risk and square-off
• Paper path and files

After a run, Kestrel serialises everything into canonical text. A fill line reads: fill, timestamp, alpha name, symbol, B or S, quantity, price, commission. For example, fill, a timestamp, sma underscore a, SYMA, B, ten, one hundred one point two five, zero point zero zero. That is illustrative. Then come order lines, equity lines, and the final state.

• fill <ts> alpha symbol side qty px commission
• Order lines, equity lines
• Final state

Equity lines are printed with percent dot six f, six decimals. This is a deliberate choice. Printing fixed decimals makes the text identical on every run, and six places is fine enough to catch a real numerical change while ignoring noise that the formatter would hide anyway. The text, not the in-memory doubles, is what gets compared.

• %.6f equity lines
• Text is the contract

The canonical text is hashed with SHA-256 and compared with the stored value in tests slash fixtures slash golden, one json file per scenario. A hash is tiny, so the fixture stays small and reviewable in a diff. And because SHA-256 changes completely if even one character differs, a single changed fill price is impossible to miss.

• SHA-256 of canonical text
• Stored in G*.json
• One character changes everything

A bare hash mismatch would be useless for debugging. So on failure, Kestrel writes the actual text to the TMPDIR folder and prints the first forty differing lines. You can see right away that, say, the three hundredth fill moved by one tick, instead of staring at two unrelated hex strings.

• Actual text saved to TMPDIR
• First 40 differing lines

Sometimes behaviour should change, for instance when you fix a fee rule. Then you set KESTREL underscore GOLDEN underscore WRITE to one, and the tests re-record the stored hashes. The new files appear in your diff, which forces a reviewer to read them. The rule is to re-record only when you can explain every changed line.

• KESTREL_GOLDEN_WRITE=1
• New hashes show in review
• Explain every changed line

This gives Kestrel a clear definition of a safe refactor. After every change, the report says G one to G four byte-identical. That means the machinery moved but the behaviour did not. G four also passed twenty consecutive runs, which is a useful check that nothing in it depends on timing, thread scheduling, or leftover state.

• Report: G1-G4 byte-identical
• G4: 20 consecutive runs

Here is a worked example with illustrative numbers. Suppose a refactor changes how commissions are summed, adding costs in a different order. Floating-point addition is not associative, so a total could shift in the last digits. The fill lines still match, but one equity line differs at the sixth decimal. The golden test flags it, and you decide whether to accept or fix.

• Summation order changes
• Sixth decimal differs
• Test flags it

A golden suite is only useful if people run it, so speed matters. There is a skipped benchmark case, skipped by default so it does not slow the normal run. On the author's measurements it handles about one point six million bars per second, and about one hundred sixty backtests per second of five thousand bars. Those are rough figures, not guarantees.

• Bench case skipped by default
• About 1.6 M bars/s
• About 160 backtests/s

Golden tests catch change, but they do not say what is correct. For that you need unit tests with answers worked out independently. Kestrel has black underscore scholes underscore test, india underscore fees underscore test, fill underscore model underscore test, metrics underscore test, test underscore store underscore bar underscore timing, and test underscore real underscore venue underscore guard.

• Pricing, fees, fills
• Metrics, store timing
• Real venue guard

A golden test can lock in a wrong answer. If the first recording was buggy, the hash will faithfully protect the bug. That is why metrics underscore test uses a Sharpe ratio computed by hand, and why the fee and pricing tests use independently derived values. Golden tests say nothing changed; unit tests say it was right to begin with.

• Golden: nothing changed
• Unit: it was right
• Hand-computed expectations

Two of these tests guard boundaries. test underscore store underscore bar underscore timing checks how stored bars are timed, which protects against lookahead. And test underscore real underscore venue underscore guard checks that the guarded path to a real venue refuses to act without its explicit permissions. Safety properties deserve tests just like arithmetic does. The live path is design, not proven in production.

• Bar timing: no lookahead
• Real venue guard refuses

Testing proves code. But paper and live sessions also need records of what actually happened. Kestrel's RunArchive writes each run into a folder: data root, paper underscore runs, the IST date, the trader name, and then a time stamp as hour minute second. What the order service itself puts in the archive is covered in episode twenty-four.

• paper_runs/<IST date>/<trader>/
• <HHMMSS>[-n]

Two processes might start in the same second. How do they avoid sharing a folder? createRunDir tries to make the directory. If it fails because it already exists, it tries the same name with a dash two, then dash three, and so on, until a mkdir succeeds. Any other error is a real failure.

• Try HHMMSS first
• EEXIST: try -2, -3, ...
• Other errors: fail

In pseudo-form: for n from one upward, name is the time stamp if n is one, otherwise the time stamp, dash, n. If mkdir of that name returns zero, return it. If errno is not EEXIST, fail. That is the whole algorithm. No lock files, no shared counters, no coordination between processes at all.

• for n = 1..
• mkdir(name) == 0: return
• errno != EEXIST: fail

The key is that mkdir is atomic. The operating system guarantees that when two processes create the same directory, exactly one succeeds and the other sees EEXIST. A check-then-create approach would have a gap between the check and the create, where both could pass. Here the check and the claim are one operation, so two processes can never claim the same run id.

• mkdir is atomic
• Exactly one winner
• No check-then-create gap

Inside a run folder are run dot json, run dot log, fills dot csv, orders dot csv, equity dot csv, trades dot csv, and summary dot json. Fills carry the timestamp, strategy, symbol, side, quantity, price, notional, and each cost component, brokerage, STT, exchange, SEBI, stamp, IPFT and GST, plus order id and liquidity.

• run.json, run.log
• fills, orders, equity, trades
• summary.json

trades dot csv holds round trips, each with an exit reason and a restored flag that says whether the position was carried over from an earlier session. Cost rates in the fills reflect the code's defaults as of twenty twenty-six, and they change over time. The run dot json file records the run's status, which matters when something goes wrong.

• Round trips with exit reason
• Restored flag
• run.json status

Now the third pitfall: disk input and output on the engine thread. Writing a file can take a surprising time if the disk is busy. If the engine waited for it, a slow disk would delay order handling during live markets. So the engine thread must never do file work directly.

• Disk I/O can be slow
• Engine thread must not wait

Engine-thread observers only enqueue rows, under a mutex. A separate writer thread takes rows from the queue and does all the input and output. The engine's cost is a short lock and a push; the slow part happens elsewhere. This is the classic producer and consumer pattern, applied to protect the one thread that must stay fast.

• Observers enqueue under mutex
• Writer thread does I/O

Not all rows are equal. Fills and orders are never dropped, because they are the record of what happened. Sampler rows, like periodic equity snapshots, are droppable. The queue is bounded at four thousand ninety-six entries. If it fills, sampler rows are dropped with a throttled warning, so a slow disk never stalls trading.

• Fills, orders: never dropped
• Samples dropped when full
• Bounded queue of 4096

The writer samples every sixty seconds. It also writes a heartbeat and rewrites the summary every five minutes. So even if the process dies midway, the folder holds fills up to the last flush and a summary no more than five minutes stale. The archive is useful exactly when things go wrong.

• Sample every 60 s
• Heartbeat and summary every 5 min

If startup fails, run dot json is written with status crashed. If a run dies later, its status stays running, so readers use the heartbeat. A running run with a heartbeat older than one hundred eighty seconds is treated as crashed, unless the process id is alive on the same host. That avoids declaring a slow but healthy run dead.

• Startup failure: status crashed
• Stale over 180 s: crashed
• Unless pid alive on same host

To recap. Tests are doctest executables per module, run by ctest, fully offline with fake transports, blanked environment variables and temporary data roots. Fixed seeds keep fixtures reproducible. Golden tests hash canonical text so refactors are provably byte-identical. Unit tests with hand-worked answers say the behaviour was right. The run archive uses atomic mkdir for unique ids and a writer thread that never stalls the engine.

• Offline, seeded tests
• Golden hashes
• Atomic mkdir, writer thread

Everything so far ran inside one process. Next, in episode seventeen, we ask why the live system is split into many processes, and what each one is responsible for. This series is educational and not financial advice.

• Next: why many processes
• Subscribe for episode 17