Inside Kestrel: The Per-Bar Pipeline, Derived Bars and Expiry

A single one-minute bar arrives at the engine. Within a few microseconds, orders may fill, an option price may be born from nothing, an expiring contract may be settled, and a strategy may decide to trade. The question is: in what order? Get that order wrong, and your backtest quietly cheats.

• One bar in
• Many things must happen
• Order is everything

In the previous episode we saw the event bus, the k-way merge, and why the engine replays the same bars in the same order every time. Today we zoom into what the engine does with each bar once the merge hands it over. This function is called handleBar, and it is the heart of Kestrel.

• Episode 5: the merge
• Episode 6: handleBar
• The heart of Kestrel

By the end you will know the order of steps in handleBar and the reason for each one. You will understand the venue pull model, how option and future bars are derived from an underlying bar, how expiry settlement works, and how the paper trading loop differs from a backtest. Educational only, and any numbers are illustrative.

• Step order and why
• Pull model and pump
• Derived bars and expiry
• The paper loop

Picture the pipeline as a vertical line of stations. Each bar enters at the top and lights up every station in turn. Settle expiries. Handle the instrument and update the market. Derive option and future bars. Advance the clock and let the venue fill. Check equity. Square off. Run the alphas. Record an equity point.

• Settle expiries
• Update market, derive bars
• Venue fills, marks
• Equity, square-off, alphas

Step one. If the clock has advanced since the last bar, the engine settles anything that expired up to the instant just before this bar began, that is bar start minus one, and then notifies equity. Settling first means a dead contract can never be traded, marked, or priced by the bar that follows.

• Only if time advanced
• settleExpired(bar.start - 1)
• Then notify equity

Step two handles bad input. If the instrument is unknown, invalid, or already settled, the bar is dropped. Otherwise the market state is updated with this bar. Only after that does anything else look at prices, so every later station sees a consistent market.

• Unknown, invalid, settled
• Drop or update market
• Consistent view for later steps

Step three. Derived bars are computed right now, synchronously, breadth first. They are never posted to the event bus. That detail matters: an option priced off this underlying bar exists in the very same step, not one event later, so the venue and the alphas see them together.

• Computed synchronously
• Breadth-first
• Never posted to the bus

Step four moves the clock: now becomes the larger of its old value and the bar's end time. Then the gateway receives onBars. The venue fills working orders using earlier bars, marks are updated, and execution reports are pumped through the system. Notice the phrase earlier bars. It is the key to the whole design.

• now_ = max(now_, bar.end)
• Venue fills from EARLIER bars
• Marks update, reports pumped

Step five runs onEquity, which drives the daily-loss kill switch. Step six is the square-off tick, which closes positions at the configured time of day, unless the run is still in warmup. These guards run before any strategy code. A strategy never gets to act on a day the risk layer has already halted.

• onEquity: daily-loss kill switch
• Square-off tick
• Skipped during warmup

Only now, in step seven, do the alphas receive onBar. After they run, step eight records an equity point. Alphas are last on purpose. By the time a strategy sees a bar, fills, marks, risk and square-off for that bar are all settled, so the strategy sees the world exactly as it is.

• Alphas' onBar runs last
• Then an equity point
• World is settled first

Why does the order matter so much? Imagine alphas ran before fills. A strategy sees the bar's close, submits a buy, and the venue fills it at a price from that same bar. That trade used information that did not exist when the order was placed. It is lookahead bias, and it flatters every backtest it touches.

• Alpha sees close
• Order fills on same bar
• That is lookahead

Here is the correct behaviour with illustrative times. At nine twenty, a bar ending at nine twenty-one reaches the alphas. The alpha decides to buy. In step four, this bar's fills have already run, so the order cannot fill here. The order sits in the venue. When the next bar, ending nine twenty-two, arrives, step four finally considers it.

• Bar ends 09:21: alpha decides
• Order rests in the venue
• Bar ends 09:22: order can fill

So the rule is simple. An alpha decides on a bar that has already been used for fills, and its order can only fill on a later bar. How the fill price is chosen is a separate topic, covered in episode ten. Here we only care that the pipeline order makes the wrong answer impossible.

• Decide on bar t
• Fill on bar t+1 or later
• Price rules: episode 10

Strategy code can throw. So callAlpha wraps every callback in a try and catch. If an alpha throws, it is faulted: disabled, with its open orders cancelled. Then pumpVenue runs, so the resulting reports apply immediately. One broken alpha cannot crash the run or leave orphan orders behind.

• try/catch around callbacks
• Exception: alpha disabled
• Orders cancelled, then pumpVenue()

Now the second big idea. Suppose a venue called back into the engine whenever an order filled. The alpha's onFill handler might submit a new order, while the venue is still looping over its working orders. The container changes under the iterator. That bug class is nasty, because it depends on timing and data.

• Venue calls back into engine
• Alpha submits inside onFill
• Venue mid-iteration: trouble

Kestrel's answer is a pull model. Venues never call back into the engine. They only queue execution reports, each one either an OrderUpdate or a Fill. The execution core later drains that queue by calling poll. The venue finishes its own work first, and only then does anyone react.

• Venues never call back
• They queue ExecutionReports
• Core drains via poll()

When a report is drained, it is applied in a fixed order. First the order management system. Then positions. Then risk counters. Then the journal. Finally the listener, which notifies observers and the owning alpha's onFill or onOrderUpdate. By the time any strategy code runs, all the books already agree.

• 1. OMS
• 2. Positions
• 3. Risk counters
• 4. Journal
• 5. Listener, alpha callbacks

Here is the real pump. The first line returns early if we are already pumping, or there is no venue. Then the flag is set, and a tiny struct named Reset clears it in its destructor. That is RAII: even if a handler throws, the flag cannot stay stuck on.

• Early return if re-entered
• pumping_ = true
• RAII Reset clears the flag

Next comes an endless loop. Clear the local reports, ask the venue to poll, and if nothing came back, break. Otherwise visit each report with std visit over the variant of OrderUpdate and Fill, calling handleOrderUpdate or handleFill. Why loop again? Handling a report can cause new reports, so we drain until the venue is quiet.

• for (;;) poll until empty
• std::visit with Overloaded
• OrderUpdate or Fill

What if a handler, say an alpha's onFill, submits an order and that triggers another pump call? The flag is already set, so the nested call returns immediately. The new reports are not lost, because the outer loop will poll again. No nested iteration, no mutation under an iterator, one clean drain.

• Nested pump() returns at once
• Outer loop polls again
• No nested iteration

Now step three in detail. The store holds bars for underlyings, but a backtest also needs bars for options and futures. Kestrel derives them, with the source tag derived. Every derived bar is computed from an underlying bar, using the time remaining to expiry. If the bar is at or after expiry, the derivation is skipped.

• Source tag: derived
• Computed from underlying bar
• Skip at or after expiry

For a future, first compute T, the year fraction from the bar's end to expiry. Then each of open, high, low and close is transformed as F equals S times e to the r T. Illustrative numbers: S of twenty-four thousand, r of seven percent, thirty days. T is about 0.082, and F comes to roughly twenty-four thousand one hundred and thirty-eight.

• T = yearFraction(bar.end, expiry)
• F = S e^(rT)
• Applied to O, H, L, C

For an option, Black-Scholes is evaluated at each of the underlying's open, high, low and close, with volatility taken from pricing dot volatility. The formula itself is episode twelve. What matters here is the mapping. Four underlying prices produce four option prices, which then become the option bar's own open, high, low and close.

• Price at each underlying O/H/L/C
• Vol from pricing.volatility
• Formula: episode 12

Here is a subtle trap. A put gains value when the underlying falls. So a put's high does not come from the underlying high. It comes from the underlying low. Take an illustrative underlying bar: open twenty-four thousand, high twenty-four twenty, low twenty-three eighty-five, close twenty-four ten. The put's high is priced at twenty-three eighty-five.

• Put value rises as S falls
• Put high uses underlying LOW
• Call high uses underlying HIGH

Raw model prices are not tradable prices. Open and close are rounded to the nearest tick. The high is rounded up and the low is rounded down, so the rounded bar always contains the real range. Everything is clamped to be at least zero, and volume is set to zero, since a derived bar has no real trading behind it.

• O and C: nearest tick
• H up, L down
• Clamp at zero; volume 0

Derived bars can themselves have derived children, so the list grows while we walk it. This simplified sketch uses an index loop, not a range-for. A range-for over a vector you push back into invalidates iterators when it reallocates, which is undefined behaviour. The index loop stays valid, and the walk is naturally breadth first.

• Simplified sketch
• Index loop, not range-for
• push_back may reallocate

Contracts end, and the books must end with them. When settleExpired runs, the engine first cancels the contract's working orders. Then it finds a settlement price. For an option, that is intrinsic value computed from the underlying's last price. For a future, the underlying's last price, else its own last price, else its mark.

• Cancel working orders
• Option: intrinsic from underlying last
• Future: underlying last, own last, mark

Next, for every strategy holding a non-flat position, the engine generates a synthetic Fill at the settlement price. Its order id is kNoOrder, because no real order exists, and its commission is zero. Illustrative: a long call, settlement price intrinsic of forty. The position closes through the same fill path as any trade, so positions and P and L stay consistent.

• One Fill per non-flat strategy
• orderId = kNoOrder
• Commission 0

Be honest about the edge. Settlement uses the first bar at or after expiry. If the data is sparse around that moment, the underlying price used may be slightly stale or later than the true expiry instant. It is a deliberate simplification of the design, and worth knowing when you read an expiry-day result.

• Uses first bar at or after expiry
• Sparse data: price may lag
• Simplification, not exact

Step eight records equity points, one per simulated second, but a long run would make millions of points. So the curve is capped at four thousand. When it fills, the engine keeps only the even-indexed points and doubles the sampling interval. The curve always spans the whole run, at a resolution that adapts to its length.

• One point per sim second
• Cap: 4000 points
• Keep even indices, double interval

Paper trading reuses the same pipeline, but time comes from the world, not from a file. First the engine replays warmup bars from the store. Then one SourceRunner thread per source posts BarEvents onto the bus. The engine thread loops on waitAndDrain with a fifty millisecond timeout, so it wakes both for bars and for the clock.

• Replay warmup bars from the store
• One SourceRunner thread per source
• waitAndDrain(50ms)

After each drain, the loop pumps the venue, runs squareOffTick with the larger of simulated now and the wall clock, and settles expiries about once a second. Using the wall clock matters: if no bar arrives, square-off time still comes. Without it, a quiet feed could leave positions open past the cutoff.

• Pump the venue
• squareOffTick(max(now, wall))
• Settle about once a second

One more safeguard. The submitGate rejects orders with the reason warmup until the instrument has a bar that ended after the run started. Replayed history is for building indicators, not for trading. An alpha cannot trade on a price that is hours old and was only loaded to prime its state.

• Reject reason: warmup
• Needs a bar ending after run start
• History primes, never trades

Two pitfalls stand out. First, a range-for over a vector that you push back into, which is undefined behaviour when it reallocates. Second, letting alphas see a bar before fills have run, which is the lookahead trap. Both look harmless in code review, and both are prevented here by structure, not by discipline.

• No range-for on a growing vector
• Alphas never before fills
• Structure beats discipline

Let us recap. Handlebar runs: settle expiries, filter and update the market, derive bars, advance the clock and let the venue fill, check equity, square off, run alphas, record equity. Venues queue reports and the pump drains them with a re-entrancy guard. Derived bars and expiry settlement keep options and futures consistent.

• Eight steps, fixed order
• Pull model with pump()
• Derived bars, expiry fills

Everything an alpha does plugs into step seven of this pipeline. In the next episode we open the alpha framework: its interfaces, its gates, and how target positions become orders.

• Next: the alpha framework
• Interfaces, gates, targets
• Subscribe for episode 7