Kestrel Order Service: Sessions, Shadow Books, Crash Recovery

Imagine your strategy process is holding fifty shares of RELIANCE, with a limit order resting in the market. At two minutes past eleven, the process crashes. Who owns that position now? Who cancels that order? And when you restart, how does the strategy learn what it already holds?

• 50 RELIANCE held
• Working limit order
• Process crashes at 11:02

In the previous episode we made a single order placement safe: uncertain outcomes, tags and a write-ahead ledger. Today we zoom out to the service that owns all of that. We will cover sessions, the order protocol, idempotent submits, the trader's gateway, resync, and the two crash scenarios.

• Episode 23: safe placement
• Today: the order service
• Sessions, resync, crashes

The core design rule is simple: the authoritative book lives in kestrel_orders, not in the strategy. The trader process runs alphas and keeps only a shadow book, a local copy it uses to make decisions. It contains no broker code at all. If the trader is the only place a position exists, a crash erases it.

• kestrel_orders: authoritative
• kestrel_trader: shadow only
• No broker code in trader

The first pitfall is exactly that: keeping positions only inside the strategy process. After a crash, the restarted process believes it is flat while the account is not. It may buy again, doubling exposure, or never square off. Putting the truth in a separate service removes that whole class of bug.

• Crash erases local state
• Restart believes it is flat
• Truth must outlive the strategy

The order link uses the same four-byte length-prefixed JSON frames as every local link in Kestrel. Each message carries a field called t, its type. The runner side speaks hello, new, cancel, cancelAll, flatten, killSwitch, and bye. The service side answers with welcome, refused, ack, and events.

• Same framing as other links
• Runner: hello, new, cancel, bye
• Service: welcome, refused, ack

A session begins with hello. It carries the strategy identifier, a run identifier, the process id, the mode, the account, the costs, the risk limits, the alphas, the instruments, the capital, and an archive directory. In other words, the runner describes itself completely, and the service decides whether to accept it. Account and token values are placeholders, never shown.

• strategyId, runId, pid, mode
• costs, risk, alphas, instruments
• capital, archive dir

If the service accepts, it replies with welcome, and welcome carries the full state: positions, cash, open orders, today's fills, the kill switch and the carry. The runner does not have to ask for anything else. That single message is the foundation of recovery, which we will return to shortly.

• welcome = full state
• Positions, cash, open orders
• Fills today, kill switch, carry

The service can also say no, with a refused message and a code. The codes include duplicate session, auth, live disabled, account, login, and instrument conflict. After that, the stream is acknowledgements plus events: order, fill, risk, blocked and settled. Every refusal is explicit, so a runner never has to guess why it was turned away.

• refused{code}
• duplicate_session, auth, live_disabled
• account, login, instrument_conflict

Put it together. The trader sends hello, receives welcome, then sends new with a client order id. The service answers with ack, and later pushes an order event and a fill event as the venue acts. Notice that submit is a request and response, while fills arrive asynchronously as events.

• hello, welcome
• new, ack
• order and fill events

The service keeps exactly one session per strategy name. If a second live runner says hello with the same name while the first is alive, it gets refused with duplicate session. This prevents two copies of one strategy from trading the same book at once, a classic accident after a hurried restart.

• One session per strategy name
• Second live runner refused
• Prevents double trading

There is a deliberate exception. A hello with a new run identifier takes over the session, because the old process is evidently gone or replaced. And if the config digest has changed, the service does not trust old in-memory state. It rebuilds the book from the journal, so the new configuration starts from recorded facts.

• New runId takes over
• Changed digest: rebuild from journal
• Facts over assumptions

Inside the service, Session submit is a synchronous call answered from memory. It does not wait on the broker to build its reply. First it checks whether this client order id has been seen. Then it applies gates, and finally it hands the order to ExecutionCore submit, where risk is checked.

• Answered from memory
• Check clOrdId first
• Gates, then ExecutionCore risk

Here is the idempotent part. Suppose the runner sends order r7-12 to buy ten shares, and the ack is lost in transit. The runner retries r7-12. The service finds it in its table and returns the stored ack, marked duplicate true. Only one order exists. Ten shares, not twenty.

• Send r7-12: buy 10
• Retry same clOrdId
• Stored ack, duplicate: true

The second pitfall is retrying without a stable identifier. If every retry mints a fresh id, a lost reply turns into a second order. Retries are only safe when the receiver can recognise them. So the client order id must be assigned before sending, and reused on every retry.

• Fresh id per retry: double order
• Stable id: safe retry
• Assign id before sending

In paper mode, orders that pass risk go to a PaperVenue. That is the same SimulatedVenue from the backtester, fed by the service's own market data link. There is no transport hop to a broker at all. So paper trading exercises the same fill rules as research, with live bars.

• PaperVenue = SimulatedVenue
• Fed by service's own md link
• No broker transport

On the trader side, the engine talks to an order gateway interface, as in every mode. In paper and live, that gateway is RemoteOrderGateway. It keeps a local book, positions and risk, and exposes them read-only as a ShadowBook. Alphas can read it, but cannot change it, only the service's events do.

• Same IOrderGateway interface
• Local book exposed read-only
• ShadowBook updated by events

Each client order id is the run identifier, a dash, and a local counter. So run r7, order twelve, becomes r7 dash twelve. The run identifier keeps ids unique across restarts, and the counter keeps them unique within a run. That is what makes the idempotent lookup safe.

• clOrdId = runId + '-' + id
• Unique across restarts
• Unique within a run

Now the critical code, lightly simplified. The gateway calls roundTrip and waits up to the submit timeout, one thousand milliseconds by default. If it is not done, three things happen at once: the gateway sets degraded, remembers this order as timed out, and sends a cancel for that same client order id.

• roundTrip waits 1000 ms
• degraded = true
• Defensive cancel sent

Then it logs a warning, builds a local order marked Rejected with the reason order service timeout, and returns a failed SubmitResult. The strategy sees a clean rejection. It never has to wonder whether the order might exist, because the cancel is already on its way.

• Warn log with order id
• Local status Rejected
• Reason: order service timeout

Why this shape? The timeout is ambiguous: the order may or may not have reached the exchange. Failing closed means we assume the worst, cancel defensively, and stop trading. While degraded, every further submit is rejected until a late ack proves the link is healthy again.

• Timeout is ambiguous
• Cancel defensively
• Reject until late ack arrives

Every time a welcome arrives, the trader resyncs. That means the first connection, a reconnect after a blip, and a restart all take the same code path. One path is easier to test than three. And it is the path that rebuilds the shadow from the service's truth.

• Every welcome triggers resync
• First connect = reconnect = restart
• One path to test

Positions and cash are reset from the service, overwriting local guesses. Open orders are reconciled: if an order is gone from the service, its result is applied from closed orders, or it is marked Cancelled. The kill switch and carry are copied. And if the clock differs by more than one second, it warns.

• Positions and cash reset
• Gone orders: closedOrders or Cancelled
• Kill switch and carry copied
• Clock skew over 1 s warned

Fills need special care. The welcome includes all of today's fills, but some may already be applied locally. So each fill is applied only if its fill id is not in the seenFills set. Suppose fill f3 buys twenty shares and is already seen. Replaying it would turn a position of fifty into seventy. The set prevents that.

• fillsToday in welcome
• Apply only if fillId unseen
• f3 replay would make 50 into 70

Now the crashes. If the trader dies without sending bye, the service notices the disconnect. It waits a grace period, ten seconds by default, controlled by the disconnect cancel open after seconds setting. After that, the strategy's working orders are cancelled.

• No bye received
• Grace period: 10 s
• Then working orders cancelled

Crucially, the service does not stop acting for the dead trader. Square-off still runs, settlement still runs, and end-of-day files are still written. The run archive is finished with a crashed status, so the record shows honestly what happened. The position is managed even though no strategy is alive.

• Square-off continues
• Settlement and EOD continue
• Archive finished as crashed

Let us run our example. The trader holds fifty RELIANCE and has a working limit order. At eleven oh two it crashes. The service sees the broken link and starts its ten second timer. Nothing else changes yet: the position of fifty is safely recorded in the service's book.

• 11:02 crash
• Position 50 in service book
• Limit order still working

At eleven oh two and ten seconds the timer expires. The service cancels the working limit order, so nothing sits in the market with no one watching. The position of fifty remains, to be handled by square-off at the normal time, or by the restarted strategy.

• 11:02:10 cancel working limit
• Position 50 remains
• No orphan order in the market

At eleven oh five the trader restarts and sends hello with a new run id, so it takes over the session. The welcome carries a position of fifty and today's fills. The fresh shadow book has an empty seenFills set, so it applies each fill once and lands on fifty. It never reaches one hundred.

• 11:05 hello with new runId
• Welcome: position 50 + fills
• Shadow lands on 50

The other direction: the service itself restarts. For each strategy that has fills in today's journal, or orders in the ledger, it rebuilds from the session file, with the token stripped, plus the journal. Hellos from runners are deferred until that restore finishes, so no one sees half-built state.

• Rebuild from .session.json
• Token stripped
• Journal replay
• Hellos deferred until done

The journal is written with an fsync per fill, so a fill that was acknowledged is on disk. Replay applies fills in order: buy fifty at one price, say, and the position becomes fifty with that average cost. Same code, same inputs, same book. That determinism is the payoff of recording facts rather than state.

• fsync per fill
• Replay fills in order
• Same inputs, same book

Each strategy gets a small set of files. End-of-day results go in a folder per alpha, one csv per date. The journal is a jsonl file per date. Then there is dot session dot json for session state, and a live jsonl file per date. Together they let you audit or rebuild any day.

• EOD alpha/date.csv
• journal/date.jsonl
• session.json
• live/date.jsonl

For operators, the service exposes a status API on loopback port seven one zero two. GET endpoints list sessions and positions. POST endpoints trigger the kill switch and flatten. Because it is bound to loopback, only local processes can reach it, a topic from episode eighteen.

• 127.0.0.1:7102
• GET sessions, positions
• POST killswitch, flatten

Recap the rules. Truth lives in the service; the trader holds a shadow. Submits are idempotent by client order id. A timeout fails closed, with a defensive cancel and degraded mode. Every welcome triggers the same resync. And fills are journalled before they are trusted.

• Service owns truth
• Idempotent clOrdId
• Fail closed on timeout
• Resync on every welcome

A fair caveat: these live order paths are heavily guarded, but they are presented here as design, not as proven in production. The examples use illustrative numbers, and nothing here is financial advice. Test any such system thoroughly with paper trading before trusting it.

• Design, not production proof
• Illustrative numbers
• Not financial advice

In the next episode we look at the ten layers of safety that stand between a strategy and a real order.

• Next: ten safety layers
• Subscribe for episode 25