Why Kestrel Runs as Many Processes: The Live Architecture

Imagine nine trading strategies, each running inside its own copy of the same program, and each one quietly polling the broker for fifty symbols every second. Nothing looks wrong, until the broker decides you have been rude. This episode is about why Kestrel, when it moves from backtest to paper and live, stops being one program and becomes several.

• Nine strategies
• Fifty symbols each
• One broker account

In the previous episode we looked at golden tests, fakes and the run archive, the tools that keep the offline engine honest. Today we leave the offline world. Paper and live trading are the same engine, but they run as several cooperating local processes, and the reasons are worth understanding.

• Episode 16 recap
• Offline engine is tested
• Now: paper and live

We will map the processes and what each one owns. Then we go through six concrete reasons for the split, see how the processes find each other, and study the small piece of signal handling that makes shutdown predictable. We finish with the composition root that wires one trader run together.

• The process map
• Six reasons
• Discovery and signals

Start with the picture. In a backtest, everything lives in one process: the engine, the order gateway, the simulated venue. For paper and live, that single block is cut along its natural seams. Market data goes into one process, the authoritative order book into another, and each strategy group gets a process of its own.

• Backtest: one process
• Paper and live: split
• Cut along natural seams

kestrel_md holds the one FYERS market-data connection. Upstream it uses a websocket, called fyers underscore ws, with a REST fallback called fyers underscore rest. It keeps subscription reference counts, the last bar and quote per symbol, and it records bars under the provider fyers underscore live and quotes under fyers underscore quotes. Its status page is on loopback port 8079.

• One FYERS connection
• Websocket plus REST fallback
• Status: 127.0.0.1:8079

kestrel_orders is the authoritative book for every paper and live strategy. It contains two venues: PaperVenue, which fills simulated orders, and FyersRouter, which routes live orders to the broker. It listens on a Unix socket named orders dot sock under the run directory, plus loopback port 7010, with status on 7102.

• Authoritative book
• PaperVenue and FyersRouter
• orders.sock, 7010, status 7102

kestrel_trader runs one strategy group. It has an engine thread, the alphas, and only a shadow book, a local copy of what the service says is true. Paper groups get control ports from 8081 to 8092. Crucially, it links no FYERS code at all.

• One per strategy group
• Shadow book only
• Control ports 8081 to 8092

Two more pieces round out the map. kestrel_portal serves the research API on port 8093, and ui slash server dot py is a small Python gateway for the browser on port 8090. There is also an optional kestrel_ivsurface, which is simply another market-data client. Notice that it is a client, not a second connection to the broker.

• kestrel_portal: 8093
• ui/server.py: 8090
• kestrel_ivsurface: optional md client

Put it together. Data flows from FYERS into kestrel_md, then out to many traders. Traders send their intentions to kestrel_orders, and orders is the only process that can ever talk to the broker for trading. The details of the framing on these links belong to episode 18, so here we only care about who talks to whom.

• Data flows down
• Intentions flow up
• Orders owns the broker

Reason one: rate limits belong to the account, not to your process. FYERS allows 10 requests per second, 200 per minute and 100,000 per day, and repeated per-minute violations can block the user for the rest of the day. Also, only one websocket is allowed per API key anyway, so a second feed in another process is not just wasteful, it may simply not work.

• 10 per second
• 200 per minute, 100,000 per day
• One websocket per API key

Take nine paper groups, each polling 50 symbols once per second inside its own process. That is 9 times 50, so 450 symbol requests every second, against a ceiling of 10 requests per second. Even if batching helps, the cost still scales with the number of groups. These numbers are illustrative, but the multiplication is the point.

• 9 groups times 50 symbols
• 450 lookups per second
• Cost scales with N

Now route the same nine groups through kestrel_md. The service holds one upstream connection and keeps a reference count per symbol. When the tenth trader subscribes to NIFTY, the count goes from nine to ten and zero extra calls leave the machine. The cost to the broker is constant, no matter how many consumers you add.

• One upstream connection
• NIFTY refcount 9 to 10
• Zero extra broker calls

Reason two: blast radius. Alpha code is the part of the system that changes most often, and it can contain bugs. A slow implied volatility solver, an out-of-range index, an unhandled exception: in one process any of these can take down everything. In separate processes, only the trader that owns the alpha dies.

• Alphas change often
• Bugs and slow solvers happen
• Crash stays local

Reason three: live safety by construction. Only kestrel_orders contains order-sending code. kestrel_trader links no FYERS code, so even a catastrophic bug in an alpha has no function to call that could place a real order. Paper mode goes further: its venue has no transport at all, so there is nothing to send with.

• Only orders sends orders
• Trader links no broker code
• PaperVenue has no transport

Reason four: surviving crashes. The book lives in the service, not in the trader. If a trader dies, positions are still there. The service still squares off at the end of the session, still settles expiries, and still writes its end of day files. When the trader restarts, it performs a full resync and rebuilds its shadow book from the truth.

• Book lives in the service
• Square-off and EOD continue
• Restart means full resync

Reason five: the same engine runs everywhere. The engine talks to an IOrderGateway, and that is the only seam. In a backtest it is a LocalOrderGateway that owns the book in process. In a trader it is a RemoteOrderGateway that forwards to the service. Strategy code cannot tell the difference, which is why a backtest means something for paper.

• Engine talks to IOrderGateway
• LocalOrderGateway in backtests
• RemoteOrderGateway in traders

Reason six is about the browser. Every service speaks JSON on loopback, and the browser talks to exactly one origin, the Python gateway. Because the gateway proxies requests, there is no cross-origin dance to configure and no service needs to be exposed beyond the machine.

• JSON on loopback
• Browser sees one origin
• No CORS configuration

To recap the six reasons: per-account rate limits, a small blast radius, safety by construction, crash survival, the same engine everywhere, and one browser origin. None of them is about speed. Every one is about making failure boring and bounded.

• Rate limits, blast radius
• Safety, crash survival
• Same engine, one origin

With many processes, you need discovery. Each service writes a small JSON file into the run directory under traders. A trader is named name dot json, while md, orders and research use role at name dot json. The file records the process id, the control bind address and the control port, and it is removed on exit.

• run/traders/ directory
• name.json or role@name.json
• pid, controlBind, controlPort

All of this hangs off dataRoot. It is the environment variable KESTREL underscore DATA underscore ROOT if set, otherwise the data folder under the current working directory. Every process that agrees on dataRoot agrees on where the sockets and registry files live, without any central coordinator.

• KESTREL_DATA_ROOT, if set
• Else cwd/data
• No central coordinator

The browser gateway does not trust the registry blindly. It ignores entries whose process id is dead, entries bound to anything other than loopback, entries with duplicate ports, and entries that claim the gateway's own port. A stale file from a crash, or a malformed one, simply never shows up in the interface.

• Dead pid: ignored
• Non-loopback bind: ignored
• Duplicate or own port: ignored

Now shutdown. A multi-threaded program that receives control C has a problem: the signal can land on any thread, at any moment, and a signal handler may only call a tiny set of async-signal-safe functions. Doing real cleanup from a handler is a classic source of rare, unreproducible bugs.

• Signal lands on any thread
• Handlers are very restricted
• Cleanup in handlers is fragile

Kestrel's answer is to avoid handlers entirely. Before any thread exists, main blocks SIGINT and SIGTERM with pthread underscore sigmask. Because threads inherit the signal mask of the thread that creates them, every thread in the process starts with those signals blocked. Also, SIGPIPE is ignored, so a write to a closed socket becomes an error code instead of a kill.

• Block before threads start
• Mask is inherited by all threads
• SIGPIPE ignored

Here is the simplified setup. We build a set with SIGINT and SIGTERM, and block it. Then a dedicated watcher thread loops on sigtimedwait with a 200 millisecond timeout. When a signal arrives, a counter increments. On the first, we call requestStop for a graceful shutdown. On the second, we call underscore Exit with 130.

• sigtimedwait, 200 ms
• First signal: requestStop()
• Second signal: _Exit(130)

Look at what the highlighted line buys us. The signals are blocked in every thread, so no asynchronous handler ever runs. The watcher pulls the signal out synchronously, like reading from a queue, in ordinary code. That means requestStop can use normal locks and flags, and there are no async-signal-safety worries at all.

• No async handler ever runs
• Signal read like a queue
• Ordinary code may be used

Why two stages? The first signal asks nicely: threads finish, recorders flush, files are closed. If something hangs and you press control C again, the second signal calls underscore Exit with code 130 immediately. kestrel_md adds a 2.5 second watchdog that forces exit after its recorder has flushed, so a stuck upstream cannot hold the process open forever.

• First: graceful stop
• Second: immediate exit 130
• md watchdog: 2.5 seconds

Exit codes are also part of the contract. Zero means success. One means a runtime failure. Two means a configuration or usage error, such as a bad flag or an invalid config file. A supervisor or a shell script can tell these apart without parsing any log text.

• 0: ok
• 1: runtime failure
• 2: config or usage error

Now, who wires a trader together? RunHost, the composition root of a trader run. It sets up the log sink, reserves the archive directory, builds the engine with a RemoteOrderGateway, starts the control API, and writes the registry entry. Everything that has a lifetime is created in one place, in a clear order.

• Log sink
• Archive directory reservation
• Engine, control API, registry

One subtle choice: the archive directory is sent to the service in the hello message. The trader does not write the CSV files; the service does. That way the files come from the authoritative book, not from a shadow copy that might have missed something. And if the run fails, RunHost writes a crashed run dot json, so a failed run is still recorded.

• Archive dir sent in hello
• Service writes the CSVs
• Failure leaves crashed run.json

The browser gateway is deliberately small: Python's standard library http server, nothing more. Requests to slash svc, then role, then name, then api, are proxied, but only to ports that appear in the registry on loopback. The browser cannot ask the gateway to fetch an arbitrary address.

• stdlib http.server
• /svc/role/name/api/...
• Only registered loopback ports

A word of honesty. This architecture is a design, and parts of it are untested against reality. The FYERS binary websocket decoder was tested only against a fake server, its specification was reverse engineered from the official SDKs, and some protocol fields are marked unverified. The live order paths are heavily guarded, but they should be read as design, not as proven in production.

• Websocket tested on a fake server
• Some fields UNVERIFIED
• Live paths: design, not proven

Three mistakes to avoid when you build something like this. Do not give each strategy its own broker connection, because limits are per account. Do not install real signal handlers that do real work. And do not let the strategy process hold the only copy of the book, or a crash will erase the truth.

• One broker connection per strategy
• Real work in signal handlers
• Book stored only in trader

Kestrel splits paper and live into kestrel_md, kestrel_orders and kestrel_trader, plus a portal and a browser gateway. The reasons are rate limits, blast radius, safety by construction, crash survival, one engine with swappable gateways, and one origin for the browser. Services find each other through registry files, and signals are handled by a watcher thread using sigtimedwait.

• Three core processes
• Six reasons
• Registry files and signal watcher

Next time, in episode 18, we look at the bytes on those local sockets and how they are secured: framing, and loopback security. This is educational content, not financial advice, and every number here is illustrative.

• Next: wire framing
• Loopback security
• Subscribe