Kestrel in One Picture: One Engine, Three Modes

Here is a quiet way for a backtest to lie. The research code and the production code are two different programs. They look similar, they were written by the same person, and yet they disagree in small ways that nobody notices until real money, or real paper, is on the line.

• Two programs, quiet differences
• Results that cannot be trusted

This series walks through the code of Kestrel, a C++20 trading platform for Indian markets. Its central rule is simple to say and hard to keep: backtest, paper trading and live trading run the same engine code. Only one small piece changes between them. By the end of this episode you will know which piece.

• Episode 1 of 26
• Whole system, no code yet

A note before we start. This channel is educational. Nothing here is financial advice, and any backtest number you see is illustrative. We will look at the whole picture first: modes, processes, the dependency graph, and the journey of one bar. Details come in later episodes, and I will point to them by number.

• Educational only
• Four views of one system

Imagine a backtester that computes position size one way, and a live bot that rounds quantities slightly differently. Or a research loop that applies risk checks, while the production path applies them in another order. Each difference is tiny. Together they mean the thing you tested is not the thing you are running.

• Different ordering of checks
• Different rounding of quantities

Kestrel removes the difference instead of managing it. There is exactly one engine. It receives bars, runs a fixed pipeline, lets strategies decide, and hands orders to an interface called the order gateway. In every mode the engine is the same compiled code. What sits behind the gateway is the only thing that changes.

• One engine
• Orders go through one interface

To be precise, one engine does not make a backtest predictive. Markets change, and fill models are approximations. What it removes is a whole class of bugs, where the test and the deployment disagree about what the code does. That is a narrower claim, and a defensible one.

• Removes a class of bugs
• Promises nothing about results

Kestrel is a C++20 research, backtesting, paper and live trading platform for the Indian exchanges: NSE, BSE and MCX. It talks to the market through the FYERS broker API. The code lives in the namespace kestrel, and every directory under src is one static library, named kestrel underscore and the directory name.

• C++20, namespace kestrel::
• src/<dir> becomes kestrel_<dir>

Research needs no network. Historical bars live in a local store of gzip compressed CSV files, one file per symbol per day. A backtest reads those files and nothing else. You can run it on a plane. Episode four is entirely about that store and how it avoids ever being corrupted.

• Local gzip CSV history store
• No network needed to backtest

The libraries are named for their jobs: common, config, data, store, history, engine, alpha, trading, oms, sim, pricing, gateway, wire, net, md, md fyers, broker, ordersvc, portal, archive and runhost. You do not need to memorise them. The point is that each is a separate library, and what each may depend on is written down in the build.

• 21 static libraries
• Dependencies declared, not implied

First mode: backtest. Bars come from the history store into the Engine. The Engine talks to a LocalOrderGateway wrapped around a SimulatedVenue, which fills orders using an Indian cost model. Everything runs on a single thread, so the same inputs always produce the same outputs. Determinism is a feature you will see paid for in episode five.

• Store, Engine, LocalOrderGateway
• SimulatedVenue, one thread

Second mode: paper trading. Now the bars are live, delivered by a service called kestrel md. The strategy sends orders to another service, kestrel orders, which fills them with a PaperVenue. Real market data, pretend money. The engine still does not know the difference, because it only ever sees the gateway interface.

• Live bars from kestrel_md
• kestrel_orders fills with PaperVenue

Third mode: live. Everything is identical to paper, except that kestrel orders routes the order to FYERS through a component called FyersRouter. Same bars, same engine, same strategy code. And I want to be honest here: this is the design. The live order path is heavily guarded, but it is not presented as proven in production.

• Same as paper, but FyersRouter
• Presented as design

Put the three modes side by side. In every column the engine box is the same colour, the same text, the same code. Only the gateway box changes: a local gateway around a simulated venue, then a remote gateway to a paper venue, then a remote gateway to a live router. That is the entire difference between research and production.

• Engine identical in every column
• Only the gateway changes

Now the process map. Paper and live run as several cooperating local processes, each with one job. There are five Kestrel binaries: kestrel md, kestrel orders, kestrel trader, kestrel portal and kestrel history. Plus one small Python script, ui slash server dot py, which is a browser gateway.

• Five binaries, one Python script
• Each process owns one job

kestrel md holds the single FYERS market data connection: a binary websocket with a REST fallback. It turns that stream into bars and fans them out to subscribers over a Unix socket at run slash md dot sock under the data root. One connection to the broker, however many strategies are listening.

• Single FYERS data connection
• Bars fan out over md.sock

kestrel orders is the authority. It owns sessions, the order management system, positions, risk checks, square off, end of day files, journals and run archives. It listens on orders dot sock and on loopback TCP port seven zero one zero. Every strategy's real book is here, and nowhere else.

• OMS, positions, risk, square-off
• orders.sock and 127.0.0.1:7010

kestrel trader runs one strategy group: the engine and its alphas. It keeps only a shadow book, a local copy of what kestrel orders reports. It contains no broker code at all. Control ports sit in the range eight zero eight one to eight zero nine two, so several trader processes can run side by side.

• Runs alphas, keeps a shadow book
• No FYERS code inside

The rest are support roles. kestrel portal is the research JSON API on port eight zero nine three: backtest jobs, results and downloads. kestrel history downloads bars and performs the daily FYERS login. And ui slash server dot py, written with only the Python standard library, is a small gateway for your browser on port eight zero nine zero.

• portal: research API on 8093
• history: downloads and daily login

Connect them and the picture is complete. FYERS to kestrel md. kestrel md to each trader over md dot sock. Each trader to kestrel orders. kestrel orders to FYERS only for live orders. All local links use the same simple format: a four byte length, then a JSON body. Episode eighteen covers that framing, and episode seventeen explains why we split into processes.

• Local links: length-prefixed JSON
• Details in episodes 17 and 18

Diagrams drift out of date. A build file does not, because the build fails if it is wrong. So in Kestrel the dependency graph is the architecture. Each module declares exactly which other modules it may link. If a dependency is not declared, the code that needs it simply cannot compile.

• Declared dependencies only
• Wrong link means failed build

Here are four lines from the top level CMakeLists. Each calls kestrel module with a name and its public dependencies. Pricing depends only on common. Sim, the simulator, depends on trading and pricing. Alpha depends on common, pricing and sim. And engine depends on data, sim, gateway, alpha, store and archive.

• Four module declarations
• PUBLIC_DEPS lists allowed links

Draw those lines as arrows and a graph grows upward from common. Pricing sits on common. Sim sits on trading and pricing. Alpha sits on common, pricing and sim. Engine sits on top of data, sim, gateway, alpha, store and archive. Dependencies only point downward, so there are no cycles, and lower layers never know the higher ones exist.

• Arrows point only downward
• No cycles, lower layers stay ignorant

Now the important detail. Look at the engine line. It lists gateway, which is an interface, the IOrderGateway. It does not list broker. It does not list md fyers. The engine can send orders only to something that implements the interface, and it has no idea whether that something is a simulator or a router.

• Engine links gateway, an interface
• Engine does not link broker

Follow that logic to the trader process. kestrel trader links neither broker nor md fyers. So a strategy process physically cannot place a FYERS order. There is no function to call and no symbol to link against. Even a bug, or a careless edit in an alpha, cannot send an order to the broker from that process.

• Trader links no broker code
• Safety enforced by the linker

There is a second boundary of the same kind. The alpha library never includes the store. Strategy code cannot open history files directly. It sees only what the engine hands it, one bar at a time. That is how you prevent an alpha from peeking at tomorrow's data. The rule is enforced by structure, not by promises.

• alpha never includes store/
• No peeking at future data

Time to watch one bar travel. This is a preview; each hop gets its own episode. The path in a backtest goes: store file, StoreSource, k way merge, EventBus, Engine handleBar, venue fills, positions, the alpha's onBar, then the OrderManager, and finally the venue again.

• Nine hops, one direction
• Preview only

Take a one minute RELIANCE bar, the one starting at nine fifteen in the morning India time. India has no daylight saving, so IST is a fixed five hours thirty minutes ahead of UTC, and the file stores that bar at three forty five UTC. StoreSource reads the gzip file and produces a Bar with instrument, start, end, open, high, low, close and volume.

• 09:15 IST = 03:45 UTC
• StoreSource yields a Bar

A real run has many symbols, each with its own file. The k way merge combines these sources into one stream ordered by time, with a deterministic tie break. The bar is then published on the EventBus. Same data in, same order out, every run. Episode five takes this apart.

• Many sources, one ordered stream
• Deterministic order every run

The EventBus delivers the bar to Engine handleBar. The first thing that happens is not strategy logic. Orders already waiting are given a chance to fill against this bar, in the venue. Then positions are updated and marked. Only after that does any alpha get to speak. Episodes six and ten explain why that order matters.

• Venue fills come first
• Alphas speak last

Now the alpha's onBar runs. It sees the bar and expresses what it wants. The OrderManager turns that into a normalised, risk checked order and sends it through the gateway to the venue. In a backtest that is the SimulatedVenue, and the order waits for the next bar to be filled, never the current one.

• Alpha decides, OrderManager sends
• Fill happens on a later bar

In paper mode the beginning and end of the trip change, and nothing in the middle does. The bar arrives from kestrel md instead of the store. The order leaves through a RemoteOrderGateway to kestrel orders instead of a local venue. The engine in between runs the very same handleBar.

• Only the ends of the path change
• handleBar is identical

Two honest warnings. First, the FYERS binary websocket decoder was tested only against a fake server. Its specification was reverse engineered from the official SDKs, and some protocol fields are marked unverified. The code logs a one time protocol facts line on the first real connection. Second, live order paths are design, not a track record.

• Decoder tested against a fake server
• Live path is design, not proof

Three misreadings to avoid. One engine does not mean one process; paper and live use several. Same code does not mean same results, since a simulated fill is still a model. And no broker code in the trader does not mean no risk; it means the risk is concentrated in one service where it can be guarded properly.

• One engine, several processes
• Simulated fills are still models

Here is the road ahead. Episodes two to sixteen are the research core: the build system, vocabulary, the history store, the engine, alphas, the order management system and risk, positions, the simulated venue, Indian costs, option pricing, metrics, portal jobs and golden tests. Episodes seventeen to twenty six cover services and the broker, ending with a day in paper trading.

• Episodes 2-16: research core
• Episodes 17-26: services and broker

To recap. Kestrel runs backtest, paper and live on one engine; only the order gateway changes. Six programs divide the work: md for data, orders for the authoritative book, trader for alphas, portal for research, history for downloads, and a Python browser gateway. The dependency graph enforces safety: a trader cannot place a broker order, and an alpha cannot read the store.

• Same engine in every mode
• Graph enforces the boundaries

Next time, in episode two, we look at how the code is built and the strict house rules that every file obeys. Remember, this is educational material and not financial advice. Thanks for watching.

• Next: build system and house rules
• Subscribe for the series