One Connection, Many Traders: Inside Kestrel's Market-Data Service

Imagine ten strategies, all hungry for the same live prices, and a broker that allows exactly one market data connection. How do you serve everyone without opening ten connections, and without letting the slowest reader hurt the rest? That is the job of kestrel_md.

• One broker connection
• Many local consumers

Last time, in episode 20, we decoded the FYERS binary protocol into ticks. Today we start from decoded quotes and follow them all the way to every trader. We cover subscriptions, fan out, bars, and what happens when the upstream connection fails.

• Input: decoded quotes
• Output: bars for everyone

We will look at the protocol, the single threaded server, the subscription book, and fan out with slow consumer protection. Then the client side, the one second bar builder, and finally websocket reconnects with REST fallback. A short honest note first: the binary decoder was tested only against a fake server, so treat the live feed as design, not proven in production.

• Protocol to fallback
• Decoder: fake-server tested

The md protocol rides on the length prefixed JSON frames from episode 18. Symbols look like NSE dot RELIANCE dash EQ.

• Field t names the type
• Symbols: NSE.RELIANCE-EQ

The first message from a client must be hello. The server answers welcome, or refused if it cannot take you. Unknown keys are ignored, so old and new versions can talk, but malformed JSON closes the connection immediately. Strict about structure, relaxed about extras.

• First message: hello
• Unknown keys ignored
• Malformed JSON closes

If some symbols were refused, the ack carries a rejected list of objects, each with a symbol and a reason. Then comes a snapshot, and from then on bar messages with the symbol s, start, end, and open, high, low, close and volume. Times are int64 milliseconds in JSON.

• ack lists rejected symbols
• bar: s, start, end, o h l c v

Connections can die silently, so both sides use a heartbeat. A ping goes out every five seconds. If a client stays silent for twenty seconds, the server treats it as dropped and cleans up. Without this, a crashed trader would keep its subscriptions alive forever.

• Ping every 5 s
• Silent 20 s = dropped

MdServer is built around a single poll loop. That loop owns every piece of client state: sockets, queues, subscriptions. Because only one thread touches that state, there are no locks to get wrong. This removes a whole class of races, at the price of discipline: nothing slow may run inside the loop.

• One poll loop
• No locks on client state

But validating a symbol can be slow, because the symbol master lookup may download a file. So a separate validation worker does that work. It hands results back through a mutex protected inbox, and then wakes the loop by writing a byte into a self pipe.

• Slow lookups off the loop
• Inbox + self-pipe wake()

Why the pipe? The loop sleeps inside poll until a socket is ready. A worker thread has no socket event to offer, so it writes one byte into a pipe that poll is also watching. The loop wakes, drains the inbox, and goes back to sleep. Simple, portable, and no busy waiting.

• poll watches the pipe too
• One byte wakes the loop

Now the heart of sharing: the SubscriptionBook. It keeps two maps. byClient_ says which symbols each client wants. bySymbol_ says which clients want each symbol. The second map is effectively a reference count: a symbol is active while at least one client is in its set.

• byClient_: client to symbols
• bySymbol_: symbol to clients

The upstream connection should care only about the set of active symbols, not about who asked. If client B also subscribes to RELIANCE, the count goes from one to two, and nothing changes upstream. Only when the set itself changes, a symbol appearing or vanishing, do we touch the broker.

• Refcount 1 to 2: no change
• New symbol: upstream subscribe

Here is takeDiff. It builds a set called now from every key in bySymbol_. If now equals applied_, the set the upstream already has, it returns nullopt: nothing to do. That one comparison is what makes redundant subscriptions free.

• Build the current set
• Equal to applied_? Return nullopt

Otherwise it computes two set differences. Symbols in now but not in applied_ are added. Symbols in applied_ but not in now are removed. It also stores the full active list, moves now into applied_, and returns the diff. Sorted sets make set_difference correct and the output deterministic.

• added = now minus applied_
• removed = applied_ minus now
• Then applied_ = now

Ten traders watching the same index would mean ten upstream subscriptions, duplicate quotes, and a hit on broker limits. And when one client leaves, it might unsubscribe a symbol others still need. The refcounted set avoids both problems.

• Naive: duplicate subscriptions
• Book: one upstream per symbol

When a bar arrives, the server encodes it to a frame exactly once, into a shared pointer. Then it enqueues that same pointer to every subscriber of the symbol. With ten clients, the JSON is built once, not ten times, and the memory is shared until the last client has written it out.

• Encode one frame
• Share the pointer

Each client has its own outbound queue, and clients drain at different speeds. If one trader stalls, its queue grows. Letting it grow forever would eat memory and eventually threaten the whole service. So there is a hard line: more than twenty thousand queued frames, and the client is killed with the reason too slow.

• Queue limit: 20,000 frames
• Over limit: killed, too slow

Killing sounds harsh, but it is safe. The client reconnects, sends hello again, and gets a fresh snapshot, so it resynchronises instead of replaying thousands of stale frames. The slow reader pays the price. Everyone else keeps getting data on time.

• Reconnect, then snapshot
• Others never wait

There are more bounds. The upstream inbox holds at most two hundred thousand bars, and when full it drops the oldest first. At most sixty four clients may connect. And quotes are coalesced per symbol, emitted every one thousand milliseconds, and only if the value changed.

• Inbox: 200,000, drop oldest
• Max 64 clients
• Quotes: 1000 ms, if changed

On the other side of the socket sits MdClientSource, used inside traders and the order service. To the engine it looks like any bar source, but it reads from md dot sock. Its first rule: open never throws. If the service is not up yet, the source reports a status of connecting and keeps trying.

• Looks like a bar source
• open() never throws

Reconnects wait half a second, then double up to a cap of five seconds. A successful connection resets the delay. A refusal does not reset it, so the delay keeps growing. Each new session re-sends hello and then every symbol it wants, because the server forgets everything when a connection ends.

• 0.5 s doubling to 5 s
• Success resets; refusal keeps growing

If the server rejects a symbol for capacity, the client retries those symbols every sixty seconds. And delivery is monotonic: a bar whose end time is not after the last delivered end is dropped. That makes replays after a reconnect harmless, because the engine never sees time go backwards.

• Capacity rejects: retry every 60 s
• Drop bars with non-increasing end

The feed gives last traded prices, not bars. The QuoteBarBuilder turns them into one second open, high, low, close bars. Each tick updates the current second: the first price opens, the extremes stretch the high and low, the latest price closes. When the second ends, the bar is emitted.

• LTP ticks in
• 1 s OHLC bars out

Quiet instruments may not trade for several seconds. To keep downstream clocks moving, the builder emits flat keep alive bars, but only inside the trading session and only within one hundred twenty seconds of the last price change. After that, silence is silence. Bar times use the local clock.

• Flat bars only in session
• Stop 120 s after last change

Volume is trickier. The feed reports vol traded today, a cumulative number. The bar volume must be the difference between observations. So the builder remembers the last cumulative value per symbol in a seen map. The very first observation is only a baseline, and contributes zero.

• Cumulative total in
• Delta out; first is baseline

In simplified form: if there is a seen value and the new total is at least that value, the delta is added to the bar. If the new total is smaller, it counts as a reset only when the IST date has changed. Otherwise the smaller value is ignored. Then seen is updated.

• Larger or equal: add delta
• Smaller: reset only on new IST day

Let us run numbers. First observation: one million. That is the baseline, so volume zero. Second: one million two hundred fifty, delta plus two hundred fifty. Third: one million one hundred, smaller, same day, so ignored and the baseline stays at one million two hundred fifty.

• 1,000,000: baseline, 0
• 1,000,250: plus 250
• 1,000,100: ignored

Fourth: one million four hundred. Compared with the stored one million two hundred fifty, the delta is one hundred fifty. Notice we did not compare against the ignored one million one hundred, which would have double counted. The bars total four hundred, which matches the real rise from one million to one million four hundred.

• 1,000,400: plus 150
• Total 400 matches the rise

Why the baseline rule? Suppose the first observation of the day were treated as volume. A symbol that already traded a million shares would show a fake giant bar of one million in its first second. Strategies using volume would see nonsense. Baselines first, deltas after, is the guard.

• First total is not volume
• Baseline prevents fake spike

Now the upstream side. Before every connection attempt, the websocket reader re-reads the token file, so a fresh daily login is picked up without a restart. During market hours, if nothing arrives for thirty seconds, the connection is declared dead. Retries back off at one, two, five, ten, then thirty seconds.

• Re-read token each attempt
• Dead after 30 s silence
• Backoff 1, 2, 5, 10, 30 s

If the broker refuses the token, hammering it again would be pointless and could look abusive. So a refused token is not retried for ten minutes. How tokens are obtained is the topic of the next episode. Here we only need to know that refusal is treated differently from a network error.

• Refusal: wait 10 minutes
• Network error: backoff ladder

After three failed connects, or a refusal, the service does not go dark. It switches to REST polling and reports the state degraded. Meanwhile the socket is retried every sixty seconds. When it recovers, the state returns to live, and rebaseVolume is called so the next cumulative volume is a fresh baseline, not a huge delta.

• 3 failures or refusal: degraded
• Retry socket every 60 s
• Recover: rebaseVolume()

Switching paths could emit the same second twice, once from each source. A per-symbol lastEnd gate prevents this: a bar is accepted only if its end is after the last end already published for that symbol. Whichever path delivers first wins, and the other path's copy is silently dropped.

• Per-symbol lastEnd gate
• Switching never duplicates

The REST poller asks for at most fifty symbols per call. It budgets two thirds of the per minute limit, capped at two calls per second, leaving headroom for other users of the same quota. The limiter itself is episode 22. If the broker answers with code minus three hundred, some symbol is bad.

• Max 50 symbols per call
• Budget: 2/3 of limit, max 2/s

One invalid symbol would fail the whole batch. So on code minus three hundred, the poller bisects: it splits the batch in half, retries each half, and recurses until the offending symbols are isolated. Good symbols keep flowing, and only the bad ones are set aside.

• Split the failing batch
• Recurse to isolate bad symbols

Everything that flows through is also recorded. The BarRecorder writes bars to the history store under the provider fyers live, so today's live session becomes tomorrow's backtest data. On SIGINT it is flushed first, before anything else shuts down, so a Ctrl C does not lose the last bars.

• Provider: fyers_live
• Flushed first on SIGINT

One poll loop owns all client state. The SubscriptionBook refcounts symbols, and upstream hears only when the active set changes. Frames are encoded once and shared, and slow clients are cut at twenty thousand frames. Bars come from quotes with baseline safe volume deltas, and failover through REST never duplicates a bar.

• Refcounts, encode once
• Baselines, lastEnd gate

Everything here was educational design, with numbers that are illustrative and a decoder that has only met a fake server. Next time, episode 22: talking to the broker REST API safely, with the daily login and rate limits.

• Next: login and rate limits
• Subscribe for episode 22