Decoding the FYERS Binary Market-Data Protocol, Byte by Byte

Here are seventeen bytes. Zero zero, zero F, zero six, and so on. Somewhere inside them is the last traded price of a stock, and the only way to get it is to read them correctly. In this episode we decode the FYERS binary market-data protocol, one byte at a time.

• 00 0F 06 00 01 55 ...
• What price is this?

In the previous episode we built a hand-written websocket, which delivers whole binary messages to us. This episode begins where that one stops. We take a message payload and turn it into quotes. Bar building comes in the next episode, so we stay strictly with bytes.

• Episode 19: websocket framing
• Episode 20: what is inside

First, an honest framing, and it goes on screen too. The specification in the project docs was reverse-engineered from the official Python and Node SDKs. Some points are marked UNVERIFIED. The decoder was tested only against a fake server. On the first real connection, the code logs a single line of protocol facts, so the open questions can be settled against reality.

• Reverse-engineered from official SDKs
• Some points UNVERIFIED
• Tested against a fake server

The endpoint is socket dot fyers dot in, port 443, path slash hsm slash v1 dash 5 slash prod. We will take these in order.

• socket.fyers.in:443
• /hsm/v1-5/prod

The key comes from the access token itself. Strip the app id prefix, the part ending in a colon. What remains is a JWT with three dot-separated parts. Base64url-decode the middle one, the payload, and read two fields: hsm_key, and exp, the expiry. No network call is needed. The token itself is never shown here, only placeholders like token env var.

• Strip the APPID: prefix
• Base64url-decode the payload
• Read hsm_key and exp

Next we need topics. A REST call to data slash symbol-token, in batches of five hundred symbols, returns an fytoken for each. The first four characters name the segment: one zero one zero is nse cash, one zero one one is nse futures and options, one one two zero is mcx. Everything from character ten onward is the exchange token.

• REST symbol-token, batches of 500
• fytoken[0:4] is the segment
• fytoken[10:] is the token

From segment and token we assemble a topic string. A stock feed looks like s f, pipe, nse underscore c m, pipe, 3045. An index feed uses i f, and ends with the index name, such as Nifty 50.

• sf|nse_cm|3045
• if|nse_cm|Nifty 50

Then the conversation. Send auth, and wait for an ack carrying the letter K. Set the mode for channel eleven: full, F, or lite, L. And send a three byte ping, zero zero, zero one, zero B, every ten seconds.

• auth, wait for ack K
• mode: full F or lite L
• subscribe in chunks of 1500
• ping every 10 seconds

Every message we send follows one grammar, in big-endian byte order. A sixteen-bit length, which equals the total size minus two, because it does not count itself. Then one byte for type, one byte for field count, and then the fields.

• All big-endian
• u16 length = total minus 2
• u8 type, u8 field_count

Each field is tiny: one byte for its id, two bytes for its length, then that many bytes of value. This is a type-length-value shape, and it makes client messages easy to build.

• u8 id
• u16 len
• bytes

Let us test the grammar on the smallest message, the ping. Zero zero, zero one, zero B. The length is one, and the total is three bytes, so three minus two is one. The single remaining byte is type eleven, and there are no fields at all.

• 00 01 0B
• length 1, type 11

Now the auth message. Type one, with four fields. Field one is the hsm key. Field two is the single letter P. Field three is the single byte zero one. Field four is a source string. We build it on screen with placeholders, because the key is a secret, and the source value comes from configuration.

• type 1, 4 fields
• 1 hsm_key, 2 P
• 3 0x01, 4 source

A detail worth knowing: the official Python SDK writes the wrong length prefixes in some messages. The Node SDK writes them as the grammar says. Kestrel matches the Node SDK. When references disagree, choose one, record the choice, and let the first real connection tell you whether it was right.

• Python SDK writes wrong prefixes
• Kestrel matches the Node SDK

Now the server side. Every incoming message is dispatched on the byte at offset two, right after the length. If it is three, it is an ack. If it is six, it is market data. Anything else is not ours to guess about, so it is reported and skipped.

• Byte 2 is the type
• 3 ack, 6 data

The auth ack has two fields. Field one is a letter: K means accepted, N means refused. Field two is an unsigned thirty-two-bit number called ack count. Hold on to it. It tells us how often the server expects us to acknowledge data messages, and it changes the layout of every data message that follows.

• field 1: K or N
• field 2: u32 ack_count

A data message starts with its length and the type, six. Then, only if ack count is greater than zero, a thirty-two-bit message number. Then a sixteen-bit record count, followed by that many records. Notice the conditional field: the same bytes mean different things depending on state learned earlier.

• u16 len, u8 6
• u32 message_number if ack_count > 0
• u16 record_count

Records come in kinds, marked by a letter. Capital S, hex five three, is a snapshot. It carries a topic id, the topic name, a list of signed thirty-two-bit values, a multiplier, and a precision. Its most important side effect: it binds that topic id to the topic for this connection. Ids are just small numbers; the snapshot is how we learn what they mean.

• 'S' 0x53
• topic_id, name, i32 values
• multiplier, precision

After the snapshot, the stream is mostly updates. Capital U, hex five five. A sixteen-bit topic id, an eight-bit field count, and then that many signed thirty-two-bit values. Updates carry no name and no scale, which is exactly why the cached snapshot matters.

• 'U' 0x55
• u16 topic_id, u8 field_count
• i32 values

Capital L, hex four C, is the lite record: last traded price only. And one special value matters everywhere. The value hex eight zero zero zero zero zero zero zero, the smallest signed integer, means unchanged. It is a sentinel: do not overwrite what you already know.

• 'L' 0x4C: LTP only
• 0x80000000 = unchanged

To get a price, divide the raw integer by ten to the power of precision, times the multiplier. The stock feed fields are indexed: zero is last traded price, one is volume traded today, six bid, seven ask, twelve open interest, thirteen low, fourteen high, nineteen open. Volume and open interest are counts, so they are not scaled like prices.

• price = raw / (10^precision x multiplier)
• Fields: 0 ltp, 1 volume, 6 bid, 7 ask
• 12 OI, 13 low, 14 high, 19 open

Back to our seventeen bytes. Zero zero, zero F: the length is fifteen. Count the bytes after those two: fifteen. Matches. Next, zero six: type six, a data message. Our ack count is zero, so there is no message number. Then zero zero, zero one: one record.

• 00 0F 06 00 01 55 00 07
• 02 00 04 57 86 80 00 00 00

The record begins with five five: capital U, an update. Then zero zero, zero seven: topic id seven. Then zero two: two fields follow, each four bytes. The decoder checks that eight bytes are really available before touching them. We will see that check in code shortly.

• 55 = U
• topic 7, 2 fields

Field zero is zero zero, zero four, five seven, eight six. That is four times sixty-five thousand five hundred thirty-six, plus five seven eight six hex, which is twenty-two thousand four hundred six. Total: two hundred eighty-four thousand five hundred fifty. The cached precision is two and the multiplier is one, so divide by one hundred. Last traded price: two thousand eight hundred forty-five rupees and fifty paise.

• 0x00045786 = 284550
• precision 2, multiplier 1
• LTP = 2845.50

Field one is eight zero, zero zero, zero zero, zero zero. That is the smallest signed integer, the unchanged marker. So the volume in our cache is left exactly as it was. Notice what the update did: it patched one number and left the rest alone. A feed of small deltas, applied to a remembered state.

• 80 00 00 00 = INT32_MIN
• Volume stays as cached

Here is the update handler. It reads the id and the count as optionals. Then, in one condition, it asks three questions: did the id read succeed, did the count read succeed, and are there count times four bytes left? If any answer is no, it fails with trunc-update, a truncated update record, and stops.

• Read id and count
• Check bytes are available

Next, look up the id. If it is unknown, perhaps because a snapshot was missed, we skip exactly count times four bytes, so the reader stays aligned with the next record, warn once, and return true. This is not a fatal error. It just means we cannot interpret this update, but the rest of the message is still fine.

• Look up the id in byId
• Skip the values, warn once
• Return true: stream stays aligned

If the id is known, we take a reference to its state and loop over the fields. Each value is read. If the index fits in the stored values and the value is not the no-value sentinel, we overwrite that slot. Finally, one quote is built from the whole state and pushed to the output. The scaling by precision and multiplier lives in that quote step, applied once, at the edge.

• Reference to TopicState
• Skip the sentinel
• Push one quote

Why can that code dereference r dot i32 without checking? Because has already proved the bytes exist. The Reader behind it is bounds-checked: every read returns an optional, and an empty optional means the buffer ended. Bytes from a network are untrusted input, so no read is allowed to run past the end.

• Bounds-checked Reader
• Every read returns std::optional

Records have a kind letter but no length of their own. If we meet a kind we do not know, we cannot know how many bytes to skip. So the decoder drops the rest of that message, reports it, and moves on to the next message. Guessing here would turn one unknown byte into a stream of fake prices.

• Unknown record type: drop the rest
• No length per record

Here is a genuine ambiguity. When ack count is zero, the two SDKs disagree about whether a message number is present. So the decoder sniffs both offsets: it checks which layout produces a record count and record types that parse cleanly to the end of the message. It then warns once, and that is one of the facts the first real connection should settle.

• SDKs disagree when ack_count = 0
• Try both offsets
• Warn once

When ack count is greater than zero, the server wants to hear from us. Every ack-count-th message produces an ack, carrying the message number, so the server knows we are keeping up. The decoder emits it as an output, and the connection layer sends it. Decoding stays pure: bytes in, quotes and replies out.

• Every ack_count-th message gets an ack
• Counter keeps the server happy

Topic ids are bound per connection. Id seven on this socket may be a different instrument on the next one. So on reconnect, the topic id map is cleared, and the new snapshots rebuild it. Until then, updates for unknown ids are ignored with a warning, rather than being applied to the wrong instrument.

• Topic ids are per connection
• Clear the cache on reconnect

The first pitfall: trusting lengths. A count byte says forty fields, but only six bytes remain. Code that loops forty times reads past the end of the buffer. Every length is a claim from outside, so verify that claim against the bytes actually present before acting on it.

• Lengths come from the wire
• Check before you read

The second pitfall: keeping topic ids across reconnects. The failure is silent. A price arrives, it parses perfectly, and it is attached to the wrong instrument. Nothing crashes, which is exactly what makes it dangerous. Clearing the map on reconnect turns a silent error into a visible warning.

• Ids kept across reconnects
• Wrong price on the wrong symbol

The third pitfall: scaling the wrong things. Divide a volume by a hundred, and your traded quantity is nonsense. Worse, treat the sentinel as a number: minus two billion divided by a hundred is a very confident, very wrong price. Check the sentinel first, scale only price fields, and keep the raw integers in the cache.

• Do not scale volume like a price
• Do not apply the sentinel as a number

Let us recap. Snapshots bind topic ids and updates patch a cached state, with the unchanged sentinel respected. Every read is bounds-checked, unknown records drop the rest, and ids are cleared on reconnect.

• Key, topics, auth, mode, subscribe, ping
• Snapshots bind, updates patch
• Check bounds, clear on reconnect

And once more, honestly: this decoder has met only a fake server. The protocol facts line on the first real connection is how reality gets its vote. That is also why robust, warn-once handling matters more here than clever handling. This is educational, not financial advice.

• Fake server only so far
• One-time protocol facts line

We now have clean quotes, and that is where the decoder's job ends. In the next episode, we look at kestrel underscore m d: how quotes become bars, how subscriptions are counted, and how bars are fanned out to the other processes.

• Episode 21: kestrel_md
• Subscriptions, fan-out, bars, fallback