Kestrel Broker Plumbing: Daily Login, Secret Redaction, Rate Limiters

Before a trading system sends a single order, it has to do three unglamorous things. It has to prove who it is. It has to make sure that proof never leaks into a log. And it has to stay politely under the broker's request limits. Get any of them wrong, and your day ends early.

• Login
• Redaction
• Rate limits

Last episode, kestrel_md fanned market data out to strategies and fell back to REST when the websocket failed. This episode is the plumbing underneath every broker call. We will cover the daily login, redaction, two cooperating rate limiters, and finally the symbol master and history downloads.

• kestrel_md recap
• Shared FyersClient
• Broker plumbing

The FYERS access token expires every day at six thirty in the morning, Indian time. There is no refresh token. So each morning the operator runs kestrel_history with the fyers-login flag. It is a deliberate human step, not something hidden in a cron job.

• Expiry: 06:30 IST daily
• No refresh token
• Operator runs login

The login starts by building an authorization URL. It carries a random state value: thirty-two hex characters. Kestrel keeps that value in memory. Whatever comes back from the browser must carry the same state, or it is rejected.

• Auth URL
• 32-hex random state
• Kept in memory

The redirect points at the loopback address, 127.0.0.1. Kestrel starts a one-shot local HttpServer, the same one we met in episode nineteen, and waits up to five minutes for the browser to arrive. It handles one request, then shuts down. No long-lived listener is left behind.

• Loopback redirect
• Waits 5 minutes
• Serves one request

When the redirect arrives, Kestrel compares the returned state with the one it generated. A mismatch means the request did not come from the login we started. This is the classic cross-site request forgery defence. Without it, any web page could push an auth code into your process.

• Compare state values
• Mismatch: reject
• CSRF defence

With a valid code, Kestrel posts to the validate-authcode endpoint. It must send an app id hash. That is the hex encoding of the SHA-256 of the app id, a colon, and the secret. For example, the string is the app id placeholder, colon, secret placeholder, and the output is sixty-four hex characters.

• sha256(APP_ID:SECRET)
• Hex encode
• POST validate-authcode

A token in hand is not yet a token that works. Kestrel calls the profile endpoint with it, as a verification step. Only if that succeeds does it persist the token. This way a typo or a half-working login never overwrites yesterday's file with garbage.

• Call profile
• Success: write file
• Failure: keep old file

Writing the token file has two requirements. It must be private, and it must be atomic. Kestrel creates the directory with mode zero seven hundred. It opens a temporary file with O_EXCL and mode zero six hundred, so only the owner can read it, and it refuses to reuse an existing file.

• Directory mode 0700
• O_EXCL, mode 0600
• Owner-only access

Then it writes, calls fsync so the bytes reach disk, and renames the temp file over the real name. Rename is atomic on a POSIX filesystem. A reader sees either the old complete token or the new complete token, never half of one. A crash in the middle leaves the old file intact.

• write, fsync, rename
• Readers never see partial
• Crash-safe

Most credential leaks are not hacks. They are logs. An error message that includes the request headers. A debug line that prints a response body. A crash report that quotes a URL. Once a token is in a log file, it is in backups, in chat pastes, and in screenshots.

• Leaks happen in logs
• Errors, bodies, URLs
• Logs get copied

So Kestrel redacts at the boundary. The value of the Authorization header becomes three asterisks. So do the token and the secret as known values. Anything that formats a request for display goes through the redactor first.

• Authorization becomes ***
• Known token and secret too
• Redact before display

Known values are not enough, because secrets also appear inside free text. So the redactor scrubs the text that follows keywords such as access token or secret key. This applies to log messages, error strings, and response bodies alike. The cost is a slightly over-eager scrub. The benefit is no leak by omission.

• Keywords trigger scrubbing
• Messages, errors, bodies
• Over-scrub beats leaking

Every broker request goes through one function, FyersClient call. It builds the headers, asks the limiters for admission, performs the request through libcurl with a twenty second timeout, and parses the JSON that comes back. One door in means one place to enforce the rules.

• Headers
• Limiters admit
• libcurl, 20 s timeout
• Parse JSON

When the broker answers with HTTP 429, or a body with code minus 429, Kestrel logs that it is not retrying, and records a strike. There are no automatic retries anywhere in the broker layer. A retry loop against a rate limit is how a small problem becomes a day-long block.

• HTTP 429 or code -429
• Log: not retrying
• Record a strike

The process limiter has conservative defaults: four requests per second, and ninety per minute. Environment settings can raise these, but they are clamped at eight and one hundred eighty. There are also daily caps: sixty thousand calls overall and nine thousand transactional. These are the code's defaults as of 2026.

• 4/s and 90/min
• Env clamp: 8 and 180
• Daily: 60000 and 9000

Any 429 starts a cooldown of at least sixty-one seconds, or the Retry-After value if the broker supplies a longer one. Why sixty-one and not sixty? Because the broker's minute window may not line up with ours, and a one-second margin is cheap.

• Cooldown at least 61 s
• Or Retry-After
• 1 s safety margin

Now the harsh rule. After two strikes within one IST day, every call fails immediately until midnight IST. Kestrel calls this the strike breaker. FYERS blocks an account for the day after more than three per-minute violations, so Kestrel stops itself at two, leaving one in reserve.

• 2 strikes in an IST day
• All calls fail until 00:00
• Broker blocks after more than 3

How does the limiter decide when a call may go? It keeps a sliding log of send times, and it hands out slots in the future. Instead of asking whether I can send now, a caller asks for the next legal slot, then sleeps until that moment.

• Log of send times
• Reserve a future slot
• Sleep until the slot

Here is the core of gateSlot. It builds two windows: per second and per minute. For each one, if the log has at least limit entries, the candidate slot is the entry limit places from the end, plus the span. Take the maximum across windows. Also remember how many entries to keep.

• Two windows
• slots[size - L] + S
• Max over windows

Two final lines matter. If the limiter has been used and the log is not empty, the slot is never earlier than the previous slot, so reservations stay ordered. Then the slot is pushed, and the oldest entries are popped until only the largest limit's worth remain.

• Never before previous slot
• push_back the slot
• Trim to keep

Take a limit of four per second. The log holds slots at zero, point one, point two, and point three seconds. A fifth call arrives at point four. The slot four places from the end is zero point zero. Add one second span, and the fifth call gets slot one point zero.

• Slots: 0.0, 0.1, 0.2, 0.3
• Call at 0.4 s
• slots[size - 4] + 1 = 1.0

Now a sixth call. The fourth most recent slot is point one. Add one second, and the slot is one point one. Notice what happened: the calls spread out naturally. Never more than four in any one-second window, and no call is rejected. It is only delayed.

• Fourth most recent: 0.1
• 0.1 + 1 = 1.1
• Delayed, never dropped

Concurrency adds a subtlety. The slot is reserved while holding the mutex, but the sleeping happens outside it, so other threads can reserve their own slots. After waking, the code re-checks conditions before sending, such as a cooldown or breaker that triggered meanwhile. If the clock jumps by more than two hours, the log is cleared.

• Reserve under mutex
• Sleep outside the lock
• Re-check before sending
• Clock jump over 2 h clears log

Here is the catch. The broker limits per account, not per process. Kestrel runs kestrel_md, kestrel_orders, kestrel_history and more. If each respects four per second on its own, together they might send far more. A per-process limit alone is the wrong abstraction.

• Broker limits per account
• Many Kestrel processes
• Sum exceeds the limit

So all processes share one fixed-size binary file under the data root, in run, fyers_rate, named by host. It holds a magic number, a version, a count, a cooldown-until time, a strike day, strikes, and two hundred fifty-six sorted slots. Defaults are eight per second and one hundred fifty per minute.

• One file per host
• Header plus 256 slots
• 8/s and 150/min

Every read-modify-write happens under flock with an exclusive lock. Process A takes the lock, reads the slots, reserves a slot, writes back, and releases. Process B waits for the lock only for that short moment. Crucially, the lock is never held across a sleep, so nobody blocks the others while waiting for their own slot.

• flock(LOCK_EX)
• Short read-modify-write
• Never held across a sleep

If a process dies while holding the lock, the kernel releases it, so no stale lock can freeze trading. A sanitize step drops slots that are stale or in the future, which protects against a corrupted file or a wild clock. And a 429 in any process sets the shared cooldown, so all processes pause together.

• Kernel frees lock on death
• Sanitize stale and future slots
• A 429 pauses everyone

Now the data side. FYERS publishes public CSV files, the symbol master, for NSE cash, NSE derivatives, BSE cash, BSE derivatives and MCX commodities. Each has twenty-one columns. Kestrel caches them with a twelve hour time to live, so a daily login day does not hammer the download.

• Five public CSVs
• 21 columns each
• 12 hour TTL

A cached file on disk is only trusted after validation. Kestrel checks the column count, and a floor on the row count; for NSE derivatives that floor is twenty thousand rows. A truncated or error page saved as a CSV fails the check. If a fresh download fails, a stale but valid copy is used as fallback.

• Check column count
• Row floor: NSE_FO 20000
• Stale fallback

For history, the broker caps how much one request may cover. Kestrel's windows are three hundred sixty-six days for daily bars, thirty days for minute bars, and five days for second bars. Longer ranges are split into consecutive windows, fetched one by one through the same limiters.

• 1D: 366 days
• Minutes: 30 days
• Seconds: 5 days

The arithmetic is simple. A year of one-minute bars is about three hundred sixty-five days divided by thirty, which is just over twelve, so about thirteen requests. At four per second that is trivial. A sixteen-entry LRU cache avoids refetching windows already seen in this run.

• 365 / 30 is about 12.2
• About 13 requests
• 16-entry LRU

Each day of history ends up in one of four states. Complete means the full session arrived. Partial means some bars are missing. Empty means a day with no data, such as a holiday. Failed means the request itself failed. Keeping these distinct stops a failure from being stored as an empty day, which would silently poison later backtests.

• Complete, Partial
• Empty, Failed
• Failure is not emptiness

Three pitfalls sum up the episode. First, retrying 429s in a loop; Kestrel records a strike and stops. Second, logging raw responses; redact at the boundary. Third, per-process limits when the broker limits per account; share the state through a locked file.

• No retry loops
• Never log raw responses
• Limit per account

To recap. The token dies at six thirty daily, so the operator logs in with a state-checked flow and writes the token atomically and privately. Secrets are redacted everywhere. Calls are admitted by a sliding-log limiter, then an account-wide shared one, with a strike breaker. Symbol masters and history are cached and validated.

• Safe login
• Redact everywhere
• Limit across processes

Next episode we ask a harder question: what happens when you send an order and the network might lie to you? Uncertain outcomes, tags, and a write-ahead ledger. This is educational content, not financial advice.

• Episode 23 next
• Uncertain outcomes
• Subscribe