Wire Framing and Loopback Security: Inside Kestrel's Local Sockets

Imagine a four byte header arrives on a local socket, and it says the message that follows is four gigabytes long. A naive server starts allocating, or waits patiently for bytes that will never come. Kestrel rejects it before a single byte of body arrives. This episode is about how.

• Hostile length header
• Rejected before the body

In the previous episode we saw why Kestrel runs as several cooperating local processes. Those processes only talk over Unix sockets and loopback TCP. So the links between them are the spine of the live system, and they must be both correct and hard to abuse.

• Local links are the spine
• Correct and hard to abuse

We will make four things. A streaming framer that turns a byte stream into messages. A Unix socket listener that is never reachable with loose permissions. A token check that does not leak through timing. And a guard that stops a malicious web page in the operator's own browser from talking to our local server.

• Framer
• Socket hardening
• Token compare
• HTTP guard

A socket gives you a stream of bytes, not messages. Two sends of ten bytes might arrive as one read of twenty, or as seven reads of three. So the receiver needs a rule that says where one message ends and the next begins. The classic answer is to put the length in front.

• Streams have no message edges
• Chunking is arbitrary

A Kestrel frame is four bytes of big-endian length, followed by that many bytes of UTF-8 JSON. The maximum is one mebibyte. And an empty frame, a length of zero, is a protocol error, because no legitimate message is empty. Both rules are enforced by the reader, not by convention.

• 4-byte big-endian length
• UTF-8 JSON payload
• Max 1 MiB, never empty

Big-endian means the most significant byte comes first, the same order a person writes a number. Reading the header does not depend on the machine's native byte order, so both ends agree regardless of hardware. It also makes the hex dump readable by eye, which is handy when you debug a socket at night.

• Most significant byte first
• Independent of host byte order

FrameReader is a streaming parser. You append bytes with feed, whatever chunk size the network gave you. Then you call next. It returns NeedMore until a whole frame is present, and then it returns the payload. It never blocks, never reads the socket itself, and so it is trivial to test with arbitrary chunking.

• feed() appends bytes
• next() returns NeedMore or payload
• No I/O inside the reader

Here is the heart of next, simplified. First, if fewer than four bytes are available, we cannot even read the length, so NeedMore. Second, if the length exceeds the maximum, error. Third, if the length is zero, error. Only fourth do we ask whether the whole body has arrived, and if not, NeedMore.

• avail < 4: NeedMore
• n > max: Error
• n == 0: Error
• avail < 4 + n: NeedMore

Notice the order. The size check happens before we wait for the body. If the checks were swapped, a hostile length of four gigabytes would make us return NeedMore and keep buffering, hoping for the rest. The attacker would simply drip bytes and make us hold memory. Checking first turns that attack into an immediate error.

• Size check before waiting
• No memory held for a lie

Let us do it with real bytes. The message curly brace, quote t quote, colon, quote ping quote, curly brace is exactly twelve bytes of JSON. Twelve is hex zero C. So the frame starts with 00 00 00 0C, then 7B 22 74 22 and on through the closing brace 7D. Sixteen bytes in total.

• Payload: 12 bytes
• Header: 00 00 00 0C
• Frame: 16 bytes

Now the network is unkind and delivers only three bytes first: 00 00 00. We feed them. We call next. Available is three, which is less than four, so the answer is NeedMore. Nothing is consumed, nothing is lost. The three bytes simply wait in the buffer.

• feed 3 bytes
• avail = 3 < 4
• Result: NeedMore

Then the remaining thirteen bytes arrive. The buffer now holds sixteen. Available is sixteen, so we can read the length: twelve. It is under the maximum and not zero. Sixteen is at least four plus twelve, so the body is complete. next returns the twelve payload bytes, and the cursor advances past the frame.

• avail = 16 >= 4 + 12
• Payload returned
• Cursor advances by 16

Now the attacker. Four bytes arrive: FF FF FF FF. As a big-endian number that is about four point three billion, far above one mebibyte. Available is four, so we pass the first check, read the length, and the second check fires. Error. No body ever arrived, and none was awaited.

• FF FF FF FF = 4,294,967,295
• Over the 1 MiB limit
• Error, no body needed

One more detail. As frames are consumed, a cursor moves forward through the buffer. Erasing the front after every frame would copy the remaining bytes each time. So feed compacts only when the consumed prefix is at least half the buffer. Each byte is moved a bounded number of times, which makes the cost amortised constant per byte.

• Cursor marks consumed prefix
• Compact at half the buffer
• Amortised O(1)

On top of the reader sits FrameChannel. It owns a non-blocking file descriptor, a FrameReader for input, and an outbound queue for output. Everything the md service and the order service do with a client goes through this one small class, so the careful handling lives in one place.

• Non-blocking fd
• FrameReader inbound
• Outbound deque

The outbound queue holds shared pointers to const strings. Why? Because kestrel_md fans one bar out to many subscribers. We encode the frame once into a string, and every subscriber's queue holds a pointer to the same bytes. One encode, N sends, no per-client copies, and the data cannot change underneath anyone.

• shared_ptr to const string
• Encode once
• Every subscriber shares it

A non-blocking send may write only part of a frame. So flush loops over the queue with a byte offset into the front frame. It calls send with MSG_NOSIGNAL, so a closed peer gives an error code instead of killing the whole process with a signal. And when the kernel says EAGAIN, we stop and wait for the socket to be writable.

• Byte offset into front frame
• MSG_NOSIGNAL: no SIGPIPE
• Stop on EAGAIN

Reading has a limit too. A single call to read does at most sixteen reads of sixty-four kibibytes. That is one mebibyte per call. A client that floods the socket cannot keep the poll loop busy forever; after its quota the loop moves on to the other clients and comes back next round.

• 16 x 64 KiB per read call
• Busy client cannot starve others

A Unix socket lives in the filesystem, which means filesystem permissions decide who can connect. That is a gift, since no other local user can touch it if we set the mode right. But it also brings classic file problems: stale files, races between creation and chmod, and deleting something that is not ours.

• Permissions decide who connects
• Stale files, races, wrong deletes

The listener starts by making the socket directory with mode 0700, so only our user can enter it. Then it looks at the path. If something exists and it is not a socket, we refuse, because we will not delete a regular file someone placed there. If it is a socket, we try to connect to it.

• Directory mode 0700
• Not a socket: refuse
• Socket: probe it

If a live server answers the probe, we refuse with the message: another kestrel_md is already listening. Two market data services would split subscriptions and confuse every client. If nobody answers, the socket is a leftover from a crash, so we unlink it and carry on. The probe separates a running instance from a corpse.

• Live server: refuse
• No answer: unlink stale socket

Now the clever part. If we bind in the target directory and then chmod, there is a window where the socket exists with permissions from the umask. Changing the umask would affect the whole process. Instead, Kestrel makes a private mkdtemp directory, binds there, calls chmod 0600, and then renames the socket into place. Rename is atomic.

• Bind in private temp dir
• chmod 0600, then rename
• Never visible with loose mode

On shutdown the listener unlinks the socket path, but only if the inode is still ours. It remembers the identity of the file it created. If a new instance has since replaced it, the path now points to someone else's socket, and deleting it would cut off a healthy server. We compare first and leave other people's sockets alone.

• Remember our inode
• Unlink only if it matches

Clients connect in non-blocking mode with a two second deadline. A blocking connect to a dead address can hang a thread for a long time, and shutdown would hang with it. So Kestrel polls in slices of ten milliseconds, and between slices it calls a cancelled callback. A stop request is noticed within ten milliseconds.

• 2 s deadline
• 10 ms poll slices
• cancelled() callback

Once connected, TCP sockets get TCP_NODELAY, so small frames are not held back by Nagle's algorithm waiting to coalesce. A tick that is late is a tick that is wrong. They also get SO_KEEPALIVE, so a peer that vanished without closing is eventually detected rather than assumed alive forever.

• TCP_NODELAY: no batching delay
• SO_KEEPALIVE: detect dead peers

A hostname is not an address. localhost might resolve to several. The function isLoopbackHost resolves the name and requires every returned address to be in 127 slash 8, or be colon colon one, or be a v4-mapped 127 address. If even one result points elsewhere, the host is not treated as loopback. Any one wrong answer is enough to fail.

• Resolve the name
• Every address must be loopback
• One outsider fails the host

Secrets are the next topic. Tokens live only in environment variables. Config files name the variable, using a field called tokenEnv, and never contain the value. The config loaders go further: they reject any key literally named token. So a secret cannot be committed by accident, because the parser refuses to accept it.

• Env vars only
• Config names, never holds
• Key named token: rejected

The policy is simple. Unix sockets are protected by mode 0600, so no token is needed. Loopback TCP without a token is allowed, since only local processes can reach it. But non-loopback TCP without a token of at least thirty-two bytes is refused at startup, and the process will not even begin listening.

• Unix socket: mode 0600
• Loopback TCP: token optional
• Non-loopback: 32+ byte token

The token is checked in the hello message. The obvious way is a string comparison, but memcmp and equals stop at the first differing byte. A guess with a correct first byte takes slightly longer to reject than a wrong first byte. Measured over many tries, an attacker can learn a secret one byte at a time.

• Early exit leaks timing
• Learn the secret byte by byte

Here is the real function. It starts diff at zero if the lengths match and one if they do not. Then it loops over every byte of the expected token, and ors the xor of each pair into diff. If the given token is too short, it compares against a scrambled copy of the expected byte instead. At the end, diff equal to zero means equal.

• Length mismatch sets diff to 1
• Loop covers every expected byte
• No early exit

Two details matter. The loop runs for the length of the expected token, never the given one, so the work does not depend on what the attacker sends. And for a short guess, the filler byte is derived from the expected byte, so the loop body stays the same and no out-of-range read happens. Equal length matters, but it is not hidden; the token length is not the secret.

• Work follows expected length
• Padding keeps the loop uniform
• Timing is flat

Now a different threat. Our local HTTP servers, the portal and the browser gateway, listen on loopback. You might think nothing outside can reach them. But the operator's own browser can. A malicious web page, call it evil dot example, runs in a tab, and the browser happily sends requests to 127.0.0.1 on its behalf.

• Loopback is not a wall
• The browser is the bridge

RequestGuard applies three rules. One, the Host header must be exactly 127.0.0.1 colon port, localhost colon port, or bracket colon colon one bracket colon port. Two, if an Origin header is present, it must equal http colon slash slash plus that host. Three, a POST must be application slash json.

• Host must match exactly
• Origin must equal http:// + host
• POST must be application/json

The first attack is DNS rebinding. The attacker's domain first resolves to their server, then flips to 127.0.0.1. The browser now believes evil dot example is the same origin as our server, and sends requests with Host set to evil dot example. Our exact Host check sees a name that is not ours, and rejects.

• Domain flips to 127.0.0.1
• Host still says evil.example
• Exact Host check rejects

The second attack skips rebinding. The page just submits a form or fetches to http colon slash slash 127.0.0.1 colon 8093. The Host is correct now, because the URL is ours. But the browser attaches an Origin header naming the attacker's page. The guard requires Origin to equal our own address, so wall two rejects it.

• Host is correct
• Origin names the attacker
• Origin must equal our host

The third attack tries to hide the Origin. Older style form posts can be sent cross-site without a preflight, but only with three content types, such as form encoded or plain text. They cannot send application slash json. Requiring JSON forces the browser to ask permission first with a CORS preflight, which our server never grants. Wall three.

• Forms cannot send JSON
• JSON forces a CORS preflight
• Preflight is never granted

Put together, the evil tab tries rebinding, a cross-site post, and a plain form. Each hits a different wall. The browser gateway, ui slash server dot py, implements the identical three rules in Python, so the same attacker meets the same walls whichever door they try. Two implementations, one specification.

• Rebinding: Host wall
• Cross-site POST: Origin wall
• Plain form: Content-Type wall

Three mistakes to avoid. Comparing secrets with memcmp, which leaks timing. Trusting the Host header without checking it exactly, which invites rebinding. And deleting a socket path on shutdown without checking that the inode is yours, which can disconnect a healthy replacement. Each has a one-line cause and a small fix.

• memcmp for secrets
• Trusting Host
• Deleting another's socket

A caution about scope. These measures defend local links. They do not make the system safe by themselves, and they are one layer among several that episode 25 will list. Loopback with no token is allowed by design, which means any local process of the same machine can talk to it. That is a deliberate trade, not an oversight.

• One layer among several
• Loopback without token is by design

To recap. A frame is a big-endian length plus JSON, capped at one mebibyte, with the size check before the wait. FrameChannel encodes once, sends many, and flushes carefully. Unix sockets are bound privately, chmoded, renamed and unlinked only if ours. Tokens live in the environment and are compared in constant time. And RequestGuard blocks hostile pages with three rules.

• Frames: length first, check first
• Sockets: private bind, rename
• Tokens: env, constant time
• HTTP: Host, Origin, JSON

Next time, in episode 19, we build the HTTP server itself, load libcurl at runtime with dlopen, and write a websocket client by hand. This series is educational only, and nothing here is financial advice. Thanks for watching.

• Next: episode 19
• Subscribe for the series