The History Store: Bar Files That Never Corrupt (Kestrel Ep. 4)

Every backtest rests on one quiet assumption: that the bars on disk are exactly the bars the market produced. If a file is half written, or every bar is off by a minute, nothing downstream will warn you. The numbers will simply be wrong, and they will look perfectly plausible.

• Wrong data looks plausible
• Nothing downstream warns you

In the previous episode we built the core vocabulary: instruments, ticks, IST time and the run config. Now we meet the first real component that touches disk, the history store. By the end you will understand the file format, a subtle bar width bug that once made a strategy act one bar late, and how files are written so that a crash never leaves a corrupt one.

• Episode 3 recap: vocabulary
• Today: format, bar width, safe writes

Kestrel research runs fully offline from a local history store. There is no network call in the middle of a backtest, which means a run is repeatable: same files in, same result out. That promise is only as strong as the files themselves, so the store has to be simple to read, and very hard to damage.

• Offline research
• Same files, same result

Here is the layout. Under data, then store, there is one directory per provider. Inside it, one directory per symbol. Inside that, one gzip compressed CSV file per symbol per UTC day, named by date, plus a manifest dot json that describes the dataset.

• data/store/provider/SYMBOL/day.csv.gz
• manifest.json per dataset

There are four providers. Fyers holds one minute bars. Fyers underscore five s holds five second bars. Fyers underscore one d holds daily bars. And fyers underscore live holds one second bars recorded from the live feed. Each resolution gets its own directory, so the different bar sizes can never be mixed in one folder.

• fyers: 1-minute
• fyers_5s: 5-second
• fyers_1d: daily
• fyers_live: recorded 1 s bars

One file per symbol per UTC day keeps every file small and independent. Rewriting a single day never touches the others, and a reader can open exactly the days in its window and nothing more. Notice the day is a UTC day, because timestamps are stored in UTC. We will see in a moment how that lines up with an Indian session.

• Small independent files
• Day boundary is UTC

Open a file and the header reads symbol, timestamp underscore ms, open, high, low, close, volume. One row per bar. The timestamp is in milliseconds since the Unix epoch, and, crucially, it is the bar's start, not its end. Numbers are written with percent point fifteen g, enough digits to round-trip a double exactly.

• symbol,timestamp_ms,open,high,low,close,volume
• timestamp = bar START
• %.15g round-trips doubles

Take a row for NSE dot RELIANCE dash EQ with timestamp one seven six seven three two five eight zero zero zero zero zero. Convert it: that is January second, 2026, 03:50 UTC. Add five hours thirty minutes and it is 09:20 in India. The bar started at 09:20 IST.

• 1767325800000 ms
• = 2026-01-02 03:50:00 UTC
• = 09:20 IST

The engine does not work on rows. It works on Bar structs, which carry an instrument, a start, an end, and open, high, low, close and volume. The file has no end column, so the reader must compute it. The rule is end equals start plus the store bar width for that provider. For a one minute bar, 09:20 plus sixty seconds gives an end of 09:21.

• Bar = instrument, start, end, OHLCV
• end = start + width

The width comes from one small function. Store bar width takes the provider name. If it is fyers, sixty seconds. If it is fyers underscore five s, five seconds. Anything else falls back to the default store bar interval, which is one second, the width of the recorded live bars.

• Provider decides the width
• Default: kStoreBarInterval

This function is the fix for a real bug. Before it existed, every bar was read with a one second width, whatever the provider. A one minute bar that started at 09:20 was given an end of 09:20 and one second. The data was right. The interpretation was wrong, and it was wrong in a way that no checksum could ever catch.

• Every bar read with 1 s width
• File correct, meaning wrong

Why does the end matter so much? Because bar end drives engine ordering. A bar is only known once it is complete, so the engine delivers it at its end time. Episode 5 covers the merge in detail. For now, remember that the end timestamp decides when a strategy gets to see a bar.

• Bar is known at its END
• End decides delivery time

Consider a strategy that resamples one minute bars. It waits to see a minute complete before acting. With the wrong one second width, the bar for 09:20 was delivered at 09:20 and one second, but the resampler only knew the minute was finished when the next minute's first bar arrived at 09:21 and one second. So it acted a whole bar late.

• Resampler waits for minute close
• Learned it only at next arrival

The lesson is that a unit has to travel with the data. A start timestamp alone is ambiguous, because it does not say how long the bar lasted. So the width is derived from the provider, in one place. And a regression test named test store bar timing guards it, so the one second assumption can never quietly return.

• Start alone is ambiguous
• One place derives width
• test_store_bar_timing

Next, the manifest. Each dataset has a manifest dot json that records the state of every day. Besides the state it keeps the bar count, the first and last bar start, the SHA-256 of the file, a source hash, the fetch time, and a free text note. It is the store's own table of contents, and its memory of what it knows.

• Bar count, first and last start
• File SHA-256, source hash
• Fetch time, note

A day is in one of four states. Complete means the session closed and the data is whole. Partial means the session had not closed plus two hours when it was fetched, so more bars may still be missing. Empty means a holiday, a day with genuinely no bars. And failed means the fetch did not work, and it will be retried on the next run.

• complete
• partial: session close + 2 h
• empty: holiday
• failed: retried next run

The distinction between empty and failed is not cosmetic. A holiday returns no data, and that is the correct answer, so recording it as empty means we never ask again. A failure also returns no data, but it is not an answer at all. Treating those two the same would either re-download holidays forever, or silently bless a hole in the history.

• Empty is an answer
• Failed is no answer

The first pitfall is trusting a partial day. If you download at midday, you get a morning, and a file that looks perfectly healthy. A backtest that reads it would trade a truncated session and report on it as if it were whole. That is why partial is its own state, and why readers have a way to refuse it, which we will see shortly.

• Midday download = half a session
• Looks healthy, is not

Now the heart of the title: files that never corrupt. The naive way to save a file is to open it and write over the old one. If the process crashes, or the power fails, halfway through, you are left with a file that is part new and part old, or just cut short. Gzip would fail to decode it, or worse, decode a truncated prefix.

• Open, overwrite, hope
• Crash leaves half a file

Kestrel never writes in place. Step one, write the complete new content to a uniquely named temporary file in the same directory. Unique, so two writers cannot collide on one tmp name. The real file is not touched at all while this happens, so any reader still sees the old version, whole.

• Unique tmp file, same directory
• Real file untouched

Step two, fsync the temporary file, which forces its bytes out of the operating system cache onto storage. Step three, rename it over the real name. A rename within one filesystem is atomic: a reader sees the old file or the new file, never a mixture. Step four, fsync the directory, so the rename itself survives a power cut.

• 1. write tmp
• 2. fsync file
• 3. rename (atomic)
• 4. fsync directory

Now inject a crash at each step. Crash while writing the temporary file: the real file is intact, and a stray tmp file is left to ignore. Crash after fsync but before the rename: still the old file, intact. Crash after the rename: the new file, complete. At no moment does any reader or any restart see a half file.

• Before rename: old file intact
• After rename: new file complete
• Never a half file

The second hazard is concurrency. The downloader, the portal, and any number of backtests share one store at the same time. Two writers updating the manifest could each read it, change a different day, and the last one to save would erase the other's change. That is the classic lost update.

• Downloader, portal, backtests
• Lost update on manifest

The fix has two layers. Inside one process, a mutex stops threads from interleaving. Across processes, the manifest update takes flock with LOCK underscore EX on a lock file, an exclusive lock that the operating system releases even if the holder crashes. Under that lock the update reads the latest manifest, applies its change, and saves it atomically as before.

• In-process mutex
• flock(LOCK_EX) on a lock file
• Read, modify, atomic write

When the downloader fetches a month, it should not take and release the lock for every single day. Write days batches many days under one lock acquisition, so the manifest is read once, updated for all of them, and written once. Fewer lock round trips, and other users wait less.

• writeDays batches many days
• One lock, one manifest write

That covers the third pitfall: writing in place. If you take only one habit from this episode, take this one: write somewhere new, make it durable, then switch names atomically. It is a pattern you can reuse for configuration files, journals, anything a restart must trust.

• Write new, sync, swap names
• Reuse for any file a restart trusts

Now reading. A StoreCursor iterates one symbol over a window, from inclusive to exclusive, one day file at a time. It only opens the next day when it needs it. If one day is bad, say a file fails to decode, the cursor logs a warning and carries on with the next day, instead of failing the whole run.

• Window [from, to)
• One day file at a time
• Bad day: warn and continue

A backtest usually has several symbols. StoreSource opens one cursor per symbol and merges them with a min-heap keyed by the pair bar end and stream index. The smallest end comes out first. Streams are sorted by instrument id before the heap is built, so when two bars end at the same instant, the tie is broken the same way every run.

• Min-heap by (bar.end, stream index)
• Streams sorted by instrument id
• Deterministic ties

Real data has holes. A quiet instrument may print no bar for a minute. Some strategies want a regular series, so StoreSource has an option, fill gaps. When the next real bar starts later than the last bar plus one width, it manufactures flat bars: open, high, low and close all equal the last close, with zero volume.

• fillGaps option
• Flat bars at last close
• Volume zero

Here is the code, from StoreSource advance, simplified. If fill gaps is on, we have a last bar, and the stashed real bar starts after the last start plus width, we build a synthetic bar at last start plus width. Otherwise we take the real bar, and fetch the next one from the cursor into the stash.

• Condition: gap larger than one width
• Else: emit the real bar

Look closely at the two branches. In the synthetic branch, the stash is never touched. The real bar stays parked while fake bars are emitted ahead of it, and each time around, last start moves forward by one width until the gap closes. Only then does the else branch consume the real bar. Forget that, and you would lose a real bar for every gap.

• Synthetic branch leaves stash alone
• Real bar emitted when gap closes

A concrete case with one minute bars. A bar starts at 09:22, and the next real bar starts at 09:25. Last start plus width is 09:23, and 09:25 is later, so we emit a flat bar starting 09:23. Now last start is 09:23, plus width is 09:24, still earlier than 09:25, so another flat bar at 09:24. Then 09:25 is not later than 09:25, and the real bar finally comes out.

• 09:22 real
• 09:23, 09:24 synthetic
• 09:25 real

Two switches protect you from silent damage. Require complete turns any partial or missing day in the window into a config error, so a run on truncated data refuses to start. And check data refuses an empty window, because a backtest over zero bars would report a clean, confident, meaningless result. Fail early and loudly.

• requireComplete: partial or missing = ConfigError
• checkData: empty window refused

The engine does not know about files. It talks to the IDataSource interface: an id, a flag saying whether it is realtime, open with the instruments, next which returns an optional Bar, close, a status, and an optional quote lookup. When next returns nothing, the stream is over.

• id, isRealtime, open, close
• next() returns optional Bar
• status, optional quote(id)

The store is historical only. The market data client is realtime only. The config must match the mode, so a backtest cannot point at a live feed, and a live run cannot quietly replay files. That keeps the one-engine promise from episode 1 honest: the source differs, the engine does not.

• Store: historical only
• Market data: realtime only
• Config must match mode

To recap. Files are gzip CSV, one per symbol per UTC day, one directory per resolution. The timestamp is the bar start, and the end is start plus a width taken from the provider, a lesson learned from a bug that made strategies act one bar late. A manifest tracks complete, partial, empty and failed days. Writes go through tmp, fsync, rename and directory fsync, under a mutex and flock. Readers merge with a min-heap and refuse partial data on request.

• Layout and resolution directories
• Width from provider
• Atomic writes and locks
• Deterministic merge

Remember the pitfalls: never mix resolutions, never trust a partial day, and never write in place. Next time, in episode 5, we follow these bars into the engine: the event bus, the k-way merge, and what determinism really requires. This is educational content, not financial advice.

• Next: Episode 5, the engine loop
• Subscribe for the series