The gate trusted the table, not the files

Data pipeline · July–Sept 2026

My backtester refuses any request that overlaps missing data. It checked a gap table that had drifted from the parquet files, so on 7 of 9 crypto venues it passed everything, including a 166-day outage.

How I noticed

The results were too good. A strategy was profitable within a couple of days, and an edge shouldn’t be that easy to find. My AI agent’s review of the pipeline found nothing. I pointed it at the gap and coverage records and the pitfalls to check. I directed the tests that confirmed it and designed the fix.

What happened

July. The gate checked only a request’s start and end. I added a gaps table and made DataHub.get() reject any request that overlaps a row. Tests passed.
August. I recomputed everything from the files. The gaps table had rows for the three sources the store held when I built it. Venues added later never had their gaps written.
A gap that was never recorded can’t overlap anything, so the gate passed every request. On kraken/BTC/USD, a request for 2025-06 → 2026-06 came back half empty with no error.

What I guessed first

The July fix was enough. The gate’s logic was right, but it was only as good as the table behind it, and no test checked that table against the files.
Record every missing minute as a gap. Binance writes a bar every minute even when nothing trades, so a missing minute is missing data. Kraken writes a bar only on a trade, so a missing minute is usually a quiet one. Kraken BTC/USD has 263,141 of them. Blocking on all of them makes the venue unusable and buries the one that matters, the 166-day outage.

The fix

One module decides what a gap is, so the pipeline, the catalog rebuild and the validator can’t disagree. A series is grid only if it writes empty bars and covers 99.5% of its own span. Checking one signal wasn’t enough: KuCoin writes some empty bars but misses 5% of minutes, and wrote 219,585 false gaps. The 99.5% is a judgement call, set above KuCoin’s 95.7%. On today’s store no series falls between 98.4% and 99.6%. Grid venues block on any missing bar. Trade-driven venues block on holes of 60 minutes or more, because an hour with no trade on a listed pair is an outage, not a quiet patch. The catalog is rebuilt from the files, not trusted.

def emission_style(series_dir: Path) -> str:
    df = pl.read_parquet(series_dir / "1m.parquet",
                         columns=["timestamp", "volume"])
    if not int((df["volume"] == 0).sum()):
        return "trade-driven"          # never stamps an empty minute
    epoch = df["timestamp"].dt.epoch("s")
    implied = (int(epoch.max() - epoch.min()) // 60) + 1
    # Grid only if it stamps empty minutes AND covers its own span.
    grid = len(df) / implied >= GRID_COVERAGE_MIN
    return "grid" if grid else "trade-driven"

hub/pipeline/gap_detection.py, trimmed

Still open. The rule classifies each series, not each venue. A Binance series with a long outage drops below 99.5% and gets the looser trade-driven rule, so its sub-hour holes don’t block. Classifying by venue would close that.

Numbers

	Before	After
Sources with gap rows	3	all
Validator ERROR findings	138	0
False blocking gaps (KuCoin)	219,585	0

How measured. The validator recomputes coverage and gaps for 733 series (16 GB of parquet) from the files. On an M4 MacBook (10 cores, 24 GB) it runs in 44 s and peaks at 2.6 GB of memory.

What it did to a backtest. I can’t reproduce the original results: the engine was rebuilt, and I deleted the old tests with it because they encoded the wrong behaviour. As an illustration on today’s engine, I recreated the 166-day hole (since partly backfilled) and ran a plain 200-hour SMA trend rule on 13 months of Kraken BTC/USD with the gate bypassed. It got 55% of the expected bars and no error. The strategy happened to be flat when the outage began, so the 23.6% price step across the hole never hit its P&L. The damage was quieter: a result labelled 13 months covered seven. The fixed gate refuses the request.

The repo is private while the work is ongoing; I’m happy to walk through it live.

What I’d do differently

Test the invariant, not the code path. One check that gap rows agree with the files would have caught this in July.
When fixing a reader, find who writes the table. I fixed the gate and assumed the data behind it.
Treat a fast edge as a bug report. Suspect the data before the strategy.

Oct 2026 · Ongoing, a few hours a week: adding venues and data sources, and exploring news-driven strategies.