diff --git a/src/content/posts/your-build-cache-is-lying-to-you.md b/src/content/posts/your-build-cache-is-lying-to-you.md new file mode 100644 index 0000000..210344f --- /dev/null +++ b/src/content/posts/your-build-cache-is-lying-to-you.md @@ -0,0 +1,234 @@ +--- +title: "Your Build Cache Is Lying to You" +date: 2026-05-27 +type: post +tags: [build-systems, caching, correctness, rage] +summary: "Every JS monorepo build cache works the same way: hash what you told it to hash. If you forgot to tell it about a file, the cache lies. Silently. For months." +--- + +Your CI passed. Your tests passed. The build cache said nothing changed. It was wrong. + +You shipped a broken build to production because a cache returned stale output from three weeks ago. The bug took two days to find. The fix was one line. The root cause was not your code. It was your build system's trust model. + +This is not a theoretical failure mode. I've seen it at Microsoft, in monorepos with hundreds of packages, where a stale cache returned test results that passed against an old version of a shared config. The tests looked green. The config had changed. Nobody declared it as an input. The cache never knew. + +## The enemy: declared-input trust + +Every mainstream JS monorepo build cache works the same way. Turborepo, Nx, and lage (which I co-created at Microsoft) all share the same architecture: you declare what files a task reads, the system hashes those files, and the hash becomes the cache key. If the hash matches a stored entry, the cache returns the stored output without running the task. + +The contract is simple. You tell the cache what matters. The cache trusts you. + +The problem is also simple. You will forget something. Not today. Not on the obvious files. You'll forget on the weird one: the shared `tsconfig.base.json` three directories up, the `.env.local` that a test helper reads at runtime, the codegen config that `graphql-codegen` pulls in during a build step. You won't forget because you're careless. You'll forget because the relationship between that file and your task is invisible until something breaks. + +Here's what that looks like in code. Turborepo computes a cache key like this: + +``` +cache_key = blake3( + command // "tsc -b" + + input_globs // "src/**/*.ts", "tsconfig.json" + + lockfile_hash // yarn.lock content hash + + env_vars // NODE_ENV, etc. +) +``` + +Nx does the same thing with "named inputs." Lage does the same thing with `environmentGlob` and package content hashes. The surface syntax differs. The trust model is identical: **hash what the developer declared, skip what they didn't.** + +## The lie in practice + +Here's a concrete scenario. Your team has a monorepo with 200 packages. Package `api` has a build task that runs `tsc -b`. The cache key includes `src/**/*.ts` and `tsconfig.json`. Builds are fast, CI is green, everyone is happy. + +Then a developer on another team adds a feature flag system. The flags live in `config/feature-flags.json` at the repo root. The `api` package reads this file at build time through a small codegen step baked into its `tsc` plugin. The developer who added the flags updated the build script. They did not update the Turborepo pipeline to declare `config/feature-flags.json` as an input to `api#build`. + +What happens next: + +1. CI runs. `api#build` executes with the new codegen. The cache stores the output, keyed to the current hash of `src/**/*.ts` + `tsconfig.json`. +2. A week later, someone changes `config/feature-flags.json` to disable a flag. +3. CI runs again. Turborepo hashes `src/**/*.ts` + `tsconfig.json`. Nothing changed in those files. Cache hit. Turborepo returns the stored output from step 1. +4. The build output still has the old feature flags baked in. Tests run against the stale output. They pass, because the tests don't check flag values. +5. The stale build ships to production. The feature that should be disabled is still enabled. + +The developer who changed the flag file did everything right. The developer who set up the codegen did everything right except declare one input. The cache did exactly what it was told. The result is a production bug that traces back to a missing line in `turbo.json`. + +No tool warned about this. No test caught it. The cache returned a stale result and reported it as a cache hit. The system lied, and the lie was silent. + +
+ + + The Silent Lie + + Developer changes + feature-flags.json + + + Cache check: + hash(src/**/*.ts + + tsconfig.json) + + + MATCH + (stale!) + + + Returns stale + build output + + + Bug ships + to prod + + feature-flags.json was never in the declared set. + The cache didn't know it existed. + + What the cache hashed: + + src/index.ts + src/routes.ts + src/utils.ts + tsconfig.json + What the task actually read: + + src/index.ts + src/routes.ts + src/utils.ts + tsconfig.json + feature-flags.json ← missed + + the gap + + + + + + + + + +
+ +## Why this is structural, not accidental + +The stale cache bug above looks like a mistake someone made. It's tempting to say "just be more careful about declaring inputs." That's the official answer from every tool that uses this model. Turborepo's docs say to list all relevant files in your `inputs` configuration. Nx says to define "named inputs" correctly. + +This advice is correct and useless. It's correct because yes, if you perfectly declare every file every task reads, the cache will be correct. It's useless because no human, across hundreds of packages maintained by dozens of developers over years, will sustain perfect declarations. The failure rate approaches 100% given enough time and enough packages. + +The deeper problem is that "declare your inputs" conflates two architecturally different things: + +**Trust declared inputs.** The developer tells the system what matters. The system believes them. If they're wrong, the system is wrong. This is the Turborepo/Nx/lage model. + +**Verify observed inputs.** The system watches what the task actually reads at runtime. If the developer forgot to declare a file, the system catches it anyway because it observed the file access. This is the BuildXL model. It's also what [rage](https://github.com/kenotron-ms/rage) implements. + +These are not two flavors of the same thing. They are fundamentally different trust architectures. One makes correctness a property of human discipline. The other makes correctness a property of the mechanism. + +
+ + + Declared Inputs + Turborepo / Nx / lage + Observed Inputs + BuildXL / rage + + + Developer declares: + + src/**/*.ts + + tsconfig.json + + package.json + + ?? missing file ?? + Cache key covers 3 of 4 actual inputs. + Result: silent stale cache. + + Sandbox observes at runtime: + + src/**/*.ts + + tsconfig.json + + package.json + + feature-flags.json ← caught + Cache key covers all 4 actual inputs. + Result: correct cache, always. + + The declared model asks: "did the developer list everything?" + The observed model asks: "what did the process actually read?" + +
+ +## The receipts + +Here's what Turborepo's cache key computation looks like, simplified from the [Turborepo source](https://github.com/vercel/turborepo): + +``` +// Turborepo: single-phase, declared inputs only +cache_key = blake3( + task_command, + sorted(hash(file) for file in resolve_globs(declared_input_globs)), + hash(lockfile_relevant_section), + sorted(env_var_pairs), +) +// If file X was read by the task but not in declared_input_globs, +// it is invisible. A change to X produces the same cache_key. +// The cache returns stale output. No warning. +``` + +Now here's what a two-phase fingerprint looks like. This is the model [rage](https://github.com/kenotron-ms/rage) uses, borrowed directly from [BuildXL](https://github.com/microsoft/BuildXL): + +``` +// Phase 1: Weak Fingerprint (cheap, runs every lookup) +weak_fp = blake3( + task_command, + hash(tool_binary), // compiler upgrade = new key + sorted(hash(file) for file in resolve_globs(declared_input_globs)), + sorted(env_var_pairs), + sorted(dep_abi_fingerprints), // upstream package ABI hashes +) + +// Phase 2: Strong Fingerprint (runs only on WF match) +pathset = pathset_store.lookup(weak_fp) // files the sandbox OBSERVED + // on a previous run of this task +strong_fp = blake3( + weak_fp, + sorted(hash(content(file)) for file in pathset.reads), +) +// Cache hit only if strong_fp matches a stored entry. +``` + +The difference is one word: `pathset`. The weak fingerprint uses declared inputs, same as Turborepo. The strong fingerprint adds the files the task *actually read*, as reported by a file-access sandbox that monitored the process. If a task reads `feature-flags.json` and nobody declared it, the sandbox catches the read. The pathset includes it. The strong fingerprint includes its content hash. Next time `feature-flags.json` changes, the strong fingerprint changes, the cache misses, and the task re-runs. + +The failure mode is inverted. In Turborepo, a missing declaration is a silent stale cache. In rage, a missing declaration is a one-time cache miss while the sandbox observes the read. The next run is correct and cached. You don't need perfect declarations. The mechanism compensates. + +## The cost of observation + +This is not free. The sandbox adds overhead. On macOS, rage uses `DYLD_INSERT_LIBRARIES` to inject a shared library that intercepts filesystem calls. On Linux, it uses eBPF tracepoints. Both record every file the task reads and writes, then ship that log back to the cache layer. + +The overhead is real: single-digit milliseconds per task on warm caches, more on cold runs. Turborepo's flat hash is faster when every input is correctly declared. If you have a small monorepo and a disciplined team that maintains perfect input declarations, the simpler model wins on raw speed. + +But "perfect declarations" is a bet against human nature. I maintained [lage](https://github.com/microsoft/lage) at Microsoft for years. I watched teams get input declarations wrong. Not because they were bad engineers. Because the codebase evolved, new files appeared, implicit dependencies formed, and nobody updated the pipeline config. The stale cache bugs were always silent, always delayed, always expensive to diagnose. + +The sandbox makes the mechanism do the work that humans can't sustain. The cost is milliseconds. The payoff is correctness you don't have to think about. + +## What the sandbox actually sees + +When rage runs a task, the sandbox records a `Pathset`: + +```rust +pub struct Pathset { + pub reads: BTreeSet, // every file the task read + pub writes: BTreeSet, // every file the task wrote +} +``` + +This pathset is stored alongside the weak fingerprint. A single weak fingerprint can have multiple pathsets: the same `tsc -b` command might follow different code paths under different conditions and read different files. The strong fingerprint phase disambiguates them. + +On cache lookup, rage iterates through candidate pathsets for the matching weak fingerprint, hashes the current content of each pathset's read files, and checks whether any resulting strong fingerprint matches a stored output. A hit means the task's actual inputs (not just declared inputs) are unchanged. A miss means something the task reads has changed, and it needs to re-run. + +The cache tells the truth because the sandbox saw the truth. + +## The next post + +The two-phase fingerprint and the sandbox that feeds it are the core of what makes rage's cache correct. How the weak fingerprint and strong fingerprint interact, how pathsets accumulate and resolve, how ABI fingerprints cut off unnecessary downstream rebuilds. That's [Post 2: "Two-Phase Fingerprinting: BuildXL's Deep Magic."](/posts/two-phase-fingerprinting-buildxls-deep-magic) + +For now, the point is simpler. If your build cache hashes only what you declared, it is exactly as correct as your declarations are complete. In a 200-package monorepo maintained by 40 engineers over 3 years, your declarations are not complete. Your cache is lying to you. You just haven't caught it yet. \ No newline at end of file