ken/crashtestdev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: publish "node_modules in a Content-Addressed Store" (Rage series #5)

Browse Source
This commit is contained in:
Ken
2026-05-27 05:37:08 +00:00
parent ba575005cf
commit e178f02abe
1 changed files with 261 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/content/posts/node-modules-in-a-content-addressed-store.md
+261
View File
@@ -0,0 +1,261 @@
---
title: "node_modules in a Content-Addressed Store"
date: 2026-05-27
type: post
tags: [build-systems, node-modules, caching, ci, rage]
summary: "Your CI caches node_modules as a tarball. One package changes, you re-upload everything. Per-package content-addressed storage with hardlink restore is the fix."
---
Ninety seconds. That's how long `yarn install` takes on a fresh CI runner for a monorepo with 1,500 packages. Every run. Every PR. Every merge queue check. The developer who triggered the build has already moved on to another task. The CI runner is still downloading tarballs.
Teams know this hurts. The fix everyone reaches for: cache `node_modules` as a tarball. GitHub Actions provides `actions/cache`. Azure Pipelines provides `Cache@2`. The recipe is everywhere: hash the lockfile, tar up `node_modules`, upload it, restore on the next run. Lockfile unchanged? Skip install entirely. Ninety seconds becomes two seconds. Problem solved.
Until one package changes.
## The enemy: the tarball model
When you cache `node_modules` as a single tarball keyed by the lockfile hash, you are making a bet: the lockfile changes rarely enough that re-uploading 1.4 GB is acceptable when it does. That bet loses in any actively maintained monorepo. A Dependabot PR bumps one transitive dependency. The lockfile hash changes. The entire tarball is invalidated. Re-download all 1,500 packages. Re-tar. Re-upload. Back to ninety seconds.
That's the obvious failure. The non-obvious ones are worse.
**Symlinks break cross-platform.** pnpm's `node_modules` is a symlink graph. `node_modules/.pnpm/react@18.2.0/node_modules/react/` is the real location; `node_modules/react/` is a symlink to it. Cross-platform tar implementations disagree on how to handle absolute vs relative symlink targets. Restore a pnpm `node_modules` tarball on a runner with a different path prefix and the symlinks point nowhere. The install "succeeded." The packages are missing.
**Update granularity is all-or-nothing.** One package changed out of 1,500. The tarball model forces you to re-upload all 1,500. Per-package CAS uploads only the one that changed. The difference is 1.4 GB vs 2 MB.
**ABI fragility.** Native addons compiled on macOS segfault on Linux. A tarball does not encode what platform it was built for. Restore a tarball from a macOS CI runner onto a Linux runner and `esbuild` crashes with an illegal instruction. You discover this at runtime, not at restore time.
**Debugging opacity.** When a tarball restore fails, you get "tar: Unexpected EOF" or a silent corruption. There is no way to know which of the 1,500 packages is wrong. With per-package CAS, a failure points at the failing key, the expected hash, and the on-disk state.
**Storage scaling.** A monorepo's lifetime worth of tarballs is gigabytes. Its lifetime worth of unique package versions in CAS is megabytes, because most package versions are reused across builds.
This is not a new observation. [BuildXL](https://github.com/microsoft/BuildXL) uses per-pip CAS. Bazel `rules_js` uses per-package CAS. pnpm's local store is itself a per-package CAS. The consensus is clear. CI cache actions ignore it.
<div class="breakout">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 440" role="img" aria-label="Diagram: Tarball vs CAS model for caching node_modules, showing what happens when 1 of 1500 packages changes">
<rect width="840" height="440" fill="#F5F0E8" rx="4"/>
<text x="420" y="30" text-anchor="middle" font-family="Georgia, serif" font-size="16" font-weight="bold" fill="#2B2421">When 1 of 1,500 Packages Changes</text>
<line x1="420" y1="44" x2="420" y2="430" stroke="#2B2421" stroke-width="1" stroke-dasharray="4,4"/>
<text x="210" y="62" text-anchor="middle" font-family="Georgia, serif" font-size="14" font-weight="bold" fill="#A0522D">Tarball Model</text>
<text x="210" y="78" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">CI cache actions</text>
<text x="630" y="62" text-anchor="middle" font-family="Georgia, serif" font-size="14" font-weight="bold" fill="#2B2421">Per-Package CAS</text>
<text x="630" y="78" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">rage / BuildXL / Bazel rules_js</text>
<rect x="60" y="96" width="300" height="52" rx="4" fill="#A0522D" stroke="none"/>
<text x="210" y="118" text-anchor="middle" font-family="monospace" font-size="12" fill="#F5F0E8">lockfile hash changed</text>
<text x="210" y="136" text-anchor="middle" font-family="monospace" font-size="12" fill="#F5F0E8">entire cache key invalidated</text>
<line x1="210" y1="148" x2="210" y2="170" stroke="#A0522D" stroke-width="1.5" marker-end="url(#arrowTar)"/>
<rect x="60" y="172" width="300" height="40" rx="4" fill="none" stroke="#A0522D" stroke-width="1.5"/>
<text x="210" y="197" text-anchor="middle" font-family="monospace" font-size="12" fill="#A0522D">re-download all 1,500 packages</text>
<line x1="210" y1="212" x2="210" y2="232" stroke="#A0522D" stroke-width="1.5" marker-end="url(#arrowTar)"/>
<rect x="60" y="234" width="300" height="40" rx="4" fill="none" stroke="#A0522D" stroke-width="1.5"/>
<text x="210" y="259" text-anchor="middle" font-family="monospace" font-size="12" fill="#A0522D">re-tar 1.4 GB, re-upload</text>
<line x1="210" y1="274" x2="210" y2="294" stroke="#A0522D" stroke-width="1.5" marker-end="url(#arrowTar)"/>
<rect x="100" y="296" width="220" height="44" rx="4" fill="#A0522D" stroke="none"/>
<text x="210" y="318" text-anchor="middle" font-family="Georgia, serif" font-size="16" font-weight="bold" fill="#F5F0E8">~90 seconds</text>
<text x="210" y="334" text-anchor="middle" font-family="monospace" font-size="10" fill="#F5F0E8">every time, for 1 changed package</text>
<rect x="480" y="96" width="300" height="52" rx="4" fill="none" stroke="#2B2421" stroke-width="1.5"/>
<text x="630" y="118" text-anchor="middle" font-family="monospace" font-size="12" fill="#2B2421">1,500 integrity hashes checked</text>
<text x="630" y="136" text-anchor="middle" font-family="monospace" font-size="12" fill="#2B2421">1 key is new, 1,499 match</text>
<line x1="630" y1="148" x2="630" y2="170" stroke="#2B2421" stroke-width="1.5" marker-end="url(#arrowCAS)"/>
<rect x="480" y="172" width="300" height="40" rx="4" fill="none" stroke="#2B2421" stroke-width="1.5"/>
<text x="630" y="192" text-anchor="middle" font-family="monospace" font-size="12" fill="#2B2421">1 download (the changed package)</text>
<line x1="630" y1="212" x2="630" y2="232" stroke="#2B2421" stroke-width="1.5" marker-end="url(#arrowCAS)"/>
<rect x="480" y="234" width="300" height="40" rx="4" fill="none" stroke="#2B2421" stroke-width="1.5"/>
<text x="630" y="259" text-anchor="middle" font-family="monospace" font-size="12" fill="#2B2421">1,499 hardlinks from CAS</text>
<line x1="630" y1="274" x2="630" y2="294" stroke="#2B2421" stroke-width="1.5" marker-end="url(#arrowCAS)"/>
<rect x="520" y="296" width="220" height="44" rx="4" fill="#2B2421" stroke="none"/>
<text x="630" y="318" text-anchor="middle" font-family="Georgia, serif" font-size="16" font-weight="bold" fill="#F5F0E8">~200 ms</text>
<text x="630" y="334" text-anchor="middle" font-family="monospace" font-size="10" fill="#F5F0E8">1 miss + 1,499 hardlinks</text>
<line x1="60" y1="370" x2="780" y2="370" stroke="#2B2421" stroke-width="0.5"/>
<text x="420" y="394" text-anchor="middle" font-family="Georgia, serif" font-size="13" fill="#2B2421">The tarball model re-uploads everything on any change.</text>
<text x="420" y="414" text-anchor="middle" font-family="Georgia, serif" font-size="13" fill="#2B2421">The CAS model re-downloads only what changed. The rest are directory entries.</text>
<defs>
<marker id="arrowTar" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto">
<polygon points="0 0, 8 3, 0 6" fill="#A0522D"/>
</marker>
<marker id="arrowCAS" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto">
<polygon points="0 0, 8 3, 0 6" fill="#2B2421"/>
</marker>
</defs>
</svg>
</div>
## The insight: lockfiles already contain the keys
Every modern lockfile already contains a content hash for every external package:
```
# pnpm-lock.yaml
integrity: sha512-abc123...
# yarn.lock (classic)
integrity sha512-abc123...
# yarn.lock (berry)
checksum: 10c0/sha512hex...
# package-lock.json
"integrity": "sha512-abc123..."
```
These are computed by the package manager from the tarball bytes downloaded from the registry. They are deterministic and machine-independent. The package manager already verifies them on every install.
The CAS key is trivial:
```
cas_key = blake3(integrity_string)
```
The value is the tarball bytes, copied from the package manager's local cache. No re-download from the registry. On restore, extract the tarball into `node_modules/{name}/`. That's it.
This is what [rage](https://github.com/kenotron-ms/rage) does. The CAS is monotonic: once a package version enters the store, it stays until garbage-collected. A monorepo's lockfile churn over months builds a CAS that contains every version ever pinned. Restore is milliseconds of hardlink syscalls, not seconds of network I/O.
## The bug that drove the v2 design
The first implementation cached `yarn install` as a skip marker. Here's what it looked like in [`crates/scheduler/`](https://github.com/kenotron-ms/rage/tree/main/crates/scheduler):
```rust
let marker = cache.dir().join(format!("root-{fp}.done"));
if marker.exists() {
eprintln!("[rage] {}#{} ✓ (cached)", ..., ...);
return Ok(()); // restores nothing
}
// run the install
let _ = std::fs::write(&marker, b""); // empty file as the "artifact"
```
The marker file was a zero-byte `.done` file. Its existence meant "install already ran." The cache had cached nothing. It had cached the *fact* that an install happened, not the *result* of the install.
On a fresh CI runner with the same lockfile as a previous build: the marker file existed in the shared cache. The install was skipped. `node_modules/` did not exist. The next task ran `tsc` and got `command not found`. The cache reported a hit. The build crashed with a missing binary.
This is the structural problem with marker-based caching. The marker is a statement about history ("an install ran"), not a statement about state ("these packages exist on disk"). On a machine that shares cache metadata but not filesystem state, the distinction is fatal.
The fix: cache the package contents themselves, at per-package granularity, and restore them on hit. The marker becomes a summary of what needs to be on disk, not a replacement for it.
## Postinstall: the three-axis cache key
Package tarballs are platform-independent. Postinstall scripts are not. A postinstall can compile a native addon with `node-gyp`, download a platform-specific binary, or write to `~/.cache/`. The tarball cache restores source files faithfully. It cannot restore the side effects of running a postinstall script.
This is where the three-axis cache key comes in. From [`crates/scheduler/src/postinstall_cache.rs`](https://github.com/kenotron-ms/rage/blob/main/crates/scheduler/src/postinstall_cache.rs):
```rust
pub fn postinstall_cas_key(task: &PostinstallTask) -> [u8; 32] {
let platform = format!("{}-{}", std::env::consts::OS, std::env::consts::ARCH);
let node_version = read_node_version();
let input = format!(
"{}:{}:{}",
task.tarball_integrity, platform, node_version
);
blake3::hash(input.as_bytes()).into()
}
```
Three axes. Each one earns its place.
**`tarball_integrity`** is the lockfile hash for this exact package version. Bump `esbuild` from 0.21.4 to 0.21.5 and this axis changes. The old postinstall output is dead.
**`platform`** is `darwin-aarch64`, `linux-x86_64`, or whatever the runner reports. A native binary compiled on macOS cannot run on Linux. Without this axis, a macOS CI run would cache a Mach-O binary and a Linux runner would try to execute it. Segfault.
**`node_version`** is the running Node.js major version. Native addons compiled against the Node 18 ABI fail to load on Node 20. The V8 module ABI version changes across major Node releases. Without this axis, upgrading Node breaks every native addon in the cache. Silently.
A change in any axis invalidates the cache for that package's postinstall. There is no way to restore a `darwin-aarch64` postinstall on `linux-x86_64` by accident.
<div class="breakout">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 380" role="img" aria-label="Diagram: Postinstall cache key showing three axes: tarball integrity, platform, and node version">
<rect width="840" height="380" fill="#F5F0E8" rx="4"/>
<text x="420" y="30" text-anchor="middle" font-family="Georgia, serif" font-size="16" font-weight="bold" fill="#2B2421">Postinstall Cache Key: Three Axes</text>
<text x="420" y="50" text-anchor="middle" font-family="monospace" font-size="12" fill="#2B2421">blake3( tarball_integrity : platform : node_version )</text>
<rect x="40" y="80" width="230" height="140" rx="4" fill="none" stroke="#2B2421" stroke-width="1.5"/>
<text x="155" y="104" text-anchor="middle" font-family="Georgia, serif" font-size="14" font-weight="bold" fill="#2B2421">tarball_integrity</text>
<text x="155" y="124" text-anchor="middle" font-family="Georgia, serif" font-size="12" fill="#2B2421">Exact package version</text>
<text x="60" y="150" font-family="monospace" font-size="11" fill="#2B2421">sha512-a1b2c3...</text>
<rect x="50" y="168" width="210" height="36" rx="3" fill="none" stroke="#A0522D" stroke-width="1" stroke-dasharray="5,3"/>
<text x="155" y="184" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#A0522D">Bump esbuild 0.21.4 to 0.21.5?</text>
<text x="155" y="198" text-anchor="middle" font-family="Georgia, serif" font-size="11" font-weight="bold" fill="#A0522D">This axis changes. Cache miss.</text>
<rect x="305" y="80" width="230" height="140" rx="4" fill="none" stroke="#2B2421" stroke-width="1.5"/>
<text x="420" y="104" text-anchor="middle" font-family="Georgia, serif" font-size="14" font-weight="bold" fill="#2B2421">platform</text>
<text x="420" y="124" text-anchor="middle" font-family="Georgia, serif" font-size="12" fill="#2B2421">OS + architecture</text>
<text x="325" y="150" font-family="monospace" font-size="11" fill="#2B2421">darwin-aarch64</text>
<text x="325" y="168" font-family="monospace" font-size="11" fill="#2B2421">linux-x86_64</text>
<rect x="315" y="178" width="210" height="36" rx="3" fill="none" stroke="#A0522D" stroke-width="1" stroke-dasharray="5,3"/>
<text x="420" y="194" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#A0522D">Build on Mac, run on Linux?</text>
<text x="420" y="208" text-anchor="middle" font-family="Georgia, serif" font-size="11" font-weight="bold" fill="#A0522D">This axis changes. Cache miss.</text>
<rect x="570" y="80" width="230" height="140" rx="4" fill="none" stroke="#2B2421" stroke-width="1.5"/>
<text x="685" y="104" text-anchor="middle" font-family="Georgia, serif" font-size="14" font-weight="bold" fill="#2B2421">node_version</text>
<text x="685" y="124" text-anchor="middle" font-family="Georgia, serif" font-size="12" fill="#2B2421">ABI compatibility</text>
<text x="590" y="150" font-family="monospace" font-size="11" fill="#2B2421">Node 18 (ABI 108)</text>
<text x="590" y="168" font-family="monospace" font-size="11" fill="#2B2421">Node 20 (ABI 115)</text>
<rect x="580" y="178" width="210" height="36" rx="3" fill="none" stroke="#A0522D" stroke-width="1" stroke-dasharray="5,3"/>
<text x="685" y="194" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#A0522D">Upgrade Node 18 to 20?</text>
<text x="685" y="208" text-anchor="middle" font-family="Georgia, serif" font-size="11" font-weight="bold" fill="#A0522D">This axis changes. Cache miss.</text>
<rect x="120" y="250" width="600" height="52" rx="4" fill="#2B2421" stroke="none"/>
<text x="420" y="272" text-anchor="middle" font-family="monospace" font-size="13" fill="#F5F0E8">blake3("sha512-a1b2c3:darwin-aarch64:18.19.0")</text>
<text x="420" y="290" text-anchor="middle" font-family="Georgia, serif" font-size="12" fill="#F5F0E8">= one unique CAS key per (version, platform, node)</text>
<text x="420" y="330" text-anchor="middle" font-family="Georgia, serif" font-size="13" fill="#2B2421">All three axes must match for a cache hit.</text>
<text x="420" y="350" text-anchor="middle" font-family="Georgia, serif" font-size="13" fill="#2B2421">A change in any axis forces a re-run of the postinstall script.</text>
<text x="420" y="370" text-anchor="middle" font-family="Georgia, serif" font-size="13" font-weight="bold" fill="#A0522D">No accidental cross-platform restore. Ever.</text>
</svg>
</div>
## The manifest model
Restoring a postinstall is not "copy files back." It's a structured materialization from a manifest that records three things naive file copying ignores.
The manifest entry, from the [rage source](https://github.com/kenotron-ms/rage/blob/main/crates/scheduler/src/postinstall_cache.rs):
```rust
pub enum FileKind {
Regular,
Symlink(PathBuf),
}
pub struct ManifestEntry {
pub rel_path: PathBuf,
pub content_hash: [u8; 32], // blake3; zeroed for symlinks
pub mode: u32, // st_mode & 0o777; zero for symlinks
pub kind: FileKind,
}
```
**Executable bits.** A native addon compiled by `node-gyp` is `0o755`. The mode is stored in the manifest and restored via `set_permissions`. Without this, `esbuild`'s binary loses its execute bit on restore and every bundler invocation fails with `EACCES`.
**Symlinks as symlinks.** A postinstall that creates `bin/foo -> ../target/release/foo` is recorded as a `Symlink(PathBuf)` entry. On restore, it becomes a symlink, not a copy. This matters for tools like Playwright that build a symlink tree to platform-specific browser binaries.
**Hardlinks from CAS.** Each regular file is restored as a hardlink into `~/.rage/artifacts/content/`. Zero-copy at the byte level. The kernel writes a directory entry, not file content. This is why restore is measured in milliseconds, not seconds.
The restore is a walk over the manifest:
```rust
for entry in &manifest {
let dest = target_dir.join(&entry.rel_path);
std::fs::create_dir_all(dest.parent().unwrap())?;
match &entry.kind {
FileKind::Regular => {
let cas_path = store.cas_file_path(&entry.content_hash);
let _ = std::fs::remove_file(&dest);
if std::fs::hard_link(&cas_path, &dest).is_err() {
std::fs::copy(&cas_path, &dest)?; // cross-device fallback
}
std::fs::set_permissions(
&dest,
Permissions::from_mode(entry.mode),
)?;
}
FileKind::Symlink(target) => {
let _ = std::fs::remove_file(&dest);
std::os::unix::fs::symlink(target, &dest)?;
}
}
}
```
The `hard_link` call is the fast path. If the CAS and the target directory are on the same filesystem (they usually are), the link succeeds and no bytes are copied. If they're on different filesystems (`EXDEV`), it falls back to `copy`. Either way, the mode and symlink structure are preserved exactly.
## The numbers
A 200-package monorepo bumps `esbuild` from 0.21.4 to 0.21.5. One lockfile entry changes.
With the tarball model: lockfile hash changed, entire cache invalidated, re-download all 200 packages, re-tar, re-upload. Back to 90 seconds.
With per-package CAS: 1 tarball miss, 199 hardlink hits. 1 postinstall key changes (esbuild's integrity differs), 199 postinstall keys are unchanged. The scheduler reruns esbuild's postinstall, restores the other 199 from CAS via hardlinks. Total time: 300ms.
The code is at [github.com/kenotron-ms/rage](https://github.com/kenotron-ms/rage). The install caching design doc is at [`docs/architecture/INSTALL-CACHING.md`](https://github.com/kenotron-ms/rage/blob/main/docs/architecture/INSTALL-CACHING.md).