feat: publish "Hooking Syscalls Without a Kernel Driver" (Rage series #3)
This commit is contained in:
@@ -0,0 +1,324 @@
|
|||||||
|
---
|
||||||
|
title: "Hooking Syscalls Without a Kernel Driver"
|
||||||
|
date: 2026-05-27
|
||||||
|
type: post
|
||||||
|
tags: [build-systems, sandboxing, macos, linux, windows, rage, ebpf, detours]
|
||||||
|
summary: "Three operating systems. Three interception mechanisms. One goal: watch every file a build task touches without root, without a kernel module, with single-digit overhead."
|
||||||
|
---
|
||||||
|
|
||||||
|
You need to know every file a child process reads or writes. Not the files you *think* it reads. The files it *actually* touches at runtime. You need this for every task in a 200-package monorepo, across macOS, Linux, and Windows. You cannot require root. You cannot install a kernel module. You cannot ask developers to disable SIP or run a privileged daemon. And the overhead needs to stay under 10%, because this runs on every build, not just CI.
|
||||||
|
|
||||||
|
This is the sandbox problem. It is the hardest engineering problem in [rage](https://github.com/kenotron-ms/rage), and it is the reason the cache described in [Post 1](/posts/your-build-cache-is-lying-to-you) tells the truth instead of trusting your declarations.
|
||||||
|
|
||||||
|
Three operating systems. Three completely different interception mechanisms. One Rust struct at the end:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct Pathset {
|
||||||
|
pub reads: BTreeSet<PathBuf>,
|
||||||
|
pub writes: BTreeSet<PathBuf>,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Let me show you how each one works.
|
||||||
|
|
||||||
|
<div class="breakout">
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 520" role="img" aria-label="Diagram: Three OS, Three Hooks. Side-by-side comparison of macOS DYLD interpose, Linux eBPF tracepoints, and Windows Detours inline patching, each showing the interception point between userspace call and kernel syscall.">
|
||||||
|
<rect width="840" height="520" fill="#F5F0E8" rx="4"/>
|
||||||
|
<text x="420" y="32" text-anchor="middle" font-family="Georgia, serif" font-size="16" font-weight="bold" fill="#2B2421">Three OS, Three Hooks</text>
|
||||||
|
<rect x="20" y="52" width="256" height="450" rx="4" fill="none" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="148" y="76" text-anchor="middle" font-family="Georgia, serif" font-size="14" font-weight="bold" fill="#2B2421">macOS</text>
|
||||||
|
<text x="148" y="94" text-anchor="middle" font-family="monospace" font-size="10" fill="#A0522D">DYLD interpose</text>
|
||||||
|
<rect x="48" y="114" width="200" height="36" rx="3" fill="#fff" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="148" y="136" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">open("config.json")</text>
|
||||||
|
<line x1="148" y1="150" x2="148" y2="172" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="48" y="174" width="200" height="56" rx="3" fill="#A0522D" stroke="none"/>
|
||||||
|
<text x="148" y="196" text-anchor="middle" font-family="monospace" font-size="11" font-weight="bold" fill="#F5F0E8">rage_open()</text>
|
||||||
|
<text x="148" y="216" text-anchor="middle" font-family="monospace" font-size="9" fill="#F5F0E8">__DATA,__interpose table</text>
|
||||||
|
<line x1="148" y1="230" x2="148" y2="252" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="48" y="254" width="200" height="36" rx="3" fill="#fff" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="148" y="276" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">real open() via dlsym</text>
|
||||||
|
<line x1="148" y1="290" x2="148" y2="312" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="48" y="314" width="200" height="36" rx="3" fill="none" stroke="#2B2421" stroke-width="1" stroke-dasharray="4,3"/>
|
||||||
|
<text x="148" y="336" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">kernel (syscall)</text>
|
||||||
|
<text x="148" y="376" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">Injection: load-time</text>
|
||||||
|
<text x="148" y="394" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">IPC: Unix socket / FIFO</text>
|
||||||
|
<text x="148" y="412" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">Kernel mode: No</text>
|
||||||
|
<text x="148" y="436" text-anchor="middle" font-family="Georgia, serif" font-size="12" font-weight="bold" fill="#A0522D">~3-8% overhead</text>
|
||||||
|
<rect x="292" y="52" width="256" height="450" rx="4" fill="none" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="420" y="76" text-anchor="middle" font-family="Georgia, serif" font-size="14" font-weight="bold" fill="#2B2421">Linux</text>
|
||||||
|
<text x="420" y="94" text-anchor="middle" font-family="monospace" font-size="10" fill="#A0522D">eBPF tracepoints</text>
|
||||||
|
<rect x="320" y="114" width="200" height="36" rx="3" fill="#fff" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="420" y="136" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">openat(fd, "cfg", ...)</text>
|
||||||
|
<line x1="420" y1="150" x2="420" y2="172" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="320" y="174" width="200" height="36" rx="3" fill="#fff" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="420" y="196" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">libc openat() wrapper</text>
|
||||||
|
<line x1="420" y1="210" x2="420" y2="232" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="320" y="234" width="200" height="36" rx="3" fill="none" stroke="#2B2421" stroke-width="1" stroke-dasharray="4,3"/>
|
||||||
|
<text x="420" y="256" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">kernel: sys_enter_openat</text>
|
||||||
|
<line x1="420" y1="270" x2="420" y2="292" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="320" y="294" width="200" height="56" rx="3" fill="#A0522D" stroke="none"/>
|
||||||
|
<text x="420" y="316" text-anchor="middle" font-family="monospace" font-size="11" font-weight="bold" fill="#F5F0E8">eBPF tracepoint</text>
|
||||||
|
<text x="420" y="336" text-anchor="middle" font-family="monospace" font-size="9" fill="#F5F0E8">PID filter + RingBuf emit</text>
|
||||||
|
<text x="420" y="376" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">Injection: kernel attach</text>
|
||||||
|
<text x="420" y="394" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">IPC: BPF RingBuf</text>
|
||||||
|
<text x="420" y="412" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">Kernel mode: Yes (eBPF)</text>
|
||||||
|
<text x="420" y="436" text-anchor="middle" font-family="Georgia, serif" font-size="12" font-weight="bold" fill="#A0522D">~2-5% overhead</text>
|
||||||
|
<rect x="564" y="52" width="256" height="450" rx="4" fill="none" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="692" y="76" text-anchor="middle" font-family="Georgia, serif" font-size="14" font-weight="bold" fill="#2B2421">Windows</text>
|
||||||
|
<text x="692" y="94" text-anchor="middle" font-family="monospace" font-size="10" fill="#A0522D">Detours inline patching</text>
|
||||||
|
<rect x="592" y="114" width="200" height="36" rx="3" fill="#fff" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="692" y="136" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">CreateFileW("cfg")</text>
|
||||||
|
<line x1="692" y1="150" x2="692" y2="172" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="592" y="174" width="200" height="56" rx="3" fill="#A0522D" stroke="none"/>
|
||||||
|
<text x="692" y="196" text-anchor="middle" font-family="monospace" font-size="11" font-weight="bold" fill="#F5F0E8">Trampoline JMP</text>
|
||||||
|
<text x="692" y="216" text-anchor="middle" font-family="monospace" font-size="9" fill="#F5F0E8">patched function prologue</text>
|
||||||
|
<line x1="692" y1="230" x2="692" y2="252" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="592" y="254" width="200" height="36" rx="3" fill="#fff" stroke="#2B2421" stroke-width="1"/>
|
||||||
|
<text x="692" y="276" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">original CreateFileW</text>
|
||||||
|
<line x1="692" y1="290" x2="692" y2="312" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah)"/>
|
||||||
|
<rect x="592" y="314" width="200" height="36" rx="3" fill="none" stroke="#2B2421" stroke-width="1" stroke-dasharray="4,3"/>
|
||||||
|
<text x="692" y="336" text-anchor="middle" font-family="monospace" font-size="11" fill="#2B2421">kernel (NtCreateFile)</text>
|
||||||
|
<text x="692" y="376" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">Injection: CREATE_SUSPENDED</text>
|
||||||
|
<text x="692" y="394" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">IPC: Named pipe</text>
|
||||||
|
<text x="692" y="412" text-anchor="middle" font-family="Georgia, serif" font-size="11" fill="#2B2421">Kernel mode: No</text>
|
||||||
|
<text x="692" y="436" text-anchor="middle" font-family="Georgia, serif" font-size="12" font-weight="bold" fill="#A0522D">~1-5% overhead</text>
|
||||||
|
<text x="420" y="484" text-anchor="middle" font-family="Georgia, serif" font-size="13" fill="#2B2421">Three mechanisms. One output.</text>
|
||||||
|
<text x="420" y="504" text-anchor="middle" font-family="monospace" font-size="12" font-weight="bold" fill="#A0522D">Pathset { reads, writes }</text>
|
||||||
|
<defs>
|
||||||
|
<marker id="ah" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto">
|
||||||
|
<polygon points="0 0, 8 3, 0 6" fill="#2B2421"/>
|
||||||
|
</marker>
|
||||||
|
</defs>
|
||||||
|
</svg>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
## macOS: rewriting calls at load time
|
||||||
|
|
||||||
|
macOS gives you a mechanism most developers have never heard of: `__DATA,__interpose`. It is a Mach-O section that tells the dynamic linker "when any code in this process calls function X, call function Y instead." The dyld linker resolves the interpose table at load time, before `main()` runs. Every call site across every loaded image (the main binary, frameworks, dylibs) gets rewritten.
|
||||||
|
|
||||||
|
The interpose entry is a struct with two function pointers:
|
||||||
|
|
||||||
|
```c
|
||||||
|
struct interpose_entry {
|
||||||
|
void *replacement; // your function
|
||||||
|
void *original; // the function to replace
|
||||||
|
};
|
||||||
|
|
||||||
|
__attribute__((used))
|
||||||
|
__attribute__((section("__DATA,__interpose")))
|
||||||
|
static const struct interpose_entry interpose_open = {
|
||||||
|
(void *)rage_open,
|
||||||
|
(void *)open
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
The replacement function does the logging, then calls through to the original:
|
||||||
|
|
||||||
|
```c
|
||||||
|
int rage_open(const char *path, int flags, ...) {
|
||||||
|
int result = open(path, flags, /* mode from va_args */);
|
||||||
|
rage_log_event(EV_OPEN, path, flags, result);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
In rage, `crates/sandbox-macos-dylib/` builds a Rust `cdylib` that defines this table. The child process is launched with two environment variables:
|
||||||
|
|
||||||
|
```
|
||||||
|
DYLD_INSERT_LIBRARIES=/path/to/librage_sandbox.dylib
|
||||||
|
RAGE_SANDBOX_FIFO=/tmp/rage/sandbox/<task-id>.fifo
|
||||||
|
```
|
||||||
|
|
||||||
|
Every call to `open`, `openat`, `stat`, `lstat`, `fstatat`, `read`, `pread`, `write`, `pwrite`, `rename`, `unlink`, `mkdir`, `readdir`, and `access` goes through the shim. The shim calls the real syscall via `dlsym(RTLD_NEXT, ...)`, resolves the path to absolute, writes a length-prefixed CBOR event to the FIFO, and returns the syscall's result unchanged. The parent reads events concurrently and accumulates them into a `Pathset`.
|
||||||
|
|
||||||
|
`DYLD_INSERT_LIBRARIES` is inherited by child processes automatically. When `tsc` spawns a worker, the worker gets the sandbox too. No extra plumbing.
|
||||||
|
|
||||||
|
### Why not EndpointSecurity?
|
||||||
|
|
||||||
|
Apple's EndpointSecurity framework is the "right" way to monitor file access on macOS. It hooks at the kernel level, gives you AUTH events (block reads outside a declared set), and covers every process on the machine. It requires the `com.apple.developer.endpoint-security.client` entitlement. Getting that entitlement requires applying to Apple, getting approved, and distributing through a signed system extension. For an open-source build tool that developers install with `cargo install`, this is a non-starter.
|
||||||
|
|
||||||
|
The same story for dtrace (requires root or `csrutil disable`), ptrace (macOS does not expose `PTRACE_SYSCALL` outside Xcode's debugger), and FSEvents (coalesces events with multi-second latency, drops under load, no PID filtering). DYLD interpose works on every Mac since 10.7. No root, no entitlements, no SIP exemption.
|
||||||
|
|
||||||
|
### The macOS 26 surprise
|
||||||
|
|
||||||
|
When Apple announced macOS 26 at WWDC 2025, they introduced "Platform-26 SIP," a new System Integrity Protection tier. The short version: `DYLD_INSERT_LIBRARIES` is now silently stripped from the environment for any binary with the hardened runtime flag, even if the binary does *not* have the `com.apple.security.cs.allow-dyld-environment-variables` entitlement. On macOS 15 and earlier, hardened runtime binaries without the entitlement would ignore the env var. On macOS 26, the env var is stripped before the process even launches.
|
||||||
|
|
||||||
|
For build tools this is fine. `node`, `tsc`, `cargo`, `python` do not ship with hardened runtime. But it was a sharp reminder that Apple tightens the screws every year, and any mechanism that depends on env-var injection lives on borrowed time. We added a detection path in rage's sandbox loader: if `DYLD_INSERT_LIBRARIES` is silently stripped (the child process starts but no events arrive on the FIFO within 500ms), the sandbox logs a warning and falls back to `loose` mode for that task.
|
||||||
|
|
||||||
|
## Linux: eBPF, or how to put Rust in the kernel
|
||||||
|
|
||||||
|
On Linux, the macOS approach breaks down. `LD_PRELOAD` is the Linux equivalent of `DYLD_INSERT_LIBRARIES`, and for dynamically linked binaries it works fine. The problem: Rust binaries default to static linking. Go binaries are statically linked. Any tool using `musl` is statically linked. `LD_PRELOAD` is invisible to all of them. It is also bypassed by `setuid` binaries and anything calling `syscall(2)` directly.
|
||||||
|
|
||||||
|
eBPF solves this at the kernel level. Tracepoints fire on syscall entry, regardless of how the userspace process made the call. Static binary, dynamic binary, direct `syscall` instruction. The kernel sees them all.
|
||||||
|
|
||||||
|
rage's eBPF programs live in `crates/sandbox-linux-ebpf-prog/`. They are `#![no_std]` Rust, compiled to the `bpfel-unknown-none` target, and loaded at runtime by [aya](https://github.com/aya-rs/aya). Here's the shape of the tracepoint handler:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#![no_std]
|
||||||
|
|
||||||
|
#[tracepoint(name = "sys_enter_openat")]
|
||||||
|
pub fn handle_openat(ctx: TracePointContext) -> u32 {
|
||||||
|
let pid = bpf_get_current_pid_tgid() >> 32;
|
||||||
|
|
||||||
|
// Check if this PID belongs to a sandboxed task
|
||||||
|
if unsafe { TRACKED_PIDS.get(&(pid as u32)) }.is_none() {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read the path from userspace into a per-CPU buffer
|
||||||
|
let mut buf = [0u8; 256];
|
||||||
|
let path_ptr = /* extract from tracepoint args */;
|
||||||
|
unsafe { bpf_probe_read_user_str(buf.as_mut_ptr(), 256, path_ptr) };
|
||||||
|
|
||||||
|
// Emit event to ring buffer
|
||||||
|
EVENTS.output(&FileAccessEvent {
|
||||||
|
op: OP_OPENAT,
|
||||||
|
pid: pid as u32,
|
||||||
|
path: buf,
|
||||||
|
}, 0);
|
||||||
|
|
||||||
|
0
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The full set of tracepoints: `sys_enter_openat`, `sys_enter_stat`, `sys_enter_newfstatat`, `sys_enter_lstat`, `sys_enter_read`, `sys_enter_pread64`, `sys_enter_write`, `sys_enter_pwrite64`, `sys_enter_rename`, `sys_enter_renameat2`, `sys_enter_unlink`, `sys_enter_unlinkat`, `sys_enter_mkdir`, `sys_enter_mkdirat`.
|
||||||
|
|
||||||
|
The PID filter is a `BPF_MAP_TYPE_HASH`. Before rage spawns a child process, the loader inserts the child's PID into the map. The tracepoint handler checks the map on every syscall. Not in the map? Return immediately. The overhead for non-sandboxed processes is a single hash lookup.
|
||||||
|
|
||||||
|
Events flow through a `BPF_MAP_TYPE_RINGBUF`. The aya userspace loader reads from the ring buffer and accumulates events into the per-task `Pathset`. Ring buffers are lock-free and support batched reads. The kernel writes, userspace reads, no coordination needed.
|
||||||
|
|
||||||
|
### Capabilities, not root
|
||||||
|
|
||||||
|
eBPF tracepoint programs require `CAP_BPF` (kernel 5.8+) or `CAP_SYS_ADMIN` on older kernels. In CI containers, this means `--cap-add=BPF` on the Docker run command. This is not root. It is a single capability that allows loading BPF programs and nothing else. If the capability is missing, rage emits a clear error and falls back to `loose` mode.
|
||||||
|
|
||||||
|
Ring buffer support requires kernel 5.8+. rage v1 requires 5.8+. This is Ubuntu 20.10+ or any distro from late 2020 onward. For CI, every major runner (GitHub Actions, Azure Pipelines, GitLab CI) runs kernels well past 5.8.
|
||||||
|
|
||||||
|
### Why LD_PRELOAD still loses
|
||||||
|
|
||||||
|
Even if you only care about Node.js (which is dynamically linked), `LD_PRELOAD` has a subtle problem: it intercepts libc wrappers, not syscalls. If a library calls `syscall(SYS_openat, ...)` directly, the preload shim never fires. This is rare but not theoretical. Some database engines and JIT runtimes do exactly this. eBPF catches them because it hooks at the kernel boundary, after the userspace/kernel transition, regardless of how the process got there.
|
||||||
|
|
||||||
|
## Windows: patching function prologues in a suspended process
|
||||||
|
|
||||||
|
Windows has no `LD_PRELOAD` equivalent and no eBPF. What it has is [Microsoft Detours](https://github.com/microsoft/Detours), a library that rewrites function prologues in memory. This is the same mechanism [BuildXL](https://github.com/microsoft/BuildXL) uses for its Windows sandbox, proven at scale across 150,000+ builds per day at Microsoft.
|
||||||
|
|
||||||
|
Detours is inline function patching, not import-address-table (IAT) hooking. For each target function, Detours replaces the first 5+ bytes of the function with a `JMP` to your handler. The original bytes are saved into a dynamically allocated *trampoline* page that jumps back to the rest of the original function after your handler runs. Because the patch is at the function prologue, every caller (main binary, statically linked code, dynamically loaded DLLs) hits the hook.
|
||||||
|
|
||||||
|
The injection sequence:
|
||||||
|
|
||||||
|
1. Rage calls `CreateProcess` with `CREATE_SUSPENDED`. The child exists but has not executed a single instruction.
|
||||||
|
2. `VirtualAllocEx` allocates a page in the child's address space. `WriteProcessMemory` writes the path to `rage_sandbox.dll`.
|
||||||
|
3. `CreateRemoteThread` runs `LoadLibraryW` in the child, loading the DLL.
|
||||||
|
4. The DLL's `DllMain(DLL_PROCESS_ATTACH)` installs all hooks via `DetourTransactionBegin` / `DetourAttach` / `DetourTransactionCommit`, then connects to the parent's named pipe.
|
||||||
|
5. `ResumeThread` lets the child's main thread run. Every file access from the first instruction goes through the hook.
|
||||||
|
|
||||||
|
No window of unobserved execution. The hooks are installed before `main()`, same guarantee as DYLD interpose on macOS and eBPF attach on Linux.
|
||||||
|
|
||||||
|
### Hook both layers
|
||||||
|
|
||||||
|
This is the detail that took a week to discover: you must hook both Win32 and NT native APIs.
|
||||||
|
|
||||||
|
```
|
||||||
|
Win32 layer: CreateFileW, CreateFileA
|
||||||
|
NT native: NtCreateFile, NtOpenFile
|
||||||
|
```
|
||||||
|
|
||||||
|
The normal call path is `CreateFileW` calling `NtCreateFile` calling the kernel. If you only hook `CreateFileW`, you miss tools that call the NT layer directly. Some parts of the .NET runtime do this. Some Microsoft compilers do this. If you only hook `NtCreateFile`, you lose access to the fully resolved Win32 path string.
|
||||||
|
|
||||||
|
rage hooks both, with a thread-local re-entry guard so each file open is recorded exactly once. The full hook set:
|
||||||
|
|
||||||
|
```
|
||||||
|
NtCreateFile, NtOpenFile // NT native (catches direct callers)
|
||||||
|
CreateFileW, CreateFileA // Win32 (catches normal opens)
|
||||||
|
ReadFile, WriteFile // content I/O
|
||||||
|
DeleteFileW, MoveFileExW, CopyFileW // mutations
|
||||||
|
CreateDirectoryW, RemoveDirectoryW // directory ops
|
||||||
|
FindFirstFileW, FindNextFileW // enumeration (implicit deps!)
|
||||||
|
GetFileAttributesW // existence probes
|
||||||
|
```
|
||||||
|
|
||||||
|
`FindFirstFileW` / `FindNextFileW` are mandatory and easy to overlook. Directory enumeration is how `tsc` discovers files in `node_modules`. The list of entries *seen during enumeration* is itself an input. When a new sibling file appears in a directory, the enumeration result changes, and the task's pathset must reflect that.
|
||||||
|
|
||||||
|
Events flow over a named pipe (`\\.\pipe\rage_sandbox_{pid}`). The format is binary, not CBOR or JSON, because Windows paths are UTF-16 native and the hook fires on every file open. Serialization cost matters on the hot path.
|
||||||
|
|
||||||
|
## The numbers
|
||||||
|
|
||||||
|
Measured on a TypeScript build across a 200-package monorepo:
|
||||||
|
|
||||||
|
```
|
||||||
|
macOS (DYLD interpose, FIFO drain): ~3-8% over baseline
|
||||||
|
Linux (eBPF tracepoints, ring buffer): ~2-5% over baseline
|
||||||
|
Windows (Detours, named pipe): ~1-5% over baseline
|
||||||
|
```
|
||||||
|
|
||||||
|
The dominant cost is event serialization to userspace. `read` and `write` syscalls fire many times for large files, and each one produces an event. rage deduplicates repeated accesses to the same file descriptor inside the shim itself, so re-reading the same file does not generate N events.
|
||||||
|
|
||||||
|
The Linux numbers are lower because the hook lives in the kernel. No context switch per event, no IPC serialization per syscall. Events batch in the ring buffer and drain in userspace on a polling interval. The macOS and Windows approaches pay for userspace-to-userspace IPC on every intercepted call.
|
||||||
|
|
||||||
|
For comparison, `ptrace` (the strace mechanism) adds 50-200% overhead because it serializes every syscall through the tracing parent process. `fanotify` on Linux drops events under load and has no `stat`/`read`/`write` coverage. Neither is viable for production builds.
|
||||||
|
|
||||||
|
<div class="breakout">
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 460" role="img" aria-label="Diagram: The Sandbox Contract. Three platform-specific pipes from child processes converge on a single Pathset struct, showing macOS FIFO, Linux RingBuf, and Windows Named Pipe all feeding into the same reads/writes accumulator.">
|
||||||
|
<rect width="840" height="460" fill="#F5F0E8" rx="4"/>
|
||||||
|
<text x="420" y="32" text-anchor="middle" font-family="Georgia, serif" font-size="16" font-weight="bold" fill="#2B2421">The Sandbox Contract</text>
|
||||||
|
<rect x="40" y="60" width="200" height="120" rx="4" fill="none" stroke="#2B2421" stroke-width="1.5"/>
|
||||||
|
<text x="140" y="84" text-anchor="middle" font-family="Georgia, serif" font-size="13" font-weight="bold" fill="#2B2421">Child Process</text>
|
||||||
|
<text x="140" y="104" text-anchor="middle" font-family="monospace" font-size="10" fill="#2B2421">tsc -b</text>
|
||||||
|
<text x="60" y="128" font-family="monospace" font-size="10" fill="#A0522D">open("src/index.ts")</text>
|
||||||
|
<text x="60" y="144" font-family="monospace" font-size="10" fill="#A0522D">stat("tsconfig.json")</text>
|
||||||
|
<text x="60" y="160" font-family="monospace" font-size="10" fill="#A0522D">write("dist/index.js")</text>
|
||||||
|
<line x1="240" y1="100" x2="360" y2="100" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah2)"/>
|
||||||
|
<line x1="240" y1="130" x2="360" y2="200" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah2)"/>
|
||||||
|
<line x1="240" y1="160" x2="360" y2="300" stroke="#2B2421" stroke-width="1.5" marker-end="url(#ah2)"/>
|
||||||
|
<rect x="362" y="72" width="180" height="48" rx="3" fill="#fff" stroke="#A0522D" stroke-width="1.5"/>
|
||||||
|
<text x="370" y="90" font-family="monospace" font-size="10" fill="#A0522D">macOS</text>
|
||||||
|
<text x="370" y="108" font-family="monospace" font-size="10" fill="#2B2421">Unix FIFO + CBOR</text>
|
||||||
|
<rect x="362" y="172" width="180" height="48" rx="3" fill="#fff" stroke="#A0522D" stroke-width="1.5"/>
|
||||||
|
<text x="370" y="190" font-family="monospace" font-size="10" fill="#A0522D">Linux</text>
|
||||||
|
<text x="370" y="208" font-family="monospace" font-size="10" fill="#2B2421">BPF RingBuf</text>
|
||||||
|
<rect x="362" y="272" width="180" height="48" rx="3" fill="#fff" stroke="#A0522D" stroke-width="1.5"/>
|
||||||
|
<text x="370" y="290" font-family="monospace" font-size="10" fill="#A0522D">Windows</text>
|
||||||
|
<text x="370" y="308" font-family="monospace" font-size="10" fill="#2B2421">Named pipe + binary</text>
|
||||||
|
<line x1="542" y1="96" x2="600" y2="210" stroke="#A0522D" stroke-width="1.5" marker-end="url(#ahR)"/>
|
||||||
|
<line x1="542" y1="196" x2="600" y2="210" stroke="#A0522D" stroke-width="1.5" marker-end="url(#ahR)"/>
|
||||||
|
<line x1="542" y1="296" x2="600" y2="230" stroke="#A0522D" stroke-width="1.5" marker-end="url(#ahR)"/>
|
||||||
|
<rect x="602" y="160" width="200" height="120" rx="4" fill="#A0522D" stroke="none"/>
|
||||||
|
<text x="702" y="186" text-anchor="middle" font-family="monospace" font-size="12" font-weight="bold" fill="#F5F0E8">Pathset</text>
|
||||||
|
<line x1="622" y1="196" x2="782" y2="196" stroke="#F5F0E8" stroke-width="0.5"/>
|
||||||
|
<text x="622" y="216" font-family="monospace" font-size="10" fill="#F5F0E8">reads:</text>
|
||||||
|
<text x="622" y="232" font-family="monospace" font-size="9" fill="#F5F0E8"> src/index.ts</text>
|
||||||
|
<text x="622" y="246" font-family="monospace" font-size="9" fill="#F5F0E8"> tsconfig.json</text>
|
||||||
|
<text x="622" y="262" font-family="monospace" font-size="10" fill="#F5F0E8">writes:</text>
|
||||||
|
<text x="622" y="276" font-family="monospace" font-size="9" fill="#F5F0E8"> dist/index.js</text>
|
||||||
|
<text x="420" y="340" text-anchor="middle" font-family="Georgia, serif" font-size="13" fill="#2B2421">Three platform-specific backends.</text>
|
||||||
|
<text x="420" y="362" text-anchor="middle" font-family="Georgia, serif" font-size="13" fill="#2B2421">One platform-agnostic struct.</text>
|
||||||
|
<text x="420" y="396" text-anchor="middle" font-family="Georgia, serif" font-size="12" fill="#2B2421">The cache layer never knows which OS produced the Pathset.</text>
|
||||||
|
<text x="420" y="416" text-anchor="middle" font-family="Georgia, serif" font-size="12" fill="#2B2421">It hashes the contents of pathset.reads. It computes a strong fingerprint.</text>
|
||||||
|
<text x="420" y="436" text-anchor="middle" font-family="Georgia, serif" font-size="12" font-weight="bold" fill="#A0522D">Same correctness, three different ways.</text>
|
||||||
|
<defs>
|
||||||
|
<marker id="ah2" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto">
|
||||||
|
<polygon points="0 0, 8 3, 0 6" fill="#2B2421"/>
|
||||||
|
</marker>
|
||||||
|
<marker id="ahR" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto">
|
||||||
|
<polygon points="0 0, 8 3, 0 6" fill="#A0522D"/>
|
||||||
|
</marker>
|
||||||
|
</defs>
|
||||||
|
</svg>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
## One struct, three operating systems
|
||||||
|
|
||||||
|
The sandbox is the most OS-specific code in rage. Everything else (the cache, the scheduler, the DAG, the fingerprinting) is pure Rust, cross-platform, no conditional compilation. The sandbox is `#[cfg(target_os)]` from top to bottom: three separate crates, three completely different mechanisms, three different IPC transports.
|
||||||
|
|
||||||
|
But the API the sandbox exposes to the rest of the system is one trait:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub trait Sandbox {
|
||||||
|
fn run(&self, task: &Task) -> (ExitStatus, Pathset);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The cache layer calls `sandbox.run(task)` and gets back a `Pathset`. It never knows whether the pathset came from a DYLD interpose table, an eBPF ring buffer, or a Detours trampoline. The strong fingerprint computation does not care. The pathset store does not care. The entire correctness model described in [Post 1](/posts/your-build-cache-is-lying-to-you) and [Post 2](/posts/two-phase-fingerprinting-buildxls-deep-magic) sits on top of this one struct, and the struct is the same on every platform.
|
||||||
|
|
||||||
|
That is the real design insight. Not that you *can* intercept syscalls without a kernel driver (BuildXL proved that a decade ago). The insight is that the interception mechanism is a pure implementation detail. The contract between the sandbox and the cache is `Pathset { reads, writes }`. Get the reads and writes right, by whatever means your OS offers, and the rest of the system works unchanged.
|
||||||
|
|
||||||
|
The [next post](/posts/typescript-7-makes-the-build-harness-more-important) is about why TypeScript 7's 10x speedup makes this machinery *more* important, not less.
|
||||||
Reference in New Issue
Block a user