eBPF · Rust · aya · Linux ≥ 5.15 (CO-RE/BTF)
The machine stalled. Blackbox watched it happen.
Blackbox records every context switch on a Linux host with eBPF,
keeps a bounded history in memory, and the moment PSI pressure or an
OOM kill says a stall happened, it writes a self-describing trace of the
last seconds: who was running, who was waiting, in what order.
The post-mortem problem
A server degrades, deadlocks or gets OOM-killed in the middle of the night. By the time you look: the evidence is gone. `dmesg` gives you one kernel line, `journalctl` gives you logs written after the stall, and the processes responsible are already dead or restarted.
The one record that survives a stall is the scheduler's: the kernel's per-CPU run queues keep a perfect memory of the last seconds of execution. Blackbox reads that memory continuously, so when the stall ends (or a systemd-shot waits) the trace is already on disk.
What it records
sched_switch
Every context switch, machine-wide: timestamp, previous task state, prev/next pid and comm, CPU id. One 64-byte record per switch, drained continuously from an 8 MiB kernel ring buffer.
Lifecycle
Process forks, execs and exits (via tracepoints), on a second 512 KiB ring buffer. A dump tells you which binary path appeared just before the stall.
Triggers
PSI pressure on CPU / memory / I/O with hysteresis, plus the kernel's oom_kill counter in /proc/vmstat. Any trigger fires an automatic dump.
assert! keeps the wire struct and userspace decode byte-identical.PSI hysteresis: no dump storms
Fires for a reason, and says why
Every dump records what triggered it (manual, one of three PSI resources, or the OOM counter) and the resource's pressure readings at that moment. Reports group events into per-CPU timelines by process, so the “who was running when it went sideways” answer is the first thing you see.
oom_kill counter
in /proc/vmstat fires a dump: the victim's last switches are
exactly the evidence a later coredump loses.
How it works
Self-describing dumps
JSON with a schema version, its own completeness counters (events_evicted, window.truncated) and the trigger context. A trace with holes says so instead of looking authoritative.
Two outputs
blackbox report prints a text timeline; --perfetto emits Chrome JSON you can drag onto ui.perfetto.dev for zooming/panning.
Clean life
start · status · dump · stop over one control socket. systemd unit included, with Restart=on-failure and a sandbox.
Install & workflow
- InstallDownload the release tarball for your kernel (CO-RE means one object works across kernels with BTF).
- Start it
sudo blackbox startattaches the tracepoints and starts collecting. - Forget itPSI pressure and OOM kills trigger dumps automatically into the dump directory.
- Later
blackbox report dump.json, or paste the Perfetto export into ui.perfetto.dev and zoom into the stall. - Stop
blackbox stopshuts it down cleanly (joined workers, socket removed).
# i386/x86_64 release tarball; one portable CO-RE object $ tar -xzf blackbox-v0.2.3-x86_64*.tar.gz $ cd blackbox-v0.2.3-* $ sudo install -m755 blackbox blackboxd /usr/local/bin/ $ sudo install -d /usr/local/lib/blackbox /etc/blackbox $ sudo install -m644 blackbox-bpf.o /usr/local/lib/blackbox/ $ sudo install -m644 examples/config.toml /etc/blackbox/config.toml $ sudo blackbox start blackboxd started (pid 190312), listening on /run/blackbox/blackboxd.sock $ blackbox status bpf collection attached scope machine-wide (no pid filter) $ blackbox dump -o /tmp/out.json dump written to /tmp/out.json $ blackbox report /tmp/out.json $ blackbox report /tmp/out.json --perfetto > out.pftrace $ blackbox stop # clean shutdown blackboxd: stop requested blackboxd stopped
CAP_BPF+CAP_PERFMON)
for the one privileged act: attaching. Everything after that runs unprivileged.
Cost: memory & CPU, honestly
Memory, measured
| Mode | RSS (measured) | Where |
|---|---|---|
| Development / no BPF | ~3.3 MB | userspace |
| Collection attached | ~55 MB | userspace |
| EVENTS ring | 8 MiB | kernel |
| LIFECYCLE ring | 512 KiB | kernel |
The history window starts at 512 KB (8192 slots) and grows lazily to at most
max_events × 64 B (up to 16 MB at the 250k default). Flat whether
idle or at a full window.
min(max_events / ingest-rate, max_seconds).
At 500k switches/s the 250k cap holds only ~0.5 s, exactly when you need the
trace most. Raise max_events, or scope with
filter_pids to cut ingest.
CPU, measured, worst case
The numbers above are from the project's own benchmark on a noisy laptop (i5-1135G7, kernel 6.8). A tracepoint still fires on every switch even when the filter drops the record; the hook cost is the floor.
What you actually get from a trace
Who held the CPU
Per-CPU, per-process timelines of runnable vs. blocked states, and you spot the task hogging a core while others starve, or the epoll loop spinning.
Who was waiting
prev_state tells you each switch's reason: uninterruptible sleep under memory pressure, runnable but not picked. The pattern of starvation is the diagnosis.
What died right before
OOM-triggered dumps pin the victim's pid and comm, plus which children were forked/exec’d in the window: the blast radius in one view.
Good for ·
- Nightly-load failures with no obvious log cause
- “It hangs for 30 s then recovers”: the window shows the interlock
- CI boxes under memory pressure obvious in
oom_kill - Teaching: a live machine's scheduler as a visible, explainable trace
Not (yet) for ·
- Continuous profiling: it's a recorder, not a flamegraph tool
- Multi-minute post-mortems without raising
max_events - I/O root-causing: no block-layer sources yet (roadmap)
- History across daemon restarts (in-memory window only)
Security stance
Least privilege
Attaching BPF is the only privileged act. PSI reads, dumps, reports run with no capabilities. The CO-RE object never needs a kernel recompile.
Privacy by design
Only scheduler metadata: comm (15 chars), pids, state, timestamps, cpu. No argv, no environment, no memory contents. An exec path up to 15 chars.
No network, gated socket
A local Unix socket only. Status is open to local users; dump/stop require the daemon owner or root via SO_PEERCRED (v0.2.3+), with [ipc].restrict_mutations to relax on single-user boxes.
Honest limits
| Limit | What it means | Mitigation |
|---|---|---|
| Machine-wide overhead | Tracepoint fires on every switch; ~0.6 core + 26–63% worst case on switch-heavy load | filter_pids → +1.5–4.7%; scope to the box that matters |
| Bounded window | 250k events / 30 s default; short at 500k switches/s | Raise max_events, PID-filter ingest |
| History in RAM only | Daemon restart loses the window, and the crash you care about may kill the daemon | Persistent dump is the long-term answer (roadmap) |
| I/O blind spots | Shows blocked-without-CPU, not the block reason | Block-layer sources planned |
| Young project | v0.2 · 30+ commits · single maintainer | 180 tests, schema-versioned dumps, honest docs |