The fleet's segment exports (50 to 500 MB each with the old [0, last] export, never pruned) filled 42 GB on the hub
(586 segment dirs, 60 GB disk, 100 percent) and 3 to 46 GB on every standing box between about 13:00Z and 20:44Z on
6 October 2026, 4 to 6 GB an hour a box; the hub's node died on the full disk at 20:42:31Z (its two earlier deaths
at 19:58:08Z and 20:01:55Z were the sync KeyNotFound). The app's prover now:
- enforces a cap on <app dir>/proving before every export: entries older than IGNEUM_PROVING_DIR_MAX_DAYS (7) go,
then the oldest until the total is under IGNEUM_PROVING_DIR_MAX_GB (20); the plan is a pure function with a
unit test (provingdir::prune_plan); the defaults leave 80 GB of a 100 GB box to the node and the system
- skips the export with a logged line when the disk is under 10 percent free (statvfs on unix, GetDiskFreeSpaceExW
on Windows), and the Prove page says so
- deletes a segment's directory (fixtures, proofs, logs) the moment its record is submitted, on the first try or on
a retry; seq.json was already removed after the cut
The fleet's kit (box-prover.py, gpu-fleet f9ad70e) carries the same rule on the boxes since 20:5xZ.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1ac60d5e2c