Main, 7 October 2026: a 100 s hook gate on the merge commit against a master that moves every minute lost six pushes in a row.
Self-test: a fixture repository (unstamped branch, stamped branch, stale stamp, wrong tip, plain commit, dirty tree).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main, 7 October 2026, 15:07 UK: one global exclusive measure flock across unrelated measurements stalled build-1 at load 120 with
free slots (a stopped probe held it 5.5 h; an exclusive waiter queued every new shared taker) and build-2 behind a one-core VDF
bench. lease.sh (installed at /srv/builds/_bin/lease by provision.sh and by hand on both boxes) takes one flock per core for a
pinned measurement and the quiet file for a whole-box one; remote-run.sh takes quiet shared only for unbounded runs, excludes
leased cores from its set, and its keeper refreshes the holder file and calls lease reap (a STOPPED holder of a lease, quiet or a
slot for 5 minutes is killed with a line in _log/reaped.log). Keepers close the lock descriptors they inherit (an orphaned sleep
held a slot and the worktree lock 20 s past the release; the slot self-test had rotted on that since the worktree lock landed).
tools/ci/box-locks-check.sh runs lease.sh --self-test and remote-run.sh --self-test-slots on build-1. provision.sh also carries
the 16 libraries headless Chromium needs (installed by hand on both boxes at 14:5x UK) and a headless self-test step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
the project lead, 7 October 2026 15:02 UK: build-1 at load 139 / 114 / 90 with both slots held and a 1 h 40 min queue while build-2 read 4.5 with
free slots, because the class router pinned each class to its box. Now lib.sh bs_route_spill reads the preferred box (free slots,
1-minute load) with one ssh and hands the job to the other box when the preferred one has no free slot or sits above load 64 and the
other qualifies; neither qualifying queues on the class's own box. The decision travels as BR_ROUTE_* into the JSONL "route" object
for the dashboard. build-2's slots file reads 3; everything on box 2 runs at nice 10 / 32 cores / -j 32, and a bounded run takes
the band its slot owns so three never share a core. Self-test tools/ci/route-spill-check.sh (thirteen cases) in the gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The dashboard lane, 7 October 2026 10:39Z: both slots of build-1 flock-held and EMPTY while two suites ran. Cause: a run from a
worktree without last night's append-mode fix opens a busy sibling's slot file with > on every probe. The holder now keeps its own
line: a keeper re-writes it within BR_KEEP_S (20 s) whenever the file is empty, until release; remote-run.sh --self-test-keeper
(in the gate) truncates a held line and sees it return, and sees nothing written after release; live on build-1 at 11:19Z (the
line came back in 25 s). The first version deadlocked the runner's bare wait with the keeper (build-2's first run hung 15 min
after its test passed): the keeper stops before the wait. The two running suites' -j 90 came from explicit --jobs 90: a bounded
class now clamps it to its cap with a log line (pass --priority gate for the full set). run-from-mac.sh --box N: the host file
was suffixed twice (build-server-2-2) and every box's mirror would have shared one remote name; one remote per box (build-N).
build-2's first green run: a suite at nice 10 on 32 cores, jobs 32, 986 s cold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's order of 7 October 2026 after a load of 190 on 96 threads (a release join bench and the 0.3.19 app gate starving each
other, 'builds' of 16 minutes). tools/build-remote.sh resolves a class from the cargo subcommand and --priority: test and bench are
the bounded class (nice 10, the last 32 cores, -j 32) unless --priority gate (nice 0, the full set, the box's own jobs rule);
builds and checks are unchanged. remote-run.sh applies renice and taskset to the command's subshell, caps the jobs, lets a queued
gate (gate-pending-<pid>) take the next slot ahead of suites and benches, and prints nice and cores in the RESULT line and the
JSONL line (nice, cores, priority). The slot label carries '; kind=<k> nice=<n> cores=<c>' before '; agent=', so the dashboard's
job card shows why a job is slow. --plan prints the resolved class without the box; tools/ci/build-kind-default-check.sh (in the
gate) holds the five shapes. Smoke on the box: a suite at jobs=32 nice=10 cores=32, a gate at nice=0 cores=96.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh checkout_tree ran `git clean -fd` on the box mirror before every build from any agent, so the attack rows
lost attack-f3/, attack-f1-venv/ and tools/attack/*/target to each other's builds (7 October 2026, 09:2x UK). The clean now
also spares the fixed prefixes attack-*, scratch-*, target-attack-*, .build-remote.log and every glob in the mirror-local
.igneum-scratch-spare (one per line, # comments, the file itself spared), keeps the target and stamp excludes and still runs
without -x. The clean-tree test asks `git clean -nd` with the same excludes instead of filtering the status list, so a spared
dir is not "not clean". --self-test: a fixed-prefix dir at the root and nested, a declared dir and the spare file survive, an
undeclared dir is removed. tools/ci/scratch-spare-check.sh in the pre-push gate fails when the clean line loses the spare
arguments, spare_args stops reading the file, a fixed prefix goes, or -x appears. docs/plans/build-server.md R4a says how a
lane declares its prefix.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three classes from the 0.3.17 night. (1) bash reads a script incrementally: tools/build-remote.sh was edited while a four-minute remote
build ran, the running copy continued at shifted bytes and died with a syntax error after the build had succeeded on the box; the
four long-running tools (build-remote, cross-remote, workers-remote, move-hand) now keep their body in one brace block ending in exit,
parsed whole before a line runs; tools/ci/whole-body-check.sh (in the gate, self-test with a block-less copy) holds the shape.
(2) A fork build pairs with the igneum-pow of the igneum worktree it sits in: a fork at 12153428 under a master worktree failed in
kaspa-pow four minutes in (no chain_program_shadow; master's igneum-pow predates release-0.3.17's); build-remote.sh says the
pairing on its first line ('pairs with igneum 6f8d7a7e (detached): igneum-pow 0.2.0') and the JSONL line carries pairs_with.
The first version of that line used '[ -n ... ] && echo' inside an assignment's $( ) and set -e ended the script on the false
status; fixed. (3) move-hand.sh restart <hand> [--digest <hex>] [--go]: after binary installed a release, restart ONE unit and read
it back (first exec line, commit string in the running binary, digest against the wanted one, igneum_getNodeInfo powEngine over the
node's loopback EVM RPC); the digest readers tolerate a missing line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The box's 34 red rows of 6 October classified (docs/analysis/ci-failures-2026-10-06.md section 6): 22 iterations, 12 in three real classes (instant deaths with nothing kept, a shared worktree directory, an unread dry run). remote-run.sh now: pre-flight (subcommand, manifest, -p package, --features) refuses in a second with exit 3 and a class; the last 400 lines of every run kept in /srv/builds/_log/runs; a class on every row (compile-error, link-error, test-failure, instant, no-test-matched, slot-timeout, no-dir, preflight-*); a cargo test whose filter matched no test exits 3; a per-worktree lock in checkout and run mode; every red row appended to /srv/ci-red/red.jsonl as source box. red-watch.mjs never posts a box row alone and sends one digest a day (counts per class with each class's guard); the timer runs tick. Shared group cired on the box so the runner and build append to one file. Shown in a sandbox on the box: pass, failing test, empty filter, bad package, bad feature, missing subcommand, compile error, broken manifest, two concurrent runs of one worktree.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pgrep -x git over the whole machine kept the lock whenever any git ran (the pre-push hook's own git push, another agent's build) and the checkout died with "index.lock: File exists". The fact is the lock's age. Class Q in docs/analysis/ci-failures-2026-10-06.md with the GIT_DIR leak of the previous commit.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's rule of 6 October 2026 from the 0.3.14 repro (docs/evidence/reproduced/0.3.14.md): prost's protowire.rs embeds OUT_DIR,
libmimalloc-sys embeds __DATE__/__TIME__, sccache hid both. lib.sh bs_repro_env exports SOURCE_DATE_EPOCH=<author time> TZ=UTC in
front of every remote command (build-remote.sh, cross-remote.sh, workers-remote.sh); remote-run.sh exports BR_SDE too and logs it
as source_date_epoch; proto-cuda/windows-node/cross-build.sh exports the same; the PC job carries node.commit_time in the manifest
(push-build-inputs.sh) and exports it before every cargo build of a stage (jobbuild.rs, unit test asserts it; 4 of 4 pass on the
box). Target dirs stay one fixed path per target. tools/build-remote.sh --self-test-repro [--full] from a fork worktree, run on
the box: igneum-miner twice a minute apart without sccache (RUSTC_WRAPPER=/usr/bin/env, an empty value is unset to cargo) MATCH
91e130f5..., a per-run target path differs (OUT_DIR shown); --full: kaspad with libmimalloc-sys recompiled a minute later MATCH
70219bc2..., without the epoch differs (__DATE__ shown).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- provision.sh step_runner: actions/runner 2.338.0 (sha256 checked) at /opt/actions-runner under a dedicated user `runner`
(no sudo, not in build's group), rustup 1.99.0 pinned with both targets, sccache against /srv/sccache in READ_ONLY mode
on its own server port, Node 22 and mingw from the system, GitHub's svc.sh unit with a Nice 10 drop-in; registered on
igneum-network/igneum as igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) through
infra/build-server/runner/register.sh (gh as igneum-labs, the token on ssh stdin, never logged). Idempotent after
the env files moved behind svc.sh install (its env.sh rewrites them). libicu74 and python3-numpy added to APT.
- main's slots ruling: SLOTS default 2; remote-run.sh sets CARGO_BUILD_JOBS 90 when it holds the only taken slot and 45
when both are held, BR_MEASURE=1 takes the `measure` file exclusively and excludes builds (builds hold it shared),
lock files open in append mode (the old `exec {fd}>` truncated a busy slot's holder line on every probe), env
IGNEUM_BUILD_SLOTS_DIR and IGNEUM_BUILD_LOG_DIR win over the profile, `--self-test-slots` with five cases (the old
script fails it with JOBS=none); build-remote.sh and cross-remote.sh pass -j only when --jobs is given.
- infra/build-server/prover/cpu-trial.sh: the SP1 CPU prover on one fixture shard under the measure hold with a VmHWM
poller; 6 Oct 2026 run: core 34.2 s, compressed 85.9 s, peak RSS 28.2 GB on 96 threads, so no standing CPU prover.
- docs/plans/ci-self-hosted.md: the proposed runs-on change for ci.yml behind the repository variable IGNEUM_CI_RUNNER
(GitHub-hosted is the fallback), and why windows.yml cannot move to a Linux box. Workflows untouched.
- docs/plans/build-server.md section 7.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
provision.sh step_cuda: NVIDIA's ubuntu2404 apt repository, cuda-nvrtc-dev-12-8, cuda-cudart-dev-12-8, cuda-driver-dev-12-8
(the libcuda stub) and opencl-c-headers; no nvcc (no build file calls it), no driver. tools/workers-remote.sh builds
igneum-worker-cuda and igneum-worker-opencl on the box from proto-cuda and proto-opencl with clang++ (static libstdc++),
prints sha256 and the glibc ceiling (2.38: fine for Ubuntu 24.04 hosts, not for 22.04 containers or HiveOS, where the Mac's
zig build stays). Proof: igneum-worker-cuda 1,613,992 B sha256 6db8a9ad..., igneum-worker-opencl 124,904 B sha256 0dea75bb...
remote-run.sh evaluates the command string in a subshell (a leaked set -e killed the runner after a successful build).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
the project lead's decision of 6 October 2026: the Mac runs nothing the network depends on. docs/plans/hands-on-build-1.md plans node 1
and the observer (its node and tools/observer) as systemd units on the box, one hand at a time, with ports, DNS names,
exec recovery, rollback and the decisions for main; infra/build-server/hands/install-hands.sh writes users, dirs, run
scripts and units (inert, run on the box); move-hand.sh (dry run by default) builds 0.3.15 on the box, copies the override
and snapshot, rsyncs each data dir hot then stopped, starts the unit and prints the hand's first executing line, copies the
observer env by scp (mode 600), stops the Mac's observer first, unloads the launchd agents last. CLAUDE.md's running-agents
rule carries the decision. lib.sh: a per-worktree lock on the box across sync, build and fetch (the shipper's collision at
18:48:56Z). remote-run.sh: the clean-tree test excuses stamps and target dirs at any depth with --untracked-files=all (a
nested stamp failed every checkout of /srv/builds/igneum for twenty minutes) and removes a stale .git/index.lock; the
self-test covers both. Plan rows for the three and the cuda prover pair built for the shipper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The second build of every repo-kind crate in a subdirectory failed (6 October 2026, 18:51 UTC, the pool build: "tree not
clean after reset: ?? pool/.build-remote-sha-target"): checkout_tree keeps the stamps with `git clean -e` at any depth but
its status check matched them at the root only. The check now reads `git status --porcelain --untracked-files=all` (an
all-untracked directory is listed file by file, not as "?? sub/") and accepts target dirs, sccache and both stamps under
any path. The self-test carries the subdirectory case (stamp and target dir kept, overlay file removed) and the known-failed
case (a stray untracked file still fails the check).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh gains a checkout mode used by lib.sh: git checkout -- . and git clean -fd (target dirs, the sha stamps and
ignored files kept), fetch, branch at the commit, then a clean-tree check; the overlay follows. Before this, the rsync of
uncommitted files stayed in the box's tree and the next commit's git checkout -B refused with 'local changes would be
overwritten'. remote-run.sh --self-test reproduces the dirty tree (edited tracked file, untracked file, target dir, sha
stamp), shows the plain checkout refusing and the mode landing clean on the new commit; it runs in ci.yml and passed on the
Mac and the box; a live dirty-then-clean pair on the fork worktree passed too. Plan: section 5, gotchas of the first day.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh writes /srv/builds/_locks/wait-<pid> in the slot-file line format while it waits and removes it when the
slot is taken or the wait is given up (shown: present during a held slot, gone after). provision.sh installs Caddy with a
Caddyfile that serves only /srv/workers/workers.json and headline.json (every other path 404, CORS for dl.igneum.network,
no-store, Let's Encrypt only) and opens 80 and 443. Both asked for by the worker-dashboard agent, approved by main.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in
/srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot,
wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h
slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and
build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash
(no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that
imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental
7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
