the project lead, 7 October 2026 15:02 UK: build-1 at load 139 / 114 / 90 with both slots held and a 1 h 40 min queue while build-2 read 4.5 with
free slots, because the class router pinned each class to its box. Now lib.sh bs_route_spill reads the preferred box (free slots,
1-minute load) with one ssh and hands the job to the other box when the preferred one has no free slot or sits above load 64 and the
other qualifies; neither qualifying queues on the class's own box. The decision travels as BR_ROUTE_* into the JSONL "route" object
for the dashboard. build-2's slots file reads 3; everything on box 2 runs at nice 10 / 32 cores / -j 32, and a bounded run takes
the band its slot owns so three never share a core. Self-test tools/ci/route-spill-check.sh (thirteen cases) in the gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's order of 7 October 2026 (igneum-build-2, AX162-1, and igneum-build-3, AX102-1, on order). lib.sh: bs_box_file N and
bs_route <class>; a class whose box has no host file yet falls back to box 1 and says so. run-from-mac.sh --box N <ip> provisions
igneum-build-N and writes build-server-N. tools/build-remote.sh --box N overrides the route; --priority gate always runs on box 1;
the proving crate routes to box 3. README.md: the kind map and the project lead's rule, no mining on any Hetzner box, ever (nodes, builds, tests,
benchmarks and CPU proving only; the pool's fast-time network mines on rented GPU pods, never on build-3). capacity/run.sh refuses a
job that would start igneum-miner mine or a GPU worker, whatever SEQUENCE says.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's order of 7 October 2026 after a load of 190 on 96 threads (a release join bench and the 0.3.19 app gate starving each
other, 'builds' of 16 minutes). tools/build-remote.sh resolves a class from the cargo subcommand and --priority: test and bench are
the bounded class (nice 10, the last 32 cores, -j 32) unless --priority gate (nice 0, the full set, the box's own jobs rule);
builds and checks are unchanged. remote-run.sh applies renice and taskset to the command's subshell, caps the jobs, lets a queued
gate (gate-pending-<pid>) take the next slot ahead of suites and benches, and prints nice and cores in the RESULT line and the
JSONL line (nice, cores, priority). The slot label carries '; kind=<k> nice=<n> cores=<c>' before '; agent=', so the dashboard's
job card shows why a job is slow. --plan prints the resolved class without the box; tools/ci/build-kind-default-check.sh (in the
gate) holds the five shapes. Smoke on the box: a suite at jobs=32 nice=10 cores=32, a gate at nice=0 cores=96.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The b3c228fa builds under the 0.3.16 app tree 5d118f58 (no rust-toolchain.toml yet) died right after the pairing line with no
message: sed on the missing file failed, pipefail carried its status into the assignment, set -e ended the script. A guard and a
tolerant pipeline; checked alive against a tree without the file.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three classes from the 0.3.17 night. (1) bash reads a script incrementally: tools/build-remote.sh was edited while a four-minute remote
build ran, the running copy continued at shifted bytes and died with a syntax error after the build had succeeded on the box; the
four long-running tools (build-remote, cross-remote, workers-remote, move-hand) now keep their body in one brace block ending in exit,
parsed whole before a line runs; tools/ci/whole-body-check.sh (in the gate, self-test with a block-less copy) holds the shape.
(2) A fork build pairs with the igneum-pow of the igneum worktree it sits in: a fork at 12153428 under a master worktree failed in
kaspa-pow four minutes in (no chain_program_shadow; master's igneum-pow predates release-0.3.17's); build-remote.sh says the
pairing on its first line ('pairs with igneum 6f8d7a7e (detached): igneum-pow 0.2.0') and the JSONL line carries pairs_with.
The first version of that line used '[ -n ... ] && echo' inside an assignment's $( ) and set -e ended the script on the false
status; fixed. (3) move-hand.sh restart <hand> [--digest <hex>] [--go]: after binary installed a release, restart ONE unit and read
it back (first exec line, commit string in the running binary, digest against the wanted one, igneum_getNodeInfo powEngine over the
node's loopback EVM RPC); the digest readers tolerate a missing line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's order of 7 October 2026. The repo root carries rust-toolchain.toml (channel 1.99.0, targets x86_64-pc-windows-gnu and
x86_64-unknown-linux-gnu); rustup resolves the nearest file walking up from the crate, so the fork worktrees under vendor/ are
covered, and the fork's master carries its own copy (vendor/igneum-node 37f1206b). lib.sh bs_toolchain_check reads the pin
and refuses a build when the pin, the Mac's rustc as resolved in the crate dir, or the box's rustc differ (the message says
the three commands that align them); run-from-mac.sh passes the pin to provision.sh as RUST_TOOLCHAIN. The 1.99.0 toolchain
with both targets is installed on the Mac so no agent's build stalls on rustup's auto-install. Verified: the Mac resolves
1.99.0 in a fork worktree, the box runs 1.99.0, the check prints 'pinned 1.99.0 by rust-toolchain.toml'.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/plans/hands-on-build-1.md section 6: the move of 6 October 2026 23:06 to 23:22 UTC, observer node, observer, node 1, unload,
each hand's first executing line, digest eada4bda MATCH, IBD and acceptance, the open readback (the Mac's Miner app has not
started its own node since 26610/26611 were freed). lib.sh: the shipper's class from the 0.3.17 tree, the fork's igneum-exec
embeds proving/igneum-prove/elf/*.vk by include_bytes! five levels up, which is no path dependency; every .rs in the trees that
travel is scanned for include_bytes!/include_str! paths leaving the crate's repository and their directories join the overlay;
proving/igneum-prove/elf is the fixed fallback for a fork build. move-hand.sh: igneumd --version exits 1 (tolerated; it ended
the binary step before the override was written), and the launchd pid lookup used \s in macOS awk (printed 'pid none').
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's rule of 6 October 2026 from the 0.3.14 repro (docs/evidence/reproduced/0.3.14.md): prost's protowire.rs embeds OUT_DIR,
libmimalloc-sys embeds __DATE__/__TIME__, sccache hid both. lib.sh bs_repro_env exports SOURCE_DATE_EPOCH=<author time> TZ=UTC in
front of every remote command (build-remote.sh, cross-remote.sh, workers-remote.sh); remote-run.sh exports BR_SDE too and logs it
as source_date_epoch; proto-cuda/windows-node/cross-build.sh exports the same; the PC job carries node.commit_time in the manifest
(push-build-inputs.sh) and exports it before every cargo build of a stage (jobbuild.rs, unit test asserts it; 4 of 4 pass on the
box). Target dirs stay one fixed path per target. tools/build-remote.sh --self-test-repro [--full] from a fork worktree, run on
the box: igneum-miner twice a minute apart without sccache (RUSTC_WRAPPER=/usr/bin/env, an empty value is unset to cargo) MATCH
91e130f5..., a per-run target path differs (OUT_DIR shown); --full: kaspad with libmimalloc-sys recompiled a minute later MATCH
70219bc2..., without the epoch differs (__DATE__ shown).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
the project lead's decision of 6 October 2026: the Mac runs nothing the network depends on. docs/plans/hands-on-build-1.md plans node 1
and the observer (its node and tools/observer) as systemd units on the box, one hand at a time, with ports, DNS names,
exec recovery, rollback and the decisions for main; infra/build-server/hands/install-hands.sh writes users, dirs, run
scripts and units (inert, run on the box); move-hand.sh (dry run by default) builds 0.3.15 on the box, copies the override
and snapshot, rsyncs each data dir hot then stopped, starts the unit and prints the hand's first executing line, copies the
observer env by scp (mode 600), stops the Mac's observer first, unloads the launchd agents last. CLAUDE.md's running-agents
rule carries the decision. lib.sh: a per-worktree lock on the box across sync, build and fetch (the shipper's collision at
18:48:56Z). remote-run.sh: the clean-tree test excuses stamps and target dirs at any depth with --untracked-files=all (a
nested stamp failed every checkout of /srv/builds/igneum for twenty minutes) and removes a stale .git/index.lock; the
self-test covers both. Plan rows for the three and the cuda prover pair built for the shipper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a member of the fork's workspace that inherits
from the fork's root manifest; syncing that one directory left cargo without a workspace root on the box. lib.sh now groups
path dependencies by git top level: a repository under vendor/ is pushed to its mirror (a fork worktree to
/srv/igneum-node.git, a repository of its own to /srv/<name>.git, created on first use), checked out whole at
/srv/builds/<worktree>/vendor/<name> and overlaid whole; run-from-mac.sh wires every vendor repository the Cargo.toml files
reach. libprotobuf-dev added (sp1-prover-types imports google/protobuf/empty.proto). build-remote.sh no longer fails on a
default artefact the caller's own -p selection did not build. Proof on the box: igneum-prove-host 71,943,192 B, sha256
e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, 1 min 05 s warm. Plan: gotcha rows for the case, the
protoc miss and one lost ssh session (collector cleared by test).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh gains a checkout mode used by lib.sh: git checkout -- . and git clean -fd (target dirs, the sha stamps and
ignored files kept), fetch, branch at the commit, then a clean-tree check; the overlay follows. Before this, the rsync of
uncommitted files stayed in the box's tree and the next commit's git checkout -B refused with 'local changes would be
overwritten'. remote-run.sh --self-test reproduces the dirty tree (edited tracked file, untracked file, target dir, sha
stamp), shows the plain checkout refusing and the mode landing clean on the new commit; it runs in ci.yml and passed on the
Mac and the box; a live dirty-then-clean pair on the fork worktree passed too. Plan: section 5, gotchas of the first day.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in
/srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot,
wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h
slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and
build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash
(no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that
imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental
7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build-remote.sh runs a cargo command on igneum-build-1 from any crate directory of any worktree: HEAD through the bare
mirror (a real .git for kaspa-build-info), uncommitted changes by rsync --checksum with the written files re-stamped, a
remote slot (/srv/builds/_locks, never the Mac's), sccache, -j 90, artefacts back into target-remote/ with size and sha256.
cross-remote.sh is the Windows cross-build with the PC job's Ubuntu mingw-posix recipe plus the Mac's static flags, DLL
list and sha256 per exe, --compare against the Mac's exes. run-from-mac.sh ships provision.sh, writes
~/.config/igneum/build-server, adds the build remotes and pushes every branch of both repos. shellcheck and the CI checks clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
