the project lead's decision of 6 October 2026: the Mac runs nothing the network depends on. docs/plans/hands-on-build-1.md plans node 1
and the observer (its node and tools/observer) as systemd units on the box, one hand at a time, with ports, DNS names,
exec recovery, rollback and the decisions for main; infra/build-server/hands/install-hands.sh writes users, dirs, run
scripts and units (inert, run on the box); move-hand.sh (dry run by default) builds 0.3.15 on the box, copies the override
and snapshot, rsyncs each data dir hot then stopped, starts the unit and prints the hand's first executing line, copies the
observer env by scp (mode 600), stops the Mac's observer first, unloads the launchd agents last. CLAUDE.md's running-agents
rule carries the decision. lib.sh: a per-worktree lock on the box across sync, build and fetch (the shipper's collision at
18:48:56Z). remote-run.sh: the clean-tree test excuses stamps and target dirs at any depth with --untracked-files=all (a
nested stamp failed every checkout of /srv/builds/igneum for twenty minutes) and removes a stale .git/index.lock; the
self-test covers both. Plan rows for the three and the cuda prover pair built for the shipper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The second build of every repo-kind crate in a subdirectory failed (6 October 2026, 18:51 UTC, the pool build: "tree not
clean after reset: ?? pool/.build-remote-sha-target"): checkout_tree keeps the stamps with `git clean -e` at any depth but
its status check matched them at the root only. The check now reads `git status --porcelain --untracked-files=all` (an
all-untracked directory is listed file by file, not as "?? sub/") and accepts target dirs, sccache and both stamps under
any path. The self-test carries the subdirectory case (stamp and target dir kept, overlay file removed) and the known-failed
case (a stray untracked file still fails the check).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a member of the fork's workspace that inherits
from the fork's root manifest; syncing that one directory left cargo without a workspace root on the box. lib.sh now groups
path dependencies by git top level: a repository under vendor/ is pushed to its mirror (a fork worktree to
/srv/igneum-node.git, a repository of its own to /srv/<name>.git, created on first use), checked out whole at
/srv/builds/<worktree>/vendor/<name> and overlaid whole; run-from-mac.sh wires every vendor repository the Cargo.toml files
reach. libprotobuf-dev added (sp1-prover-types imports google/protobuf/empty.proto). build-remote.sh no longer fails on a
default artefact the caller's own -p selection did not build. Proof on the box: igneum-prove-host 71,943,192 B, sha256
e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, 1 min 05 s warm. Plan: gotcha rows for the case, the
protoc miss and one lost ssh session (collector cleared by test).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh gains a checkout mode used by lib.sh: git checkout -- . and git clean -fd (target dirs, the sha stamps and
ignored files kept), fetch, branch at the commit, then a clean-tree check; the overlay follows. Before this, the rsync of
uncommitted files stayed in the box's tree and the next commit's git checkout -B refused with 'local changes would be
overwritten'. remote-run.sh --self-test reproduces the dirty tree (edited tracked file, untracked file, target dir, sha
stamp), shows the plain checkout refusing and the mode landing clean on the new commit; it runs in ci.yml and passed on the
Mac and the box; a live dirty-then-clean pair on the fork worktree passed too. Plan: section 5, gotchas of the first day.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tools/workers/collect.mjs runs every 30 s on the box (infra/build-server/workers/igneum-workers.{service,timer},
install.sh) and writes /srv/workers/workers.json from the machine itself: /proc/stat deltas per core, meminfo, df on
/srv, net bytes, hwmon temperatures, the flock state of /srv/builds/_locks, cargo processes with worktree, target and
start time, flock waiters, /srv/builds/_log/builds.jsonl (the format agreed with the build-server agent), sccache
--show-stats, headline.json. tools/workers/push.mjs (launchd every 60 s) adds the Mac's with-lock slots and waiters
and the two PCs' relay job states from the intake, drops them on the box so its file is whole, merges the box's file
and writes the fleet folder's workers.json; it deploys only when the live copy is over 6 min old, otherwise the
fleet orchestrator's 5-minute deploy carries it. The page (tools/workers/page/workers.html, shape.js) tries
https://build.igneum.network/workers.json first and falls back to the folder copy; core strip, arcs, now building,
queue, recently done with closing lines, headline timings, analytics; UK time with UTC tooltips; phone width.
node --test tools/workers/test: 13 tests over /proc/stat, lock dir, JSONL and sccache fixtures and the page shaping.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh writes /srv/builds/_locks/wait-<pid> in the slot-file line format while it waits and removes it when the
slot is taken or the wait is given up (shown: present during a held slot, gone after). provision.sh installs Caddy with a
Caddyfile that serves only /srv/workers/workers.json and headline.json (every other path 404, CORS for dl.igneum.network,
no-store, Let's Encrypt only) and opens 80 and 443. Both asked for by the worker-dashboard agent, approved by main.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in
/srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot,
wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h
slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and
build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash
(no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that
imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental
7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build-remote.sh runs a cargo command on igneum-build-1 from any crate directory of any worktree: HEAD through the bare
mirror (a real .git for kaspa-build-info), uncommitted changes by rsync --checksum with the written files re-stamped, a
remote slot (/srv/builds/_locks, never the Mac's), sccache, -j 90, artefacts back into target-remote/ with size and sha256.
cross-remote.sh is the Windows cross-build with the PC job's Ubuntu mingw-posix recipe plus the Mac's static flags, DLL
list and sha256 per exe, --compare against the Mac's exes. run-from-mac.sh ships provision.sh, writes
~/.config/igneum/build-server, adds the build remotes and pushes every branch of both repos. shellcheck and the CI checks clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Idempotent provisioning of the Hetzner AX162 build box: install mode (rescue system, Ubuntu 24.04, software RAID 1 over
the two NVMe drives, no swap, rescue keys taken over, run 6 October 2026 17:16 to 17:20 UTC) and provision mode (user build,
compilers, mingw-w64 posix, rustup 1.99.0 with the Windows target, sccache 100 GB, Node 22, bare mirrors, /srv/builds, sshd
key-only, ufw 22 + seed p2p ports). shellcheck clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
