Three classes from the 0.3.17 night. (1) bash reads a script incrementally: tools/build-remote.sh was edited while a four-minute remote
build ran, the running copy continued at shifted bytes and died with a syntax error after the build had succeeded on the box; the
four long-running tools (build-remote, cross-remote, workers-remote, move-hand) now keep their body in one brace block ending in exit,
parsed whole before a line runs; tools/ci/whole-body-check.sh (in the gate, self-test with a block-less copy) holds the shape.
(2) A fork build pairs with the igneum-pow of the igneum worktree it sits in: a fork at 12153428 under a master worktree failed in
kaspa-pow four minutes in (no chain_program_shadow; master's igneum-pow predates release-0.3.17's); build-remote.sh says the
pairing on its first line ('pairs with igneum 6f8d7a7e (detached): igneum-pow 0.2.0') and the JSONL line carries pairs_with.
The first version of that line used '[ -n ... ] && echo' inside an assignment's $( ) and set -e ended the script on the false
status; fixed. (3) move-hand.sh restart <hand> [--digest <hex>] [--go]: after binary installed a release, restart ONE unit and read
it back (first exec line, commit string in the running binary, digest against the wanted one, igneum_getNodeInfo powEngine over the
node's loopback EVM RPC); the digest readers tolerate a missing line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The shipper's report: with no arguments, cross-remote.sh died under the Mac's bash 3.2 with 'CARGO_ARGS: unbound variable' and
its chain kept the previous exes. Cause: a comment appended to the defaults line in the box-work merge turned OUT, COMPARE,
TARGET_DIR and CARGO_ARGS=() into comment text, so the array was never declared (the same shape build-remote.sh had at 19:5x).
Fix: the line split, and the default tests use [ -n "${CARGO_ARGS[*]:-}" ] in both tools, safe whether or not the array
exists. Proof: the default cross-build from a fork worktree under /bin/bash 3.2.57 built both exes (igneumd.exe 8f7e2ae3...,
igneum-miner.exe 6f8b49d8...). tools/ci/defaults-line-check.sh fails CI on a line where a comment start is followed by an
assignment list (quoted strings, ${...}, $# and [^#] dropped first); self-test fires on the swallowed shape and passes
${a#b}, sed s#x#y#, $# loops and prose mentions; the tree is clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's rule of 6 October 2026 from the 0.3.14 repro (docs/evidence/reproduced/0.3.14.md): prost's protowire.rs embeds OUT_DIR,
libmimalloc-sys embeds __DATE__/__TIME__, sccache hid both. lib.sh bs_repro_env exports SOURCE_DATE_EPOCH=<author time> TZ=UTC in
front of every remote command (build-remote.sh, cross-remote.sh, workers-remote.sh); remote-run.sh exports BR_SDE too and logs it
as source_date_epoch; proto-cuda/windows-node/cross-build.sh exports the same; the PC job carries node.commit_time in the manifest
(push-build-inputs.sh) and exports it before every cargo build of a stage (jobbuild.rs, unit test asserts it; 4 of 4 pass on the
box). Target dirs stay one fixed path per target. tools/build-remote.sh --self-test-repro [--full] from a fork worktree, run on
the box: igneum-miner twice a minute apart without sccache (RUSTC_WRAPPER=/usr/bin/env, an empty value is unset to cargo) MATCH
91e130f5..., a per-run target path differs (OUT_DIR shown); --full: kaspad with libmimalloc-sys recompiled a minute later MATCH
70219bc2..., without the epoch differs (__DATE__ shown).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- provision.sh step_runner: actions/runner 2.338.0 (sha256 checked) at /opt/actions-runner under a dedicated user `runner`
(no sudo, not in build's group), rustup 1.99.0 pinned with both targets, sccache against /srv/sccache in READ_ONLY mode
on its own server port, Node 22 and mingw from the system, GitHub's svc.sh unit with a Nice 10 drop-in; registered on
igneum-network/igneum as igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) through
infra/build-server/runner/register.sh (gh as igneum-labs, the token on ssh stdin, never logged). Idempotent after
the env files moved behind svc.sh install (its env.sh rewrites them). libicu74 and python3-numpy added to APT.
- main's slots ruling: SLOTS default 2; remote-run.sh sets CARGO_BUILD_JOBS 90 when it holds the only taken slot and 45
when both are held, BR_MEASURE=1 takes the `measure` file exclusively and excludes builds (builds hold it shared),
lock files open in append mode (the old `exec {fd}>` truncated a busy slot's holder line on every probe), env
IGNEUM_BUILD_SLOTS_DIR and IGNEUM_BUILD_LOG_DIR win over the profile, `--self-test-slots` with five cases (the old
script fails it with JOBS=none); build-remote.sh and cross-remote.sh pass -j only when --jobs is given.
- infra/build-server/prover/cpu-trial.sh: the SP1 CPU prover on one fixture shard under the measure hold with a VmHWM
poller; 6 Oct 2026 run: core 34.2 s, compressed 85.9 s, peak RSS 28.2 GB on 96 threads, so no standing CPU prover.
- docs/plans/ci-self-hosted.md: the proposed runs-on change for ci.yml behind the repository variable IGNEUM_CI_RUNNER
(GitHub-hosted is the fallback), and why windows.yml cannot move to a Linux box. Workflows untouched.
- docs/plans/build-server.md section 7.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
the project lead's decision of 6 October 2026: the Mac runs nothing the network depends on. docs/plans/hands-on-build-1.md plans node 1
and the observer (its node and tools/observer) as systemd units on the box, one hand at a time, with ports, DNS names,
exec recovery, rollback and the decisions for main; infra/build-server/hands/install-hands.sh writes users, dirs, run
scripts and units (inert, run on the box); move-hand.sh (dry run by default) builds 0.3.15 on the box, copies the override
and snapshot, rsyncs each data dir hot then stopped, starts the unit and prints the hand's first executing line, copies the
observer env by scp (mode 600), stops the Mac's observer first, unloads the launchd agents last. CLAUDE.md's running-agents
rule carries the decision. lib.sh: a per-worktree lock on the box across sync, build and fetch (the shipper's collision at
18:48:56Z). remote-run.sh: the clean-tree test excuses stamps and target dirs at any depth with --untracked-files=all (a
nested stamp failed every checkout of /srv/builds/igneum for twenty minutes) and removes a stale .git/index.lock; the
self-test covers both. Plan rows for the three and the cuda prover pair built for the shipper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's decision of 6 October 2026. CLAUDE.md 'Running agents on this Mac': every Linux and Windows cargo build and every
Linux test suite goes to igneum-build-1 through tools/build-remote.sh and tools/cross-remote.sh; the PCs keep GPU and
Windows-runtime jobs; the Mac keeps macOS binaries, the DMG and Metal tests under the lock. -Wl,--no-insert-timestamp in
cross-remote.sh, proto-cuda/windows-node/cross-build.sh and jobbuild.rs (two box builds byte-identical, verified).
The empty-commit class: tools/ci/commit-string-check.sh (self-test in ci.yml, shown firing on a Mac worktree build and
passing on a box build) runs on every igneumd from the three build scripts; push-build-inputs.sh adds node.commit_full,
the PC job writes a minimal node/.git from it at extract and cleans kaspa-build-info on a new commit (4 jobbuild tests
pass, run on the box); cross-build.sh refuses a worktree and cleans on a new commit. Plan: second worktree's clean build
1 min 18 s with sccache 604 hits of 993.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in
/srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot,
wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h
slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and
build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash
(no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that
imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental
7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build-remote.sh runs a cargo command on igneum-build-1 from any crate directory of any worktree: HEAD through the bare
mirror (a real .git for kaspa-build-info), uncommitted changes by rsync --checksum with the written files re-stamped, a
remote slot (/srv/builds/_locks, never the Mac's), sccache, -j 90, artefacts back into target-remote/ with size and sha256.
cross-remote.sh is the Windows cross-build with the PC job's Ubuntu mingw-posix recipe plus the Mac's static flags, DLL
list and sha256 per exe, --compare against the Mac's exes. run-from-mac.sh ships provision.sh, writes
~/.config/igneum/build-server, adds the build remotes and pushes every branch of both repos. shellcheck and the CI checks clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
