remote-run.sh checkout_tree ran `git clean -fd` on the box mirror before every build from any agent, so the attack rows
lost attack-f3/, attack-f1-venv/ and tools/attack/*/target to each other's builds (7 October 2026, 09:2x UK). The clean now
also spares the fixed prefixes attack-*, scratch-*, target-attack-*, .build-remote.log and every glob in the mirror-local
.igneum-scratch-spare (one per line, # comments, the file itself spared), keeps the target and stamp excludes and still runs
without -x. The clean-tree test asks `git clean -nd` with the same excludes instead of filtering the status list, so a spared
dir is not "not clean". --self-test: a fixed-prefix dir at the root and nested, a declared dir and the spare file survive, an
undeclared dir is removed. tools/ci/scratch-spare-check.sh in the pre-push gate fails when the clean line loses the spare
arguments, spare_args stops reading the file, a fixed prefix goes, or -x appears. docs/plans/build-server.md R4a says how a
lane declares its prefix.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
the project lead added the read-only deploy key on 7 October 2026 (SHA256:51ice3W8...). The sync still said 'mirror' because ssh -T to GitHub
exits 1 after its greeting and observer-sync.sh runs under pipefail, so su ... | grep -q reported failure although grep had
matched (the same line by hand, without pipefail, said authenticated; shown under the unit's environment with systemd-run -v).
The probe's output is captured first. First github pass 07:30:54 UTC: the clone moved from the mirror's d79f4a6 to origin/master
8b878ba4, the observer restarted on it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The shipper's read-back for 0.3.18 (7 October 2026): igneum_getNodeInfo exists again and must answer powEngine igneum-pow (a stub
is a FAIL) with a blockrate object, which the mover prints; a tree before 0.3.18 answers -32601 and the engine is still read from
the binary's igneum-pow source paths. The parser is checked against the three answer shapes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The b3c228fa builds under the 0.3.16 app tree 5d118f58 (no rust-toolchain.toml yet) died right after the pairing line with no
message: sed on the missing file failed, pipefail carried its status into the assignment, set -e ended the script. A guard and a
tolerant pipeline; checked alive against a tree without the file.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three classes from the 0.3.17 night. (1) bash reads a script incrementally: tools/build-remote.sh was edited while a four-minute remote
build ran, the running copy continued at shifted bytes and died with a syntax error after the build had succeeded on the box; the
four long-running tools (build-remote, cross-remote, workers-remote, move-hand) now keep their body in one brace block ending in exit,
parsed whole before a line runs; tools/ci/whole-body-check.sh (in the gate, self-test with a block-less copy) holds the shape.
(2) A fork build pairs with the igneum-pow of the igneum worktree it sits in: a fork at 12153428 under a master worktree failed in
kaspa-pow four minutes in (no chain_program_shadow; master's igneum-pow predates release-0.3.17's); build-remote.sh says the
pairing on its first line ('pairs with igneum 6f8d7a7e (detached): igneum-pow 0.2.0') and the JSONL line carries pairs_with.
The first version of that line used '[ -n ... ] && echo' inside an assignment's $( ) and set -e ended the script on the false
status; fixed. (3) move-hand.sh restart <hand> [--digest <hex>] [--go]: after binary installed a release, restart ONE unit and read
it back (first exec line, commit string in the running binary, digest against the wanted one, igneum_getNodeInfo powEngine over the
node's loopback EVM RPC); the digest readers tolerate a missing line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's order of 7 October 2026 after RunPod's Ubuntu 22.04 canaries (glibc 2.35) refused the box's GLIBC_2.38 binaries and HiveOS
turned out Ubuntu 20.04 based (2.31). The table lives once, in tools/ci/glibc-ceiling-check.sh (--class, --ceiling-of; self-test
covers the table, an unknown class and a 2.34 need against hive); tools/build-remote.sh --ship hive|rig|seed|linux (default seed)
and tools/workers-remote.sh --class (default rig) build with zig at the class's glibc and check against it. provision.sh installs
docker.io (user build in the docker group) for the proof. Proof: fork 3bfe346f at class hive, igneumd and igneum-miner need
GLIBC_2.30; the workers at class rig need GLIBC_2.17; all four run in ubuntu:20.04 (ldd 2.31) and ubuntu:22.04 (ldd 2.35) and
print their version or usage lines (the OpenCL worker its own no-libOpenCL message, the binary running in a GPU-less container).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit's plan edit aborted on a changed anchor, so section 5c was missing; written now with the public half
(ssh-ed25519 ... igneum-build-1 observer read-only) and the CI-runner row closed. observer-sync.sh ran git rev-parse as root
against the build-owned clone (safe.directory refused it, 'up to date at' with no commit); every git call runs as build now.
Verified on the box: one sync pass prints 'source: mirror (the deploy key is not accepted by GitHub yet ...)' and the commit.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's order of 7 October 2026. install-hands.sh creates /srv/observer/.ssh/deploy_igneum (ed25519, user build, mode 600; the private
half never leaves the box and nothing prints it) and the ssh alias github-igneum-observer; observer-sync.sh tests the key with
ssh -T on every pass, pulls from GitHub when it is accepted and from the mirror otherwise, saying which. docs/plans/build-server.md
5c: the four steps for the project lead (print the public half, Settings > Deploy keys > Add, read-only, start one sync pass) and the public
half itself. Verified on the box: key created, alias written, one sync pass reads 'source: mirror (the deploy key is not accepted by
GitHub yet)'. The CI-runner row is closed (the box-work agent's runner service).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's order of 7 October 2026. The repo root carries rust-toolchain.toml (channel 1.99.0, targets x86_64-pc-windows-gnu and
x86_64-unknown-linux-gnu); rustup resolves the nearest file walking up from the crate, so the fork worktrees under vendor/ are
covered, and the fork's master carries its own copy (vendor/igneum-node 37f1206b). lib.sh bs_toolchain_check reads the pin
and refuses a build when the pin, the Mac's rustc as resolved in the crate dir, or the box's rustc differ (the message says
the three commands that align them); run-from-mac.sh passes the pin to provision.sh as RUST_TOOLCHAIN. The 1.99.0 toolchain
with both targets is installed on the Mac so no agent's build stalls on rustup's auto-install. Verified: the Mac resolves
1.99.0 in a fork worktree, the box runs 1.99.0, the check prints 'pinned 1.99.0 by rust-toolchain.toml'.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's order of 7 October 2026 after a seed took 14 restarts and three minutes down on a glibc 2.39 binary. provision.sh:
step_zig (zig 0.17.0 from ziglang.org, sha256 from the official download index) and cargo-zigbuild 0.23.4 in the cargo tools.
tools/build-remote.sh --ship [--glibc 2.36]: cargo zigbuild --target x86_64-unknown-linux-gnu.2.36 with zig as the C/C++
toolchain (the Mac's infra/cross/build-linux.sh recipe), artefacts from the target-triple dir, each checked by
tools/ci/glibc-ceiling-check.sh (need at most the ceiling; self-test fires on 2.38 against 2.36, passes 2.34 and 2.36;
--symbols reads a saved objdump -T text so CI needs no ELF tools). tools/workers-remote.sh builds the two GPU workers with
zig at 2.36 by default (GLIBC=native for clang). Proof on the box: fork 3bfe346f igneumd needs GLIBC_2.34 (47,023,120 B,
sha256 345dfb95...), igneum-miner GLIBC_2.34 (9,248,808 B, d09dc27b...), 3 min 17 s cold through zig; the workers at 2.36.
Rule: anything that ships to a seed or a HiveOS rig is built with --ship; a plain build (glibc 2.39) is for the box and
Ubuntu 24.04 hosts only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/plans/hands-on-build-1.md section 6: the move of 6 October 2026 23:06 to 23:22 UTC, observer node, observer, node 1, unload,
each hand's first executing line, digest eada4bda MATCH, IBD and acceptance, the open readback (the Mac's Miner app has not
started its own node since 26610/26611 were freed). lib.sh: the shipper's class from the 0.3.17 tree, the fork's igneum-exec
embeds proving/igneum-prove/elf/*.vk by include_bytes! five levels up, which is no path dependency; every .rs in the trees that
travel is scanned for include_bytes!/include_str! paths leaving the crate's repository and their directories join the overlay;
proving/igneum-prove/elf is the fixed fallback for a fork build. move-hand.sh: igneumd --version exits 1 (tolerated; it ended
the binary step before the override was written), and the launchd pid lookup used \s in macOS awk (printed 'pid none').
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's hold of 6 October 2026 (23:xx UK): the hands move only on the shipper's 'publish 2 live: digest <x>' line, each hand read back by
commit string and consensus digest. move-hand.sh binary now derives --node from the fork tree behind the Mac's node1 launchd agent
(igneum-wt-ship0315/vendor/igneum-node-0315 at f1ea7a38 tonight; the box's igneumd rebuilt from it is sha256 7f0bde70...), warns when the
Mac's override still has 13 fields (publish 2 writes the sixteen-field object when it restarts the hands), and after each hand's first
executing line prints the box binary's commit string and the hand's digest against the Mac hand's last digest (MATCH or DIFFER, DIFFER
stops the sequence). Dry run clean against the live Mac.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The fleet's 22:09 UK incident (a Mac-side pkill -f <log file name> matched nothing, the roll-everything script lived on and wiped a held box) and the day's two pgrep self-matches are one class. The check flags pgrep -f / pkill -f with a plain literal (every one on a line), any pgrep/pkill on a file-name shape, and ps | grep with a literal; it allows the bracket form, -x, -F pidfile, kill $(cat pidfile), a variable and a full path; 11 banned and 16 allowed shapes in its self-test; 0.15 s over the tree. The 25 pkill -f sp1-gpu-server inside bash -c bodies (which matched the calling bash) are pkill -x; the other 11 literals take the bracket form; prover-socket-check accepts both. Row R in the record; the CLAUDE.md rule names the check and covers pkill and file names.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The box's 34 red rows of 6 October classified (docs/analysis/ci-failures-2026-10-06.md section 6): 22 iterations, 12 in three real classes (instant deaths with nothing kept, a shared worktree directory, an unread dry run). remote-run.sh now: pre-flight (subcommand, manifest, -p package, --features) refuses in a second with exit 3 and a class; the last 400 lines of every run kept in /srv/builds/_log/runs; a class on every row (compile-error, link-error, test-failure, instant, no-test-matched, slot-timeout, no-dir, preflight-*); a cargo test whose filter matched no test exits 3; a per-worktree lock in checkout and run mode; every red row appended to /srv/ci-red/red.jsonl as source box. red-watch.mjs never posts a box row alone and sends one digest a day (counts per class with each class's guard); the timer runs tick. Shared group cired on the box so the runner and build append to one file. Shown in a sandbox on the box: pass, failing test, empty filter, bad package, bad feature, missing subcommand, compile error, broken manifest, two concurrent runs of one worktree.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- pow-fuzz: list saved mismatches from the directory, not an ls|node pipe (pipefail ran
the || echo too, giving invalid JSON [][]); drop the dead inner accumulation line
- sync-fuzz: merge the override with python, not node (node rounds the u64::MAX activation
fields to a float the Rust parser rejects); retention-period-days 2 (the 2-day minimum);
grep -c without || echo (pipefail doubled the count)
- clippy-audit: cap_cargo_t (timeout cannot run the cap_cargo shell function); grep -c fix
- all jobs sync the checkout unconditionally and lib.sh defaults the branch vars so a
standalone job or smoke never trips set -u on CAP_NODE_BRANCH
Smokes on igneum-build-1, all 0 panics: pow-fuzz 98 rounds / 19,600 programs / 78,400
units; sync-fuzz 142,883 requests, node alive, every kind disconnected or answered;
sim-sweeps 5 rows; model-sweeps 7 sections; clippy-audit 69 branches, 1,192 warnings,
1 error (ca2-coord), 0 advisories.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pgrep -x git over the whole machine kept the lock whenever any git ran (the pre-push hook's own git push, another agent's build) and the checkout died with "index.lock: File exists". The fact is the lock's age. Class Q in docs/analysis/ci-failures-2026-10-06.md with the GIT_DIR leak of the previous commit.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
168 non-green runs since the first workflow run were classified (docs/analysis/ci-failures-2026-10-06.md): 102 were tree checks that finish in under 25 s on the pushing machine, 40 were GitHub-side refusals nobody saw.
tools/ci/pre-push.sh is the one list of fast checks; ci.yml's site job calls it with --ci and the pre-push hook with --hook (full gate for master and release-*, structural checks for other refs; never writes into the worktree). tools/ci/export-exclude.txt lists research documents outside the public export list, pruned by identity-check.sh and by the mirror's sync.sh (self-test: an excluded path may quote the patterns, an exported one may not); polish.md and this record are its first entries, which makes master green. tools/ci/windows-paths-check.sh (colon, trailing dot or space, reserved names, over 240 characters) runs as the pre-commit hook on staged paths and in the gate. tools/ci/red-watch.mjs plus the red job on the box's runner record one line per failed master or release-* run to /srv/ci-red/red.jsonl; igneum-ci-red.timer posts each once to the updates channel. pow and sims read IGNEUM_CI_RUNNER for the box. no-foreign-tree-writes.sh no longer exits silently on its warning pipeline under pipefail.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
infra/build-server/capacity: igneum-capacity.service (user build, Nice 19, SCHED_IDLE,
CPUQuota leaving 8 threads free) runs run.sh, a controller over a priority queue of jobs
that pauses (SIGSTOP) the instant a build slot or the measure hold is taken (5 s poll of
/srv/builds/_locks) and resumes after. Jobs, each with a dry-run and a 10-min smoke:
pow fuzz (continuous, seed base advances), sync-request fuzz against a throwaway pruned
node on loopback (the Horizon 28-unwrap gate, igneum-p2p-probe sync-fuzz), GHOSTDAG and
finality sweeps seeds 1 to 1,000, model.py sweeps cached, clippy+audit per recent branch.
Summaries to /srv/workers/capacity.json; the worker dashboard gains a Background lane
(collect.mjs doc.background, page renderBackground). Night battery stops and restarts the
layer. docs/plans/build-server.md section 8. igneum-pow fuzz tests and ghostdag_sim.py /
attacks.py gain a seed-base knob for the continuous sweeps.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The build-server agent's two findings on rebuild-on-box.sh (6 Oct 2026, 20:1xZ): tools/ci/copied-sources-check.sh named it
(a tar on a code line plus cargo build with no touch), and `RUSTC_WRAPPER=` did not switch sccache off, so some passes of the
first runs took hits. Both 0.3.14 and 0.3.15 are re-run with this version before their evidence files are trusted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's rule of 6 October 2026 from the 0.3.14 repro (docs/evidence/reproduced/0.3.14.md): prost's protowire.rs embeds OUT_DIR,
libmimalloc-sys embeds __DATE__/__TIME__, sccache hid both. lib.sh bs_repro_env exports SOURCE_DATE_EPOCH=<author time> TZ=UTC in
front of every remote command (build-remote.sh, cross-remote.sh, workers-remote.sh); remote-run.sh exports BR_SDE too and logs it
as source_date_epoch; proto-cuda/windows-node/cross-build.sh exports the same; the PC job carries node.commit_time in the manifest
(push-build-inputs.sh) and exports it before every cargo build of a stage (jobbuild.rs, unit test asserts it; 4 of 4 pass on the
box). Target dirs stay one fixed path per target. tools/build-remote.sh --self-test-repro [--full] from a fork worktree, run on
the box: igneum-miner twice a minute apart without sccache (RUSTC_WRAPPER=/usr/bin/env, an empty value is unset to cargo) MATCH
91e130f5..., a per-run target path differs (OUT_DIR shown); --full: kaspad with libmimalloc-sys recompiled a minute later MATCH
70219bc2..., without the epoch differs (__DATE__ shown).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- rebuild-on-box.sh: the DIFFER reason comes from the binaries (glibc need of both, the build path each embeds, the
mimalloc clock string, the PE timestamp, the commit string), never from an assumed story; the three string helpers run
their producer under `|| true` so a consumer that stops early (grep -m1, awk exit) no longer trips pipefail into the
fallback and a second line in a table cell.
- docs/evidence/reproduced/0.3.15.md, and the 0.3.14 file re-reported with the same column.
- docs/plans/build-server.md 7.3: the 0.3.15 row and what the two files mean for shipping from the box.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- infra/build-server/repro/rebuild-on-box.sh: one target path per target with each pass's artefacts copied aside (prost's
protowire.rs embeds its OUT_DIR path, so a pass in a dir of another name differs), SOURCE_DATE_EPOCH from the node commit
and TZ=UTC (libmimalloc-sys compiles mimalloc's C with __DATE__ and __TIME__), --reuse for a re-report, reason text
for a shipped file that is not on hand (innoextract 1.9 cannot open the Inno Setup 6 installer; the 0.3.14 HiveOS
tarball left dl/public when 0.3.15 published).
- tools/repro/rebuild-release.sh: the plan's hashes always travel (the miners' 8-hex prefixes too), bash 3.2 empty-array
fix, the box half's exit code is the script's.
- docs/evidence/reproduced/0.3.14.md: igneumd 03f35e05..., igneum-miner 900c1f0b..., igneumd.exe 166e604e...,
igneum-miner.exe fefd266c... identical across two clean passes; DIFFER against the shipped 934f393c... (zig, glibc 2.36)
and 44fa74c0... (Homebrew mingw before the timestamp fix, from a worktree).
- docs/plans/build-server.md 7.2 (night battery: timer, dry run 3 min 54 s, 10 pass, the fork's 22 cargo-audit
advisories as the one FAIL and what follows) and 7.3 (repro: results, the two classes, SOURCE_DATE_EPOCH proposed for
every build script).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- infra/build-server/night/night-battery.sh: checkout of master and the newest release-*-node fork branch under
/srv/builds/_night, cargo test --release --no-fail-fast per crate (repo and fork), igneum-pow fuzz at 10x, the three
simulators in full, the fast-time harnesses (finality-attacks, harness s3 s4, exec-sync reorg) on binaries built into
target-integration, clippy per crate dir, cargo audit per Cargo.lock; docs/benchmarks/night/<date>.md with a pass/fail
table and "new since last night"; committed as igneum-labs on night-battery and pushed to the mirror, never master.
NIGHT_SUBSET=1 is the dry-run subset. The unit runs it through remote-run.sh so the slot spans the invocation.
- igneum-night-battery.{service,timer}: 02:00 Europe/London, User build, Nice 19, idle IO, 8 h limit, not Persistent.
- provision.sh: step_cargo_tools (cargo-audit), step_night (files from the mirror at NIGHT_REF, timer enabled), innoextract.
- tools/repro/rebuild-release.sh + infra/build-server/repro/rebuild-on-box.sh: pins from docs/plans/release-<v>.md
("(node <sha>, app <sha>)"), shipped hashes from the public downloads (the HiveOS tarball, the installer via
innoextract) or --shipped, a clean clone of repo and fork on the box, two passes per target without sccache under build
slots, MATCH or DIFFER per artefact against the shipped bytes and against the other pass, with the reason read off the
binaries (glibc version needed, PE timestamp, commit string); evidence into docs/evidence/reproduced/<version>.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- provision.sh step_runner: actions/runner 2.338.0 (sha256 checked) at /opt/actions-runner under a dedicated user `runner`
(no sudo, not in build's group), rustup 1.99.0 pinned with both targets, sccache against /srv/sccache in READ_ONLY mode
on its own server port, Node 22 and mingw from the system, GitHub's svc.sh unit with a Nice 10 drop-in; registered on
igneum-network/igneum as igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) through
infra/build-server/runner/register.sh (gh as igneum-labs, the token on ssh stdin, never logged). Idempotent after
the env files moved behind svc.sh install (its env.sh rewrites them). libicu74 and python3-numpy added to APT.
- main's slots ruling: SLOTS default 2; remote-run.sh sets CARGO_BUILD_JOBS 90 when it holds the only taken slot and 45
when both are held, BR_MEASURE=1 takes the `measure` file exclusively and excludes builds (builds hold it shared),
lock files open in append mode (the old `exec {fd}>` truncated a busy slot's holder line on every probe), env
IGNEUM_BUILD_SLOTS_DIR and IGNEUM_BUILD_LOG_DIR win over the profile, `--self-test-slots` with five cases (the old
script fails it with JOBS=none); build-remote.sh and cross-remote.sh pass -j only when --jobs is given.
- infra/build-server/prover/cpu-trial.sh: the SP1 CPU prover on one fixture shard under the measure hold with a VmHWM
poller; 6 Oct 2026 run: core 34.2 s, compressed 85.9 s, peak RSS 28.2 GB on 96 threads, so no standing CPU prover.
- docs/plans/ci-self-hosted.md: the proposed runs-on change for ci.yml behind the repository variable IGNEUM_CI_RUNNER
(GitHub-hosted is the fallback), and why windows.yml cannot move to a Linux box. Workflows untouched.
- docs/plans/build-server.md section 7.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
provision.sh step_cuda: NVIDIA's ubuntu2404 apt repository, cuda-nvrtc-dev-12-8, cuda-cudart-dev-12-8, cuda-driver-dev-12-8
(the libcuda stub) and opencl-c-headers; no nvcc (no build file calls it), no driver. tools/workers-remote.sh builds
igneum-worker-cuda and igneum-worker-opencl on the box from proto-cuda and proto-opencl with clang++ (static libstdc++),
prints sha256 and the glibc ceiling (2.38: fine for Ubuntu 24.04 hosts, not for 22.04 containers or HiveOS, where the Mac's
zig build stays). Proof: igneum-worker-cuda 1,613,992 B sha256 6db8a9ad..., igneum-worker-opencl 124,904 B sha256 0dea75bb...
remote-run.sh evaluates the command string in a subshell (a leaked set -e killed the runner after a successful build).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's ruling (6 October 2026, 20:1x UK): the box installs with the NUMBERS key alone so the 21:00 UK pulse comes from
it. install.sh accepts at least one key and names the missing ones; every tick's log line ends with "missing <keys>"; the
watcher without an incidents webhook logs its open or resolve and still advances its state, so the key landing later
does not flood the channel. Test added (31 passing).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tools/community/discord-hooks.mjs (Node 22, standard library): one ember embed per post to #numbers, #announcements and
#incidents. Network pulse every 6 h (03/09/15/21 UK) from /api/stats, /api/live, /api/supply with the 6 h window from its
own snapshot and a Devnet 2 line that uses the fleet file's numbers and gate word only; a 24 h digest and the reddit kit's
weekly template at 09:00 UK; a release subcommand for the shipper (three downloads with sha256, node commit, digest, up to
five plain "what changed" lines read from the release plan); incident open and resolve by hand; a watcher that opens one
incident per condition (finality paused 5 min, median proof lag 15 min, observer silent 3 min) and resolves after 2 clear
minutes, and never opens on a condition already firing when it first looks (the pulse says "finality paused since" instead).
Guards before every post: the founder's name and logins, hosts, machine ids, paths, IP addresses, standalone 32-hex
tokens, dl.igneum.network outside /public/, webhook URLs, mentions, the tools/ci/forbidden-strings.txt patterns; Discord's
limits refused rather than cut; idempotency keys per slot in a state file; exponential backoff on 429; dry run by default
with a Discord-like preview in tools/community/out/preview.html. 30 tests on fixtures cut from the live API.
infra/build-server/discord-hooks: a tick every minute on igneum-build-1 (the Mac sleeps). install.sh copies the webhook file
to /srv/discord-hooks/env (mode 600, over ssh stdin) and refuses without IGNEUM_SECRET_ON_BOX_OK=1; the recorded exception
to rule R6 is in docs/plans/build-server.md (main's ruling, 6 October 2026).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
the project lead's decision of 6 October 2026: the Mac runs nothing the network depends on. docs/plans/hands-on-build-1.md plans node 1
and the observer (its node and tools/observer) as systemd units on the box, one hand at a time, with ports, DNS names,
exec recovery, rollback and the decisions for main; infra/build-server/hands/install-hands.sh writes users, dirs, run
scripts and units (inert, run on the box); move-hand.sh (dry run by default) builds 0.3.15 on the box, copies the override
and snapshot, rsyncs each data dir hot then stopped, starts the unit and prints the hand's first executing line, copies the
observer env by scp (mode 600), stops the Mac's observer first, unloads the launchd agents last. CLAUDE.md's running-agents
rule carries the decision. lib.sh: a per-worktree lock on the box across sync, build and fetch (the shipper's collision at
18:48:56Z). remote-run.sh: the clean-tree test excuses stamps and target dirs at any depth with --untracked-files=all (a
nested stamp failed every checkout of /srv/builds/igneum for twenty minutes) and removes a stale .git/index.lock; the
self-test covers both. Plan rows for the three and the cuda prover pair built for the shipper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The second build of every repo-kind crate in a subdirectory failed (6 October 2026, 18:51 UTC, the pool build: "tree not
clean after reset: ?? pool/.build-remote-sha-target"): checkout_tree keeps the stamps with `git clean -e` at any depth but
its status check matched them at the root only. The check now reads `git status --porcelain --untracked-files=all` (an
all-untracked directory is listed file by file, not as "?? sub/") and accepts target dirs, sccache and both stamps under
any path. The self-test carries the subdirectory case (stamp and target dir kept, overlay file removed) and the known-failed
case (a stray untracked file still fails the check).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a member of the fork's workspace that inherits
from the fork's root manifest; syncing that one directory left cargo without a workspace root on the box. lib.sh now groups
path dependencies by git top level: a repository under vendor/ is pushed to its mirror (a fork worktree to
/srv/igneum-node.git, a repository of its own to /srv/<name>.git, created on first use), checked out whole at
/srv/builds/<worktree>/vendor/<name> and overlaid whole; run-from-mac.sh wires every vendor repository the Cargo.toml files
reach. libprotobuf-dev added (sp1-prover-types imports google/protobuf/empty.proto). build-remote.sh no longer fails on a
default artefact the caller's own -p selection did not build. Proof on the box: igneum-prove-host 71,943,192 B, sha256
e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, 1 min 05 s warm. Plan: gotcha rows for the case, the
protoc miss and one lost ssh session (collector cleared by test).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The node pin moves to 4c6b129d (release-0.3.14-node), which is what the live payload inputs carry, so windows-ci's payload-inputs step is green again (red since e9eca23 on master's 0.3.13 pin against the 0.3.14 inputs).
Conflicts: infra/fast-time/override-60x.json keeps master's side (the fresh-rule field was already there at u64::MAX; the release line would be a duplicate key); tools/ci/playbook-quit-check.sh keeps master's rule 2 and pre-rule list with the release side's Ember allow entry and the 0.3.15 expiry check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh gains a checkout mode used by lib.sh: git checkout -- . and git clean -fd (target dirs, the sha stamps and
ignored files kept), fetch, branch at the commit, then a clean-tree check; the overlay follows. Before this, the rsync of
uncommitted files stayed in the box's tree and the next commit's git checkout -B refused with 'local changes would be
overwritten'. remote-run.sh --self-test reproduces the dirty tree (edited tracked file, untracked file, target dir, sha
stamp), shows the plain checkout refusing and the mode landing clean on the new commit; it runs in ci.yml and passed on the
Mac and the box; a live dirty-then-clean pair on the fork worktree passed too. Plan: section 5, gotchas of the first day.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tools/workers/collect.mjs runs every 30 s on the box (infra/build-server/workers/igneum-workers.{service,timer},
install.sh) and writes /srv/workers/workers.json from the machine itself: /proc/stat deltas per core, meminfo, df on
/srv, net bytes, hwmon temperatures, the flock state of /srv/builds/_locks, cargo processes with worktree, target and
start time, flock waiters, /srv/builds/_log/builds.jsonl (the format agreed with the build-server agent), sccache
--show-stats, headline.json. tools/workers/push.mjs (launchd every 60 s) adds the Mac's with-lock slots and waiters
and the two PCs' relay job states from the intake, drops them on the box so its file is whole, merges the box's file
and writes the fleet folder's workers.json; it deploys only when the live copy is over 6 min old, otherwise the
fleet orchestrator's 5-minute deploy carries it. The page (tools/workers/page/workers.html, shape.js) tries
https://build.igneum.network/workers.json first and falls back to the folder copy; core strip, arcs, now building,
queue, recently done with closing lines, headline timings, analytics; UK time with UTC tooltips; phone width.
node --test tools/workers/test: 13 tests over /proc/stat, lock dir, JSONL and sccache fixtures and the page shaping.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh writes /srv/builds/_locks/wait-<pid> in the slot-file line format while it waits and removes it when the
slot is taken or the wait is given up (shown: present during a held slot, gone after). provision.sh installs Caddy with a
Caddyfile that serves only /srv/workers/workers.json and headline.json (every other path 404, CORS for dl.igneum.network,
no-store, Let's Encrypt only) and opens 80 and 443. Both asked for by the worker-dashboard agent, approved by main.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in
/srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot,
wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h
slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and
build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash
(no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that
imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental
7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build-remote.sh runs a cargo command on igneum-build-1 from any crate directory of any worktree: HEAD through the bare
mirror (a real .git for kaspa-build-info), uncommitted changes by rsync --checksum with the written files re-stamped, a
remote slot (/srv/builds/_locks, never the Mac's), sccache, -j 90, artefacts back into target-remote/ with size and sha256.
cross-remote.sh is the Windows cross-build with the PC job's Ubuntu mingw-posix recipe plus the Mac's static flags, DLL
list and sha256 per exe, --compare against the Mac's exes. run-from-mac.sh ships provision.sh, writes
~/.config/igneum/build-server, adds the build remotes and pushes every branch of both repos. shellcheck and the CI checks clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Idempotent provisioning of the Hetzner AX162 build box: install mode (rescue system, Ubuntu 24.04, software RAID 1 over
the two NVMe drives, no swap, rescue keys taken over, run 6 October 2026 17:16 to 17:20 UTC) and provision mode (user build,
compilers, mingw-w64 posix, rustup 1.99.0 with the Windows target, sccache 100 GB, Node 22, bare mirrors, /srv/builds, sshd
key-only, ufw 22 + seed p2p ports). shellcheck clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On 47ede3f, by hand, the app half of the exec fix (fork exec-sync-0313 1f59c5d0 is the node half):
- app/igneum-app/src/prover.rs: igneum_exportSegments [n-1, n] for a shard and [first-1, last] for a segment,
never from 0 (on a restarted node the records below the restart carry zero roots and the exporter refused every
cut, the fleet 16:02Z); exec_boundary() reads igneum_getExecStatus.restartNumber (or the startedFrom text on a
0.3.13 node) and the prover claims no shard and no segment below it
- proving/igneum-prove/export: seeds the port from the export's preState (the node's account dump after the first
segment), checks its root against the node's there and replays from the next segment; without a dump the
restart-aware replay (execRestart) and the genesis replay stay
- tools/exec-sync/net.mjs (15 checks: the persisted state, the file, the wrong pin, another chain, the unreadable
flag file, the account-dump cut) and tools/exec-sync/reorg.mjs (18 checks: a 300-block reorg from the ring and
from the persisted generation)
- infra/fast-time/override-60x.json: the duplicate proving_v1 block from the 0.3.12 merge removed (the
consensus-core test fast_time_60x_file_is_the_devnet_at_60x failed on it)
The three UI files are untouched (byte-equal to 47ede3f); the igneum-prove-r0 tree is not in this cut.
Green on this tip (6 October 2026): cargo test in app/igneum-app 28 + 8; node --test app/igneum-app/ui 35; the 13
CI checks of ci.yml that run on this Mac; tools/exec-sync/net.mjs 15/15 in 97.2 s against this exporter and the
fork's igneumd (IGNEUM_EXEC_BIN, IGNEUM_EXPORTER).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
