- infra/build-server/repro/rebuild-on-box.sh: one target path per target with each pass's artefacts copied aside (prost's
protowire.rs embeds its OUT_DIR path, so a pass in a dir of another name differs), SOURCE_DATE_EPOCH from the node commit
and TZ=UTC (libmimalloc-sys compiles mimalloc's C with __DATE__ and __TIME__), --reuse for a re-report, reason text
for a shipped file that is not on hand (innoextract 1.9 cannot open the Inno Setup 6 installer; the 0.3.14 HiveOS
tarball left dl/public when 0.3.15 published).
- tools/repro/rebuild-release.sh: the plan's hashes always travel (the miners' 8-hex prefixes too), bash 3.2 empty-array
fix, the box half's exit code is the script's.
- docs/evidence/reproduced/0.3.14.md: igneumd 03f35e05..., igneum-miner 900c1f0b..., igneumd.exe 166e604e...,
igneum-miner.exe fefd266c... identical across two clean passes; DIFFER against the shipped 934f393c... (zig, glibc 2.36)
and 44fa74c0... (Homebrew mingw before the timestamp fix, from a worktree).
- docs/plans/build-server.md 7.2 (night battery: timer, dry run 3 min 54 s, 10 pass, the fork's 22 cargo-audit
advisories as the one FAIL and what follows) and 7.3 (repro: results, the two classes, SOURCE_DATE_EPOCH proposed for
every build script).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- infra/build-server/night/night-battery.sh: checkout of master and the newest release-*-node fork branch under
/srv/builds/_night, cargo test --release --no-fail-fast per crate (repo and fork), igneum-pow fuzz at 10x, the three
simulators in full, the fast-time harnesses (finality-attacks, harness s3 s4, exec-sync reorg) on binaries built into
target-integration, clippy per crate dir, cargo audit per Cargo.lock; docs/benchmarks/night/<date>.md with a pass/fail
table and "new since last night"; committed as igneum-labs on night-battery and pushed to the mirror, never master.
NIGHT_SUBSET=1 is the dry-run subset. The unit runs it through remote-run.sh so the slot spans the invocation.
- igneum-night-battery.{service,timer}: 02:00 Europe/London, User build, Nice 19, idle IO, 8 h limit, not Persistent.
- provision.sh: step_cargo_tools (cargo-audit), step_night (files from the mirror at NIGHT_REF, timer enabled), innoextract.
- tools/repro/rebuild-release.sh + infra/build-server/repro/rebuild-on-box.sh: pins from docs/plans/release-<v>.md
("(node <sha>, app <sha>)"), shipped hashes from the public downloads (the HiveOS tarball, the installer via
innoextract) or --shipped, a clean clone of repo and fork on the box, two passes per target without sccache under build
slots, MATCH or DIFFER per artefact against the shipped bytes and against the other pass, with the reason read off the
binaries (glibc version needed, PE timestamp, commit string); evidence into docs/evidence/reproduced/<version>.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- provision.sh step_runner: actions/runner 2.338.0 (sha256 checked) at /opt/actions-runner under a dedicated user `runner`
(no sudo, not in build's group), rustup 1.99.0 pinned with both targets, sccache against /srv/sccache in READ_ONLY mode
on its own server port, Node 22 and mingw from the system, GitHub's svc.sh unit with a Nice 10 drop-in; registered on
igneum-network/igneum as igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) through
infra/build-server/runner/register.sh (gh as igneum-labs, the token on ssh stdin, never logged). Idempotent after
the env files moved behind svc.sh install (its env.sh rewrites them). libicu74 and python3-numpy added to APT.
- main's slots ruling: SLOTS default 2; remote-run.sh sets CARGO_BUILD_JOBS 90 when it holds the only taken slot and 45
when both are held, BR_MEASURE=1 takes the `measure` file exclusively and excludes builds (builds hold it shared),
lock files open in append mode (the old `exec {fd}>` truncated a busy slot's holder line on every probe), env
IGNEUM_BUILD_SLOTS_DIR and IGNEUM_BUILD_LOG_DIR win over the profile, `--self-test-slots` with five cases (the old
script fails it with JOBS=none); build-remote.sh and cross-remote.sh pass -j only when --jobs is given.
- infra/build-server/prover/cpu-trial.sh: the SP1 CPU prover on one fixture shard under the measure hold with a VmHWM
poller; 6 Oct 2026 run: core 34.2 s, compressed 85.9 s, peak RSS 28.2 GB on 96 threads, so no standing CPU prover.
- docs/plans/ci-self-hosted.md: the proposed runs-on change for ci.yml behind the repository variable IGNEUM_CI_RUNNER
(GitHub-hosted is the fallback), and why windows.yml cannot move to a Linux box. Workflows untouched.
- docs/plans/build-server.md section 7.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
provision.sh step_cuda: NVIDIA's ubuntu2404 apt repository, cuda-nvrtc-dev-12-8, cuda-cudart-dev-12-8, cuda-driver-dev-12-8
(the libcuda stub) and opencl-c-headers; no nvcc (no build file calls it), no driver. tools/workers-remote.sh builds
igneum-worker-cuda and igneum-worker-opencl on the box from proto-cuda and proto-opencl with clang++ (static libstdc++),
prints sha256 and the glibc ceiling (2.38: fine for Ubuntu 24.04 hosts, not for 22.04 containers or HiveOS, where the Mac's
zig build stays). Proof: igneum-worker-cuda 1,613,992 B sha256 6db8a9ad..., igneum-worker-opencl 124,904 B sha256 0dea75bb...
remote-run.sh evaluates the command string in a subshell (a leaked set -e killed the runner after a successful build).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The public /ledger page (site/ledger.html) carried ~/.config/igneum paths five times, "pid 33114", --rpclisten=0.0.0.0:26610,
"PC 2" fifteen times, "the Mac" nineteen times and 202 repository file paths, because tools/ledger-page.mjs scrubbed with its
own short list and never ran the forbidden-strings hard stop (found by the site audit of 6 October 2026, docs/plans/site-ui-3-audit.md).
Now: the generator's scrub replaces every config or home-directory path with "a config file" or "a home-directory file", a pid with
"the process", a listen flag or 0.0.0.0 address with its plain words, "PC 1" and "PC 2" with "the Windows machine", "the Mac" with
"the Apple M5 Max" (the bench log's rule), and every repository file path with "a repository file"; the page's own source note
names no path. After rendering, the page is grepped with tools/ci/forbidden-strings.txt plus the leak classes and the render
exits 1 on a hit. The pattern list gains ~/.config, pid N, 0.0.0.0:port, PC 1 and PC 2 and --rpclisten=, and the identity grep
now covers site/ledger.html (html added to its file types). Regenerated: 167 entries, 0 leaks, identity grep 0 hits over 231
files, link check 0 broken.
Consequence per reader: the ledger keeps every criticism, status and answer; what a reader loses is the exact repository path
of a fix, which meant nothing outside the private repository. Nothing else on the site changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's ruling (6 October 2026, 20:1x UK): the box installs with the NUMBERS key alone so the 21:00 UK pulse comes from
it. install.sh accepts at least one key and names the missing ones; every tick's log line ends with "missing <keys>"; the
watcher without an incidents webhook logs its open or resolve and still advances its state, so the key landing later
does not flood the channel. Test added (31 passing).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tools/community/discord-hooks.mjs (Node 22, standard library): one ember embed per post to #numbers, #announcements and
#incidents. Network pulse every 6 h (03/09/15/21 UK) from /api/stats, /api/live, /api/supply with the 6 h window from its
own snapshot and a Devnet 2 line that uses the fleet file's numbers and gate word only; a 24 h digest and the reddit kit's
weekly template at 09:00 UK; a release subcommand for the shipper (three downloads with sha256, node commit, digest, up to
five plain "what changed" lines read from the release plan); incident open and resolve by hand; a watcher that opens one
incident per condition (finality paused 5 min, median proof lag 15 min, observer silent 3 min) and resolves after 2 clear
minutes, and never opens on a condition already firing when it first looks (the pulse says "finality paused since" instead).
Guards before every post: the founder's name and logins, hosts, machine ids, paths, IP addresses, standalone 32-hex
tokens, dl.igneum.network outside /public/, webhook URLs, mentions, the tools/ci/forbidden-strings.txt patterns; Discord's
limits refused rather than cut; idempotency keys per slot in a state file; exponential backoff on 429; dry run by default
with a Discord-like preview in tools/community/out/preview.html. 30 tests on fixtures cut from the live API.
infra/build-server/discord-hooks: a tick every minute on igneum-build-1 (the Mac sleeps). install.sh copies the webhook file
to /srv/discord-hooks/env (mode 600, over ssh stdin) and refuses without IGNEUM_SECRET_ON_BOX_OK=1; the recorded exception
to rule R6 is in docs/plans/build-server.md (main's ruling, 6 October 2026).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
the project lead's decision of 6 October 2026: the Mac runs nothing the network depends on. docs/plans/hands-on-build-1.md plans node 1
and the observer (its node and tools/observer) as systemd units on the box, one hand at a time, with ports, DNS names,
exec recovery, rollback and the decisions for main; infra/build-server/hands/install-hands.sh writes users, dirs, run
scripts and units (inert, run on the box); move-hand.sh (dry run by default) builds 0.3.15 on the box, copies the override
and snapshot, rsyncs each data dir hot then stopped, starts the unit and prints the hand's first executing line, copies the
observer env by scp (mode 600), stops the Mac's observer first, unloads the launchd agents last. CLAUDE.md's running-agents
rule carries the decision. lib.sh: a per-worktree lock on the box across sync, build and fetch (the shipper's collision at
18:48:56Z). remote-run.sh: the clean-tree test excuses stamps and target dirs at any depth with --untracked-files=all (a
nested stamp failed every checkout of /srv/builds/igneum for twenty minutes) and removes a stale .git/index.lock; the
self-test covers both. Plan rows for the three and the cuda prover pair built for the shipper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a member of the fork's workspace that inherits
from the fork's root manifest; syncing that one directory left cargo without a workspace root on the box. lib.sh now groups
path dependencies by git top level: a repository under vendor/ is pushed to its mirror (a fork worktree to
/srv/igneum-node.git, a repository of its own to /srv/<name>.git, created on first use), checked out whole at
/srv/builds/<worktree>/vendor/<name> and overlaid whole; run-from-mac.sh wires every vendor repository the Cargo.toml files
reach. libprotobuf-dev added (sp1-prover-types imports google/protobuf/empty.proto). build-remote.sh no longer fails on a
default artefact the caller's own -p selection did not build. Proof on the box: igneum-prove-host 71,943,192 B, sha256
e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, 1 min 05 s warm. Plan: gotcha rows for the case, the
protoc miss and one lost ssh session (collector cleared by test).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On 6 October 2026 the pre-push hook's site build fetched the live downloads index (0.3.14 since 17:47Z) and rewrote site/downloads.json and the stamped pages in five worktrees that had nothing to do with the release, as uncommitted edits to tracked files. A plain build and the hook now read live and warn when the snapshot lags; the ship step refreshes it with the flag and commits it. tools/ci/no-foreign-tree-writes.sh fails a script that builds a path from a worktree name, a glob over Projects or a worktree-list loop that writes; the eight absolute defaults into the shared checkout are listed as warnings until they move to env-only defaults. The journey, bench and index rebuilt from the merged tree.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The node pin moves to 4c6b129d (release-0.3.14-node), which is what the live payload inputs carry, so windows-ci's payload-inputs step is green again (red since e9eca23 on master's 0.3.13 pin against the 0.3.14 inputs).
Conflicts: infra/fast-time/override-60x.json keeps master's side (the fresh-rule field was already there at u64::MAX; the release line would be a duplicate key); tools/ci/playbook-quit-check.sh keeps master's rule 2 and pre-rule list with the release side's Ember allow entry and the 0.3.15 expiry check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tools/workers/collect.mjs runs every 30 s on the box (infra/build-server/workers/igneum-workers.{service,timer},
install.sh) and writes /srv/workers/workers.json from the machine itself: /proc/stat deltas per core, meminfo, df on
/srv, net bytes, hwmon temperatures, the flock state of /srv/builds/_locks, cargo processes with worktree, target and
start time, flock waiters, /srv/builds/_log/builds.jsonl (the format agreed with the build-server agent), sccache
--show-stats, headline.json. tools/workers/push.mjs (launchd every 60 s) adds the Mac's with-lock slots and waiters
and the two PCs' relay job states from the intake, drops them on the box so its file is whole, merges the box's file
and writes the fleet folder's workers.json; it deploys only when the live copy is over 6 min old, otherwise the
fleet orchestrator's 5-minute deploy carries it. The page (tools/workers/page/workers.html, shape.js) tries
https://build.igneum.network/workers.json first and falls back to the folder copy; core strip, arcs, now building,
queue, recently done with closing lines, headline timings, analytics; UK time with UTC tooltips; phone width.
node --test tools/workers/test: 13 tests over /proc/stat, lock dir, JSONL and sccache fixtures and the page shaping.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's decision of 6 October 2026. CLAUDE.md 'Running agents on this Mac': every Linux and Windows cargo build and every
Linux test suite goes to igneum-build-1 through tools/build-remote.sh and tools/cross-remote.sh; the PCs keep GPU and
Windows-runtime jobs; the Mac keeps macOS binaries, the DMG and Metal tests under the lock. -Wl,--no-insert-timestamp in
cross-remote.sh, proto-cuda/windows-node/cross-build.sh and jobbuild.rs (two box builds byte-identical, verified).
The empty-commit class: tools/ci/commit-string-check.sh (self-test in ci.yml, shown firing on a Mac worktree build and
passing on a box build) runs on every igneumd from the three build scripts; push-build-inputs.sh adds node.commit_full,
the PC job writes a minimal node/.git from it at extract and cleans kaspa-build-info on a new commit (4 jobbuild tests
pass, run on the box); cross-build.sh refuses a worktree and cleans on a new commit. Plan: second worktree's clean build
1 min 18 s with sccache 604 hits of 993.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in
/srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot,
wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h
slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and
build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash
(no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that
imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental
7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build-remote.sh runs a cargo command on igneum-build-1 from any crate directory of any worktree: HEAD through the bare
mirror (a real .git for kaspa-build-info), uncommitted changes by rsync --checksum with the written files re-stamped, a
remote slot (/srv/builds/_locks, never the Mac's), sccache, -j 90, artefacts back into target-remote/ with size and sha256.
cross-remote.sh is the Windows cross-build with the PC job's Ubuntu mingw-posix recipe plus the Mac's static flags, DLL
list and sha256 per exe, --compare against the Mac's exes. run-from-mac.sh ships provision.sh, writes
~/.config/igneum/build-server, adds the build remotes and pushes every branch of both repos. shellcheck and the CI checks clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On 47ede3f, by hand, the app half of the exec fix (fork exec-sync-0313 1f59c5d0 is the node half):
- app/igneum-app/src/prover.rs: igneum_exportSegments [n-1, n] for a shard and [first-1, last] for a segment,
never from 0 (on a restarted node the records below the restart carry zero roots and the exporter refused every
cut, the fleet 16:02Z); exec_boundary() reads igneum_getExecStatus.restartNumber (or the startedFrom text on a
0.3.13 node) and the prover claims no shard and no segment below it
- proving/igneum-prove/export: seeds the port from the export's preState (the node's account dump after the first
segment), checks its root against the node's there and replays from the next segment; without a dump the
restart-aware replay (execRestart) and the genesis replay stay
- tools/exec-sync/net.mjs (15 checks: the persisted state, the file, the wrong pin, another chain, the unreadable
flag file, the account-dump cut) and tools/exec-sync/reorg.mjs (18 checks: a 300-block reorg from the ring and
from the persisted generation)
- infra/fast-time/override-60x.json: the duplicate proving_v1 block from the 0.3.12 merge removed (the
consensus-core test fast_time_60x_file_is_the_devnet_at_60x failed on it)
The three UI files are untouched (byte-equal to 47ede3f); the igneum-prove-r0 tree is not in this cut.
Green on this tip (6 October 2026): cargo test in app/igneum-app 28 + 8; node --test app/igneum-app/ui 35; the 13
CI checks of ci.yml that run on this Mac; tools/exec-sync/net.mjs 15/15 in 97.2 s against this exporter and the
fork's igneumd (IGNEUM_EXEC_BIN, IGNEUM_EXPORTER).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
