pgrep -x git over the whole machine kept the lock whenever any git ran (the pre-push hook's own git push, another agent's build) and the checkout died with "index.lock: File exists". The fact is the lock's age. Class Q in docs/analysis/ci-failures-2026-10-06.md with the GIT_DIR leak of the previous commit.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
168 non-green runs since the first workflow run were classified (docs/analysis/ci-failures-2026-10-06.md): 102 were tree checks that finish in under 25 s on the pushing machine, 40 were GitHub-side refusals nobody saw.
tools/ci/pre-push.sh is the one list of fast checks; ci.yml's site job calls it with --ci and the pre-push hook with --hook (full gate for master and release-*, structural checks for other refs; never writes into the worktree). tools/ci/export-exclude.txt lists research documents outside the public export list, pruned by identity-check.sh and by the mirror's sync.sh (self-test: an excluded path may quote the patterns, an exported one may not); polish.md and this record are its first entries, which makes master green. tools/ci/windows-paths-check.sh (colon, trailing dot or space, reserved names, over 240 characters) runs as the pre-commit hook on staged paths and in the gate. tools/ci/red-watch.mjs plus the red job on the box's runner record one line per failed master or release-* run to /srv/ci-red/red.jsonl; igneum-ci-red.timer posts each once to the updates channel. pow and sims read IGNEUM_CI_RUNNER for the box. no-foreign-tree-writes.sh no longer exits silently on its warning pipeline under pipefail.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The build-server agent's two findings on rebuild-on-box.sh (6 Oct 2026, 20:1xZ): tools/ci/copied-sources-check.sh named it
(a tar on a code line plus cargo build with no touch), and `RUSTC_WRAPPER=` did not switch sccache off, so some passes of the
first runs took hits. Both 0.3.14 and 0.3.15 are re-run with this version before their evidence files are trusted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's rule of 6 October 2026 from the 0.3.14 repro (docs/evidence/reproduced/0.3.14.md): prost's protowire.rs embeds OUT_DIR,
libmimalloc-sys embeds __DATE__/__TIME__, sccache hid both. lib.sh bs_repro_env exports SOURCE_DATE_EPOCH=<author time> TZ=UTC in
front of every remote command (build-remote.sh, cross-remote.sh, workers-remote.sh); remote-run.sh exports BR_SDE too and logs it
as source_date_epoch; proto-cuda/windows-node/cross-build.sh exports the same; the PC job carries node.commit_time in the manifest
(push-build-inputs.sh) and exports it before every cargo build of a stage (jobbuild.rs, unit test asserts it; 4 of 4 pass on the
box). Target dirs stay one fixed path per target. tools/build-remote.sh --self-test-repro [--full] from a fork worktree, run on
the box: igneum-miner twice a minute apart without sccache (RUSTC_WRAPPER=/usr/bin/env, an empty value is unset to cargo) MATCH
91e130f5..., a per-run target path differs (OUT_DIR shown); --full: kaspad with libmimalloc-sys recompiled a minute later MATCH
70219bc2..., without the epoch differs (__DATE__ shown).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- rebuild-on-box.sh: the DIFFER reason comes from the binaries (glibc need of both, the build path each embeds, the
mimalloc clock string, the PE timestamp, the commit string), never from an assumed story; the three string helpers run
their producer under `|| true` so a consumer that stops early (grep -m1, awk exit) no longer trips pipefail into the
fallback and a second line in a table cell.
- docs/evidence/reproduced/0.3.15.md, and the 0.3.14 file re-reported with the same column.
- docs/plans/build-server.md 7.3: the 0.3.15 row and what the two files mean for shipping from the box.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- infra/build-server/repro/rebuild-on-box.sh: one target path per target with each pass's artefacts copied aside (prost's
protowire.rs embeds its OUT_DIR path, so a pass in a dir of another name differs), SOURCE_DATE_EPOCH from the node commit
and TZ=UTC (libmimalloc-sys compiles mimalloc's C with __DATE__ and __TIME__), --reuse for a re-report, reason text
for a shipped file that is not on hand (innoextract 1.9 cannot open the Inno Setup 6 installer; the 0.3.14 HiveOS
tarball left dl/public when 0.3.15 published).
- tools/repro/rebuild-release.sh: the plan's hashes always travel (the miners' 8-hex prefixes too), bash 3.2 empty-array
fix, the box half's exit code is the script's.
- docs/evidence/reproduced/0.3.14.md: igneumd 03f35e05..., igneum-miner 900c1f0b..., igneumd.exe 166e604e...,
igneum-miner.exe fefd266c... identical across two clean passes; DIFFER against the shipped 934f393c... (zig, glibc 2.36)
and 44fa74c0... (Homebrew mingw before the timestamp fix, from a worktree).
- docs/plans/build-server.md 7.2 (night battery: timer, dry run 3 min 54 s, 10 pass, the fork's 22 cargo-audit
advisories as the one FAIL and what follows) and 7.3 (repro: results, the two classes, SOURCE_DATE_EPOCH proposed for
every build script).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- infra/build-server/night/night-battery.sh: checkout of master and the newest release-*-node fork branch under
/srv/builds/_night, cargo test --release --no-fail-fast per crate (repo and fork), igneum-pow fuzz at 10x, the three
simulators in full, the fast-time harnesses (finality-attacks, harness s3 s4, exec-sync reorg) on binaries built into
target-integration, clippy per crate dir, cargo audit per Cargo.lock; docs/benchmarks/night/<date>.md with a pass/fail
table and "new since last night"; committed as igneum-josh on night-battery and pushed to the mirror, never master.
NIGHT_SUBSET=1 is the dry-run subset. The unit runs it through remote-run.sh so the slot spans the invocation.
- igneum-night-battery.{service,timer}: 02:00 Europe/London, User build, Nice 19, idle IO, 8 h limit, not Persistent.
- provision.sh: step_cargo_tools (cargo-audit), step_night (files from the mirror at NIGHT_REF, timer enabled), innoextract.
- tools/repro/rebuild-release.sh + infra/build-server/repro/rebuild-on-box.sh: pins from docs/plans/release-<v>.md
("(node <sha>, app <sha>)"), shipped hashes from the public downloads (the HiveOS tarball, the installer via
innoextract) or --shipped, a clean clone of repo and fork on the box, two passes per target without sccache under build
slots, MATCH or DIFFER per artefact against the shipped bytes and against the other pass, with the reason read off the
binaries (glibc version needed, PE timestamp, commit string); evidence into docs/evidence/reproduced/<version>.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- provision.sh step_runner: actions/runner 2.338.0 (sha256 checked) at /opt/actions-runner under a dedicated user `runner`
(no sudo, not in build's group), rustup 1.99.0 pinned with both targets, sccache against /srv/sccache in READ_ONLY mode
on its own server port, Node 22 and mingw from the system, GitHub's svc.sh unit with a Nice 10 drop-in; registered on
igneum-network/igneum as igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) through
infra/build-server/runner/register.sh (gh as igneum-josh, the token on ssh stdin, never logged). Idempotent after
the env files moved behind svc.sh install (its env.sh rewrites them). libicu74 and python3-numpy added to APT.
- main's slots ruling: SLOTS default 2; remote-run.sh sets CARGO_BUILD_JOBS 90 when it holds the only taken slot and 45
when both are held, BR_MEASURE=1 takes the `measure` file exclusively and excludes builds (builds hold it shared),
lock files open in append mode (the old `exec {fd}>` truncated a busy slot's holder line on every probe), env
IGNEUM_BUILD_SLOTS_DIR and IGNEUM_BUILD_LOG_DIR win over the profile, `--self-test-slots` with five cases (the old
script fails it with JOBS=none); build-remote.sh and cross-remote.sh pass -j only when --jobs is given.
- infra/build-server/prover/cpu-trial.sh: the SP1 CPU prover on one fixture shard under the measure hold with a VmHWM
poller; 6 Oct 2026 run: core 34.2 s, compressed 85.9 s, peak RSS 28.2 GB on 96 threads, so no standing CPU prover.
- docs/plans/ci-self-hosted.md: the proposed runs-on change for ci.yml behind the repository variable IGNEUM_CI_RUNNER
(GitHub-hosted is the fallback), and why windows.yml cannot move to a Linux box. Workflows untouched.
- docs/plans/build-server.md section 7.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
provision.sh step_cuda: NVIDIA's ubuntu2404 apt repository, cuda-nvrtc-dev-12-8, cuda-cudart-dev-12-8, cuda-driver-dev-12-8
(the libcuda stub) and opencl-c-headers; no nvcc (no build file calls it), no driver. tools/workers-remote.sh builds
igneum-worker-cuda and igneum-worker-opencl on the box from proto-cuda and proto-opencl with clang++ (static libstdc++),
prints sha256 and the glibc ceiling (2.38: fine for Ubuntu 24.04 hosts, not for 22.04 containers or HiveOS, where the Mac's
zig build stays). Proof: igneum-worker-cuda 1,613,992 B sha256 6db8a9ad..., igneum-worker-opencl 124,904 B sha256 0dea75bb...
remote-run.sh evaluates the command string in a subshell (a leaked set -e killed the runner after a successful build).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's ruling (6 October 2026, 20:1x UK): the box installs with the NUMBERS key alone so the 21:00 UK pulse comes from
it. install.sh accepts at least one key and names the missing ones; every tick's log line ends with "missing <keys>"; the
watcher without an incidents webhook logs its open or resolve and still advances its state, so the key landing later
does not flood the channel. Test added (31 passing).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tools/community/discord-hooks.mjs (Node 22, standard library): one ember embed per post to #numbers, #announcements and
#incidents. Network pulse every 6 h (03/09/15/21 UK) from /api/stats, /api/live, /api/supply with the 6 h window from its
own snapshot and a Devnet 2 line that uses the fleet file's numbers and gate word only; a 24 h digest and the reddit kit's
weekly template at 09:00 UK; a release subcommand for the shipper (three downloads with sha256, node commit, digest, up to
five plain "what changed" lines read from the release plan); incident open and resolve by hand; a watcher that opens one
incident per condition (finality paused 5 min, median proof lag 15 min, observer silent 3 min) and resolves after 2 clear
minutes, and never opens on a condition already firing when it first looks (the pulse says "finality paused since" instead).
Guards before every post: the founder's name and logins, hosts, machine ids, paths, IP addresses, standalone 32-hex
tokens, dl.igneum.network outside /public/, webhook URLs, mentions, the tools/ci/forbidden-strings.txt patterns; Discord's
limits refused rather than cut; idempotency keys per slot in a state file; exponential backoff on 429; dry run by default
with a Discord-like preview in tools/community/out/preview.html. 30 tests on fixtures cut from the live API.
infra/build-server/discord-hooks: a tick every minute on igneum-build-1 (the Mac sleeps). install.sh copies the webhook file
to /srv/discord-hooks/env (mode 600, over ssh stdin) and refuses without IGNEUM_SECRET_ON_BOX_OK=1; the recorded exception
to rule R6 is in docs/plans/build-server.md (main's ruling, 6 October 2026).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Josh's decision of 6 October 2026: the Mac runs nothing the network depends on. docs/plans/hands-on-build-1.md plans node 1
and the observer (its node and tools/observer) as systemd units on the box, one hand at a time, with ports, DNS names,
exec recovery, rollback and the decisions for main; infra/build-server/hands/install-hands.sh writes users, dirs, run
scripts and units (inert, run on the box); move-hand.sh (dry run by default) builds 0.3.15 on the box, copies the override
and snapshot, rsyncs each data dir hot then stopped, starts the unit and prints the hand's first executing line, copies the
observer env by scp (mode 600), stops the Mac's observer first, unloads the launchd agents last. CLAUDE.md's running-agents
rule carries the decision. lib.sh: a per-worktree lock on the box across sync, build and fetch (the shipper's collision at
18:48:56Z). remote-run.sh: the clean-tree test excuses stamps and target dirs at any depth with --untracked-files=all (a
nested stamp failed every checkout of /srv/builds/igneum for twenty minutes) and removes a stale .git/index.lock; the
self-test covers both. Plan rows for the three and the cuda prover pair built for the shipper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The second build of every repo-kind crate in a subdirectory failed (6 October 2026, 18:51 UTC, the pool build: "tree not
clean after reset: ?? pool/.build-remote-sha-target"): checkout_tree keeps the stamps with `git clean -e` at any depth but
its status check matched them at the root only. The check now reads `git status --porcelain --untracked-files=all` (an
all-untracked directory is listed file by file, not as "?? sub/") and accepts target dirs, sccache and both stamps under
any path. The self-test carries the subdirectory case (stamp and target dir kept, overlay file removed) and the known-failed
case (a stray untracked file still fails the check).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a member of the fork's workspace that inherits
from the fork's root manifest; syncing that one directory left cargo without a workspace root on the box. lib.sh now groups
path dependencies by git top level: a repository under vendor/ is pushed to its mirror (a fork worktree to
/srv/igneum-node.git, a repository of its own to /srv/<name>.git, created on first use), checked out whole at
/srv/builds/<worktree>/vendor/<name> and overlaid whole; run-from-mac.sh wires every vendor repository the Cargo.toml files
reach. libprotobuf-dev added (sp1-prover-types imports google/protobuf/empty.proto). build-remote.sh no longer fails on a
default artefact the caller's own -p selection did not build. Proof on the box: igneum-prove-host 71,943,192 B, sha256
e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, 1 min 05 s warm. Plan: gotcha rows for the case, the
protoc miss and one lost ssh session (collector cleared by test).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The node pin moves to 4c6b129d (release-0.3.14-node), which is what the live payload inputs carry, so windows-ci's payload-inputs step is green again (red since 630b537 on master's 0.3.13 pin against the 0.3.14 inputs).
Conflicts: infra/fast-time/override-60x.json keeps master's side (the fresh-rule field was already there at u64::MAX; the release line would be a duplicate key); tools/ci/playbook-quit-check.sh keeps master's rule 2 and pre-rule list with the release side's Ember allow entry and the 0.3.15 expiry check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh gains a checkout mode used by lib.sh: git checkout -- . and git clean -fd (target dirs, the sha stamps and
ignored files kept), fetch, branch at the commit, then a clean-tree check; the overlay follows. Before this, the rsync of
uncommitted files stayed in the box's tree and the next commit's git checkout -B refused with 'local changes would be
overwritten'. remote-run.sh --self-test reproduces the dirty tree (edited tracked file, untracked file, target dir, sha
stamp), shows the plain checkout refusing and the mode landing clean on the new commit; it runs in ci.yml and passed on the
Mac and the box; a live dirty-then-clean pair on the fork worktree passed too. Plan: section 5, gotchas of the first day.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tools/workers/collect.mjs runs every 30 s on the box (infra/build-server/workers/igneum-workers.{service,timer},
install.sh) and writes /srv/workers/workers.json from the machine itself: /proc/stat deltas per core, meminfo, df on
/srv, net bytes, hwmon temperatures, the flock state of /srv/builds/_locks, cargo processes with worktree, target and
start time, flock waiters, /srv/builds/_log/builds.jsonl (the format agreed with the build-server agent), sccache
--show-stats, headline.json. tools/workers/push.mjs (launchd every 60 s) adds the Mac's with-lock slots and waiters
and the two PCs' relay job states from the intake, drops them on the box so its file is whole, merges the box's file
and writes the fleet folder's workers.json; it deploys only when the live copy is over 6 min old, otherwise the
fleet orchestrator's 5-minute deploy carries it. The page (tools/workers/page/workers.html, shape.js) tries
https://build.igneum.network/workers.json first and falls back to the folder copy; core strip, arcs, now building,
queue, recently done with closing lines, headline timings, analytics; UK time with UTC tooltips; phone width.
node --test tools/workers/test: 13 tests over /proc/stat, lock dir, JSONL and sccache fixtures and the page shaping.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh writes /srv/builds/_locks/wait-<pid> in the slot-file line format while it waits and removes it when the
slot is taken or the wait is given up (shown: present during a held slot, gone after). provision.sh installs Caddy with a
Caddyfile that serves only /srv/workers/workers.json and headline.json (every other path 404, CORS for dl.igneum.network,
no-store, Let's Encrypt only) and opens 80 and 443. Both asked for by the worker-dashboard agent, approved by main.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in
/srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot,
wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h
slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and
build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash
(no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that
imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental
7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build-remote.sh runs a cargo command on igneum-build-1 from any crate directory of any worktree: HEAD through the bare
mirror (a real .git for kaspa-build-info), uncommitted changes by rsync --checksum with the written files re-stamped, a
remote slot (/srv/builds/_locks, never the Mac's), sccache, -j 90, artefacts back into target-remote/ with size and sha256.
cross-remote.sh is the Windows cross-build with the PC job's Ubuntu mingw-posix recipe plus the Mac's static flags, DLL
list and sha256 per exe, --compare against the Mac's exes. run-from-mac.sh ships provision.sh, writes
~/.config/igneum/build-server, adds the build remotes and pushes every branch of both repos. shellcheck and the CI checks clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Idempotent provisioning of the Hetzner AX162 build box: install mode (rescue system, Ubuntu 24.04, software RAID 1 over
the two NVMe drives, no swap, rescue keys taken over, run 6 October 2026 17:16 to 17:20 UTC) and provision mode (user build,
compilers, mingw-w64 posix, rustup 1.99.0 with the Windows target, sccache 100 GB, Node 22, bare mirrors, /srv/builds, sshd
key-only, ufw 22 + seed p2p ports). shellcheck clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On 220dc03, by hand, the app half of the exec fix (fork exec-sync-0313 1f59c5d0 is the node half):
- app/igneum-app/src/prover.rs: igneum_exportSegments [n-1, n] for a shard and [first-1, last] for a segment,
never from 0 (on a restarted node the records below the restart carry zero roots and the exporter refused every
cut, the fleet 16:02Z); exec_boundary() reads igneum_getExecStatus.restartNumber (or the startedFrom text on a
0.3.13 node) and the prover claims no shard and no segment below it
- proving/igneum-prove/export: seeds the port from the export's preState (the node's account dump after the first
segment), checks its root against the node's there and replays from the next segment; without a dump the
restart-aware replay (execRestart) and the genesis replay stay
- tools/exec-sync/net.mjs (15 checks: the persisted state, the file, the wrong pin, another chain, the unreadable
flag file, the account-dump cut) and tools/exec-sync/reorg.mjs (18 checks: a 300-block reorg from the ring and
from the persisted generation)
- infra/fast-time/override-60x.json: the duplicate proving_v1 block from the 0.3.12 merge removed (the
consensus-core test fast_time_60x_file_is_the_devnet_at_60x failed on it)
The three UI files are untouched (byte-equal to 220dc03); the igneum-prove-r0 tree is not in this cut.
Green on this tip (6 October 2026): cargo test in app/igneum-app 28 + 8; node --test app/igneum-app/ui 35; the 13
CI checks of ci.yml that run on this Mac; tools/exec-sync/net.mjs 15/15 in 97.2 s against this exporter and the
fork's igneumd (IGNEUM_EXEC_BIN, IGNEUM_EXPORTER).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Swift DatasetContext is the version 2 item construction; a v3 program over it would hash another dataset than
the node's (every found refused by the CPU re-check). servePackDataset compiles the pack's memhard.metal and runs
igneum_cache_fill and igneum_build as packbench does, releases the cache, and the v3 job path takes the pack
program and the pack day of its class and era only (need + error otherwise). The gate script's --metal mode drives
igneum-miner --worker <igneum-bench> --prepare-packs <dir> --exit-on-seed-change (the app's shape) on node 0 and
reports the PREPARE and prepared lines, need / mismatch / refusal lines, accepted blocks after the switch, the
CPU re-check counter and the swap line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
