# Build server: igneum-build-1 (plan, benchmarks, rules)

6 October 2026. the project lead ordered a Hetzner dedicated server at 18:2x UK: AX162-1-LTD, EPYC 9454P (48 cores, 96 threads),
128 GB, 2x 3.84 TB NVMe, Falkenstein (FSN1-DC24, Hetzner #3088308), 188.40.146.49, IPv6 2a01:4f8:2240:205a::/64.
Purpose: this Mac is the compile queue for ten agents (8-minute rebuilds under contention) and the Windows cross-builds;
the box takes over Rust builds, cross-builds and a shared compile cache, and later a CI runner and the Devnet 2 seed.

## 1. What is in place (all on branch `build-server`)

| Piece | File | State 6 Oct 2026 |
|---|---|---|
| Install + provision | `infra/build-server/provision.sh` | ran: installimage 17:16 to 17:20 UTC (Ubuntu 24.04, RAID 1, no swap), provision 17:24 to 17:27 UTC, second run 2 s with zero changes (idempotent) |
| Mac side | `infra/build-server/run-from-mac.sh <ip>` | ran: `~/.config/igneum/build-server` = `build@188.40.146.49`, `build` remotes added, 124 igneum branches and 48 fork branches pushed to the bare mirrors |
| Shared library | `infra/build-server/lib.sh` | host line, ssh options, mirror push, remote checkout, overlay, remote slot runner |
| Remote runner | `infra/build-server/remote-run.sh` | runs on the box: slot, sccache, RESULT line, one JSON line per run in `/srv/builds/_log/builds.jsonl` (the worker dashboard's feed, fields as main asked on 6 Oct) |
| Remote cargo | `tools/build-remote.sh` | any crate dir, any worktree: `tools/build-remote.sh [-- cargo args]`; `IGNEUM_AGENT=<name>` tags the slot label and the log |
| Remote Windows cross | `tools/cross-remote.sh` | fork worktree or app/igneum-app: `tools/cross-remote.sh [--compare <dir of the Mac's exes>]` |

ssh line: `ssh -i ~/.ssh/igneum_ed25519 build@188.40.146.49` (root works with the same key; passwords are off).

Box facts (provision summary): rustc 1.99.0 (the Mac's version; NO rust-toolchain file exists in the repo or the fork, the
pin is `RUST_TOOLCHAIN` in provision.sh), targets x86_64-unknown-linux-gnu and x86_64-pc-windows-gnu, sccache 0.18.0 with a
100 GB disk cache at /srv/sccache, mingw-w64 GCC 13 posix threads (the PC job's recipe), clang and lld 18.1.3, Node v22.23.3,
git 2.43.0, tmux 3.4, ufw 22 + 26611 + 26811 (the devnet and testnet seed p2p ports; RPC stays on loopback as on every seed),
md0 /boot and md1 / as RAID 1, 3.3 TB free, swap none, 1 build slot in `/srv/builds/_locks/slots`.

## 2. How a build travels

1. `bs_context` (lib.sh) reads the crate: a fork worktree under `vendor/` (kind node, mirror `/srv/igneum-node.git`) or a crate
   of the igneum repo (kind repo, mirror `/srv/igneum.git`). `cargo metadata` lists every path dependency.
2. HEAD is pushed to the mirror; the box clones or fetches it at `/srv/builds/<worktree>/<same relative path>` and checks out
   that commit. A real `.git` is needed: the fork's `kaspa-build-info` runs `git rev-parse HEAD` at build time and every release
   plan checks the commit in the binary's strings.
3. Uncommitted changes go by `rsync --checksum` without `-t` (files that changed are written with the box's clock) and the
   written files are re-stamped with `touch` (the copied-sources rule, `tools/ci/copied-sources-check.sh` passes).
4. The cargo command runs in the crate dir under one of the box's slot files (`flock` on `/srv/builds/_locks/build-<k>`,
   never the Mac's `~/.config/igneum/build-slots`), with sccache and `-j 90`, logging wall time, compile count and cache hits.
5. Artefacts come back into `<crate>/target-remote/...` with size and sha256. NEVER into `target/`: the box's native
   binaries are x86_64 Linux (glibc 2.39) and do not run on the Mac.

`/srv/builds/<worktree>` mirrors the Mac's igneum worktree ROOT because the fork's path dependency is
`../../../../igneum-pow` (vendor/igneum-node/consensus/pow/Cargo.toml).

## 3. Benchmarks

The Mac numbers are from the logs (docs/bench-log.md, docs/plans/release-0.3.10.md, release-0.3.11.md); no fresh Mac run
was made, because a Mac build would take a slot from the agents and the logs already hold several readings per case.
Mac = Apple M5 Max (18 cores), builds at `nice -n 19` with 4 to 6 jobs under the build lock, usually loaded by other agents.
Box = igneum-build-1, `-j 90`, nothing else running.

| Case | Mac (logged) | Box | Box run |
|---|---|---|---|
| Clean node build (`cargo build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow`, every crate) | 12 min 36 s (0.3.10, new target dir, -j 4); 17 min 53 s at load 140 and 8 min 58 s second time (fud ledger, -j 4); rusty-kaspa kaspad alone 2 min 36 s on an idle Mac | **1 min 27 s** cargo wall (1 min 34 s end to end from the Mac: push and sync 1 s, build, fetch of both binaries) | cold: 518 crates downloaded inside that time, sccache 0 hits / 993 misses, 563 crates compiled; igneumd 48,220,896 B, igneum-miner 9,107,232 B, ELF x86-64 PIE; `igneumd --version` runs on the box; 17:30:58 to 17:32:32 UTC |
| Incremental rebuild (one file changed, same target dir) | 2 min 08 s (0.3.10, 17:49Z); 3 min 19 s (0.3.11); 5 min 06 s (txgossip); 15 min 18 s at load 110 to 134 (M31); 35 s to 4 min (execution layer); "8 min under contention" (main, 6 Oct) | **7 s** cargo wall (10 s end to end: sync 1 s, build 6.97 s, fetch) | one line appended to kaspad/src/main.rs in the fork worktree, uncommitted, carried by the overlay (1 file written and re-stamped); 1 crate compiled, igneumd relinked; box idle (load 12 from the clean build a minute earlier); 17:33:11 to 17:33:21 UTC |
| Windows cross-build (`--target x86_64-pc-windows-gnu`, igneumd.exe + igneum-miner.exe) | 8 min 25 s clean (-j 6, 3 Oct); 4 min 49 s and 4 min 53 s with warm dependencies (4 Oct); 12 min 28 s (finality-fixes, 4 Oct) | **1 min 44 s** cargo wall (1 min 48 s end to end) | cold: 995 compiles, sccache 0 hits; igneumd.exe 49,434,624 B, igneum-miner.exe 10,201,600 B; mingw GCC 13 posix, libclang 18; igneumd.exe imports libstdc++-6.dll (this fork head predates the housekeeping commit that made the C++ runtime static), so cross-remote.sh now fetches the three GCC 13 runtime DLLs beside the exes as the PC job does; 17:33:58 to 17:35:46 UTC. Second run on the same target dir, no source change: 8 s |

Also logged for context: the Mac's Linux cross-build with zig (`infra/cross/build-linux.sh`) took 30 min 56 s cold and
3 min 20 s incremental; PC 1 built Linux + Windows node and app and ran both test suites in 7 min 38 s cold, 5 min 09 s warm
(CLAUDE.md, 5 Oct).

### What the numbers mean and what follows

| Number | Means | Done or proposed |
|---|---|---|
| Clean build 1 min 27 s against 12 to 18 min on the loaded Mac (8 to 12x) | a new worktree costs an agent a minute and a half, not a slot for a quarter of an hour; the first build of every one of the 123 worktree dirs on the box is this cold case, later ones are the 7 s case | R1 proposed; sccache was cold (0 hits of 1,982 requests) because every run so far was the first of its kind; the cache fills as agents build the same crate versions from different worktrees, so the second worktree's clean build will be mostly hits (measure when it happens, write the number here) |
| Incremental 7 s against 2 to 15 min on the Mac (the "8 min under contention") | an edit-build loop of seconds for Linux targets; end to end 10 s because sync is 1 s and the two binaries (57 MB) come back in 2 s | R1; the slot count stays 1 until two agents collide, then 2 with `-j 48` each (`SLOTS=2 run-from-mac.sh` and `--jobs 48`) |
| Windows cross 1 min 44 s against 4 min 49 s to 12 min 28 s on the Mac (3 to 7x), 8 s incremental | a Windows exe per commit is cheap enough to build on every push; the PC `build` job (7 min 38 s cold, 5 min 09 s warm for Linux + Windows + tests) stays the second source | R2 proposed |
| Mac arm64 binaries: not built here | agents who run nodes on the Mac (local devnets, the DMG) still take Mac slots; the box cannot remove that contention | R3; the real relief is to move test networks to the fleet or to a Devnet 2 seed on this box (its unit, ports and ufw are ready) |
| The commit hash is EMPTY in every Mac worktree build and was empty in the first box builds | `kaspa-build-info` (build-info/build.rs) needs `.git` to be a directory AND HEAD to be a symbolic ref to a loose branch file; a worktree's `.git` is a file and a detached HEAD is not a ref; and once it has found nothing it emits no `rerun-if-changed`, so cargo never runs it again in that target dir (release-0.3.11: `cargo clean -p kaspa-build-info`) | fixed on the box: the remote checkout is `git checkout -B <branch> <sha>` and build-remote.sh runs `cargo clean --release -p kaspa-build-info` whenever the commit differs from the last one built in that target dir (`.build-remote-sha-<target dir>`); verified 17:40 UTC: 2 string hits for 3bfe346f, binary +1,024 B. The release plans' "commit in its strings" checks were passing against builds from the fork's MAIN checkout (a real .git directory on a branch), not from worktrees. DONE (main's decision): the PC job writes a minimal `node/.git` (HEAD -> refs/heads/build holding the manifest's new `commit_full`, written by push-build-inputs.sh) at extract and cleans kaspa-build-info on a new commit (jobbuild.rs, 4 unit tests pass on the box); the Mac's cross-build.sh refuses a worktree and cleans on a new commit; the gate `tools/ci/commit-string-check.sh` runs on every igneumd the three scripts produce (self-test in ci.yml; shown firing on the Mac's worktree-built igneumd and passing on the box's). Only kaspad depends on kaspa-build-info, so igneum-miner is out of the gate's scope |
| igneumd.exe hash changed build to build with no source change (c424aae0 then bfbff015) while the Linux igneumd stayed byte-identical across three builds | the mingw linker wrote a timestamp into the PE header; the Linux ELF has none | DONE (main's decision): `-C link-arg=-Wl,--no-insert-timestamp` in cross-remote.sh, the Mac's cross-build.sh and the PC job (jobbuild.rs); verified 17:48 and 17:49 UTC: two builds, igneumd.exe c38b7570... and igneum-miner.exe c3a0fc2b... identical both times |
| Second worktree's clean build (vendor/igneum-node-v4 from the main checkout, cold target dir, warm sccache): 1 min 18 s, 604 hits of 993 compiles (61 percent), 389 misses | the first build of each of the 123 worktree dirs costs 1 min 18 s to 1 min 27 s, not the Mac's 12 to 18 min; sccache saves 9 s of the 87 because the clean build is dominated by the fork's own 74 crates and the C++ (rocksdb) objects, which differ per tree or do not cache; the cache is 1.0 GB after four builds, capped at 100 GB | nothing; the hit rate is in every RESULT line and every JSONL line |
| Disk 3.3 TB free, RAM 125 GB, load peaked at 24 during the Windows build with 96 threads | room for 2 slots and the Devnet 2 seed without contention | nothing now |

## 4. Rules (ADOPTED by main on 6 October 2026, written into CLAUDE.md "Running agents on this Mac"; R2 narrowed: the PCs keep only GPU and Windows-runtime jobs from now, not two releases)

| Rule | Text |
|---|---|
| R1 | Every agent's `cargo build`, `cargo test`, `cargo check` and `cargo clippy` for Linux goes through `tools/build-remote.sh` from the crate directory. The box takes one remote slot per build; nobody runs cargo over ssh by hand. |
| R2 | Windows exes come from `tools/cross-remote.sh` (fork worktree: igneumd.exe, igneum-miner.exe; app/igneum-app: igneum-app.exe and the two tools). The PC `build` job stays as the second source until two releases have shipped from the box. |
| R3 | The Mac keeps what only it can do: aarch64-apple-darwin binaries (the DMG, nodes that agents run locally), tests that need Metal (proto-metal, the Metal worker), and measurements. Those still use `tools/lock/with-lock.sh` and the Mac's build slots. |
| R4 | A worktree builds once on the box per commit plus overlay; the next build is incremental in `/srv/builds/<worktree>/.../target`. Nobody deletes another worktree's target dir on the box. |
| R5 | `RUST_TOOLCHAIN` in provision.sh is bumped in the same commit as the Mac's `rustup update`; build-remote.sh refuses a mismatch. Add a `rust-toolchain.toml` to the fork and the repo (none exists today) so both sides pin from one file. |
| R6 | The box is never a node host for the live devnet and never holds a secret (no `~/.config/igneum` there). The Devnet 2 seed on it runs under its own unit with `--devnet --devnet-suffix=<n>` on 26611 (ufw already open) when that work starts. |
| R6a | ONE recorded exception to R6 (main's ruling, 6 October 2026, 20:0x UK): the three Discord webhook URLs live at `/srv/discord-hooks/env` (mode 600, owner build), installed by `infra/build-server/discord-hooks/install.sh` with `IGNEUM_SECRET_ON_BOX_OK=1`, because the Mac sleeps and the bot's timer must not. The reasoning: a webhook URL signs no release, moves no funds and reaches none of the devnet's hands; whoever holds it can only post as the bot, and a leaked one is deleted in Discord in one click and the file replaced. No other secret joins it; `check` prints key names, never values. |
| R7 | CLAUDE.md line "nothing is built on a server" and "Windows builds go to the GitHub runner, Linux binaries come from infra/cross/build-linux.sh" are rewritten when R1 and R2 are adopted; until then the box is the measured option, not the rule. |

## 5. Gotchas met on the first day

| Case | What happened | Fix |
|---|---|---|
| Stale overlay blocks the next checkout (PC 1 worker, 6 Oct 2026) | build-remote.sh rsyncs uncommitted files over the box's checkout; on the next commit `git checkout -B` refused with "local changes would be overwritten" | remote-run.sh `checkout` mode: `git checkout -- .` and `git clean -fd` (target dirs, sha stamps and ignored files kept) before the branch checkout, then the overlay; `remote-run.sh --self-test` reproduces the dirty tree and shows the mode landing on the new commit clean |
| An all-identical overlay listed only directories | the first run's touch pipeline got an empty file list and failed | files only are counted and re-stamped (lib.sh bs_overlay_dir) |
| bash 3.2 on the Mac treats an empty array as unbound under `set -u` | run-from-mac.sh died on `PASS[*]` | a string instead of an array |
| `grep -q` plus `pipefail` turned a strings hit into a miss | the commit-string gate failed a stamped igneumd on its first use (SIGPIPE on `strings`) | `grep -c` |
| A path dependency inside a vendor repo (the shipper's proving build, 6 Oct 2026) | proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a MEMBER of the fork's workspace (it inherits `thiserror` from the fork's root manifest); the first design synced that one directory, so cargo found no workspace root on the box ("failed to load manifest for workspace member"), and the shipper cross-built the Linux prove-host on the Mac with cargo-zigbuild meanwhile | lib.sh groups path dependencies by git top level: one under vendor/ is a whole repository, pushed to its mirror (a fork worktree such as igneum-node-exec goes to /srv/igneum-node.git, which already held its branch; a repository of its own gets /srv/<name>.git, created on first use) and checked out whole at /srv/builds/<worktree>/vendor/<name>; run-from-mac.sh wires every vendor repo the Cargo.toml files reach (today only igneum-node-exec). The detector's first version tested "under BS_TOP" before "own repository" and missed it, since vendor/ lies under the igneum top level on disk. Then `libprotobuf-dev` was missing (sp1-prover-types's build script imports google/protobuf/empty.proto); added to provision.sh. Proof: `cd proving/igneum-prove && tools/build-remote.sh -- build --release -p igneum-prove-host`: igneum-prove-host 71,943,192 B, sha256 e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, ELF x86-64, 1 min 05 s warm (the cold run compiled 605 crates in 55 s before protoc stopped it). A Mac worktree has no vendor/ of its own, so a worktree that builds proving needs `git -C vendor/igneum-node worktree add <wt>/vendor/igneum-node-exec execution-layer` first, as the fork worktrees do |
| One remote build lost its ssh session after 75 s (18:30:50 UTC, the first full proving build) | the remote bash died with it (slot line left behind, no JSONL line); no OOM, no reboot, the retry a minute later passed | the dashboard collector's one-pass unit finished within a second of the drop, so it was tested: a 90 s remote session through the same ControlMaster path survived two collector passes triggered by hand; the collector only reads (/proc, lock files, `flock -n`, `sccache --show-stats`, `kill(pid, 0)`). One event, no cause in the journal, the retry passed. A build that must survive a dropped connection would need the remote command under setsid with the Mac reconnecting to wait; not done, open if it happens again |
| A stamp file at a nested path failed every checkout of /srv/builds/igneum (18:31 to 18:5x UTC, the shipper's 18:52 run) | a repo-kind crate keeps its `.build-remote-sha-<target>` and `target/` inside the crate dir; the checkout mode's clean-tree test only excused them at the tree root | the test is depth-agnostic and uses `--untracked-files=all` (a wholly untracked directory is otherwise collapsed to `?? dir/`); the self-test carries a nested stamp, a nested target dir and a stale `.git/index.lock`, which the mode now removes when no git runs there |
| Two runs on one worktree at once (the shipper, 18:48:56Z) | the second run's checkout replaced the first's sources mid-cargo; both died | lib.sh takes a per-worktree lock on the box (`/srv/builds/_locks/wt-<worktree>`, mkdir-atomic, holder line) across sync, build and fetch; a second run waits up to 2 h (a line every minute), a lock older than 3 h is taken over; released on EXIT. The build slot (`build-<k>`) is unchanged |
| The 0.3.15 prover pair for the PCs needs `--features igneum-prove-host/cuda` (the PCs run SP1_PROVER=cuda) | my first proving build named no feature | built on the box from master e1b5bc9: igneum-prove-host 73,161,528 B sha256 71bc2438856bb141f6cad3d18489f708568144fad5a06a002fbadefb9ce256f9, igneum-prove-export 3,609,360 B sha256 263bf4cef70af4a13a45b2e79b8dbab373282ea4c571f624d02ddb5791935361 (52 s warm, no CUDA needed at build time); handed to the shipper |
| Let's Encrypt saw NXDOMAIN for build.igneum.network | the deSEC record was minutes old; Ubuntu's Caddy then fell back to ZeroSSL and failed with HTTP 422 for ever | issuer pinned to Let's Encrypt; the retry got the certificate |

## 5a. The GPU workers (added 6 October 2026, 19:10 UTC, for the class v4 rehearsal)

| What | Fact |
|---|---|
| Headers on the box | provision.sh `step_cuda`: NVIDIA's ubuntu2404 apt repository, `cuda-nvrtc-dev-12-8`, `cuda-cudart-dev-12-8`, `cuda-driver-dev-12-8` (the libcuda stub) and Ubuntu's `opencl-c-headers`; no nvcc (no build file calls it: both workers compile from C/C++ sources with the headers and dlopen libcuda, libnvrtc and libOpenCL at run time), no driver (no GPU here). 12.8 is the minor proto-cuda/nvrtc/fetch-redist.sh pins |
| Command | `tools/workers-remote.sh [--out <dir>]` from any igneum worktree: proto-cuda and proto-opencl through the mirror and overlay, clang++ 18 with `-static-libstdc++ -static-libgcc`, both workers, sha256 and the glibc ceiling printed, artefacts in `<worktree>/infra/cross/out-workers-box/` |
| Proof (master worker.cpp at 3b2c840) | igneum-worker-cuda 1,613,992 B sha256 6db8a9ad295a9f7598a8b5500f1c271fbfdb9a8df90848b95ff340664007031e; igneum-worker-opencl 124,904 B sha256 0dea75bb8d2d54721ee44b55a3ba486241d3c6053e72a133f8f5e40d2355cbbd; 3 s on the box |
| Consequence: glibc ceiling 2.38 | runs on the fleet (Ubuntu 22.04 containers are glibc 2.35: NO, 2.38 > 2.35; Ubuntu 24.04 hosts yes). The Mac's zig build (`infra/cross/build-workers-linux.sh`, glibc 2.36) is the one for Debian 12 and HiveOS; the fleet agent must check its boxes' glibc before swapping the worker. Fix if needed: zig on the box (open row in section 6) or `clang -target x86_64-linux-gnu.2.35` via zig; both a day's work, not done |
| Runner fix found on the way | a command string carrying `set -e` leaked into remote-run.sh through `eval` and killed the runner before its RESULT line (reported as rc 101); the runner now evaluates the command in a subshell |

## 6. What the box does not do yet

| Gap | Why it matters | Next step |
|---|---|---|
| No zig / cargo-zigbuild | the devnet seed (Debian 12, glibc 2.36) takes the Mac's zig build; a native box build links glibc 2.39, which Debian 13 seeds accept and HiveOS (Ubuntu 18/20 base) does not | install zig 0.17 + cargo-zigbuild in provision.sh, add `--target x86_64-unknown-linux-gnu.2.36` mode to build-remote.sh |
| No macOS target | agents who run nodes on the Mac still build there | out of scope (needs the macOS SDK on Linux); the fleet or the box's own Devnet 2 seed takes the test-network runs instead |
| No CI runner | DONE 6 October 2026, 19:19Z (section 7): the runner `igneum-build-1` is online under user `runner`, never build; the workflow change is proposed in docs/plans/ci-self-hosted.md | main flips `IGNEUM_CI_RUNNER=box` after the shipper's cut |
| Byte identity with the Mac's exes | different C/C++ toolchain (Homebrew mingw vs Ubuntu GCC 13) and embedded source paths | not a goal; the box is identical with itself build to build, cross-remote.sh reports sha256 and the DLL list per exe |
| Robot API | `~/.config/igneum/hetzner-token` is the Cloud token (hcloud); the dedicated box lives in Robot, a separate credential | main sets the Robot server name in the UI; a webservice user goes to `~/.config/igneum/robot-credentials` when needed |

## 7. The box's second shift (6 October 2026, from 20:3x UK, the project lead: "what else can our building machine be working on?")

Four items, each its own commit on branch `box-work` with its section here. Times UTC.

### 7.1 The GitHub Actions runner (DONE 19:19Z)

| Fact | Value |
|---|---|
| Runner | `igneum-build-1`, actions/runner 2.338.0 (tarball sha256 af4b794c... checked against the release note), registered on igneum-network/igneum at 19:19:49Z, online, labels `self-hosted, Linux, X64, igneum-build-1` |
| User | `runner` (uid 1001, own group, no sudo, not in `build`'s group; /home/runner 750). Never `build`, never root. The runner's credential (`/opt/actions-runner/.credentials`, mode 600) is the only thing it holds; it signs nothing and reaches no hand |
| Service | `actions.runner.igneum-network-igneum.igneum-build-1.service` (GitHub's `svc.sh install runner`), drop-in `igneum.conf`: Nice 10, IO best-effort 7, Restart on-failure. Agents' builds (nice 0 through build-remote.sh) win the CPU over a CI job |
| Toolchains | rustup 1.99.0 pinned like the box (`RUST_TOOLCHAIN`), targets x86_64-unknown-linux-gnu and x86_64-pc-windows-gnu, clippy, rustfmt; mingw-w64 GCC 13 posix, Node 22, python3 + numpy from the system (numpy added to APT for the simulators) |
| sccache | `/usr/local/bin/sccache` (the build user's binary copied; /home/build is 750), config `/home/runner/.config/sccache/config` with `rw_mode = "READ_ONLY"` on /srv/sccache, own server port 4227. Shown: a job-shaped `cargo test --release` of igneum-pow as `runner` (99 tests pass, 42 s cold) made 6 compile requests, 0 hits, 6 cache WRITE ERRORS (the refusal, as wanted), and /srv/sccache stayed at 5,880,836 KB |
| Jobs | `.env`: RUSTC_WRAPPER, SCCACHE_CONF, SCCACHE_SERVER_PORT 4227, CARGO_INCREMENTAL 0, CARGO_BUILD_JOBS 48 (half the box); `.path`: the runner's cargo bin, /usr/local/bin, /usr/bin, /bin. A CI job takes NO build slot today (ci-self-hosted.md, open row) |
| Ephemeral | no. `--ephemeral` is for autoscaled fleets that register a fresh runner per job; one standing runner on a private repository keeps its registration and cleans `_work` per job (approximate: GitHub's docs host answered 404 to both fetches tonight, so this is the rule as remembered, labelled so) |
| Registration | `infra/build-server/runner/register.sh`: gh as igneum-labs (fails on any other active account, checks the login is igneum-labs), `POST repos/igneum-network/igneum/actions/runners/registration-token`, the token as the first stdin line to `provision.sh` on the box (never an argument, never a file, never logged; the output is filtered for it as a belt). `--status` lists the repository's runners and the unit |
| Idempotent | provision.sh runs 3 and 4 after the registration: `runner: ok`, no restart (`ActiveEnterTimestamp` unchanged). Run 2 had said `changed` because GitHub's `svc.sh install` runs `env.sh`, which rewrites `.env` and `.path`; the step now writes them AFTER the install |
| Workflows | NOT changed (the shipper owns them tonight). The proposed diff and the fallback (repository variable `IGNEUM_CI_RUNNER`; GitHub has no "else" in `runs-on`) are in `docs/plans/ci-self-hosted.md`. `windows.yml` cannot move to a Linux box (MSVC, WebView2, Inno Setup, PowerShell 5.1) |

Consequences: ci.yml's `pow` and `sims` jobs would run on a pinned 1.99.0 (GitHub's `ubuntu-latest` ships whatever stable it has), with 48 jobs; GitHub-hosted minutes on a private repository are the thing saved. A CI job on the box reads only what the checkout gives it; the mirrors and `/srv/builds` belong to `build` and are not readable by `runner` (git's safe.directory is set for the runner so a future job may clone a mirror read-only if main wants it).

### 7.0 The slots ruling (main, 20:3x UK; DONE 19:28Z, live on the box)

| Change | Where | Shown by |
|---|---|---|
| 2 build slots (`/srv/builds/_locks/slots` = 2) | provision.sh `SLOTS` default 2, applied 19:28:18Z | `dirs: changed (... slots=2)` |
| CARGO_BUILD_JOBS 90 when a build holds the only taken slot, 45 when it sees the other slot held (one second of settling after taking the slot, then a `flock -n` probe of the other file); a `-j` on the cargo line wins | remote-run.sh; tools/build-remote.sh and tools/cross-remote.sh pass `-j` only when `--jobs` is given | `remote-run.sh --self-test-slots` on the box: two concurrent fake builds get 45 each, a lone one 90 |
| A measurement (`BR_MEASURE=1`) takes the `measure` file exclusively; builds hold it shared for their whole run, so a measure waits for the running builds and blocks new ones, as with-lock.sh's `measure` on the Mac | remote-run.sh; `infra/build-server/prover/cpu-trial.sh` is its first user | the self-test: a measure blocks a build, a build blocks a measure; JSONL carries `"jobs"` and `"measure"` |
| Lock files opened in APPEND mode | remote-run.sh | the first version's `exec {fd}>build-k` truncated a BUSY slot's holder line each time another build probed it (the dashboard read empty lines for held slots); the self-test's case 5 keeps a holder line through a probe. The OLD script under the same cases: `JOBS=none JOBS=none`, FAIL (the known-failed run, 19:25Z) |
| An environment IGNEUM_BUILD_SLOTS_DIR or IGNEUM_BUILD_LOG_DIR wins over the profile | remote-run.sh | the first self-test run let the profile reset the scratch dir and took the box's REAL slot for 7 s (19:24Z, box idle) |

Open: a worktree whose remote-run.sh predates this keeps the old behaviour until it has master with it (the script is piped from each Mac worktree per build), so until every agent rebases, a build from an old worktree still asks `-j 90` beside a new one at 45. The build-server agent was told at 19:28Z.

### 7.2 The night battery (DONE 19:42Z installed, dry run 19:45 to 19:49Z)

`infra/build-server/night/night-battery.sh`, run by `igneum-night-battery.timer` at 02:00 Europe/London (the box's clock is
Europe/Berlin, so the unit names the zone: next run Wed 2026-10-07 03:00 CEST = 02:00 BST; not Persistent, a missed night is
not run by day) through `igneum-night-battery.service` (User build, Nice 19, idle IO, 8 h limit), whose ExecStart is
`remote-run.sh` with the battery as BR_CMD, so ONE build slot spans the whole invocation and one JSONL line records it.
Installed by provision.sh `step_night` from the mirror at `NIGHT_REF` (box-work tonight, master once merged); the battery
re-execs itself from master's checkout at run time. Every row: `cargo test --release --no-fail-fast` per crate (the repo's
five crates and the proving workspace's host, core and export; every fork workspace member, kaspad with `igneum-pow`),
igneum-pow's two fuzz tests at 2,000 programs (10x), the three simulators in full, the fast-time harnesses
(`tools/finality-attacks/run.mjs --fast-time`, `tools/harness/run.mjs s3 s4 --fast-time --no-bench-log`,
`tools/exec-sync/reorg.mjs`) on igneumd, igneum-miner, igneum-harness-sim and igneum-p2p-probe built into the night
checkout's `target-integration`, clippy per crate dir, `cargo audit` per Cargo.lock; then `docs/benchmarks/night/<date>.md`
(pass/fail table, "new since last night" against the newest earlier report, commits moved), committed as igneum-labs on
branch `night-battery` (rebuilt on master each night, earlier unmerged reports carried over) and force-pushed to
`/srv/igneum.git` only. Main merges: `git fetch build night-battery` from the main checkout. The fork branch defaults to the
newest `release-*-node` on the mirror (tonight release-0.3.15-node 713ef876).

Dry run (`NIGHT_SUBSET=1`, the second slot while the repro held the first, so CARGO_BUILD_JOBS 45): **3 min 54 s** wall,
10 pass, 1 FAIL, 1 skip; report `docs/benchmarks/night/2026-10-06-dryrun.md` on the mirror's night-battery branch (fbb72e5).

| Row | Result | Time | Detail |
|---|---|---|---|
| suite igneum-pow | pass | 51 s | 99 passed |
| suite fork/kaspa-pow | pass | 1 min 04 s | 7 passed |
| suite fork/igneum-miner | pass | 49 s | 18 passed |
| fuzz igneum-pow x200 | pass | 11 s | 200 mx8 programs and 200 scratch programs, 800 units each |
| sim finality_sim.py, finality_v2.py --quick, difficulty/sim.py --quick | pass | 3 s, 41 s, 5 s | 249, 117, 8 table lines |
| clippy igneum-pow | pass | 3 s | 28 warnings |
| audit igneum-pow | pass | 3 s | 0 vulnerabilities |
| audit vendor/igneum-node | **FAIL** | 2 s | 22 advisories in the fork's lock file: h2 (RUSTSEC-2026-0258, unbounded empty DATA frames), quinn-proto (2026-0185, remote memory exhaustion), rustls (2026-0285, TLS 1.3 handshake across encryption levels), ruint (2026-0220), crossbeam-epoch, anyhow (2026-0190), event-listener, faster-hex (2026-0306, AVX2 read past src), lru (2026-0253), tracing-subscriber (2025-0055), chacha20, spin; 17 unmaintained-crate warnings (async-std discontinued, atty, bincode, derivative, instant, mach, paste, proc-macro-error, rustls-pemfile) |
| harness | skip | | not in the subset; its first run is the 02:00 battery, so the first full report will show whether the Node harnesses run on Linux unchanged (c4, fud and v3.mjs default IGNEUM_NODE_ROOT to /Users/joshm/Projects/igneum/; the battery sets it) |

What the FAIL means and what follows: every node binary shipped so far (and the 0.3.15 one tonight) links h2, quinn-proto and
rustls at versions with published advisories; h2 and quinn-proto are in the gRPC and QUIC paths a peer can reach, so these are
the remote ones. The fix is a dependency bump in the fork (`cargo update -p h2 -p quinn-proto -p rustls -p ruint -p
crossbeam-epoch -p anyhow -p event-listener -p faster-hex -p lru -p tracing-subscriber`, then the suites), a consensus
engineer's hour on a quiet branch, and the row goes green by itself the next night. Until then the row stays FAIL every night
and "new since last night" stays quiet about it. The unmaintained-crate warnings are upstream rusty-kaspa's and do not fail
the row.

Expected full-run time (not measured yet): the three suites above compile the fork's test targets once (about 1 min each for
the first crates, seconds after), so 75 fork members plus the repo crates are estimated at 40 to 70 min; the full sims about
10 min (finality_v2.py is 5 min on the Mac); the harnesses 15 to 30 min; clippy and audit under 10 min. Under 2 h, inside
the 8 h limit; the first report at 02:00 BST writes the real number.

### 7.3 Reproducible builds (DONE 19:52Z; A vs B MATCH on all four, DIFFER against the shipped bytes, both explained)

`tools/repro/rebuild-release.sh <version>` (the Mac) reads the pins from `docs/plans/release-<v>.md` (the heading
"(node <sha>, app <sha>)" and the bold hashes of the Linux and Windows rows), makes sure both commits are on the mirrors, and
runs `infra/build-server/repro/rebuild-on-box.sh` on the box: a clean clone of the fork at the node commit on a branch under
a clean clone of the repo at the app commit, two clean passes per target in ONE target path each, no sccache, under build
slots through remote-run.sh, `SOURCE_DATE_EPOCH` = the node commit's time, `TZ=UTC`; the shipped hashes come token-free from
the public downloads (the HiveOS tarball for the Linux pair; the installer for the exes, when innoextract can open it) with
the plan's hashes as the fallback; the evidence goes to `docs/evidence/reproduced/<version>.md`. The 0.3.14 run (node
4c6b129d, app a90f6a5): four passes of 70 to 78 s, whole run 5 min 06 s.

| Artefact | A vs shipped | A vs B | Why the shipped bytes differ (read off the binaries) |
|---|---|---|---|
| igneumd (box 03f35e05..., 49,600,096 B) | DIFFER | **MATCH** | shipped 934f393c... was the Mac's zig build for glibc 2.36; the box's needs GLIBC_2.39 (native clang and lld). Commit string 4c6b129d in the box's: 1 hit |
| igneum-miner (box 900c1f0b..., 9,842,168 B) | DIFFER | **MATCH** | the same toolchain difference |
| igneumd.exe (box 166e604e..., 51,758,592 B) | DIFFER | **MATCH** | shipped 44fa74c0... came from the Mac's Homebrew mingw at 16:52Z, before the --no-insert-timestamp fix (17:48Z) and from a worktree (the empty-commit class); the box's is Ubuntu GCC 13 posix with a zero PE timestamp and the commit string (1 hit). innoextract 1.9 cannot open the Inno Setup 6 installer (setup loader revision 2), so the shipped exe's own header was not read |
| igneum-miner.exe (box fefd266c..., 10,994,688 B) | DIFFER | **MATCH** | the same |

Two non-determinisms found on the way, both in the SHIPPED builds too (first run 19:43Z, passes A and B differed on all four):

| Class | Fact | Fix |
|---|---|---|
| OUT_DIR path in the binary | prost's generated `protowire.rs` (kaspa-grpc-core, kaspa-p2p-lib) embeds its OUT_DIR path; a pass in a target dir of another NAME differs (igneum-miner matched byte for byte once the path was the same) | one target path per target in the repro; for cross-machine identity a `--remap-path-prefix` of the target dir and the home (not done: the Mac and the box differ in every path anyway) |
| Build clock in the binary | libmimalloc-sys compiles mimalloc's C with `__DATE__` and `__TIME__` ("Oct  6 2026", "21:38:15" sat in libmimalloc.a, next to the mimalloc option names); two builds a minute apart differ | `SOURCE_DATE_EPOCH` exported for every pass (GCC and clang take the date and time from it); PROPOSED for build-remote.sh, cross-remote.sh, cross-build.sh and the PC job: export it from the commit time so two builds of one commit give one hash. The earlier "byte-identical across three builds" on the box was under sccache, which returns the first build's object and hides this class |

What it means: the box is deterministic for a given commit and path, so a release built on it can be checked by anyone with the
same toolchain by rebuilding and comparing; the shipped 0.3.14 bytes cannot be reproduced anywhere because they came from
two Mac toolchains with a build clock inside, and that is the reason to ship from the box from 0.3.16 (R2) with
SOURCE_DATE_EPOCH set. Open: `rebuild-release.sh` for 0.3.15 the moment it ships (one command, 5 min).

### 7.4 The CPU prover trial (DONE 19:30Z; verdict: the box is NOT a prover)

`infra/build-server/prover/cpu-trial.sh` on the box under the measure hold (builds excluded), `igneum-prove-host` from master
7483fb37 (the 0.3.15 prover pair, sha256 71bc2438...; its `cuda` feature changes nothing under `SP1_PROVER=cpu`), fixture
`proving/fixtures/block-56-transfers.json` (the v0 block: 3 transfers, 600 pgas, one shard, 556,369 SP1 cycles), `--mode shard
--shard 0`, `RAYON_NUM_THREADS=96`, nice 19, box otherwise idle (load 2.6 at start). Log and results JSON:
`/srv/builds/_log/prover-trial/trial-20261006T192824Z.{log,json,txt}`; JSONL line kind `measure`.

| Stage | Box, 96 threads (EPYC 9454P) | Mac, same statement class (bench-log) |
|---|---|---|
| setup (prover client, shard and aggregator keys) | 14.4 s (client 12.6, keys 1.8) | 9.3 s per invocation in the v0 loop (4 Oct, "nearly all SP1 setup") |
| execute | 0.28 s, 556,369 cycles, 927 cycles per pgas | 0.19 s for block 78 (3 Oct) |
| core proof | **34.2 s**, 7,317,561 B, verify 0.34 s | 83 s for the 200-pgas shard on a Mac at load 40 (4 Oct); 71.7 s is main's Mac figure for this fixture (its stage not recorded here, labelled approximate) |
| compressed proof (what a record carries) | **85.9 s**, 1,272,897 B, verify 0.07 s | 272 s for the 200-pgas shard on the loaded Mac (4 Oct); 61 s for the smallest shard in the 3-node v0 loop |
| whole run, wall | 136.9 s | |
| peak RSS | 28.2 GB (VmHWM) | |
| CPU use | 64 of 96 threads busy on average (6,408 percent in `ps`) | |

What the numbers mean, per tier, and what follows:

| Number | Means | Done or proposed |
|---|---|---|
| core 34.2 s under 60 s, compressed 85.9 s over it | the proof a record carries is the compressed one, so the shard that matters takes 120 s of proving on 96 CPU threads for the SMALLEST shard the chain has (600 pgas, 0.56 M cycles); a shard at `S_p` is 60 M cycles (bench-log 4 Oct), about 100x, so hours per shard on this CPU against the 60 s proof lag the litepaper states | the 60 s test of the ask is NOT met on the stage that counts; NO standing CPU prover unit is written, nothing joins the devnet from the box (R6 holds: the box is never a node host for the live devnet) |
| 28.2 GB peak for the smallest shard | a CPU prover needs 32 GB of RAM for a toy shard; every home tier (8, 12, 16, 24 or 32 GB CARDS, 16 to 64 GB of RAM) is out of CPU proving, and the rented 4090 boxes' CPUs are not a fallback either | the app keeps "proving on the CPU (slow)" as a correctness lane only; the prover tiers are the real cards (docs/analysis, 6 Oct rented-card measurement) |
| 64 of 96 threads busy | SP1's CPU prover does not scale to the whole box; a second trial with `RAYON_NUM_THREADS=48` would show whether half the box proves as fast (then two shards side by side) | not run tonight (one slot of the box's evening); the script takes `--threads` |
| 14.4 s setup per invocation | the same per-process cost the Mac pays; a resident prover would pay it once | already the design of the app's prover loop |

The box stays a build and test machine. If main wants a CPU prover anyway for coverage (a prover that is always on, never fast),
the shape is a systemd unit as `build` with `SP1_PROVER=cpu`, a throwaway devnet key (never the OTA key, never a hand's key),
`--threads 48`, Nice 19 and the measure hold taken for the whole run, which would exclude builds for minutes at a time: that
is why it is not written.