igneum/docs/plans/build-server.md

# Build server: igneum-build-1 (plan, benchmarks, rules)

6 October 2026. the project lead ordered a Hetzner dedicated server at 18:2x UK: AX162-1-LTD, EPYC 9454P (48 cores, 96 threads),
128 GB, 2x 3.84 TB NVMe, Falkenstein (FSN1-DC24, Hetzner #3088308), 188.40.146.49, IPv6 2a01:4f8:2240:205a::/64.
Purpose: this Mac is the compile queue for ten agents (8-minute rebuilds under contention) and the Windows cross-builds;
the box takes over Rust builds, cross-builds and a shared compile cache, and later a CI runner and the Devnet 2 seed.

## 1. What is in place (all on branch `build-server`)

| Piece | File | State 6 Oct 2026 |
|---|---|---|
| Install + provision | `infra/build-server/provision.sh` | ran: installimage 17:16 to 17:20 UTC (Ubuntu 24.04, RAID 1, no swap), provision 17:24 to 17:27 UTC, second run 2 s with zero changes (idempotent) |
| Mac side | `infra/build-server/run-from-mac.sh <ip>` | ran: `~/.config/igneum/build-server` = `build@188.40.146.49`, `build` remotes added, 124 igneum branches and 48 fork branches pushed to the bare mirrors |
| Shared library | `infra/build-server/lib.sh` | host line, ssh options, mirror push, remote checkout, overlay, remote slot runner |
| Remote runner | `infra/build-server/remote-run.sh` | runs on the box: slot, sccache, RESULT line, one JSON line per run in `/srv/builds/_log/builds.jsonl` (the worker dashboard's feed, fields as main asked on 6 Oct) |
| Remote cargo | `tools/build-remote.sh` | any crate dir, any worktree: `tools/build-remote.sh [-- cargo args]`; `IGNEUM_AGENT=<name>` tags the slot label and the log |
| Remote Windows cross | `tools/cross-remote.sh` | fork worktree or app/igneum-app: `tools/cross-remote.sh [--compare <dir of the Mac's exes>]` |

ssh line: `ssh -i ~/.ssh/igneum_ed25519 build@188.40.146.49` (root works with the same key; passwords are off).

Box facts (provision summary): rustc 1.99.0 (the Mac's version; NO rust-toolchain file exists in the repo or the fork, the
pin is `RUST_TOOLCHAIN` in provision.sh), targets x86_64-unknown-linux-gnu and x86_64-pc-windows-gnu, sccache 0.18.0 with a
100 GB disk cache at /srv/sccache, mingw-w64 GCC 13 posix threads (the PC job's recipe), clang and lld 18.1.3, Node v22.23.3,
git 2.43.0, tmux 3.4, ufw 22 + 26611 + 26811 (the devnet and testnet seed p2p ports; RPC stays on loopback as on every seed),
md0 /boot and md1 / as RAID 1, 3.3 TB free, swap none, 1 build slot in `/srv/builds/_locks/slots`.

## 2. How a build travels

1. `bs_context` (lib.sh) reads the crate: a fork worktree under `vendor/` (kind node, mirror `/srv/igneum-node.git`) or a crate
   of the igneum repo (kind repo, mirror `/srv/igneum.git`). `cargo metadata` lists every path dependency.
2. HEAD is pushed to the mirror; the box clones or fetches it at `/srv/builds/<worktree>/<same relative path>` and checks out
   that commit. A real `.git` is needed: the fork's `kaspa-build-info` runs `git rev-parse HEAD` at build time and every release
   plan checks the commit in the binary's strings.
3. Uncommitted changes go by `rsync --checksum` without `-t` (files that changed are written with the box's clock) and the
   written files are re-stamped with `touch` (the copied-sources rule, `tools/ci/copied-sources-check.sh` passes).
4. The cargo command runs in the crate dir under one of the box's slot files (`flock` on `/srv/builds/_locks/build-<k>`,
   never the Mac's `~/.config/igneum/build-slots`), with sccache and `-j 90`, logging wall time, compile count and cache hits.
5. Artefacts come back into `<crate>/target-remote/...` with size and sha256. NEVER into `target/`: the box's native
   binaries are x86_64 Linux (glibc 2.39) and do not run on the Mac.

`/srv/builds/<worktree>` mirrors the Mac's igneum worktree ROOT because the fork's path dependency is
`../../../../igneum-pow` (vendor/igneum-node/consensus/pow/Cargo.toml).

## 3. Benchmarks

The Mac numbers are from the logs (docs/bench-log.md, docs/plans/release-0.3.10.md, release-0.3.11.md); no fresh Mac run
was made, because a Mac build would take a slot from the agents and the logs already hold several readings per case.
Mac = Apple M5 Max (18 cores), builds at `nice -n 19` with 4 to 6 jobs under the build lock, usually loaded by other agents.
Box = igneum-build-1, `-j 90`, nothing else running.

| Case | Mac (logged) | Box | Box run |
|---|---|---|---|
| Clean node build (`cargo build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow`, every crate) | 12 min 36 s (0.3.10, new target dir, -j 4); 17 min 53 s at load 140 and 8 min 58 s second time (fud ledger, -j 4); rusty-kaspa kaspad alone 2 min 36 s on an idle Mac | **1 min 27 s** cargo wall (1 min 34 s end to end from the Mac: push and sync 1 s, build, fetch of both binaries) | cold: 518 crates downloaded inside that time, sccache 0 hits / 993 misses, 563 crates compiled; igneumd 48,220,896 B, igneum-miner 9,107,232 B, ELF x86-64 PIE; `igneumd --version` runs on the box; 17:30:58 to 17:32:32 UTC |
| Incremental rebuild (one file changed, same target dir) | 2 min 08 s (0.3.10, 17:49Z); 3 min 19 s (0.3.11); 5 min 06 s (txgossip); 15 min 18 s at load 110 to 134 (M31); 35 s to 4 min (execution layer); "8 min under contention" (main, 6 Oct) | **7 s** cargo wall (10 s end to end: sync 1 s, build 6.97 s, fetch) | one line appended to kaspad/src/main.rs in the fork worktree, uncommitted, carried by the overlay (1 file written and re-stamped); 1 crate compiled, igneumd relinked; box idle (load 12 from the clean build a minute earlier); 17:33:11 to 17:33:21 UTC |
| Windows cross-build (`--target x86_64-pc-windows-gnu`, igneumd.exe + igneum-miner.exe) | 8 min 25 s clean (-j 6, 3 Oct); 4 min 49 s and 4 min 53 s with warm dependencies (4 Oct); 12 min 28 s (finality-fixes, 4 Oct) | **1 min 44 s** cargo wall (1 min 48 s end to end) | cold: 995 compiles, sccache 0 hits; igneumd.exe 49,434,624 B, igneum-miner.exe 10,201,600 B; mingw GCC 13 posix, libclang 18; igneumd.exe imports libstdc++-6.dll (this fork head predates the housekeeping commit that made the C++ runtime static), so cross-remote.sh now fetches the three GCC 13 runtime DLLs beside the exes as the PC job does; 17:33:58 to 17:35:46 UTC. Second run on the same target dir, no source change: 8 s |

Also logged for context: the Mac's Linux cross-build with zig (`infra/cross/build-linux.sh`) took 30 min 56 s cold and
3 min 20 s incremental; PC 1 built Linux + Windows node and app and ran both test suites in 7 min 38 s cold, 5 min 09 s warm
(CLAUDE.md, 5 Oct).

### What the numbers mean and what follows

| Number | Means | Done or proposed |
|---|---|---|
| Clean build 1 min 27 s against 12 to 18 min on the loaded Mac (8 to 12x) | a new worktree costs an agent a minute and a half, not a slot for a quarter of an hour; the first build of every one of the 123 worktree dirs on the box is this cold case, later ones are the 7 s case | R1 proposed; sccache was cold (0 hits of 1,982 requests) because every run so far was the first of its kind; the cache fills as agents build the same crate versions from different worktrees, so the second worktree's clean build will be mostly hits (measure when it happens, write the number here) |
| Incremental 7 s against 2 to 15 min on the Mac (the "8 min under contention") | an edit-build loop of seconds for Linux targets; end to end 10 s because sync is 1 s and the two binaries (57 MB) come back in 2 s | R1; the slot count stays 1 until two agents collide, then 2 with `-j 48` each (`SLOTS=2 run-from-mac.sh` and `--jobs 48`) |
| Windows cross 1 min 44 s against 4 min 49 s to 12 min 28 s on the Mac (3 to 7x), 8 s incremental | a Windows exe per commit is cheap enough to build on every push; the PC `build` job (7 min 38 s cold, 5 min 09 s warm for Linux + Windows + tests) stays the second source | R2 proposed |
| Mac arm64 binaries: not built here | agents who run nodes on the Mac (local devnets, the DMG) still take Mac slots; the box cannot remove that contention | R3; the real relief is to move test networks to the fleet or to a Devnet 2 seed on this box (its unit, ports and ufw are ready) |
| The commit hash is EMPTY in every Mac worktree build and was empty in the first box builds | `kaspa-build-info` (build-info/build.rs) needs `.git` to be a directory AND HEAD to be a symbolic ref to a loose branch file; a worktree's `.git` is a file and a detached HEAD is not a ref; and once it has found nothing it emits no `rerun-if-changed`, so cargo never runs it again in that target dir (release-0.3.11: `cargo clean -p kaspa-build-info`) | fixed on the box: the remote checkout is `git checkout -B <branch> <sha>` and build-remote.sh runs `cargo clean --release -p kaspa-build-info` whenever the commit differs from the last one built in that target dir (`.build-remote-sha-<target dir>`); verified 17:40 UTC: 2 string hits for 3bfe346f, binary +1,024 B. The release plans' "commit in its strings" checks were passing against builds from the fork's MAIN checkout (a real .git directory on a branch), not from worktrees. DONE (main's decision): the PC job writes a minimal `node/.git` (HEAD -> refs/heads/build holding the manifest's new `commit_full`, written by push-build-inputs.sh) at extract and cleans kaspa-build-info on a new commit (jobbuild.rs, 4 unit tests pass on the box); the Mac's cross-build.sh refuses a worktree and cleans on a new commit; the gate `tools/ci/commit-string-check.sh` runs on every igneumd the three scripts produce (self-test in ci.yml; shown firing on the Mac's worktree-built igneumd and passing on the box's). Only kaspad depends on kaspa-build-info, so igneum-miner is out of the gate's scope |
| igneumd.exe hash changed build to build with no source change (c424aae0 then bfbff015) while the Linux igneumd stayed byte-identical across three builds | the mingw linker wrote a timestamp into the PE header; the Linux ELF has none | DONE (main's decision): `-C link-arg=-Wl,--no-insert-timestamp` in cross-remote.sh, the Mac's cross-build.sh and the PC job (jobbuild.rs); verified 17:48 and 17:49 UTC: two builds, igneumd.exe c38b7570... and igneum-miner.exe c3a0fc2b... identical both times |
| Second worktree's clean build (vendor/igneum-node-v4 from the main checkout, cold target dir, warm sccache): 1 min 18 s, 604 hits of 993 compiles (61 percent), 389 misses | the first build of each of the 123 worktree dirs costs 1 min 18 s to 1 min 27 s, not the Mac's 12 to 18 min; sccache saves 9 s of the 87 because the clean build is dominated by the fork's own 74 crates and the C++ (rocksdb) objects, which differ per tree or do not cache; the cache is 1.0 GB after four builds, capped at 100 GB | nothing; the hit rate is in every RESULT line and every JSONL line |
| Disk 3.3 TB free, RAM 125 GB, load peaked at 24 during the Windows build with 96 threads | room for 2 slots and the Devnet 2 seed without contention | nothing now |

## 4. Rules (ADOPTED by main on 6 October 2026, written into CLAUDE.md "Running agents on this Mac"; R2 narrowed: the PCs keep only GPU and Windows-runtime jobs from now, not two releases)

| Rule | Text |
|---|---|
| R1 | Every agent's `cargo build`, `cargo test`, `cargo check` and `cargo clippy` for Linux goes through `tools/build-remote.sh` from the crate directory. The box takes one remote slot per build; nobody runs cargo over ssh by hand. |
| R2 | Windows exes come from `tools/cross-remote.sh` (fork worktree: igneumd.exe, igneum-miner.exe; app/igneum-app: igneum-app.exe and the two tools). The PC `build` job stays as the second source until two releases have shipped from the box. |
| R3 | The Mac keeps what only it can do: aarch64-apple-darwin binaries (the DMG, nodes that agents run locally), tests that need Metal (proto-metal, the Metal worker), and measurements. Those still use `tools/lock/with-lock.sh` and the Mac's build slots. |
| R4 | A worktree builds once on the box per commit plus overlay; the next build is incremental in `/srv/builds/<worktree>/.../target`. Nobody deletes another worktree's target dir on the box. |
| R5 | `RUST_TOOLCHAIN` in provision.sh is bumped in the same commit as the Mac's `rustup update`; build-remote.sh refuses a mismatch. Add a `rust-toolchain.toml` to the fork and the repo (none exists today) so both sides pin from one file. |
| R6 | The box is never a node host for the live devnet and never holds a secret (no `~/.config/igneum` there). The Devnet 2 seed on it runs under its own unit with `--devnet --devnet-suffix=<n>` on 26611 (ufw already open) when that work starts. |
| R6a | ONE recorded exception to R6 (main's ruling, 6 October 2026, 20:0x UK): the three Discord webhook URLs live at `/srv/discord-hooks/env` (mode 600, owner build), installed by `infra/build-server/discord-hooks/install.sh` with `IGNEUM_SECRET_ON_BOX_OK=1`, because the Mac sleeps and the bot's timer must not. The reasoning: a webhook URL signs no release, moves no funds and reaches none of the devnet's hands; whoever holds it can only post as the bot, and a leaked one is deleted in Discord in one click and the file replaced. No other secret joins it; `check` prints key names, never values. |
| R7 | CLAUDE.md line "nothing is built on a server" and "Windows builds go to the GitHub runner, Linux binaries come from infra/cross/build-linux.sh" are rewritten when R1 and R2 are adopted; until then the box is the measured option, not the rule. |

## 5. Gotchas met on the first day

| Case | What happened | Fix |
|---|---|---|
| Stale overlay blocks the next checkout (PC 1 worker, 6 Oct 2026) | build-remote.sh rsyncs uncommitted files over the box's checkout; on the next commit `git checkout -B` refused with "local changes would be overwritten" | remote-run.sh `checkout` mode: `git checkout -- .` and `git clean -fd` (target dirs, sha stamps and ignored files kept) before the branch checkout, then the overlay; `remote-run.sh --self-test` reproduces the dirty tree and shows the mode landing on the new commit clean |
| An all-identical overlay listed only directories | the first run's touch pipeline got an empty file list and failed | files only are counted and re-stamped (lib.sh bs_overlay_dir) |
| bash 3.2 on the Mac treats an empty array as unbound under `set -u` | run-from-mac.sh died on `PASS[*]` | a string instead of an array |
| `grep -q` plus `pipefail` turned a strings hit into a miss | the commit-string gate failed a stamped igneumd on its first use (SIGPIPE on `strings`) | `grep -c` |
| A path dependency inside a vendor repo (the shipper's proving build, 6 Oct 2026) | proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a MEMBER of the fork's workspace (it inherits `thiserror` from the fork's root manifest); the first design synced that one directory, so cargo found no workspace root on the box ("failed to load manifest for workspace member"), and the shipper cross-built the Linux prove-host on the Mac with cargo-zigbuild meanwhile | lib.sh groups path dependencies by git top level: one under vendor/ is a whole repository, pushed to its mirror (a fork worktree such as igneum-node-exec goes to /srv/igneum-node.git, which already held its branch; a repository of its own gets /srv/<name>.git, created on first use) and checked out whole at /srv/builds/<worktree>/vendor/<name>; run-from-mac.sh wires every vendor repo the Cargo.toml files reach (today only igneum-node-exec). The detector's first version tested "under BS_TOP" before "own repository" and missed it, since vendor/ lies under the igneum top level on disk. Then `libprotobuf-dev` was missing (sp1-prover-types's build script imports google/protobuf/empty.proto); added to provision.sh. Proof: `cd proving/igneum-prove && tools/build-remote.sh -- build --release -p igneum-prove-host`: igneum-prove-host 71,943,192 B, sha256 e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, ELF x86-64, 1 min 05 s warm (the cold run compiled 605 crates in 55 s before protoc stopped it). A Mac worktree has no vendor/ of its own, so a worktree that builds proving needs `git -C vendor/igneum-node worktree add <wt>/vendor/igneum-node-exec execution-layer` first, as the fork worktrees do |
| One remote build lost its ssh session after 75 s (18:30:50 UTC, the first full proving build) | the remote bash died with it (slot line left behind, no JSONL line); no OOM, no reboot, the retry a minute later passed | the dashboard collector's one-pass unit finished within a second of the drop, so it was tested: a 90 s remote session through the same ControlMaster path survived two collector passes triggered by hand; the collector only reads (/proc, lock files, `flock -n`, `sccache --show-stats`, `kill(pid, 0)`). One event, no cause in the journal, the retry passed. A build that must survive a dropped connection would need the remote command under setsid with the Mac reconnecting to wait; not done, open if it happens again |
| A stamp file at a nested path failed every checkout of /srv/builds/igneum (18:31 to 18:5x UTC, the shipper's 18:52 run) | a repo-kind crate keeps its `.build-remote-sha-<target>` and `target/` inside the crate dir; the checkout mode's clean-tree test only excused them at the tree root | the test is depth-agnostic and uses `--untracked-files=all` (a wholly untracked directory is otherwise collapsed to `?? dir/`); the self-test carries a nested stamp, a nested target dir and a stale `.git/index.lock`, which the mode now removes when no git runs there |
| Two runs on one worktree at once (the shipper, 18:48:56Z) | the second run's checkout replaced the first's sources mid-cargo; both died | lib.sh takes a per-worktree lock on the box (`/srv/builds/_locks/wt-<worktree>`, mkdir-atomic, holder line) across sync, build and fetch; a second run waits up to 2 h (a line every minute), a lock older than 3 h is taken over; released on EXIT. The build slot (`build-<k>`) is unchanged |
| The 0.3.15 prover pair for the PCs needs `--features igneum-prove-host/cuda` (the PCs run SP1_PROVER=cuda) | my first proving build named no feature | built on the box from master e1b5bc9: igneum-prove-host 73,161,528 B sha256 71bc2438856bb141f6cad3d18489f708568144fad5a06a002fbadefb9ce256f9, igneum-prove-export 3,609,360 B sha256 263bf4cef70af4a13a45b2e79b8dbab373282ea4c571f624d02ddb5791935361 (52 s warm, no CUDA needed at build time); handed to the shipper |
| Let's Encrypt saw NXDOMAIN for build.igneum.network | the deSEC record was minutes old; Ubuntu's Caddy then fell back to ZeroSSL and failed with HTTP 422 for ever | issuer pinned to Let's Encrypt; the retry got the certificate |

## 5a. The GPU workers (added 6 October 2026, 19:10 UTC, for the class v4 rehearsal)

| What | Fact |
|---|---|
| Headers on the box | provision.sh `step_cuda`: NVIDIA's ubuntu2404 apt repository, `cuda-nvrtc-dev-12-8`, `cuda-cudart-dev-12-8`, `cuda-driver-dev-12-8` (the libcuda stub) and Ubuntu's `opencl-c-headers`; no nvcc (no build file calls it: both workers compile from C/C++ sources with the headers and dlopen libcuda, libnvrtc and libOpenCL at run time), no driver (no GPU here). 12.8 is the minor proto-cuda/nvrtc/fetch-redist.sh pins |
| Command | `tools/workers-remote.sh [--out <dir>]` from any igneum worktree: proto-cuda and proto-opencl through the mirror and overlay, clang++ 18 with `-static-libstdc++ -static-libgcc`, both workers, sha256 and the glibc ceiling printed, artefacts in `<worktree>/infra/cross/out-workers-box/` |
| Proof (master worker.cpp at 3b2c840) | igneum-worker-cuda 1,613,992 B sha256 6db8a9ad295a9f7598a8b5500f1c271fbfdb9a8df90848b95ff340664007031e; igneum-worker-opencl 124,904 B sha256 0dea75bb8d2d54721ee44b55a3ba486241d3c6053e72a133f8f5e40d2355cbbd; 3 s on the box |
| Consequence: glibc ceiling 2.38 | runs on the fleet (Ubuntu 22.04 containers are glibc 2.35: NO, 2.38 > 2.35; Ubuntu 24.04 hosts yes). The Mac's zig build (`infra/cross/build-workers-linux.sh`, glibc 2.36) is the one for Debian 12 and HiveOS; the fleet agent must check its boxes' glibc before swapping the worker. Fix if needed: zig on the box (open row in section 6) or `clang -target x86_64-linux-gnu.2.35` via zig; both a day's work, not done |
| Runner fix found on the way | a command string carrying `set -e` leaked into remote-run.sh through `eval` and killed the runner before its RESULT line (reported as rc 101); the runner now evaluates the command in a subshell |

## 6. What the box does not do yet

| Gap | Why it matters | Next step |
|---|---|---|
| No zig / cargo-zigbuild | the devnet seed (Debian 12, glibc 2.36) takes the Mac's zig build; a native box build links glibc 2.39, which Debian 13 seeds accept and HiveOS (Ubuntu 18/20 base) does not | install zig 0.17 + cargo-zigbuild in provision.sh, add `--target x86_64-unknown-linux-gnu.2.36` mode to build-remote.sh |
| No macOS target | agents who run nodes on the Mac still build there | out of scope (needs the macOS SDK on Linux); the fleet or the box's own Devnet 2 seed takes the test-network runs instead |
| No CI runner | GitHub `ci.yml` and `windows.yml` run on GitHub's machines | install a self-hosted runner as user build once R1 is in |
| Byte identity with the Mac's exes | different C/C++ toolchain (Homebrew mingw vs Ubuntu GCC 13) and embedded source paths | not a goal; the box is identical with itself build to build, cross-remote.sh reports sha256 and the DLL list per exe |
| Robot API | `~/.config/igneum/hetzner-token` is the Cloud token (hcloud); the dedicated box lives in Robot, a separate credential | main sets the Robot server name in the UI; a webservice user goes to `~/.config/igneum/robot-credentials` when needed |