igneum/docs/plans/build-server.md

Build server: igneum-build-1 (plan, benchmarks, rules)

6 October 2026. the project lead ordered a Hetzner dedicated server at 18:2x UK: AX162-1-LTD, EPYC 9454P (48 cores, 96 threads), 128 GB, 2x 3.84 TB NVMe, Falkenstein (FSN1-DC24, Hetzner #3088308), 188.40.146.49, IPv6 2a01:4f8:2240:205a::/64. Purpose: this Mac is the compile queue for ten agents (8-minute rebuilds under contention) and the Windows cross-builds; the box takes over Rust builds, cross-builds and a shared compile cache, and later a CI runner and the Devnet 2 seed.

1. What is in place (all on branch build-server)

Piece	File	State 6 Oct 2026
Install + provision	infra/build-server/provision.sh	ran: installimage 17:16 to 17:20 UTC (Ubuntu 24.04, RAID 1, no swap), provision 17:24 to 17:27 UTC, second run 2 s with zero changes (idempotent)
Mac side	infra/build-server/run-from-mac.sh <ip>	ran: ~/.config/igneum/build-server = build@188.40.146.49, build remotes added, 124 igneum branches and 48 fork branches pushed to the bare mirrors
Shared library	infra/build-server/lib.sh	host line, ssh options, mirror push, remote checkout, overlay, remote slot runner
Remote runner	infra/build-server/remote-run.sh	runs on the box: slot, sccache, RESULT line, one JSON line per run in /srv/builds/_log/builds.jsonl (the worker dashboard's feed, fields as main asked on 6 Oct)
Remote cargo	tools/build-remote.sh	any crate dir, any worktree: tools/build-remote.sh [-- cargo args]; IGNEUM_AGENT=<name> tags the slot label and the log
Remote Windows cross	tools/cross-remote.sh	fork worktree or app/igneum-app: tools/cross-remote.sh [--compare <dir of the Mac's exes>]

ssh line: ssh -i ~/.ssh/igneum_ed25519 build@188.40.146.49 (root works with the same key; passwords are off).

Box facts (provision summary): rustc 1.99.0 (the Mac's version; NO rust-toolchain file exists in the repo or the fork, the pin is RUST_TOOLCHAIN in provision.sh), targets x86_64-unknown-linux-gnu and x86_64-pc-windows-gnu, sccache 0.18.0 with a 100 GB disk cache at /srv/sccache, mingw-w64 GCC 13 posix threads (the PC job's recipe), clang and lld 18.1.3, Node v22.23.3, git 2.43.0, tmux 3.4, ufw 22 + 26611 + 26811 (the devnet and testnet seed p2p ports; RPC stays on loopback as on every seed), md0 /boot and md1 / as RAID 1, 3.3 TB free, swap none, 1 build slot in /srv/builds/_locks/slots.

2. How a build travels

1. bs_context (lib.sh) reads the crate: a fork worktree under vendor/ (kind node, mirror /srv/igneum-node.git) or a crate of the igneum repo (kind repo, mirror /srv/igneum.git). cargo metadata lists every path dependency.
2. HEAD is pushed to the mirror; the box clones or fetches it at /srv/builds/<worktree>/<same relative path> and checks out that commit. A real .git is needed: the fork's kaspa-build-info runs git rev-parse HEAD at build time and every release plan checks the commit in the binary's strings.
3. Uncommitted changes go by rsync --checksum without -t (files that changed are written with the box's clock) and the written files are re-stamped with touch (the copied-sources rule, tools/ci/copied-sources-check.sh passes).
4. The cargo command runs in the crate dir under one of the box's slot files (flock on /srv/builds/_locks/build-<k>, never the Mac's ~/.config/igneum/build-slots), with sccache and -j 90, logging wall time, compile count and cache hits.
5. Artefacts come back into <crate>/target-remote/... with size and sha256. NEVER into target/: the box's native binaries are x86_64 Linux (glibc 2.39) and do not run on the Mac.

/srv/builds/<worktree> mirrors the Mac's igneum worktree ROOT because the fork's path dependency is ../../../../igneum-pow (vendor/igneum-node/consensus/pow/Cargo.toml).

3. Benchmarks

The Mac numbers are from the logs (docs/bench-log.md, docs/plans/release-0.3.10.md, release-0.3.11.md); no fresh Mac run was made, because a Mac build would take a slot from the agents and the logs already hold several readings per case. Mac = Apple M5 Max (18 cores), builds at nice -n 19 with 4 to 6 jobs under the build lock, usually loaded by other agents. Box = igneum-build-1, -j 90, nothing else running.

Case	Mac (logged)	Box	Box run
Clean node build (cargo build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow, every crate)	12 min 36 s (0.3.10, new target dir, -j 4); 17 min 53 s at load 140 and 8 min 58 s second time (fud ledger, -j 4); rusty-kaspa kaspad alone 2 min 36 s on an idle Mac	1 min 27 s cargo wall (1 min 34 s end to end from the Mac: push and sync 1 s, build, fetch of both binaries)	cold: 518 crates downloaded inside that time, sccache 0 hits / 993 misses, 563 crates compiled; igneumd 48,220,896 B, igneum-miner 9,107,232 B, ELF x86-64 PIE; igneumd --version runs on the box; 17:30:58 to 17:32:32 UTC
Incremental rebuild (one file changed, same target dir)	2 min 08 s (0.3.10, 17:49Z); 3 min 19 s (0.3.11); 5 min 06 s (txgossip); 15 min 18 s at load 110 to 134 (M31); 35 s to 4 min (execution layer); "8 min under contention" (main, 6 Oct)	7 s cargo wall (10 s end to end: sync 1 s, build 6.97 s, fetch)	one line appended to kaspad/src/main.rs in the fork worktree, uncommitted, carried by the overlay (1 file written and re-stamped); 1 crate compiled, igneumd relinked; box idle (load 12 from the clean build a minute earlier); 17:33:11 to 17:33:21 UTC
Windows cross-build (--target x86_64-pc-windows-gnu, igneumd.exe + igneum-miner.exe)	8 min 25 s clean (-j 6, 3 Oct); 4 min 49 s and 4 min 53 s with warm dependencies (4 Oct); 12 min 28 s (finality-fixes, 4 Oct)	1 min 44 s cargo wall (1 min 48 s end to end)	cold: 995 compiles, sccache 0 hits; igneumd.exe 49,434,624 B, igneum-miner.exe 10,201,600 B; mingw GCC 13 posix, libclang 18; igneumd.exe imports libstdc++-6.dll (this fork head predates the housekeeping commit that made the C++ runtime static), so cross-remote.sh now fetches the three GCC 13 runtime DLLs beside the exes as the PC job does; 17:33:58 to 17:35:46 UTC. Second run on the same target dir, no source change: 8 s

Also logged for context: the Mac's Linux cross-build with zig (infra/cross/build-linux.sh) took 30 min 56 s cold and 3 min 20 s incremental; PC 1 built Linux + Windows node and app and ran both test suites in 7 min 38 s cold, 5 min 09 s warm (CLAUDE.md, 5 Oct).

What the numbers mean and what follows

Number	Means	Done or proposed
Clean build 1 min 27 s against 12 to 18 min on the loaded Mac (8 to 12x)	a new worktree costs an agent a minute and a half, not a slot for a quarter of an hour; the first build of every one of the 123 worktree dirs on the box is this cold case, later ones are the 7 s case	R1 proposed; sccache was cold (0 hits of 1,982 requests) because every run so far was the first of its kind; the cache fills as agents build the same crate versions from different worktrees, so the second worktree's clean build will be mostly hits (measure when it happens, write the number here)
Incremental 7 s against 2 to 15 min on the Mac (the "8 min under contention")	an edit-build loop of seconds for Linux targets; end to end 10 s because sync is 1 s and the two binaries (57 MB) come back in 2 s	R1; the slot count stays 1 until two agents collide, then 2 with -j 48 each (SLOTS=2 run-from-mac.sh and --jobs 48)
Windows cross 1 min 44 s against 4 min 49 s to 12 min 28 s on the Mac (3 to 7x), 8 s incremental	a Windows exe per commit is cheap enough to build on every push; the PC build job (7 min 38 s cold, 5 min 09 s warm for Linux + Windows + tests) stays the second source	R2 proposed
Mac arm64 binaries: not built here	agents who run nodes on the Mac (local devnets, the DMG) still take Mac slots; the box cannot remove that contention	R3; the real relief is to move test networks to the fleet or to a Devnet 2 seed on this box (its unit, ports and ufw are ready)
The commit hash is EMPTY in every Mac worktree build and was empty in the first box builds	kaspa-build-info (build-info/build.rs) needs .git to be a directory AND HEAD to be a symbolic ref to a loose branch file; a worktree's .git is a file and a detached HEAD is not a ref; and once it has found nothing it emits no rerun-if-changed, so cargo never runs it again in that target dir (release-0.3.11: cargo clean -p kaspa-build-info)	fixed on the box: the remote checkout is git checkout -B <branch> <sha> and build-remote.sh runs cargo clean --release -p kaspa-build-info whenever the commit differs from the last one built in that target dir (.build-remote-sha-<target dir>); verified 17:40 UTC: 2 string hits for 3bfe346f, binary +1,024 B. The release plans' "commit in its strings" checks were passing against builds from the fork's MAIN checkout (a real .git directory on a branch), not from worktrees. Proposed: the PC build job and the Mac's release recipe adopt the same two steps, or the fork's build.rs learns to read a worktree's gitdir file
igneumd.exe hash changes build to build with no source change (c424aae0 then bfbff015) while the Linux igneumd stays byte-identical across three builds	the mingw linker writes a timestamp into the PE header; the Linux ELF has none	not changed (the Mac and PC exes have the same property); -C link-arg=-Wl,--no-insert-timestamp would make the exe reproducible and is a one-line change to cross-remote.sh, the Mac script and jobbuild.rs together if main wants reproducible Windows builds
sccache misses 1,982 of 1,982	expected for first builds; the cache is 494 MB after three builds, capped at 100 GB	nothing; the hit rate is in every RESULT line and every JSONL line, read it after the first repeat build
Disk 3.3 TB free, RAM 125 GB, load peaked at 24 during the Windows build with 96 threads	room for 2 slots and the Devnet 2 seed without contention	nothing now

4. Rules (proposed for CLAUDE.md once the box passes; main decides)

Rule	Text
R1	Every agent's cargo build, cargo test, cargo check and cargo clippy for Linux goes through tools/build-remote.sh from the crate directory. The box takes one remote slot per build; nobody runs cargo over ssh by hand.
R2	Windows exes come from tools/cross-remote.sh (fork worktree: igneumd.exe, igneum-miner.exe; app/igneum-app: igneum-app.exe and the two tools). The PC build job stays as the second source until two releases have shipped from the box.
R3	The Mac keeps what only it can do: aarch64-apple-darwin binaries (the DMG, nodes that agents run locally), tests that need Metal (proto-metal, the Metal worker), and measurements. Those still use tools/lock/with-lock.sh and the Mac's build slots.
R4	A worktree builds once on the box per commit plus overlay; the next build is incremental in /srv/builds/<worktree>/.../target. Nobody deletes another worktree's target dir on the box.
R5	RUST_TOOLCHAIN in provision.sh is bumped in the same commit as the Mac's rustup update; build-remote.sh refuses a mismatch. Add a rust-toolchain.toml to the fork and the repo (none exists today) so both sides pin from one file.
R6	The box is never a node host for the live devnet and never holds a secret (no ~/.config/igneum there). The Devnet 2 seed on it runs under its own unit with --devnet --devnet-suffix=<n> on 26611 (ufw already open) when that work starts.
R7	CLAUDE.md line "nothing is built on a server" and "Windows builds go to the GitHub runner, Linux binaries come from infra/cross/build-linux.sh" are rewritten when R1 and R2 are adopted; until then the box is the measured option, not the rule.

5. What the box does not do yet

Gap	Why it matters	Next step
No zig / cargo-zigbuild	the devnet seed (Debian 12, glibc 2.36) takes the Mac's zig build; a native box build links glibc 2.39, which Debian 13 seeds accept and HiveOS (Ubuntu 18/20 base) does not	install zig 0.17 + cargo-zigbuild in provision.sh, add --target x86_64-unknown-linux-gnu.2.36 mode to build-remote.sh
No macOS target	agents who run nodes on the Mac still build there	out of scope (needs the macOS SDK on Linux); the fleet or the box's own Devnet 2 seed takes the test-network runs instead
No CI runner	GitHub ci.yml and windows.yml run on GitHub's machines	install a self-hosted runner as user build once R1 is in
Byte identity with the Mac's exes	different C/C++ toolchain (Homebrew mingw vs Ubuntu GCC 13) and embedded source paths	not a goal; the box is identical with itself build to build, cross-remote.sh reports sha256 and the DLL list per exe
Robot API	~/.config/igneum/hetzner-token is the Cloud token (hcloud); the dedicated box lives in Robot, a separate credential	main sets the Robot server name in the UI; a webservice user goes to ~/.config/igneum/robot-credentials when needed