igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity
igneum/infra/build-server
History
igneum-labs 0deaa5489a CI queue: a second self-hosted runner (igneum-build-2 joins the pool label), docs-only pushes skip the compile jobs, the red watcher pinned to the box that posts 
31 runs were queued on igneum-build-1's one runner at 13:15 UK on 7 October 2026 and nothing had concluded since 13:03Z, so no lane could read a conclusion.
- ci.yml: a `changes` job (ubuntu-latest) classifies the push with tools/ci/docs-only-check.sh (docs/, site/, *.md only = code=false; a new branch, a pull request, a force push or an API error = code=true); pow and sims need it and run only on code=true. The site job is unchanged on ubuntu-latest for every push. The classifier's self-test is in the gate.
- provision.sh and runner/register.sh: `--host <ip>` registers another box, forwarding BOX_HOSTNAME, RUNNER_NAME, RUNNER_LABELS, RUNNER_CPUS and RUNNER_JOBS (plain words only); RUNNER_CPUS writes AllowedCPUs into the service drop-in beside Nice=10, so igneum-build-2's runner is bounded like a suite (32 cores). The pool label is igneum-build-1 (both boxes carry it); ci-red marks the box with the record file and the poster, the default labels carry it, and igneum-build-1 got it through the runners API today.
- ci-red.yml runs on the ci-red label, so the red line always lands where the poster reads it.
- CLAUDE.md: the rule reads "read the conclusion when it lands, own a red before the next push"; pushes are never held.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:20:18 +00:00
..
capacity Build boxes: one host file per box and a route by class (suites and benches to build-2, proving to build-3, the rest to build-1); the no-mining rule in a box README and as a guard in the capacity layer 2026-10-07 10:34:16 +00:00
ci-red Build box: every red run classified and kept, pre-flight before the slot, one run per worktree, box reds in the red-run file, a 09:00 UK digest 2026-10-06 21:40:07 +00:00
discord-hooks Discord webhooks: install.sh finishes on a partial credentials file (the check step exits 1 by design) 2026-10-06 19:07:26 +00:00
hands Observer sync follows GitHub: the deploy-key probe captured before grep (ssh -T exits 1 under pipefail); plan 5c done 2026-10-07 07:31:42 +00:00
night Capacity layer: idle-core background workload on igneum-build-1 that yields to builds 2026-10-06 20:58:42 +00:00
prover Box: GitHub Actions runner as user runner, two build slots with 90 or 45 jobs, the measure hold, the CPU prover trial 2026-10-06 19:32:43 +00:00
repro Repro check: sccache really off (RUSTC_WRAPPER=/usr/bin/env, an empty value falls back to the box's config), the author-time epoch of lib.sh bs_sde, and the re-stamp line the copied-sources check reads 2026-10-06 20:13:43 +00:00
runner CI queue: a second self-hosted runner (igneum-build-2 joins the pool label), docs-only pushes skip the compile jobs, the red watcher pinned to the box that posts 2026-10-07 13:20:18 +00:00
workers Worker dashboard: one server section per box, per-box live feeds, installer and pusher take build-2 and build-3 2026-10-07 10:37:33 +00:00
lib.sh Build boxes: the box probe carries its own ssh options (bash 3.2 under set -u refused the unset BS_SSH_OPTS array before bs_host built it; every unpinned route on the Mac died, the pool lane 7 Oct 2026); the router check runs the ssh path under /bin/bash 2026-10-07 13:16:39 +00:00
provision.sh CI queue: a second self-hosted runner (igneum-build-2 joins the pool label), docs-only pushes skip the compile jobs, the red watcher pinned to the box that posts 2026-10-07 13:20:18 +00:00
README.md Build boxes: one host file per box and a route by class (suites and benches to build-2, proving to build-3, the rest to build-1); the no-mining rule in a box README and as a guard in the capacity layer 2026-10-07 10:34:16 +00:00
remote-run.sh Build boxes: the class router is a preference with spill-over (a held or overloaded box hands the job to the other one); build-2 gets a third slot and every run there is bounded on its own 32-core band; the spill decision in the first route line, the slot label and the JSONL row 2026-10-07 13:08:19 +00:00
run-from-mac.sh Build boxes: the slot holder keeps its own line (a keeper, with its self-test in the gate); an explicit --jobs is clamped for bounded classes; one git remote per box; the host-file double suffix 2026-10-07 11:22:12 +00:00

README.md

The build boxes (infra/build-server)

Three Hetzner dedicated servers in Falkenstein run everything the Mac must not: builds, test suites, benchmarks, CPU proving, the devnet hands and the observer. The Mac keeps macOS binaries, the DMG and Metal tests (CLAUDE.md, "Running agents on this Mac").

No mining on any Hetzner box, ever

the project lead's rule through main, 7 October 2026: Hetzner's policies forbid crypto mining. The boxes run nodes, builds, tests, benchmarks and CPU proving only. The pool's fast-time network runs its miners on rented GPU pods (tools/fleet), never on a box; a box may run the network's nodes. The capacity layer refuses a job that would start igneum-miner mine or a GPU worker (capacity/run.sh), and no hands unit carries a miner. A node started with --enable-unsynced-mining is a node flag, not a miner; nothing feeds it blocks here.

The boxes and the kind map

Box Host file on the Mac Takes Never
igneum-build-1 (188.40.146.49, AX162-1-LTD) ~/.config/igneum/build-server release gates (--priority gate), builds and cross-builds, checks, the GPU workers' host side, the devnet hands (node 1, the observer node, the observer), the Devnet 2 seed, the CI runner, the dashboard feed suites and benches once box 2 exists
igneum-build-2 (AX162-1, on order) ~/.config/igneum/build-server-2 suites (cargo test), benches (cargo bench), the attack rows (--box 2) gates, hands
igneum-build-3 (AX102-1, on order) ~/.config/igneum/build-server-3 proving and aggregation CPU work (proving/igneum-prove builds and suites), the second prover's shadow runner, the pool's fast-time NETWORK (nodes only, --box 3) miners of any kind

tools/build-remote.sh routes by class (lib.sh bs_route): suite and bench to box 2, the proving crate to box 3, everything else to box 1; --box N overrides; a class whose box has no host file yet falls back to box 1 and says so. --priority gate always runs on box 1. Each box has its own mirrors, slots, locks and JSONL log under /srv; run-from-mac.sh --box N <ip> provisions a box and writes its host file; the dashboard collector reads every box it is told about.

Files

File What
provision.sh the box itself: install mode (rescue system, Ubuntu 24.04, RAID 1, no swap) and provision mode (user build, toolchains, the pin, sccache, zig, CUDA headers, docker, Caddy, mirrors, slots, sshd, ufw)
run-from-mac.sh ships provision.sh, writes the host file, wires the build remotes and pushes every branch
lib.sh, remote-run.sh the Mac and box halves of a remote run: sync, checkout, slots, scheduling classes, the JSONL line
hands/ the devnet hands' units, the mover and the restart read-backs
capacity/ the capacity layer (the box-work lane's): background jobs under the build slots, never a miner
repro/, night/, prover/, runner/, workers/ other lanes' pieces that live on the boxes

Plan, numbers and the gotchas: docs/plans/build-server.md; the hands: docs/plans/hands-on-build-1.md.