igneum/docs/plans/ci-self-hosted.md
igneum-josh f0e74edaae Box: GitHub Actions runner as user runner, two build slots with 90 or 45 jobs, the measure hold, the CPU prover trial
- provision.sh step_runner: actions/runner 2.338.0 (sha256 checked) at /opt/actions-runner under a dedicated user `runner`
  (no sudo, not in build's group), rustup 1.99.0 pinned with both targets, sccache against /srv/sccache in READ_ONLY mode
  on its own server port, Node 22 and mingw from the system, GitHub's svc.sh unit with a Nice 10 drop-in; registered on
  igneum-network/igneum as igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) through
  infra/build-server/runner/register.sh (gh as igneum-josh, the token on ssh stdin, never logged). Idempotent after
  the env files moved behind svc.sh install (its env.sh rewrites them). libicu74 and python3-numpy added to APT.
- main's slots ruling: SLOTS default 2; remote-run.sh sets CARGO_BUILD_JOBS 90 when it holds the only taken slot and 45
  when both are held, BR_MEASURE=1 takes the `measure` file exclusively and excludes builds (builds hold it shared),
  lock files open in append mode (the old `exec {fd}>` truncated a busy slot's holder line on every probe), env
  IGNEUM_BUILD_SLOTS_DIR and IGNEUM_BUILD_LOG_DIR win over the profile, `--self-test-slots` with five cases (the old
  script fails it with JOBS=none); build-remote.sh and cross-remote.sh pass -j only when --jobs is given.
- infra/build-server/prover/cpu-trial.sh: the SP1 CPU prover on one fixture shard under the measure hold with a VmHWM
  poller; 6 Oct 2026 run: core 34.2 s, compressed 85.9 s, peak RSS 28.2 GB on 96 threads, so no standing CPU prover.
- docs/plans/ci-self-hosted.md: the proposed runs-on change for ci.yml behind the repository variable IGNEUM_CI_RUNNER
  (GitHub-hosted is the fallback), and why windows.yml cannot move to a Linux box. Workflows untouched.
- docs/plans/build-server.md section 7.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 20:32:43 +01:00

4.8 KiB

CI on the box: the self-hosted runner and the workflow change (proposal, 6 October 2026)

The runner igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) is installed by infra/build-server/provision.sh step_runner and registered by infra/build-server/runner/register.sh (docs/plans/build-server.md section 7). The workflows are NOT changed here: the shipper owns .github/workflows tonight. This is the proposed diff for main.

1. The shape: one repository variable decides, GitHub-hosted is the fallback

GitHub has no "try this runner, else that one" in runs-on: a list of labels means ALL of them must match one runner, so [self-hosted, igneum-build-1, ubuntu-latest] would never schedule. The fallback is therefore a repository variable read in the expression. IGNEUM_CI_RUNNER = box sends the job to the box; unset or anything else keeps ubuntu-latest. Flipping it back is one click in Settings > Secrets and variables > Actions > Variables (or gh variable set IGNEUM_CI_RUNNER --body box and gh variable delete IGNEUM_CI_RUNNER as igneum-josh), with no commit and no queue lost: a job already queued for the box stays queued; the next push goes to GitHub's machines.

2. ci.yml (the two jobs that compile or compute; the site job stays on GitHub's machines)

 jobs:
   pow:
     name: igneum-pow tests, igneum-census build
-    runs-on: ubuntu-latest
+    runs-on: ${{ vars.IGNEUM_CI_RUNNER == 'box' && fromJSON('["self-hosted", "linux", "x64", "igneum-build-1"]') || 'ubuntu-latest' }}
     steps:
       - uses: actions/checkout@v4
       - name: toolchain
         run: rustc --version && cargo --version
@@
   sims:
     name: simulators, quick modes
-    runs-on: ubuntu-latest
+    runs-on: ${{ vars.IGNEUM_CI_RUNNER == 'box' && fromJSON('["self-hosted", "linux", "x64", "igneum-build-1"]') || 'ubuntu-latest' }}
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
+        if: vars.IGNEUM_CI_RUNNER != 'box'   # the box has python3 and numpy from provision.sh; setup-python would download a second Python
         with:
           python-version: '3.12'
-      - run: python3 -m pip install --quiet numpy
+      - run: python3 -m pip install --quiet numpy
+        if: vars.IGNEUM_CI_RUNNER != 'box'

What the box gives these two jobs: rustc 1.99.0 pinned (GitHub's ubuntu-latest carries whatever stable it ships; the box is the Mac's version, so CI compiles what the agents compile), sccache hits from the agents' cache (read-only), 48 cargo jobs. The site job is Node and shell checks and takes under a minute on GitHub's runners; moving it buys nothing and would put tools/ci/public-api-check.mjs (a live HTTPS check) behind the box's egress for no reason.

Why the toolchain line still runs: on the box rustc --version must print 1.99.0; a mismatch means provision.sh and the runner's rustup disagree (R5 in build-server.md) and the job should say so in its first step.

3. windows.yml: no change possible on this box

Every job of windows.yml runs on windows-latest for a reason the box cannot answer: the engine builds on the MSVC target, the window host needs the Windows SDK and WebView2, the installer needs Inno Setup, the smoke run executes the exes and the launcher under Windows PowerShell 5.1. A Linux runner has none of that. The only self-hosted option for this workflow is a Windows runner on PC 1 or PC 2 (actions/runner for Windows under a service account), which conflicts with the rule that the PCs keep only GPU and Windows-runtime JOBS through the signed job system, and is not proposed tonight.

What the box already does for Windows is upstream of this workflow: tools/cross-remote.sh builds igneumd.exe and igneum-miner.exe (the payload inputs) in 1 min 44 s, and the night battery rebuilds them for the reproducibility record.

4. What to check after the flip (main, the first run on the box)

Check Where Pass
the job landed on the box the run's "Set up job" log says Runner name: 'igneum-build-1' yes
the toolchain the toolchain step prints rustc 1.99.0 yes
sccache hits add sccache --show-stats as a step once, or read /srv/sccache size before and after: the runner's config is READ_ONLY, so the size must NOT change size unchanged
the agents were not starved /srv/builds/_log/builds.jsonl secs of the builds during the run against the same crate's earlier lines within the usual spread
the fallback gh variable delete IGNEUM_CI_RUNNER, push a no-op commit: the job runs on ubuntu-latest again yes

Open: a CI job on the box does not take a build slot (/srv/builds/_locks/build-<k>), it runs at Nice 10 with 48 jobs; if a CI job ever delays a release build visibly, the fix is a step at the top of the job that takes a slot through infra/build-server/remote-run.sh's flock, the same file the agents use.