7184e5bfc7
- provision.sh step_runner: actions/runner 2.338.0 (sha256 checked) at /opt/actions-runner under a dedicated user `runner`
(no sudo, not in build's group), rustup 1.99.0 pinned with both targets, sccache against /srv/sccache in READ_ONLY mode
on its own server port, Node 22 and mingw from the system, GitHub's svc.sh unit with a Nice 10 drop-in; registered on
igneum-network/igneum as igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) through
infra/build-server/runner/register.sh (gh as igneum-labs, the token on ssh stdin, never logged). Idempotent after
the env files moved behind svc.sh install (its env.sh rewrites them). libicu74 and python3-numpy added to APT.
- main's slots ruling: SLOTS default 2; remote-run.sh sets CARGO_BUILD_JOBS 90 when it holds the only taken slot and 45
when both are held, BR_MEASURE=1 takes the `measure` file exclusively and excludes builds (builds hold it shared),
lock files open in append mode (the old `exec {fd}>` truncated a busy slot's holder line on every probe), env
IGNEUM_BUILD_SLOTS_DIR and IGNEUM_BUILD_LOG_DIR win over the profile, `--self-test-slots` with five cases (the old
script fails it with JOBS=none); build-remote.sh and cross-remote.sh pass -j only when --jobs is given.
- infra/build-server/prover/cpu-trial.sh: the SP1 CPU prover on one fixture shard under the measure hold with a VmHWM
poller; 6 Oct 2026 run: core 34.2 s, compressed 85.9 s, peak RSS 28.2 GB on 96 threads, so no standing CPU prover.
- docs/plans/ci-self-hosted.md: the proposed runs-on change for ci.yml behind the repository variable IGNEUM_CI_RUNNER
(GitHub-hosted is the fallback), and why windows.yml cannot move to a Linux box. Workflows untouched.
- docs/plans/build-server.md section 7.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
75 lines
4.8 KiB
Markdown
75 lines
4.8 KiB
Markdown
# CI on the box: the self-hosted runner and the workflow change (proposal, 6 October 2026)
|
|
|
|
The runner `igneum-build-1` (labels `self-hosted, linux, x64, igneum-build-1`) is installed by `infra/build-server/provision.sh`
|
|
step_runner and registered by `infra/build-server/runner/register.sh` (docs/plans/build-server.md section 7). The workflows are
|
|
NOT changed here: the shipper owns `.github/workflows` tonight. This is the proposed diff for main.
|
|
|
|
## 1. The shape: one repository variable decides, GitHub-hosted is the fallback
|
|
|
|
GitHub has no "try this runner, else that one" in `runs-on`: a list of labels means ALL of them must match one runner, so
|
|
`[self-hosted, igneum-build-1, ubuntu-latest]` would never schedule. The fallback is therefore a repository variable read in
|
|
the expression. `IGNEUM_CI_RUNNER` = `box` sends the job to the box; unset or anything else keeps `ubuntu-latest`. Flipping it
|
|
back is one click in Settings > Secrets and variables > Actions > Variables (or `gh variable set IGNEUM_CI_RUNNER --body box`
|
|
and `gh variable delete IGNEUM_CI_RUNNER` as igneum-labs), with no commit and no queue lost: a job already queued for the box
|
|
stays queued; the next push goes to GitHub's machines.
|
|
|
|
## 2. ci.yml (the two jobs that compile or compute; the `site` job stays on GitHub's machines)
|
|
|
|
```diff
|
|
jobs:
|
|
pow:
|
|
name: igneum-pow tests, igneum-census build
|
|
- runs-on: ubuntu-latest
|
|
+ runs-on: ${{ vars.IGNEUM_CI_RUNNER == 'box' && fromJSON('["self-hosted", "linux", "x64", "igneum-build-1"]') || 'ubuntu-latest' }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: toolchain
|
|
run: rustc --version && cargo --version
|
|
@@
|
|
sims:
|
|
name: simulators, quick modes
|
|
- runs-on: ubuntu-latest
|
|
+ runs-on: ${{ vars.IGNEUM_CI_RUNNER == 'box' && fromJSON('["self-hosted", "linux", "x64", "igneum-build-1"]') || 'ubuntu-latest' }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-python@v5
|
|
+ if: vars.IGNEUM_CI_RUNNER != 'box' # the box has python3 and numpy from provision.sh; setup-python would download a second Python
|
|
with:
|
|
python-version: '3.12'
|
|
- - run: python3 -m pip install --quiet numpy
|
|
+ - run: python3 -m pip install --quiet numpy
|
|
+ if: vars.IGNEUM_CI_RUNNER != 'box'
|
|
```
|
|
|
|
What the box gives these two jobs: rustc 1.99.0 pinned (GitHub's `ubuntu-latest` carries whatever stable it ships; the box
|
|
is the Mac's version, so CI compiles what the agents compile), sccache hits from the agents' cache (read-only), 48 cargo
|
|
jobs. The `site` job is Node and shell checks and takes under a minute on GitHub's runners; moving it buys nothing and would
|
|
put `tools/ci/public-api-check.mjs` (a live HTTPS check) behind the box's egress for no reason.
|
|
|
|
Why the toolchain line still runs: on the box `rustc --version` must print 1.99.0; a mismatch means provision.sh and the
|
|
runner's rustup disagree (R5 in build-server.md) and the job should say so in its first step.
|
|
|
|
## 3. windows.yml: no change possible on this box
|
|
|
|
Every job of `windows.yml` runs on `windows-latest` for a reason the box cannot answer: the engine builds on the MSVC
|
|
target, the window host needs the Windows SDK and WebView2, the installer needs Inno Setup, the smoke run executes the exes
|
|
and the launcher under Windows PowerShell 5.1. A Linux runner has none of that. The only self-hosted option for this
|
|
workflow is a Windows runner on PC 1 or PC 2 (`actions/runner` for Windows under a service account), which conflicts with
|
|
the rule that the PCs keep only GPU and Windows-runtime JOBS through the signed job system, and is not proposed tonight.
|
|
|
|
What the box already does for Windows is upstream of this workflow: `tools/cross-remote.sh` builds `igneumd.exe` and
|
|
`igneum-miner.exe` (the payload inputs) in 1 min 44 s, and the night battery rebuilds them for the reproducibility record.
|
|
|
|
## 4. What to check after the flip (main, the first run on the box)
|
|
|
|
| Check | Where | Pass |
|
|
|---|---|---|
|
|
| the job landed on the box | the run's "Set up job" log says `Runner name: 'igneum-build-1'` | yes |
|
|
| the toolchain | the `toolchain` step prints `rustc 1.99.0` | yes |
|
|
| sccache hits | add `sccache --show-stats` as a step once, or read `/srv/sccache` size before and after: the runner's config is READ_ONLY, so the size must NOT change | size unchanged |
|
|
| the agents were not starved | `/srv/builds/_log/builds.jsonl` `secs` of the builds during the run against the same crate's earlier lines | within the usual spread |
|
|
| the fallback | `gh variable delete IGNEUM_CI_RUNNER`, push a no-op commit: the job runs on `ubuntu-latest` again | yes |
|
|
|
|
Open: a CI job on the box does not take a build slot (`/srv/builds/_locks/build-<k>`), it runs at Nice 10 with 48 jobs; if a
|
|
CI job ever delays a release build visibly, the fix is a step at the top of the job that takes a slot through
|
|
`infra/build-server/remote-run.sh`'s flock, the same file the agents use.
|