igneum/packaging/linux/README.md

# Igneum rig on Ubuntu 24.04

`install-rig.sh` turns an Ubuntu 24.04 machine with up to eight GPUs into an Igneum mining rig run by systemd:
one node, one miner per card (CUDA on NVIDIA, OpenCL on AMD and Intel), a prover unit, a telemetry unit, an hourly
signed-manifest update timer and a `rig-status` command. Built on 5 October 2026 from the HiveOS package
(`packaging/hive`: the hooks, the glibc ceiling, the consensus override rule, the stop path and the sync wait learnt
on PC 1 that night) for the rig the project lead is building (Threadripper PRO, 4 RTX 5090 or 4090, 2 RX 9070 XT, 2 Arc B580,
NVIDIA driver 580 or newer; HiveOS is out because its image's driver predates the RTX 50 series).

**Mining works from the package as published. Proving does not yet: the HiveOS package carries `igneumd`,
`igneum-miner` and the two workers, no `igneum-prove-host`, no `igneum-prove-export`, and its `igneum-miner` has no
`key-hash` or `sign-record` subcommand. The prover unit installs, decides by the per-card rule below, and then idles
in state `setup` naming what is missing. Until a Linux prover build is published, a rig mines only and the proving
share is earned from the app machines.** The same sentence belongs on the miners page (`site/miner.html` says the
card mines and proves) until the package carries the prover.

**Nothing here has run on a rig.** Everything below marked tested ran on this Mac (no systemd, no GPU, no Docker) on
5 October 2026; the "untested until a rig exists" list is the truth of the state.

## What the project lead types

```
git clone <the repo> && cd igneum
sudo packaging/linux/install-rig.sh --wallet 0x<40 hex> --name rig1 --allow-sidecar-sha256 \
     [--relay-key-file ~/log-intake-key] [--network devnet|testnet] [--prover auto|on|off]
rig-status
```

Asked once when not given: the payout wallet (0x and 40 hex, an EVM address he holds the key for; lower-cased and
stored) and the rig name (letters, digits, `-` and `_`, up to 32; it labels the rig's vote keys as `<name>-<card>`,
its payouts, and the console card as `<name>-<id8>`). Everything else has a default in `/etc/igneum/rig.conf`
(the file is written once and kept on re-runs; edit it, then `systemctl restart igneum-node`, the miners follow).

`--allow-sidecar-sha256` is needed today: the signed public manifest names only the mac and windows builds, so the
package's sha256 comes from `igneum-downloads.json` and the `.sha256` sidecar on the same TLS host, unsigned. The
installer says so in capitals. The follow-up that removes the flag: `packaging/ota/publish-public.sh --hive` adds a
`platforms.linux` entry (url, sha256, size, kind) to the public app manifest and re-signs it; the apps ignore the
extra key (`manifest.rs parse` reads mac and windows only), and `install-rig.sh` and `igneum-update.sh` already
prefer that entry when it exists. `--package-url --package-sha256 --package-size` is the hand path.

`--relay-key-file` installs the log-intake key (the same upload-only key every app ships) so the rig's journals reach
the console every 60 s; without it nothing leaves the rig. `--preflight-only` runs the checks alone; `--dry-run`
prints every step and only downloads into a scratch folder.

## What the installer assumes

| Assumption | Why | Checked by the preflight |
|---|---|---|
| Ubuntu 24.04 on x86_64 with systemd, OpenSSL 3 (or python3-cryptography), curl, python3, tar, flock, pciutils | the units, the Ed25519 check, the package (built with `GLIBC=2.27`: igneumd 2.27, miner 2.25, workers 2.17, read from the ELFs of 0.3.9) | yes; `--force` goes on anyway |
| NVIDIA driver 580 or newer, `libcuda.so.1` and `libnvrtc.so.12` on the library path | the project lead's floor for the RTX 50 series; the CUDA worker dlopens NVRTC 12 and compiles the hourly program (`infra/cross/build-workers-linux.sh`); a CUDA 13 toolkit's `libnvrtc.so.13` does not satisfy it | yes, failure |
| AMD: amdgpu bound, `libOpenCL.so.1`, an AMD ICD in `/etc/OpenCL/vendors` (ROCm 6.4 or newer for RDNA 4, kernel 6.11 or newer; both approximate) | the OpenCL worker compiles through the ICD | ICD and library failure, versions warning |
| Intel: xe (or i915) bound, `intel-opencl-icd` from Intel's compute-runtime repository (a 2025 release for Battlemage, kernel 6.12 or newer; approximate) | same | ICD failure, kernel warning |
| 20 GB free on /var/lib and /opt; 8 GB RAM to mine, 16 GB to prove | chain data under /var/lib/igneum/node, releases under /opt/igneum; the SP1 host side measured 2.3 GB inside WSL2 on 5 October 2026 (approximate for bare Linux) | yes |
| Ports 26611 (devnet P2P) or 26811 (testnet) free; RPC 26610/26810 and EVM RPC 26790/26890 on loopback | the apps' port layout (`config.rs evm_port` = rpc + 180) | yes; ufw rule added when ufw is active |
| The integrated GPU is not a card: AMD APUs report under 2 GiB of VRAM carve-out, an Intel iGPU sits at `0000:00:02.0`, the BMC's ASPEED VGA is vendor 1a03 | the rule the app's telemetry uses (kind integrated) rebuilt from sysfs; approximate | printed as notes |
| CUDA device index = PCI bus order (`CUDA_DEVICE_ORDER=PCI_BUS_ID` in the units); OpenCL index = the card's position among its vendor's devices in `igneum-worker-opencl --list` | OpenCL lists no bus id; two identical AMD cards are told apart by list order only, as the app does | the list is printed at install |
| Vote keys are derived from labels (`VoteSecretKey::from_label`; `--identities N` gives `<label>-1..N`) | no key store to back up; a rig's keys are its name plus its card ids; a renamed rig has new keys | n/a |

## The units

| Unit | Runs | Notes |
|---|---|---|
| `igneum-node.service` | `igneumd --devnet` (or `--testnet`) with the apps' flags, `--override-params-file /etc/igneum/override-params.json` from the verified manifest's `consensus.override`, the package's own file as the offline fallback | the HiveOS rule: without the override the devnet seed refuses the node (digest mismatch) |
| `igneum-miner@<card>.service` | one `igneum-miner` with `igneum-worker-cuda` or `igneum-worker-opencl`; waits for `synced=true` (SYNC_WAIT 3600 s), exports the pack once for all cards under a lock, re-exports on exit 42 | `PartOf=igneum-node`: a node restart restarts every miner; user igneum in video and render |
| `igneum-prover.service` | the shard loop of `app/igneum-app/src/prover.rs` (proving-v1) in bash: keys from `igneum-miner key-hash`, `igneum_getAssignedShards`, export, cut, `igneum-prove-host --mode compressed` with `SP1_PROVER=cuda` on the chosen card, `sign-record`, `igneum_submitProofRecord`; state in `/run/igneum/prover.state` | not ported: the v1 aggregator step and the paid-shard poll; pauses the card's miner through a sudoers rule when the card is under the mine-and-prove line |
| `igneum-telemetry.service` | every 5 s one line per card: NVIDIA from `nvidia-smi` (the app's query), AMD from the amdgpu sysfs in the exact `gpu-telemetry.c` line, Intel from the xe hwmon in the same shape; every 30 s the app's `status:` line, every 300 s its `stability:` line per card; every 60 s the journal tails to the intake as `nodelog-linux-<id8>`, `miner-<vendor>-<id8>-<n>`, `linux-<id8>` | the console (`relay/lib/parse.mjs`) now accepts the `linux` os; the relay must be redeployed for that |
| `igneum-update.timer` + `.service` | hourly (15 min after boot, 10 min jitter): verify the manifest; a changed `consensus.override` is written and the node restarted; a newer package (signed entry, or sidecar when `PACKAGE_SOURCE=sidecar`) is downloaded, checked, unpacked under `/opt/igneum/releases/<v>`, switched and restarted; rollback when the new node does not answer in 90 s | no restart while `prover.state` says proving; a deferral older than 6 h applies anyway (the app's `SAFE_MOMENT_PATIENCE_S`); the miners' hourly program boundary is not consulted (the app does; a follow-up) |
| `rig-status` | per card: MH/s (the last STATUS line's `now=`), W and GPU C (the last telemetry line), MH/W, accepted, rejected, unit state, STATUS age; node blocks, headers, daa, peers, sync; prover state; update line | read-only |

Layout: scripts in `/opt/igneum/bin`, releases in `/opt/igneum/releases/<version>` with `current` a symlink,
config in `/etc/igneum` (`rig.conf` 0640 root:igneum, `machine-id` 16 hex, `override-params.json`,
`log-intake-key`), data in `/var/lib/igneum` (`node`, `packs`, `proving`, `updates`), runtime state in `/run/igneum`.
Logs are journald only: `journalctl -fu igneum-miner@nvidia0`.

## The per-card rules and what they mean

| Rule | Value | Source | Consequence per tier |
|---|---|---|---|
| Identities per card | 8 for 8 GiB or more (or unknown), else 2; `IDENTITIES=N` overrides | `app/igneum-app/src/detect.rs` (proving-v1), the app's rule; the HiveOS README's "8 for a big card, 2 for a small one" now has its threshold | an 8 GB card runs 8 vote keys; a 6 GB card 2 (fewer blue blocks per key, the same hashrate) |
| Prover on by default | NVIDIA card with 11,776 MB or more (the 12 GB gate), the biggest card proves; `PROVER=on` or `off` wins | `app/igneum-app/src/provedefault.rs` (proving-v1); the brief said 16 GB, the branch's constant is 12 GB and is what the app ships, so the rig follows the branch and the line below handles the difference | 8 GB: off. 12 and 16 GB: on, with the miner paused per shard. 24 and 32 GB: on, mining and proving at once. AMD and Intel: off (no CUDA prover) |
| Mine and prove on one card | 20,480 MB or more; under it the card's miner stops for each shard (`PROVER_PAUSE_MINER=auto`; `always` and `never` force it) | `docs/bench-log.md` on proving-v1, "Step 1, the prover default and its cost": 15.6 GB measured for the miner and the prover together on one card | a 16 GB card loses its hashrate for the shard's duration (10.9 s of proving on a 5090 for one shard, bench-log; the pause is the whole export-cut-prove-sign round, longer); a 24 GB card loses nothing |
| RAM | 8 GB to mine, 16 GB to prove (warning, not failure) | 2.3 GB inside WSL2 on PC 1 (5 October 2026), the rest approximate | a 4 to 8 GB rig board mines; proving on it is a warning until measured on bare Linux |
| SP1 GPU server | downloaded by the SDK on the first `SP1_PROVER=cuda` run into the igneum user's `~/.sp1/bin` (home is `/var/lib/igneum`) | `proving/windows-wsl2/setup-wsl.sh` (134 MB for v6.8.1; 251 MB reported for CUDA 12.8; both approximate) | the first proof waits for the download; the 20 GB free-disk check covers it |
| Sync wait | 3600 s cap, miners start at `synced=true` | PC 1 under WSL2, 5 October 2026: miners started during IBD rebuilt their pack on every epoch seed move | a fresh rig mines 5 to 8 min after the node starts on the devnet (runs 2 to 4 of the HiveOS test); longer on a bigger chain |
| Package glibc | ELFs want GLIBC_2.27 (igneumd), 2.25 (miner), 2.17 (workers) | read from the 0.3.9 package tonight | runs on any Ubuntu from 18.04 up; 24.04 has 2.39 |

## Tested on 5 October 2026 (this Mac, no rig)

`packaging/linux/selftest.sh`, all passing:

| What | Result |
|---|---|
| `bash -n` and `shellcheck -x -S style` (0.11.0) on the installer, the library, the 7 runtime scripts, the checker and the self-test | clean |
| `check-units.sh`: the 6 unit files against the directive lists of systemd.unit/service/timer/exec(5), enumerated values, Exec paths against `bin/`, template `%i`, cross-references | 6 checked, 0 failures; `systemd-analyze verify` is NOT available here (no systemd, no Docker) and runs on the rig through the same script |
| `igneum-gpus.sh` on a fake sysfs tree: 2 NVIDIA, 2 AMD, 1 Intel Arc, plus an AMD APU, an Intel iGPU, an ASPEED BMC VGA and a NIC | 5 cards in PCI order per vendor, the four non-cards excluded with a note each |
| the identities rule, the vote-key labels, the prover rule (unknown VRAM off, `PROVER=on` picks nvidia0 and pauses), the OpenCL index mapping on a fake `--list` (amd1 to 1, intel0 to 2, intel1 to 3, amd2 to none) | as designed |
| the LIVE signed manifest: fetched, Ed25519 verified with the OTA public key through OpenSSL 3.6.4 (`pkeyutl -rawin`, the Ubuntu path) and through python3 cryptography (the fallback); a tampered copy and a flipped signature refused by both | version 0.3.9, override with the four fields |
| `install-rig.sh --dry-run`: the preflight (fails here, as expected), the manifest, the sidecar entry, the 24,179,978-byte `igneum-hive-0.3.9.tar.gz` downloaded and checked against the sidecar sha256 `7a58a30f...` and the size, the four binaries and `override-params.json` listed, the glibc floors read, then 31 printed steps (user, folders, machine id, release, symlink, override file, rig.conf, key, scripts, units, sudoers, enable, start) | nothing under / touched |
| `relay/test/parse.test.mjs` with the `linux` labels | 6 tests pass |

## Untested until a rig exists

- Every unit under a real systemd: start order, `PartOf` restarts, `ProtectSystem=full` and `ReadWritePaths` against
  what the binaries open, the `RuntimeDirectory` shared by five units, `SuccessExitStatus=42`, the stop path
  (TERM to `igneum-miner.sh` must end the miner and the worker; the HiveOS script needed a descendant walk).
- `systemd-analyze verify` itself (`check-units.sh` runs it when present).
- The preflight on real drivers: the driver and CUDA version parse of `nvidia-smi`, `ldconfig -p` for
  `libnvrtc.so.12`, the ROCm and Intel ICD files, `ss` and `ufw`.
- The CUDA index assumption (`CUDA_DEVICE_ORDER=PCI_BUS_ID` against the sysfs PCI order) and the OpenCL ordinal
  mapping on a real `--list` with AMD and Intel platforms both installed; whether the OpenCL worker runs at all on an
  Arc B580 (never tried on Intel; the RDNA 4 measurements are on the telemetry branch).
- The AMD sysfs telemetry on RX 9070 XT (the file names come from `proto-opencl/gpu-telemetry.c`, measured on PC 1 under
  Windows ADLX, not Linux sysfs), the Intel xe hwmon files (guessed: `power1_input` or `energy1_input`, `temp1_input`
  or `temp2_input`), the `stability:` p95 over mawk.
- The relay upload from journald tails: the label parse is tested, the upload and the console card are not; the relay
  needs a deploy for the `linux` os (`cd relay && npx vercel deploy`, `relay/README.md`).
- The whole prover unit: no Linux `igneum-prove-host` is published; the loop was ported from `prover.rs` by reading,
  the JSON shapes (`igneum_getAssignedShards`, the results file, `sign-record`'s last line) are taken from that file;
  the miner pause through sudo; `CUDA_VISIBLE_DEVICES` with SP1's GPU server.
- The update path: the hourly timer, a real override change (the node restart on the next switch), a package switch
  and the 90-s rollback; `PACKAGE_SOURCE=sidecar` on a rig means the version in `igneum-downloads.json` drives it.
- The testnet: `--network testnet` uses `--testnet`, the three DNS seeds and ports 26810/26811/26890 from
  `docs/testnet/README.md`; nothing mines there until the owner's go.

## Files

| File | What |
|---|---|
| `install-rig.sh` | the installer (`--help`) |
| `bin/igneum-rig-lib.sh` | paths, `rig.conf`, the manifest fetch and Ed25519 check, the package download and sha256 check, the inventory and prover rules, the relay upload |
| `bin/igneum-gpus.sh` | the card inventory from `/sys/bus/pci` (`IGNEUM_SYS_ROOT` for a fixture tree) |
| `bin/igneum-node.sh`, `igneum-miner.sh`, `igneum-prover.sh`, `igneum-telemetry.sh`, `igneum-update.sh` | the units' ExecStart scripts |
| `bin/rig-status` | installed to `/usr/local/bin/rig-status` |
| `units/*.service`, `units/igneum-update.timer` | installed to `/etc/systemd/system` |
| `check-units.sh` | the static unit check, plus `systemd-analyze verify` where it exists |
| `selftest.sh` | everything in the tested table above |