190 lines
16 KiB
Markdown
190 lines
16 KiB
Markdown
# Over-the-air updates (packaging/ota)
|
|
|
|
the project lead's rule (4 October 2026): every app updates itself and downloads the update without being asked. The Igneum
|
|
Miner app on Windows and macOS does, and the node, the miner and the GPU workers ship inside it, so a consensus
|
|
upgrade (a height-activated rule such as difficulty v2) reaches every node before its activation height.
|
|
|
|
The launcher packages (`proto-cuda/windows-app`, the `igneum-windows-v4.zip` console launchers, the Terminal DMGs
|
|
0.1.0 and 0.2.0) are NOT auto-updated, by design: they are the engineering path and are replaced by hand.
|
|
|
|
## The pieces
|
|
|
|
| Piece | Where | What it does |
|
|
|---|---|---|
|
|
| manifest | `dl.igneum.network/dl/<token>/igneum-app-latest.json` + `.sig` | version, per-platform file (url, sha256, size, kind), min_supported_version, notes, consensus activation height |
|
|
| signer | `app/igneum-app/src/bin/ota-sign.rs` (`igneum-ota-sign`, built with the app, never shipped) | keygen, sign, verify, sha256; includes `src/manifest.rs` so it signs what the app verifies |
|
|
| publisher | `packaging/ota/publish-manifest.sh` | copies the DMG or installer into the downloads folder, writes the canonical manifest, signs it, prints or runs the deploy |
|
|
| verifier | `app/igneum-app/src/manifest.rs` | Ed25519 check of the manifest bytes with the compiled-in public key, parse, version compare, safe-moment rule, unit tests |
|
|
| updater | `app/igneum-app/src/ota.rs` | check, download with resume, verify, stage, apply at a safe moment, rollback; `update` in `/api/state` |
|
|
| dashboard | `app/igneum-app/ui` | one banner (available, downloading, ready, applying, red bar for a close fork), Settings: Check now, Install now, automatic switch |
|
|
| Windows installer | `packaging/windows/Igneum-Miner.iss` | `CloseApplications=yes`, `RestartApplications=no`, a `[Run]` relaunch on `/IGNOTA=1` |
|
|
| CI loop | `packaging/windows/fetch-ci-artifacts.sh` | after copying the installer it calls `publish-manifest.sh --win` (the Mac entry is carried over); `--deploy` ships both |
|
|
|
|
## Keys
|
|
|
|
Generated once on the Mac (4 October 2026), never in the repo or in CI:
|
|
|
|
app/igneum-app/target/release/igneum-ota-sign keygen ~/.config/igneum/ota-signing-key ~/.config/igneum/ota-signing-key.pub
|
|
|
|
`ota-signing-key` is the 32-byte seed as hex, mode 0600. The public key is the constant `OTA_PUBLIC_KEY_HEX` in
|
|
`app/igneum-app/src/manifest.rs`; `igneum-ota-sign embedded` prints it with its fingerprint (SHA-256 of the 32 key
|
|
bytes). `publish-manifest.sh` refuses to sign when the embedded key is not the one in `~/.config/igneum`.
|
|
|
|
Key rotation: a new key means a new app build (the constant), published and signed with the OLD key, then the next
|
|
manifest signed with the new one. Apps that skipped the bridge build stop updating and show "manifest signature does
|
|
not verify"; they are updated by hand from the download page.
|
|
|
|
## Publishing a version
|
|
|
|
1. Bump `version` in `app/igneum-app/Cargo.toml` (and `app/windows/version.h`, `resources/igneum-app.rc`, as the
|
|
CI smoke run demands). Commit, push: the Windows installer builds on GitHub.
|
|
2. Mac: `packaging/mac/build-dmg.sh`, then
|
|
|
|
packaging/ota/publish-manifest.sh --version 0.3.1 --mac packaging/mac/dist/Igneum-Miner-0.3.1.dmg \
|
|
--notes "difficulty v2 and over-the-air updates" [--activation-height 120000 --deadline-note "difficulty v2"]
|
|
|
|
writes `dl/<token>/igneum-app-latest.json` with the Mac entry only and prints the deploy command. Deploying now is
|
|
fine: a Windows app finds no `windows` entry and does nothing.
|
|
3. Windows: `packaging/windows/fetch-ci-artifacts.sh --deploy` copies the installer, adds the Windows entry to the
|
|
same manifest (same version, Mac entry carried over), deploys the downloads folder.
|
|
4. Every app checks within the hour (`Settings > Check now` at once): it downloads, verifies and installs at the next
|
|
safe moment. The event feed shows each step; `app-<run>.log` has the detail.
|
|
|
|
`--min-supported 0.3.0` marks older versions unsupported: they install at once, without waiting for a safe moment,
|
|
and show the red bar. `--activation-height N` does the same once a node's DAA score is within 1,800 blocks of N.
|
|
|
|
## What the app does
|
|
|
|
Check on start (20 to 50 s in) and every 60 minutes plus up to 10 minutes of per-machine jitter; after an error,
|
|
again in 10 minutes. Both files come through curl (the engine carries no TLS stack); the signature is checked over
|
|
the manifest bytes before parsing; a version that is not newer, or a manifest without this platform, ends the round.
|
|
|
|
Download into `<app data>/app/updates/` (`~/Library/Application Support/Igneum/app/updates`,
|
|
`%LOCALAPPDATA%\igneum\app\updates`) with `curl -C -` (resume) and `--retry 3`; then the size and the sha256 from
|
|
the manifest; a file already there with the right hash is not fetched again. The banner shows the percentage.
|
|
|
|
Stage. macOS: mount the DMG (or unpack the zip), copy `Igneum Miner.app` to `.Igneum Miner.app.new` next to the
|
|
running bundle (same volume: the swap is two renames), run its engine with `--version` and demand the manifest's
|
|
version. When the folder is not writable the state is `manual`: the banner says so and offers "Open the download".
|
|
Windows: the installer is the staged file.
|
|
|
|
Safe moment (`manifest::safe_to_apply`): the network has not lost over 30% of its identities in 10 minutes, this
|
|
minute is the machine's slot, node synced, no hourly program boundary within 180 s (`program.eta_s`), no worker
|
|
starting. Urgent (fork within 1,800 blocks, or unsupported version, or Install now) skips the wait. A ready update
|
|
that found no safe moment for 6 hours applies anyway in its slot (an unsynced node mines nothing).
|
|
|
|
Apply. The engine writes `update-pending.json` (from, to, starts), starts the helper detached and leaves through its
|
|
normal quit path: miners first (8 s grace), then the node (30 s), the last log upload, `EXIT` for the window.
|
|
- macOS helper `ota-apply.sh`: waits for the engine, asks the window (`network.igneum.miner`) to quit, moves the
|
|
bundle to `Igneum Miner.app.previous`, the staged one in, strips quarantine, `open -n`. If the new engine is not
|
|
running after 30 s it opens once more; if that fails too it puts `.previous` back and reports.
|
|
- Windows helper `ota-apply.ps1` (0.3.3, after the 4 October incident below): runs `Igneum-Miner-Setup-<v>.exe
|
|
/VERYSILENT /SUPPRESSMSGBOXES /NORESTART /CLOSEAPPLICATIONS /IGNOTA=1 /LOG=...` FIRST, with the engine still
|
|
mining. The installer is per-user since 0.3.3 (`PrivilegesRequired=lowest`,
|
|
`{localappdata}\Programs\Igneum Miner`), so nothing asks for an administrator; its `PrepareToInstall` runs
|
|
`stop-igneum.ps1` (api/quit: miners first, then the node, then the window), replaces the files, and the `[Run]`
|
|
entry on `/IGNOTA=1` relaunches `igneum-app.exe --launch`. An older install in Program Files (0.3.2 and before)
|
|
still raises one UAC prompt through ShellExecute: declined, timed out or unanswered, the helper writes
|
|
`deferred:true`, the engine keeps mining, logs `OTA: waiting for administrator approval` / `OTA: administrator
|
|
approval not given ...; mining continues`, shows "waits for the next time someone is at this PC" and retries on
|
|
Install now, at the next start, or after 6 hours. The engine is never stopped before the installer is running.
|
|
- Machines take turns: an update applies only in the machine's own minute of the hour (`manifest::slot_minute`:
|
|
the first 8 hex of the machine id modulo 60; PC 2 = :58), and never while `/api/live` shows the network lost
|
|
over 30% of its identities (`state.miners_10m`) in the last 10 minutes. Urgent (fork close, unsupported) and
|
|
Install now skip both.
|
|
- Per-user install, migration: the data (`%LOCALAPPDATA%\igneum`: chain, wallet, settings) is the same folder for
|
|
both installs, nothing is copied. A hand-run installer offers to remove the Program Files copy (one
|
|
administrator prompt for its uninstaller); a silent OTA install leaves it and says nothing. The login entry
|
|
(HKCU Run) is rewritten to the new path on first start. The inbound firewall rule for `igneumd.exe` is requested
|
|
once on the first run of a per-user install (one prompt, in a thread; declined or unanswered = the node dials out
|
|
and mines without it, never asked again).
|
|
|
|
Field incident, 4 October 2026 15:40 BST: 0.3.2 published; both Windows PCs (0.3.1, nobody at either keyboard)
|
|
reached apply, stopped the miners and the node, then sat at the installer's UAC prompt. The chain fell to one
|
|
laptop. The 0.3.1 helper waits for the engine to exit before it runs the installer, so a prompt nobody answers
|
|
strands the machine; 0.3.3 inverts the order (installer first, engine stops only when the installer runs) and the
|
|
per-user installer removes the prompt altogether. The 0.3.2 engines still carry the old helper: their 0.3.3 update
|
|
raises the prompt once more (with 0.3.2's R4.3.6 rule the version is then marked failed, not retried); click Yes
|
|
once, or run the 0.3.3 installer by hand.
|
|
|
|
Rollback. The helper writes `update-result.json`; the new engine reads it on start and reports "updated to X from
|
|
Y" or the error. The new engine counts its starts in `update-pending.json` and deletes the file after 90 healthy
|
|
seconds; a third start without reaching that point restores the previous version (macOS: the `.previous` bundle;
|
|
Windows: the previous version's installer kept in `updates/`, so the FIRST update from 0.3.0 has no rollback target
|
|
on Windows, said so in the state) and shows "rolled back" in Settings.
|
|
|
|
Settings: `auto_update` (default on). Off: downloads still happen, the banner waits for Install now. The forced
|
|
screenshots: `?update=available|downloading|ready|waiting|applying|urgent|manual|error|updated` on the dashboard URL
|
|
(and `?job=running|done|failed` for the remote-job notice). The dashboard shows one notice at a time in the strip under
|
|
the header, the most important first (app/igneum-app/ui/app.js, Notices).
|
|
|
|
## Testing
|
|
|
|
Unit tests: `cargo test` in `app/igneum-app` (manifest parse, a bad signature and a tampered manifest refused,
|
|
sha256 of a file, version ordering incl. pre-releases, safe-moment rules, fork closeness, unsupported versions).
|
|
|
|
Dry run on the Mac, 4 October 2026 (`packaging/mac/README.md` has the private-devnet recipe; ports 29700+):
|
|
the 0.3.0 bundle from the tree ran under its window host against a private devnet with
|
|
`IGNEUM_APP_UPDATE_MANIFEST=http://127.0.0.1:29790/dl/<token>/igneum-app-latest.json` (a `python3 -m http.server`
|
|
over a folder written by `publish-manifest.sh --base-url ... --dest ...`; loopback http is the one non-https URL the
|
|
parser accepts), found the 0.3.1 manifest, downloaded and verified the DMG, staged the bundle, waited for the worker
|
|
to start, applied, and came back as 0.3.1 with "updated to Igneum Miner 0.3.1 from 0.3.0" in the event feed. The
|
|
screenshots are `docs/design/app-screens/update-*.png`. Windows: reviewed only, see `TEST.md`.
|
|
|
|
## Remote jobs (4 October 2026)
|
|
|
|
the project lead's rule: one app on both PCs that the Mac can send commands and files to over the line, so everything is tested
|
|
and built without a person at the PC. The channel is `igneum-jobs.json` plus `igneum-jobs.json.sig`, next to the
|
|
update manifest, signed with the same OTA key and verified by the same code. Since 0.3.6 (5 October 2026) every app
|
|
holds a long-poll on the relay's public `/wake` and fetches the file within seconds of `--deploy` (the script records
|
|
the new stamp there); a poll every 2 minutes is the fallback (it was 10 minutes before 0.3.6).
|
|
The relay (`relay/`) stays for the Mac and for humans; its PC agent is replaced by this.
|
|
|
|
| Piece | Where |
|
|
|---|---|
|
|
| model: parse, verify, targeting, expiry, the once-per-id ledger (`jobs-state.json` in the app data folder), unit tests | `app/igneum-app/src/jobs.rs` |
|
|
| runner: poll, requirement probes, the kinds, reports, the dashboard state | `app/igneum-app/src/jobrun.rs` |
|
|
| signer: `igneum-ota-sign sign-jobs` and `verify-jobs` (a bad job is refused at signing) | `app/igneum-app/src/bin/ota-sign.rs` |
|
|
| publisher: `packaging/ota/publish-jobs.sh add|list|remove|sign [--deploy]` | this folder |
|
|
| reader: the published file (signature checked), status per machine, a job's result, watch | `tools/jobs.mjs` |
|
|
|
|
A job: `{id, kind, title, created_at, expires_at, target: {machine_ids: [...] | "all", platform, requires}, params,
|
|
report: "log-intake"}`. Machine ids are the per-install id (16 hex) or its first 8 (PC 1 `ae432dc7`, PC 2
|
|
`1ccfe586`). Requirements the engine probes: `wsl`, `wsl-prover` (cargo, `~/.sp1` and `~/igneum-prove` inside
|
|
Ubuntu-24.04), `nvidia`; an unknown one is never met and the job waits until it expires.
|
|
|
|
| Kind | What the app does | Params |
|
|
|---|---|---|
|
|
| `run` | writes the script body to `<app data>/app/jobs/<id>/` and runs it (powershell or bash), output captured, exit code reported | `script`, `shell`, `elevated`, `stop_miners_first`, `timeout_minutes` (60, at most 600) |
|
|
| `fetch` | downloads by https, checks the sha256 (and size), into the job folder or a named app folder; can extract | `url`, `sha256`, `size`, `dir` jobs/prove/packs/updates, `to`, `extract`, `extract_dir`, `fresh` |
|
|
| `collect` | uploads files matching globs under the app data root, and/or a command's output, to the intake | `globs`, `command` |
|
|
| `restart` | miners, node (the miners follow) or the app (a detached helper opens it again) | `what` |
|
|
| `update-now` | the updater checks and installs a newer version at once | |
|
|
| `shard-benchmark` | miners stopped, GPU under 5%, the prove zip fetched fresh, `prove-shard.sh` inside WSL under the cap, RESULT and STAGE lines and `results/*.json` uploaded, miners back | `zip_url`, `sha256`, `size`, `fixtures`, `cap_minutes` (90), `distro`, `wsl_user` |
|
|
| `build` | mining untouched; free space checked (20 GB on the drive and in the distro), the build-inputs zip fetched (sha256), setup inside WSL2 as root (mingw-w64, clang, protoc, zstd, the Windows rust target; idempotent), sources extracted, `cargo build --release` native and for `x86_64-pc-windows-gnu`, `cargo test` for the manifest's packages, binaries zstd-compressed and sent to the relay (50 MB each) with sha256 in RESULT lines; the target dir under `/root/igneum-build` persists between jobs. `src/jobbuild.rs`, `packaging/windows/push-build-inputs.sh`, `tools/build-job.mjs` | `zip_url`, `sha256`, `size`, `targets` (linux, windows), `budget_minutes` (40), `stage_minutes` {stage: min}, `min_free_gb` (20), `tests` (true), `relay_url`, `nice` (19), `cargo_jobs`, `distro`, `wsl_user` |
|
|
|
|
Reports: every job uploads to the log intake as run_id `job-<id>-<machine id8>` with the label `job-<kind>`; the
|
|
first line is `SUMMARY {json}` (status, exit, times, the RESULT lines, per-kind extras), then the captured output.
|
|
Long jobs upload a running report every 5 minutes; collected files and result files are separate labels
|
|
(`file-<name>`, `result-<name>`, `prove-log`) under the same run_id.
|
|
|
|
Safety: the file is rejected when the signature or any job's `expires_at` or params fail (since 0.3.4 a job of a kind the app does not know is skipped instead of rejecting the file; 0.3.3 and earlier reject the whole file, so a new kind reaches the PCs in an app update before its first job is published); a job id runs once per
|
|
machine (a crash mid-job counts); nothing writes outside the app data folder except an explicit `run` script,
|
|
which is the operator's responsibility; the dashboard shows the running job and a history (id, kind, started, exit,
|
|
report uploaded); Settings has "Allow remote jobs from Igneum (signed)", ON by default on this devnet build, with
|
|
the key fingerprint, and OFF aborts the running job. `elevated` needs someone at the UAC prompt (or UAC set to
|
|
elevate without prompting); unattended it fails after the prompt times out. A `restart app` or `update-now` job
|
|
relaunches through the usual quit path.
|
|
|
|
Publishing and reading:
|
|
|
|
packaging/ota/publish-jobs.sh add --kind shard-benchmark --target 1ccfe586 --title "Shard proof run on the 5090" --deploy
|
|
packaging/ota/publish-jobs.sh add --kind run --target ae432dc7,1ccfe586 --script fix.ps1 --title "..." [--elevated] [--stop-miners]
|
|
packaging/ota/publish-jobs.sh add --kind build --target ae432dc7 --deploy (after packaging/windows/push-build-inputs.sh; or node tools/build-job.mjs run)
|
|
packaging/ota/publish-jobs.sh list | remove <id> | sign
|
|
node tools/jobs.mjs | status | <id> [--all] | watch <id>
|
|
|
|
Tested on the Mac, 4 October 2026: the unit tests (parse, bad jobs refused, signature and tampering, times,
|
|
targeting, the once-only ledger, globs), the signer with the real key, the publisher against a scratch folder, the
|
|
reader against the live intake; the crate also compiles for `x86_64-pc-windows-gnu`. Not yet run on a PC: the
|
|
first job goes to PC 2 (`1ccfe586`) once 0.3.2 is installed there (`docs/plans/shard-test-pc2.md`).
|