# Over-the-air updates (packaging/ota)

the project lead's rule (4 October 2026): every app updates itself and downloads the update without being asked. The Igneum
Miner app on Windows and macOS does, and the node, the miner and the GPU workers ship inside it, so a consensus
upgrade (a height-activated rule such as difficulty v2) reaches every node before its activation height.

The launcher packages (`proto-cuda/windows-app`, the `igneum-windows-v4.zip` console launchers, the Terminal DMGs
0.1.0 and 0.2.0) are NOT auto-updated, by design: they are the engineering path and are replaced by hand.

## The pieces

| Piece | Where | What it does |
|---|---|---|
| manifest | `dl.igneum.network/dl/<token>/igneum-app-latest.json` + `.sig` | version, per-platform file (url, sha256, size, kind), min_supported_version, notes, consensus activation height |
| signer | `app/igneum-app/src/bin/ota-sign.rs` (`igneum-ota-sign`, built with the app, never shipped) | keygen, sign, verify, sha256; includes `src/manifest.rs` so it signs what the app verifies |
| publisher | `packaging/ota/publish-manifest.sh` | copies the DMG or installer into the downloads folder, writes the canonical manifest, signs it, prints or runs the deploy |
| verifier | `app/igneum-app/src/manifest.rs` | Ed25519 check of the manifest bytes with the compiled-in public key, parse, version compare, safe-moment rule, unit tests |
| updater | `app/igneum-app/src/ota.rs` | check, download with resume, verify, stage, apply at a safe moment, rollback; `update` in `/api/state` |
| dashboard | `app/igneum-app/ui` | one banner (available, downloading, ready, applying, red bar for a close fork), Settings: Check now, Install now, automatic switch |
| Windows installer | `packaging/windows/Igneum-Miner.iss` | `CloseApplications=yes`, `RestartApplications=no`, a `[Run]` relaunch on `/IGNOTA=1` |
| CI loop | `packaging/windows/fetch-ci-artifacts.sh` | after copying the installer it calls `publish-manifest.sh --win` (the Mac entry is carried over); `--deploy` ships both |

## Keys

Generated once on the Mac (4 October 2026), never in the repo or in CI:

    app/igneum-app/target/release/igneum-ota-sign keygen ~/.config/igneum/ota-signing-key ~/.config/igneum/ota-signing-key.pub

`ota-signing-key` is the 32-byte seed as hex, mode 0600. The public key is the constant `OTA_PUBLIC_KEY_HEX` in
`app/igneum-app/src/manifest.rs`; `igneum-ota-sign embedded` prints it with its fingerprint (SHA-256 of the 32 key
bytes). `publish-manifest.sh` refuses to sign when the embedded key is not the one in `~/.config/igneum`.

Key rotation: a new key means a new app build (the constant), published and signed with the OLD key, then the next
manifest signed with the new one. Apps that skipped the bridge build stop updating and show "manifest signature does
not verify"; they are updated by hand from the download page.

## Publishing a version

1. Bump `version` in `app/igneum-app/Cargo.toml` (and `app/windows/version.h`, `resources/igneum-app.rc`, as the
   CI smoke run demands). Commit, push: the Windows installer builds on GitHub.
2. Mac: `packaging/mac/build-dmg.sh`, then

        packaging/ota/publish-manifest.sh --version 0.3.1 --mac packaging/mac/dist/Igneum-Miner-0.3.1.dmg \
            --notes "difficulty v2 and over-the-air updates" [--activation-height 120000 --deadline-note "difficulty v2"]

   writes `dl/<token>/igneum-app-latest.json` with the Mac entry only and prints the deploy command. Deploying now is
   fine: a Windows app finds no `windows` entry and does nothing.
3. Windows: `packaging/windows/fetch-ci-artifacts.sh --deploy` copies the installer, adds the Windows entry to the
   same manifest (same version, Mac entry carried over), deploys the downloads folder.
4. Every app checks within the hour (`Settings > Check now` at once): it downloads, verifies and installs at the next
   safe moment. The event feed shows each step; `app-<run>.log` has the detail.

`--min-supported 0.3.0` marks older versions unsupported: they install at once, without waiting for a safe moment,
and show the red bar. `--activation-height N` does the same once a node's DAA score is within 1,800 blocks of N.

## What the app does

Check on start (20 to 50 s in) and every 60 minutes plus up to 10 minutes of per-machine jitter; after an error,
again in 10 minutes. Both files come through curl (the engine carries no TLS stack); the signature is checked over
the manifest bytes before parsing; a version that is not newer, or a manifest without this platform, ends the round.

Download into `<app data>/app/updates/` (`~/Library/Application Support/Igneum/app/updates`,
`%LOCALAPPDATA%\igneum\app\updates`) with `curl -C -` (resume) and `--retry 3`; then the size and the sha256 from
the manifest; a file already there with the right hash is not fetched again. The banner shows the percentage.

Stage. macOS: mount the DMG (or unpack the zip), copy `Igneum Miner.app` to `.Igneum Miner.app.new` next to the
running bundle (same volume: the swap is two renames), run its engine with `--version` and demand the manifest's
version. When the folder is not writable the state is `manual`: the banner says so and offers "Open the download".
Windows: the installer is the staged file.

Safe moment (`manifest::safe_to_apply`): the network has not lost over 30% of its identities in 10 minutes, this
minute is the machine's slot, node synced, no hourly program boundary within 180 s (`program.eta_s`), no worker
starting. Urgent (fork within 1,800 blocks, or unsupported version, or Install now) skips the wait. A ready update
that found no safe moment for 6 hours applies anyway in its slot (an unsynced node mines nothing).

Apply. The engine writes `update-pending.json` (from, to, starts), starts the helper detached and leaves through its
normal quit path: miners first (8 s grace), then the node (30 s), the last log upload, `EXIT` for the window.
- macOS helper `ota-apply.sh`: waits for the engine, asks the window (`network.igneum.miner`) to quit, moves the
  bundle to `Igneum Miner.app.previous`, the staged one in, strips quarantine, `open -n`. If the new engine is not
  running after 30 s it opens once more; if that fails too it puts `.previous` back and reports.
- Windows helper `ota-apply.ps1` (0.3.3, after the 4 October incident below): runs `Igneum-Miner-Setup-<v>.exe
  /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /CLOSEAPPLICATIONS /IGNOTA=1 /LOG=...` FIRST, with the engine still
  mining. The installer is per-user since 0.3.3 (`PrivilegesRequired=lowest`,
  `{localappdata}\Programs\Igneum Miner`), so nothing asks for an administrator; its `PrepareToInstall` runs
  `stop-igneum.ps1` (api/quit: miners first, then the node, then the window), replaces the files, and the `[Run]`
  entry on `/IGNOTA=1` relaunches `igneum-app.exe --launch`. An older install in Program Files (0.3.2 and before)
  still raises one UAC prompt through ShellExecute: declined, timed out or unanswered, the helper writes
  `deferred:true`, the engine keeps mining, logs `OTA: waiting for administrator approval` / `OTA: administrator
  approval not given ...; mining continues`, shows "waits for the next time someone is at this PC" and retries on
  Install now, at the next start, or after 6 hours. The engine is never stopped before the installer is running.
- Machines take turns: an update applies only in the machine's own minute of the hour (`manifest::slot_minute`:
  the first 8 hex of the machine id modulo 60; PC 2 = :58), and never while `/api/live` shows the network lost
  over 30% of its identities (`state.miners_10m`) in the last 10 minutes. Urgent (fork close, unsupported) and
  Install now skip both.
- Per-user install, migration: the data (`%LOCALAPPDATA%\igneum`: chain, wallet, settings) is the same folder for
  both installs, nothing is copied. A hand-run installer offers to remove the Program Files copy (one
  administrator prompt for its uninstaller); a silent OTA install leaves it and says nothing. The login entry
  (HKCU Run) is rewritten to the new path on first start. The inbound firewall rule for `igneumd.exe` is requested
  once on the first run of a per-user install (one prompt, in a thread; declined or unanswered = the node dials out
  and mines without it, never asked again).

Field incident, 4 October 2026 15:40 BST: 0.3.2 published; both Windows PCs (0.3.1, nobody at either keyboard)
reached apply, stopped the miners and the node, then sat at the installer's UAC prompt. The chain fell to one
laptop. The 0.3.1 helper waits for the engine to exit before it runs the installer, so a prompt nobody answers
strands the machine; 0.3.3 inverts the order (installer first, engine stops only when the installer runs) and the
per-user installer removes the prompt altogether. The 0.3.2 engines still carry the old helper: their 0.3.3 update
raises the prompt once more (with 0.3.2's R4.3.6 rule the version is then marked failed, not retried); click Yes
once, or run the 0.3.3 installer by hand.

Rollback. The helper writes `update-result.json`; the new engine reads it on start and reports "updated to X from
Y" or the error. The new engine counts its starts in `update-pending.json` and deletes the file after 90 healthy
seconds; a third start without reaching that point restores the previous version (macOS: the `.previous` bundle;
Windows: the previous version's installer kept in `updates/`, so the FIRST update from 0.3.0 has no rollback target
on Windows, said so in the state) and shows "rolled back" in Settings.

Settings: `auto_update` (default on). Off: downloads still happen, the banner waits for Install now. The forced
screenshots: `?update=available|downloading|ready|waiting|applying|urgent|manual|error|updated` on the dashboard URL
(and `?job=running|done|failed` for the remote-job notice). The dashboard shows one notice at a time in the strip under
the header, the most important first (app/igneum-app/ui/app.js, Notices).

## Testing

Unit tests: `cargo test` in `app/igneum-app` (manifest parse, a bad signature and a tampered manifest refused,
sha256 of a file, version ordering incl. pre-releases, safe-moment rules, fork closeness, unsupported versions).

Dry run on the Mac, 4 October 2026 (`packaging/mac/README.md` has the private-devnet recipe; ports 29700+):
the 0.3.0 bundle from the tree ran under its window host against a private devnet with
`IGNEUM_APP_UPDATE_MANIFEST=http://127.0.0.1:29790/dl/<token>/igneum-app-latest.json` (a `python3 -m http.server`
over a folder written by `publish-manifest.sh --base-url ... --dest ...`; loopback http is the one non-https URL the
parser accepts), found the 0.3.1 manifest, downloaded and verified the DMG, staged the bundle, waited for the worker
to start, applied, and came back as 0.3.1 with "updated to Igneum Miner 0.3.1 from 0.3.0" in the event feed. The
screenshots are `docs/design/app-screens/update-*.png`. Windows: reviewed only, see `TEST.md`.

## Remote jobs (4 October 2026)

the project lead's rule: one app on both PCs that the Mac can send commands and files to over the line, so everything is tested
and built without a person at the PC. The channel is `igneum-jobs.json` plus `igneum-jobs.json.sig`, next to the
update manifest, signed with the same OTA key and verified by the same code. Since 0.3.6 (5 October 2026) every app
holds a long-poll on the relay's public `/wake` and fetches the file within seconds of `--deploy` (the script records
the new stamp there); a poll every 2 minutes is the fallback (it was 10 minutes before 0.3.6).
The relay (`relay/`) stays for the Mac and for humans; its PC agent is replaced by this.

| Piece | Where |
|---|---|
| model: parse, verify, targeting, expiry, the once-per-id ledger (`jobs-state.json` in the app data folder), unit tests | `app/igneum-app/src/jobs.rs` |
| runner: poll, requirement probes, the kinds, reports, the dashboard state | `app/igneum-app/src/jobrun.rs` |
| signer: `igneum-ota-sign sign-jobs` and `verify-jobs` (a bad job is refused at signing) | `app/igneum-app/src/bin/ota-sign.rs` |
| publisher: `packaging/ota/publish-jobs.sh add|list|remove|sign [--deploy]` | this folder |
| reader: the published file (signature checked), status per machine, a job's result, watch | `tools/jobs.mjs` |

A job: `{id, kind, title, created_at, expires_at, target: {machine_ids: [...] | "all", platform, requires}, params,
report: "log-intake"}`. Machine ids are the per-install id (16 hex) or its first 8 (PC 1 `ae432dc7`, PC 2
`1ccfe586`). Requirements the engine probes: `wsl`, `wsl-prover` (cargo, `~/.sp1` and `~/igneum-prove` inside
Ubuntu-24.04), `nvidia`; an unknown one is never met and the job waits until it expires.

| Kind | What the app does | Params |
|---|---|---|
| `run` | writes the script body to `<app data>/app/jobs/<id>/` and runs it (powershell or bash), output captured, exit code reported | `script`, `shell`, `elevated`, `stop_miners_first`, `timeout_minutes` (60, at most 600) |
| `fetch` | downloads by https, checks the sha256 (and size), into the job folder or a named app folder; can extract | `url`, `sha256`, `size`, `dir` jobs/prove/packs/updates, `to`, `extract`, `extract_dir`, `fresh` |
| `collect` | uploads files matching globs under the app data root, and/or a command's output, to the intake | `globs`, `command` |
| `restart` | miners, node (the miners follow) or the app (a detached helper opens it again) | `what` |
| `update-now` | the updater checks and installs a newer version at once | |
| `shard-benchmark` | miners stopped, GPU under 5%, the prove zip fetched fresh, `prove-shard.sh` inside WSL under the cap, RESULT and STAGE lines and `results/*.json` uploaded, miners back | `zip_url`, `sha256`, `size`, `fixtures`, `cap_minutes` (90), `distro`, `wsl_user` |
| `build` | mining untouched; free space checked (20 GB on the drive and in the distro), the build-inputs zip fetched (sha256), setup inside WSL2 as root (mingw-w64, clang, protoc, zstd, the Windows rust target; idempotent), sources extracted, `cargo build --release` native and for `x86_64-pc-windows-gnu`, `cargo test` for the manifest's packages, binaries zstd-compressed and sent to the relay (50 MB each) with sha256 in RESULT lines; the target dir under `/root/igneum-build` persists between jobs. `src/jobbuild.rs`, `packaging/windows/push-build-inputs.sh`, `tools/build-job.mjs` | `zip_url`, `sha256`, `size`, `targets` (linux, windows), `budget_minutes` (40), `stage_minutes` {stage: min}, `min_free_gb` (20), `tests` (true), `relay_url`, `nice` (19), `cargo_jobs`, `distro`, `wsl_user` |

Reports: every job uploads to the log intake as run_id `job-<id>-<machine id8>` with the label `job-<kind>`; the
first line is `SUMMARY {json}` (status, exit, times, the RESULT lines, per-kind extras), then the captured output.
Long jobs upload a running report every 5 minutes; collected files and result files are separate labels
(`file-<name>`, `result-<name>`, `prove-log`) under the same run_id.

Safety: the file is rejected when the signature or any job's `expires_at` or params fail (since 0.3.4 a job of a kind the app does not know is skipped instead of rejecting the file; 0.3.3 and earlier reject the whole file, so a new kind reaches the PCs in an app update before its first job is published); a job id runs once per
machine (a crash mid-job counts); nothing writes outside the app data folder except an explicit `run` script,
which is the operator's responsibility; the dashboard shows the running job and a history (id, kind, started, exit,
report uploaded); Settings has "Allow remote jobs from Igneum (signed)", ON by default on this devnet build, with
the key fingerprint, and OFF aborts the running job. `elevated` needs someone at the UAC prompt (or UAC set to
elevate without prompting); unattended it fails after the prompt times out. A `restart app` or `update-now` job
relaunches through the usual quit path.

Publishing and reading:

    packaging/ota/publish-jobs.sh add --kind shard-benchmark --target 1ccfe586 --title "Shard proof run on the 5090" --deploy
    packaging/ota/publish-jobs.sh add --kind run --target ae432dc7,1ccfe586 --script fix.ps1 --title "..." [--elevated] [--stop-miners]
    packaging/ota/publish-jobs.sh add --kind build --target ae432dc7 --deploy     (after packaging/windows/push-build-inputs.sh; or node tools/build-job.mjs run)
    packaging/ota/publish-jobs.sh list | remove <id> | sign
    node tools/jobs.mjs | status | <id> [--all] | watch <id>

Tested on the Mac, 4 October 2026: the unit tests (parse, bad jobs refused, signature and tampering, times,
targeting, the once-only ledger, globs), the signer with the real key, the publisher against a scratch folder, the
reader against the live intake; the crate also compiles for `x86_64-pc-windows-gnu`. Not yet run on a PC: the
first job goes to PC 2 (`1ccfe586`) once 0.3.2 is installed there (`docs/plans/shard-test-pc2.md`).