igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity

release-0.3.10 plan: the ship (run 37374158235 after six dispatches through GitHub's outage, the fallback rehearsed on PC 1 and stood down), the rollout with per-machine times, the hand nodes and the seed, the digest sweep, the mixed fleet, the next cut and the open items

Browse source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-05 21:57:21 +00:00
parent 54132959be
commit 26ebf9cf29
1 changed files with 216 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

223
docs/plans/release-0.3.10.md
View file

@ -1,4 +1,8 @@
# Igneum Miner 0.3.10: the certificate-driven reorg, transaction gossip and the PC-built Windows node, 5 October 2026
# Igneum Miner 0.3.10: the certificate-driven reorg, transaction gossip, the pack loader fix, the six-section miner and the PC-built Windows node, 5 October 2026
Shipped: manifest 0.3.10 published 21:32:40Z, every reachable machine on 0.3.10 by 21:49:41Z, the hand nodes and the seed on 21d4c73c by
21:50:15Z, one digest (1f4b4425...) on every node that restarted; PC 37ba0461 (the owner's laptop) mid-install and Sam's Mac quit at the
time of writing. The cut ran from 16:39Z to 21:5xZ, two hours of it GitHub's hosted-runner outage.
Release engineer, from 16:39 UTC. Worktree `/Users/joshm/Projects/igneum-wt-ship0310`, branch `release-0.3.10` from master
after the 0.3.9 merge. Fork worktree `vendor/igneum-node-0310`, branch `release-0.3.10` from a24ab01a (the 0.3.9 fork tip).
@ -106,7 +110,9 @@ manifest republished 17:59:14Z, `docs/plans/finality-v3-devnet-publish.md`): the
| PC 37ba0461 | silent (0.3.7 at the 0.3.8 cut) | | |
Master moved under the branch while it was cut (fef98a2/a850aef CLAUDE.md, 900bba7 the site's 0.3.9 download cards, 46a6a4e and
bf8d1ba the N3 record, a93199a); merged as d1f4923 (docs and site only, 5 files, no conflict).
bf8d1ba the N3 record, a93199a); merged as d1f4923 (docs and site only, 5 files, no conflict). It moved again at 19:4xZ (the coordinator's
`explorer` merge, origin/master 9746391: the observer's explorer detail, `site/api/stats.mjs` and `supply.mjs`, the explorer pages, three node
tests and `tools/ci/public-api-check.mjs` in ci.yml; no app, node or packaging file): the final merge to master lands on it (section 7b).
## 5. The node binaries and the DMG (fork 21d4c73c, app 5520fb1 and later)
@ -232,22 +238,207 @@ the app (G, H, I: the UI, the engine additions, the export lock), the two Window
| `cargo build --release` in `app/igneum-app`, then `cargo test --release -p igneum-app`, under the lock on this Mac (the coordinator's ask) | 19:24Z: ok, 103 (lib) + 27 (ota-sign) + 8 (prove-verify), 0 failed; `igneum-app 0.3.10` |
| `--mode id` on the worktree's host (unchanged prover) | shard `0x2b1a81cb...`, aggregator `0x474678f3...` |
| The DMG, `packaging/mac/build-dmg.sh` under the lock, 19:24:57 to 19:25:18Z | `Igneum-Miner-0.3.10.dmg` 151687c5672553c571af7224a8e4228348135b8a313fc89e6641db7ae21c85b3, 41,383,375 bytes (engine 0.3.10 with G, H, I; node 21d4c73c; the 0.3.9 prover), fingerprints 477bb0ef and ed9c4d2e; it replaces the e8f1f9f6... DMG of the first tree |
| PC 1 app-only job (`node tools/build-job.mjs run --target ae432dc7 --targets windows --no-node --no-tests`, the branch's new `--no-node`) | (pending) |
| PC 1 app-only job (`node tools/build-job.mjs run --target ae432dc7 --targets windows --no-node --no-tests`, the branch's new `--no-node`) | published 19:24:50Z, done 19:26:47Z in 14 s (the engine alone, 6 s on PC 1's warm cache): igneum-app.exe 4564f8502bf40fdfb61a979ff19543838ca66be95b6b9f0beeaee53330c4d284 (2,962,944), warnings only (the dead-code set of 5a plus `count` is never used); `host.cpp` (WM_GETMINMAXINFO, WM_DEVICECHANGE) compiles on the runner only: CI run 37363381420 (section 7b) |
Not in this tree (they arrived after it; next cut): `job-console` 3562f26 (its second offer, 19:2xZ: power control as a setting, default off), and
the FORK-side `pack-loop` 05ef0fa3 (`vendor/igneum-node`: the miner checks every pack and rebuilds a refused one, exit 44; a node change: the
node stays 21d4c73c). The app side of af983a7 (`watchdog.rs` `PackRebuilds`, the engine re-exporting the pack on a refusal, capped per
epoch) works with the 21d4c73c miner: it keys on the worker's refusal line, not on exit 44.
epoch) is keyed on the miner's exit 44, which only the fork-side miner emits (the pack-loop agent's correction, 19:3xZ): with the 21d4c73c miner a worker refusal line shows "program pack out of date, rebuilding" on the strip, the card and the log but does NOT re-export; with the attempt-aware workers of this release a refusal means a genuinely broken pack, so the gap is small, and the self-healing half lands with the next node cut (section 11).
(pending: the push, CI, the publish)
### 7b. The second push and CI
`git push origin release-0.3.10` at 5b0d54f (19:26:13Z; the pre-push hook flipped the two site files again, restored with `git checkout -- site/`),
`gh workflow run windows.yml --ref release-0.3.10` -> run 37363381420 (queued 19:26:19Z on 5b0d54f). The ship state file now names 5b0d54f
and that run. At 19:46Z the run was still QUEUED with no runner assigned, as were the repo's two `ci` runs (19:26Z, 19:39Z): GitHub's status
API reported Actions "degraded_performance" with an unresolved "Incident with Actions" (investigating since 19:15:17Z). Nothing in this
repository or on this Mac can shorten that: the installer comes only from the hosted `windows-latest` runner. The rollout waits for the
verdict; every other step is staged (the DMG, the signed inputs, the runbook `r0310/rollout.sh` with the exact commands).
Run 37363381420 ended at 19:41:24Z as `failure` with its first job CANCELLED by GitHub after 15 minutes queued ("The job was not acquired by
Runner of type hosted even after multiple attempts", the job's annotation) and the build job skipped: nothing of the tree ran. The workflow
was dispatched again under a watcher that dispatches again on that same annotation and stops on a green or on a failure of the tree itself:
try 2 run 37365130137 (19:42:43Z), try 3 run 37366744501 (19:57:52Z), try 4 run 37368355454 (20:13:21Z), try 5 run 37369931084
(20:28:32Z), every one cancelled by GitHub the same way after about 15 minutes queued; a follow-on watcher took over at 20:54:38Z and
dispatched try 6, run 37374158235 (20:54:55Z), which a runner acquired at 21:24:04Z and which went green at 21:30:29Z (section 7d).
Six dispatches, 2 h 04 min from the first to the green; GitHub's incident ("major outage" at 20:4xZ) was the whole of it.
### 7c. The fallback, prepared at 20:3xZ (the coordinator: a switch at 21:00Z, not a scramble; nothing built yet)
What the runner does for the installer, and what PC 1 has for each step (probe job `probe-installer-pc1-0310`, read-only, ran 20:33:5xZ
in 3 s; its `R` helper collided with PowerShell's `r` alias so every line came back inside an error message, the facts intact):
| Runner step | Needs | PC 1 (ae432dc7) | Fallback |
|---|---|---|---|
| engine (`cargo build --release --locked`, MSVC target) | Rust on Windows | not probed (the PC's build jobs build the engine in WSL on the GNU target: igneum-app.exe 4564f850..., 5a) | the GNU-target engine from job `build-20261005-192450` unless `cargo` exists on the Windows side (checked at the start of job B); recorded either way |
| packaged configuration | the intake key and the folder token as files | the installed 0.3.9 app's `igneum-app.json` at `C:\Users\Admin\AppData\Local\Programs\Igneum Miner\` carries the same manifest URL and key (`override=False`: 0.3.9 shipped no packaged override) | copy that file and add `node_override_params` = the four-field object (not a secret); no secret leaves the Mac |
| payload inputs (`payload-inputs.zip`, signature, hashes, node commit) | the dl folder URL | the URL's folder is in the packaged json | download, verify the sha256s against `payload-inputs.json` (the signature is verified by the runner's step with the key compiled into the app; on the PC the app's own `igneum-ota-sign` is not installed: recorded as a gap, the hashes stand) |
| window host (`app\windows\BUILD-APP.bat`, MSVC v143 cl.exe and rc.exe) | Visual Studio with MSVC | `vcvarsall.bat` under `C:\Program Files\Microsoft Visual Studio\...`, cl.exe 19.51.36260 | runs as on the runner |
| payload (`make-payload.sh`, bash) | Git Bash | `C:\Program Files\Git\bin\bash.exe` (and WSL) | runs as on the runner |
| installer (`build-installer.ps1`: Inno Setup 6 `ISCC.exe`, rcedit) | Inno Setup 6, rcedit-x64 | ISCC NOT FOUND; rcedit not on PATH; winget present | `build-installer.ps1` installs Inno Setup 6 through winget (a tool install on PC 1: the coordinator's call) and downloads rcedit-x64 from GitHub (`-NoRcedit` skips it: the exes then ship without the coin icon and version block, cosmetic, recorded) |
| smoke run, launcher dry run | the exes | | the same PowerShell lines in job B |
| the files back to the Mac | | `relay/clients/send.ps1` (a file up to 50 MB straight to Blob): the installer is 25 MB, the payload zip 35 MB | two `send.ps1 <file>` calls with the sha256s in the report; on the Mac `node tools/relay.mjs read <id>`, sha256 compared, `packaging/windows/check-runtime-dlls.sh` on the payload folder |
| code signing | none on the runner either | | none |
The jobs, in order (neither published until the word; both staged in the scratchpad `r0310/fb/`: the tree zip `fb-tree-0310.zip`, 709,814 bytes,
98 files from 5b0d54f by `git archive`, sha256 012c8a52c27631e8539704d703bf13eb68de0afd9cef094e68d1e6fc398f2cfc; the script `fb-installer-pc1.ps1`,
which stops on any sha mismatch, pins the inputs' node commit against `node-source.pin` as the runner's G13 step does, and takes `-NoRcedit` as its one argument):
1. `fetch` job `fb-tree-0310` to ae432dc7: a zip of the tree at 5b0d54f (`git archive`: `app/windows`, `brand/icons`, `packaging/windows`,
`proto-cuda/windows-app`, `proto-cuda/{host.cu,build.bat,README.md}`, `proto-opencl/{host.c,build.bat,README.md}`, `wsl2`, the two
worker `.rc` files), `--dir jobs --extract --extract-dir fb-0310 --fresh`.
2. `run` job `fb-installer-0310` to ae432dc7 (PowerShell, 20 min): `cargo --version` if any; the packaged json copied and extended; the
inputs zip downloaded and hash-checked; `BUILD-APP.bat`; `make-payload.sh` under Git Bash with `IGNEUM_APP_EXE` = the WSL engine
(`\\wsl$\Ubuntu-24.04\root\igneum-build\...\igneum-app.exe`, its sha256 checked against 4564f850...); `build-installer.ps1 -Payload <folder>
-Version 0.3.10` (with or without `-NoRcedit` per the word); `igneum-app.exe --version`, `Igneum Miner.exe --version`; `send.ps1` the
installer and the zip; every version and sha256 in the report.
3. On the Mac: the two files into the downloads folder, hashes equal to the report, the DLL gate, then
`node tools/ship-app.mjs 0.3.10 ... --from dmg` (preflight, then dmg already, copy, manifest, deploy, verify, console; the fetch step is
skipped by `--from`, the console item carries no run id), then the rollout as planned.
What the gate records in this plan if the fallback ships: the installer marked "PC-built on ae432dc7, GitHub run owed"; non-reproducible
(the runner's engine is the MSVC target, this one the GNU target from WSL; Inno Setup's version from winget; cl.exe 19.51.36260; Git Bash's
version; Windows 11 build 26200); the sha256 and size of the engine, the host, the payload zip and the installer; the GitHub run's id and its
own installer sha256 when it lands (they differ by construction); and the next cut goes back through the runner.
The coordinator asked at 21:3xZ whether the Mac mingw path was now faster and safer; the answer stands as below (there is no such path to an
installer), so PC 1's queue position was kept for attempt 4.
What does not exist: a Mac mingw build of `host.cpp` (the coordinator's "how 0.3.7 shipped"): 0.3.7's node exes were cross-built on the Mac
and its installer still came from the runner (release-0.3.6 plan, section 9); the host has only ever been built by `BUILD-APP.bat` with MSVC,
on the runner or on a PC. So if PC 1 lacked MSVC the next fallback would be new work, not a known path; PC 1 has MSVC, so it is not needed.
### 7d. The fallback, run (the coordinator's word at 21:00Z: GitHub's status page "Actions: major outage", the sixth queued run)
The PC 1 window: asked of the Counter ASIC coordinator (it schedules PC 1 tonight) at 21:0xZ; granted at 21:09:24Z when its reproducible
benchmark released the machine (every card restored; the RX 9070 XT back on the bus). `fb-tree-0310` (fetch) published 21:10:59Z, ran
21:11:32 to 21:11:33Z: the zip (709,814 bytes, sha256 ok) extracted into `%LOCALAPPDATA%\igneum\app\jobs\fb-tree-0310\fb-0310` (the
script's expected path was wrong and it now finds the tree by search). `fb-installer-pc1` (run, 25 min cap) published 21:19:28Z, FAILED at 21:20:05Z with exit 2 after 2 s: the script, not the build.
Its own lines: the tree found (82 files), `cargo on Windows: none` (so the engine is the WSL GNU-target build), Git Bash 5.3.15, Windows
10.0.26200, and then `engine not found at \\wsl$\Ubuntu-24.04\root\igneum-build\...\igneum-app.exe` with PowerShell's
`ItemExistsUnauthorizedAccessError`: the WSL tree belongs to root and the `\\wsl$` share refuses it to the app's non-elevated session. The
build jobs read that tree with `wsl -u root`, so the script now copies the engine out with
`wsl.exe -d Ubuntu-24.04 -u root -- bash -c "cp ... /mnt/c/.../fb-0310-work/igneum-app.exe"`. Republished as `fb-installer-pc1-2` at 21:2xZ
(the Counter ASIC coordinator kept PC 1 for it: "a 2-second exit 2 is the script, not the build", 5-minute reply window met).
`fb-installer-pc1-2` (21:23:08Z) failed at 21:23:58Z, exit 2 in 4 s: `/root/igneum-build/app/igneum-app/target/...` does not exist (the build
job keeps ONE target dir for the whole job; the acceptance script had read the node exes from `/root/igneum-build/target/...`): the script
now finds `igneum-app.exe` under `/root/igneum-build` with `find` and prints its sha256 from inside WSL. `fb-installer-pc1-3` (21:26:52Z)
failed at 21:27:20Z, exit 2 in 4 s: the inline `bash -c "..."` string lost a quote on its way through PowerShell (`unexpected EOF while
looking for matching quote`), the class the 0.3.6 cut closed for the app's own WSL calls (3811d8e, "every WSL script runs from a file") and
which this scratch script had reopened: the bash part is now written to `engine.sh` on the PC (LF, no BOM, the acceptance script's own
pattern) and run as `bash <file> <arg>`. Three 4-second failures of the script, none of the build; by the Counter ASIC coordinator's rule
its 10-minute measurement takes PC 1 first, then a read-only path probe (every path the script needs, found and printed), then attempt 4.
Before attempt 4 one more change, after reading the fixed block: `find ... | tail -1` would take the NEWEST `igneum-app.exe` under
`/root/igneum-build`, and other agents' build jobs have run on PC 1 since mine (job-console's, pack-loop's), so the sha check could stop
attempt 4 on another tree's engine. The engine this plan already holds and verified (4564f850..., 2,962,944 bytes, from PC 1's own job
`build-20261005-192450`) now travels INSIDE the tree zip (`fb-tree-0310b.zip`, with it under `app/igneum-app/target/x86_64-pc-windows-gnu/release/`),
and the script reads nothing from WSL at all; the path probe checks that file too.
Attempt 4 was never published: at 21:30:29Z GitHub's runner acquired try 6 (run 37374158235, dispatched 21:10:04Z on 5b0d54f) and it went
GREEN (the parse job 21:24:04 to 21:24:57Z; engine, window host, payload, installer, smoke run 21:25:18 to 21:30:29Z; the G13 inputs step
against the 21d4c73c inputs of 7a). The recipe's own installer therefore ships; the fallback stops here with nothing built on PC 1, PC 1
released to the Counter ASIC coordinator at 21:31Z, and the prepared pieces (the tree zip with the engine, the installer script, the two
probes, the runbook steps) kept in the scratchpad `r0310/fb/` as the rehearsed path for the next outage. Cost of the detour: three
4-second script failures on PC 1 and about 70 minutes of the shipper's attention; the gate record "PC-built, non-reproducible, GitHub run
owed" is not needed.
### 7e. The ship (21:31 to 21:40Z)
`OTA_SKIP=1 CONSOLE_SKIP=1 packaging/windows/fetch-ci-artifacts.sh 37374158235` (21:31:21Z): the installer and the payload zip into the
downloads folder; the DMG copied beside them. The payload zip holds the rebuilt workers (igneum-worker-cuda.exe 85cc357b..., 1,510,912;
igneum-worker-opencl.exe afa73a32..., 445,952: both differ from 0.3.9's f50e19f2.../1abc673e... and from the PCs' hot-fix pair), the PC-built
node 3adb01a7..., and the runner's MSVC engine 496c4883... (3,361,792). The first ship run (21:32:02Z) stopped in preflight: the tree was 4
commits behind origin/master (the coordinator's explorer merge, 9746391: docs, site, observer, ci.yml; no app, node or packaging file), so
`origin/master` was merged as ff873f6 (37 files, no conflict), pushed 21:32:27Z (the hook's site flip restored), and the state file kept
`sha` = 5b0d54f, the CI commit, as the 0.3.6 and 0.3.9 cuts did.
```
node tools/ship-app.mjs 0.3.10 --node vendor/igneum-node-0310 --branch release-0.3.10 --public \
--activation-height 135200 --deadline-note "finality v3" --notes "<the seven changelog lines; node 21d4c73c>" --from ci
```
| Step | Result |
|---|---|
| preflight | ok: tree ff873f6 clean, 0.3.10 in all 6, fork 21d4c73c, gh igneum-labs, live inputs 21d4c73c built 19:24:41Z |
| ci | already: run 37374158235 green |
| fetch, dmg, copy | already (above) |
| manifest | 0.3.10 mac+windows, signed (key 8f186e37...), verified locally; `consensus` carried over from the folder's 0.3.9 manifest: `activation_height` 135200, `deadline_note` "finality v3", `override` the four-field object; and in `dl/public/` (with the wallet 0.1.4 manifest) |
| deploy | one deploy, 21:33Z |
| verify | the token folder: `igneum-app-latest.json` 0.3.10, signature ok, mac 151687c5..., windows 24e58849...; the public folder: every file HEAD 200 with the local size (the DMG, the installer, the 0.3.9 HiveOS package, the wallet DMG, the four `/public/` aliases, the two manifests and their signatures, `igneum-downloads.json`), but the byte comparison of a json file against the edge still failed after 12 tries at 21:37Z and again on a resume from `verify`: the edge cache, the class of the 0.3.6 and 0.3.9 verifies (section 11 if it does not clear) |
| console | (pending: `--from console` after the verify clears) |
| File | sha256 | Size |
|---|---|---|
| Igneum-Miner-0.3.10.dmg | 151687c5672553c571af7224a8e4228348135b8a313fc89e6641db7ae21c85b3 | 41,383,375 |
| Igneum-Miner-Setup-0.3.10.exe | 24e58849f2b517e8c5579455e8c9d8376ff7dd27052f458ef91913cbc48abc88 | 49,858,273 |
| igneum-windows-app.zip | 16b68b7bf625a946ac62a22413982b38c6649a124d799b08424a070f1b34810e | 71,809,486 |
The installer is 30 MB larger than 0.3.9's (19,836,912): the two NVRTC DLLs (93 MB unpacked) ride with the rebuilt CUDA worker now.
`update-now-0310` to all, published 21:39:59Z (apps woken, stamp a23f594a). Timing with the Counter ASIC coordinator (it schedules PC 1 tonight): PC 1
was released by its hot-table job at 21:35:39Z, so one update-now went to every machine; its era measurement starts on PC 1's 0.3.10 STATUS line.
## 8. The machines after the publish (manifest live 21:33Z, update-now 21:39:59Z)
Baseline 21:40:31Z: Mac d937c69d app 0.3.9 (its card reads node 1, a24ab01a), PC 1 ae432dc7 0.3.9 node a24ab01a DAA 133,903 141.5 MH/s,
PC 2 1ccfe586 0.3.9 node a24ab01a DAA 133,945 0.0 MH/s (its 5090 worker off since a job's `/api/resume` at 21:25:11Z that the 0.3.9 app
answered and never acted on, section 11), Sam's Mac 3a9bf309 quit 53 min earlier, PC 37ba0461 0.3.9 node `2.1.0` 2.0 MH/s. A machine
counts as updated when its engine logs the 0.3.10 header, its node reports a DAA score and its miner a hash rate on 0.3.10. The two checks
asked by the coordinator: (1) the workers start without the seed-words error on the first try, on the rebuilt pair; (2) the Mac's card
shows node 1's digest, by design. C5: the provers' "stopped after" lines (shards aborted by the restart).
| Machine | On 0.3.10 | Its log |
|---|---|---|
| PC 1 ae432dc7 (Windows) | engine restart 21:40:41Z (run `win-ae432dc7-20261005-214041`), 42 s after the job; `[ok] updated to Igneum Miner 0.3.10 from 0.3.9` 21:40:42Z; `igneumd started` 21:40:46Z (the PC-built node from the new install path); `node proof verifier: command` and `reported: command` +6 s; `cards: NVIDIA GeForce RTX 5090 [discrete, off] \| AMD Radeon(TM) Graphics [integrated, off] \| AMD Radeon RX 9070 XT [discrete, off]` (hotplug's line: the iGPU integrated and off, the two discrete cards then started); miners started 21:40:47Z (nvidia-ae432dc7-1, the bundled `igneum-worker-cuda.exe`) and 21:40:48Z (amd-ae432dc7-3, the bundled OpenCL worker); `update to 0.3.10 complete` 21:42:12Z. Check 1 PASS: the NVIDIA miner's `epoch seed 66b26013... (daa 133967): CPU program and cache ready in 176 ms` 21:40:47Z, then `worker: info first pack packs\devnet: nvrtc 170 cache 4 dataset 23 check 249 race 37802 ms variant base; self-test PASS` and `worker: ready cuda NVIDIA_GeForce_RTX_5090 ... prepare 1 path nvrtc 12.8` at 21:41:25Z, no `epoch seed bytes do not give` line; STATUS 45.1 MH/s wall at 60 s (124.1 inside jobs), 124.3 MH/s inside jobs from 90 s on; the console 141.4 MH/s at 21:45:17Z (both cards). The node log: `igneumd/2.1.0` (no commit: the PC build, section 11), `Calibrated v1 fees ... 210000`, digest 1f4b44255fcd2ea8f75664ed47200f409186ddd2292960c9e2cf95bbbdc11505, peers node 1 (192.168.68.64) outbound and inbound and the seed 188.245.5.161, flows registered at THEIR protocol version 13 (a v14 node beside v13 peers, the mixed-fleet case of section 10, measured) | |
| Mac d937c69d | engine restart 21:40:51Z (run `mac-d937c69d-20261005-214051`), 52 s after the job; `[ok] updated to Igneum Miner 0.3.10 from 0.3.9` 21:40:52Z; `a node already answers on 127.0.0.1:26610; using it (it is not stopped by this app)`: the app attaches to node 1 as it has since 17:45Z, so its card shows node 1's version and digest (check 2, by design); `cards: Apple M5 Max [apple, off]` (its miner was off before the update too); `update to 0.3.10 complete` 21:42:22Z | |
| PC 2 1ccfe586 (Windows) | its 0.3.9 app fetched the woken jobs file at 21:40:33Z (`1 new for this machine, 1 queued`) and queued the update behind the aggregation-cost agent's running job `agg-cost-pc2-2` (one job at a time), so the update ran at 21:49:24Z when that job ended: installer downloaded and verified 21:49:27Z, `quit: stopping the miners, then the node` 21:49:29Z (no prover "stopped after" line: no shard in flight), engine restart 21:49:41Z (run `win-1ccfe586-20261005-214940`), 9 min 41 s after the job; `igneumd started` 21:49:42Z; `cards: NVIDIA GeForce RTX 5090 [discrete, off] \| AMD Radeon(TM) Graphics [integrated, off]`; miners started 21:49:44Z. Check 1 PASS: `epoch seed 66b26013... (daa 134395): CPU program and cache ready in 169 ms`, `worker: ready cuda NVIDIA_GeForce_RTX_5090 ... first pack ... self-test PASS` at 21:50:21Z, no seed-words line; STATUS 120.7 MH/s inside jobs at 60 s, 120.5 at 90 s; the console 123.2 MH/s at 21:53:04Z. This also ended the `/api/resume` no-op (its 5090 had been off since 21:25:11Z). The node log: `igneumd/2.1.0`, digest 1f4b4425..., flows at version 13 with node 1 and the seed (still a24ab01a for 10 s more) and at 14 with PC 1. The prover: `prover: host /opt/igneum/igneum-prove-host (WSL2), CUDA`, the pinned ids, then three assigned shards (22126, 51922, 84099) each failed with `Failed to create the CUDA prover impl: CudaClientError: Connect(Os { code: 13, kind: PermissionDenied })` at 21:49:56, 21:50:12 and 21:50:32Z: PC 2's prover is dark after the update (section 11) | |
| PC 37ba0461 (Windows, the US laptop) | its 0.3.9 app (run `win-37ba0461-20261005-202247`) downloaded and verified the installer at 21:40:52Z, started it at 21:40:53Z (`per-user install, no administrator prompt`), stopped its miners and node at 21:41:16Z, and no 0.3.10 engine run had reported by 21:56Z (the console: `STOPPED (update)` for 14 min): the install is in progress or stuck on the owner's machine; nothing to drive from here (section 11) | |
| Sam's Mac 3a9bf309 | quit since 20:47Z (0.3.9); takes 0.3.10 when it is started | |
C5, the shards aborted by the restart: PC 2's engine logged no prover "stopped after" line at its quit (21:49:29Z) and PC 1 runs no prover;
the Mac's prover was off. Observed count: 0. The coverage numbers measured across the restart carry no abort from it.
The ship's last step, `--from console` (21:55:08Z): item #364 "Igneum Miner 0.3.10 shipped (mac+windows)"; the tool's closing line
"manifest 0.3.10 published 2026-10-05T21:32:40Z".
## 8. The machines after the publish
(pending)
## 9. The hand nodes and the seed
## 9. The hand nodes and the seed (21:49 to 21:50Z)
(pending)
Moved while PC 2 and PC 37ba0461 were still on 0.3.9 (their updates gated by another agent's job and a slow download): the digest does
not change with 0.3.10 and a v14 node beside v13 peers is the measured mixed-fleet case (section 10), so moving them shortened the mixed
window.
| Node | Command | Result |
|---|---|---|
| The observer, then node 1 | `IGNEUMD=<fork>/target-integration/release/igneumd IGNEUMD_COMMIT=21d4c73c infra/devnet/restart-hand-nodes.sh '<the four-field object>'` (the branch's script: `grep -c` for the commit string, the object written to `/tmp/igneum-devnet/override-v3.json`, the previous file kept with a stamp) | observer restarted 21:49:38Z (pid 67770), node 1 21:49:50Z (pid 67963, caffeinate 67965); both print `Calibrated v1 fees ... from DAA score 210000` and digest 1f4b44255fcd2ea8f75664ed47200f409186ddd2292960c9e2cf95bbbdc11505; the Mac app's card follows node 1 from here (its node line reads 21d4c73c) |
| The seed 188.245.5.161 | `IGNEUMD_LINUX=infra/cross/out-0310/igneumd IGNEUMD_LINUX_SHA256=40be0e14... infra/devnet/restart-seed.sh '<the same object>'` (the zig cross-build of 21d4c73c, glibc 2.36 target; the script checks the sha on both sides, keeps the previous binary as `igneumd.prev-035` and the previous override file with a stamp) | restart 21:50:15Z, unit active, MainPID 125195, `igneumd/2.1.0-21d4c73c`, `Calibrated v1 fees ... 210000`, digest 1f4b4425... |
Peers after the restarts: the observer and node 1 connected to the seed within 20 s and register flows at protocol version 14 with each
other and the seed (node 1's inbound from the observer 21:50:08Z, node 1 to the seed 21:50:20Z, the observer to the seed 21:50:38Z; the
seed's three inbound peers from this Mac's address at 21:50:20, 21:50:27 and 21:50:38Z, all at 14). PC 1's node (started 21:40:46Z, before
the hand nodes moved) registered flows at version 13 with node 1 and the seed (then a24ab01a) and keeps them; PC 2's node (21:49:42Z) at 13
with node 1 and the seed (10 s before their restarts) and at 14 with PC 1: the mixed fleet of section 10, measured on the live network, with
no refusal and no drop.
### 9a. The digest sweep (21:50Z)
| Node | Binary | Digest | Read from |
|---|---|---|---|
| PC 1 app node ae432dc7 | the PC-built 3adb01a7... (`igneumd/2.1.0`, no commit string) | 1f4b44255fcd2ea8f75664ed47200f409186ddd2292960c9e2cf95bbbdc11505 | its node log through the log intake (run 214041) |
| PC 2 app node 1ccfe586 | the same | 1f4b4425... | its node log (run 214940) |
| The Mac app d937c69d | node 1's (attached; its card's commit string is the app's own reading from its start at 21:40:51Z, before node 1 restarted, and refreshes on the app's schedule; its status line reads node 1's height and peers) | node 1's | |
| Node 1 | 21d4c73c Mac arm64 4bb356f4... | 1f4b4425... | `/tmp/igneum-devnet/node1.out` |
| The observer | the same | 1f4b4425... | `/tmp/igneum-devnet/observer-v4.out` |
| The seed | 21d4c73c Linux 40be0e14... | 1f4b4425... | the journal through `restart-seed.sh` |
| PC 37ba0461 | (its update in progress) | (pending) | |
| Sam's Mac 3a9bf309 | quit since 20:47Z | (pending) | |
One digest, equal to the N3 publish's (`finality-v3-devnet-publish.md`) and to this cut's three readings on the fork (section 3): 0.3.10
moved no parameter, as measured.
## 10. The mixed fleet during the window
@ -267,12 +458,30 @@ publish.
## 11. Open after the cut
The next cut (0.3.11), decided by the coordinator on 5 October 2026 night:
| Branch | What | Why not 0.3.10 |
|---|---|---|
| fork `pack-loop` 05ef0fa3 (`vendor/igneum-node`) | `write_pack_checked`, `export-pack` exit 3, the force-prepare at the epoch boundary, the miner's exit 44 that unlocks the app's capped re-export | a node change after the node was frozen at 21d4c73c with its suites, harness runs and Windows acceptance done; the app side (af983a7) with the attempt-aware workers is the fix that matters tonight and it is in |
| `job-console` 13755b9, then 3562f26 and whatever follows | one hidden-console builder for every elevated launch, the spawn check in CI, PC 1 console watchers; then power control as a setting, default off (the project lead: "if we don't have to ask then don't ask") | arrived after the tree closed (both times); repoints elevated-exit's `include_str!` test at `platform.rs` at its own merge |
| `opencl-rdna4-telemetry` 7adcd4c (`igneum-wt-rdna4-telemetry`, not pushed: master + a08c371 as 38c9eec + 7adcd4c) | `igneum-gpu-telemetry.exe` (ADLX, SetupAPI bus, PDH fallback; amdgpu sysfs on Linux) built by `build-windows.sh`, shipped by `make-payload.sh` and `push-inputs.sh`; the engine runs it at `-l 5` and fills power, temperature, fan, memory clock and utilisation on the AMD card; 82 app tests pass. The 9070 XT measured on PC 1: 198.9 W, 64 C, 17.73 MH/s, 0.089 MH/W against the 5090's 0.398 MH/W (job `tele-measure-1`) | arrived after the tree closed; a new shipped exe and an engine source |
| `ember-tune` 54ff1bc (+38ef711, `igneum-wt-ember-tune`; its agent's message at 21:2xZ) | Ember Tune: `ember.rs` replaces the NVIDIA power-only run in `sweep.rs` and the AMD single-lever sweep of 720b369; sits on cherry-picks of 7adcd4c and 13755b9+3562f26, so those merge first; 93 app tests, `relay/test/ember.test.mjs` and `ui/tune-line.test.mjs` in ci.yml; design in `docs/plans/ember-tune.md` | arrived after the tree closed |
| the rest of `opencl-rdna4` a08c371 | the OpenCL worker's duplicate-platform fold, `--readback select`, `--memprobe`, `detect.rs parse_opencl_list`, the 9070 XT bench-log entry | conflicts with gpu-hotplug in `detect.rs` and `host.c`; only its EXPORT_LOCK hunk shipped (854f9a8) |
| Item | State |
|---|---|
| The per-job build-inputs zip (68f14b9, c4's tooling commit; the coordinator's check after two agents' PC jobs ran on another agent's sources through the shared `build-inputs.zip` tonight, jobs build-20261005-191656 and -192537). Checked on this tree's own jobs in the live jobs file rather than a dry run (a dry publish would leave a stray entry in the file the ship deploys): `build-20261005-182405` (PC 1) pins `params.zip_url` = `.../build-inputs-20261005182334-74461.zip`, `params.sha256` 628e6dae..., size 8,266,818; `build-20261005-183131` (PC 2) pins `build-inputs-20261005183051-83796.zip`, c83ebb2a..., 7,271,492; both shas equal the local zips of those names, and 15 per-job zips sit beside the folder default. So a job published through `tools/build-job.mjs` pins the zip it just pushed. The residual: `packaging/ota/publish-jobs.sh add --kind build` WITHOUT `--zip` still defaults to the shared `$DEST/build-inputs.zip` (line 307); nothing refuses that name. A hand-added build job can therefore still pin whatever the folder default holds | build-job.mjs path closed; the hand path is the first item of the next cut: refuse `--kind build` without `--zip`, or default to the newest per-job zip |
| `infra/cross/build-linux.sh` resolved a relative `TARGET_DIR` inside the node source after its `cd`, so the copy step shipped the previous build's bytes (twice tonight, caught by the commit string in `strings`). Fixed here (e0a5fd1). The class: any script that takes a directory argument and changes directory before using it; `proto-cuda/windows-node/cross-build.sh` takes the node worktree as `$1` and `CARGO_TARGET_DIR` from the environment (the 0.3.9 cut passed a relative one and it worked only because that script does not cd); a CI check for the shape is owed | fixed on the branch; the check owed |
| hotplug's `proto-opencl/host.c` change (the worker's `--list` prints the PCI address, the key to one row per physical card): no prebuilt OpenCL worker is in the payload inputs (the live zip holds igneumd.exe, igneum-miner.exe and three DLLs), the payload carries `host.c` and `build.bat` and the app builds the worker on the PC from them (`engine.rs build_worker_from_source`), so the change ships inside the 0.3.10 installer; whether an app that already has a worker exe rebuilds it from the newer source was not read here. Check on the console after the update: the PCs' card rows carry the PCI address (and one row per AMD card) only if the worker was rebuilt; else a rebuild is the hotplug agent's follow-up | open: told the hotplug agent |
| `kaspa-consensus` test `ban_is_decided_by_the_carrying_block_so_nodes_agree_on_every_voter_list` (c4-fix) failed once under the six-package parallel run on PC 2 with `UnexpectedDifficulty(487112096 vs 487129281)` in `mine_on_all` (section 3) and passes alone, twice. The helper builds a block on one `TestConsensus` and inserts it into others; the expected difficulty of the receiving node differs when the run is slow, which points at a wall-clock dependence (difficulty v2's sanitised clock per header). For the c4 agent: pin the clock or the timestamps in that helper, as the finality tests do for the cache queue. Until then a parallel six-package run can fail this test under load; the suites are run as consensus alone plus the other five | open |
| `site/build.mjs` is not idempotent: one bench entry's label alternates between "RTX 5090 first run" and "RTX 5090, memory-hard dataset" on every run of the same tree (three runs at 18:38Z: first-run, memory-hard, first-run), because the build reads its own `site/journey.json` back (line 248) and the label table at lines 203 and 204 matches against what the previous run wrote. Every push flips the two files through the pre-push hook, which is why the 0.3.9 and 0.3.10 cuts both met a modified tree after the push | open: build the journey from the sources only (never from the previous output), then have the hook refuse a push whose build differs from the committed site |
| The Windows payload carries the PC's three mingw DLLs (34 MB of inputs, `libstdc++-6.dll` alone 26.3 MB unstripped) although the exes import none of them since housekeeping A; the installer grew 5.2 MB | open: drop the DLLs from `push-inputs.sh` and `jobbuild.rs`'s copy once no shipped fork needs them, or strip them |
| The two worker exes' version blocks say 0.3.0 (`proto-cuda/nvrtc/igneum-worker-cuda.rc`, `igneum-worker-opencl.rc` are not among the six files `ship-app.mjs` bumps) | open: add the two `.rc` files to `VERSION_FILES` |
| The app's re-export of a refused pack (af983a7's `watchdog.rs` `PackRebuilds`, 3 per epoch) waits for the miner's exit 44, which the 21d4c73c miner never emits: a refusal in 0.3.10 shows the notice and restarts the worker, no re-export. The fork-side `pack-loop` 05ef0fa3 (the miner checks every pack, rebuilds a refused one, exit 44) is the other half | next node cut |
| C1 (the consequences reviewer, 20:0xZ): the 0.3.10 node's `igneum_exportSegments` (fork `igneum/exec/src/rpc.rs`) writes no `daaScore` and no `feesV1ActivationDaa` per segment, which the 0.3.9 exporter needs to replay both sides of the fee switch (`export/src/main.rs`: "a dump without `daaScore` is accepted only when the switch is never or 0"). Measured here: the handler (90 lines from `rpc.rs` 849 in `vendor/igneum-node-0310`) carries neither key (`daaScore` appears in the fork only in other RPCs, lines 239, 423, 819); the reviewer names `vendor/igneum-node-pv1` at eb32c645 (on 21d4c73c) as the carrier: its `rpc.rs` writes `daaScore` per segment (line 961) and `fees` and `feesV1ActivationDaa` at the top level (line 982); confirmed here at 20:4xZ (`git -C vendor/igneum-node-pv1 grep -n feesV1ActivationDaa -- igneum/exec/src/rpc.rs`: line 982); my first read of that path at 20:1xZ was wrong. So at H = 210,000 (about 19:50Z on 6 October) every 0.3.10 prover's export of a post-H block fails or meters with the wrong table and the fleet's provers go dark, unless the next node cut carries that RPC onto every prover before H, or H is republished later. The mechanism (the proving agent, 20:1xZ): the app's prover calls the exporter with no fee flags, so from H every app prover on 0.3.10 cuts with the prototype table and every statement is vetoed. The two closes: (a) the proving-v1 fork (eb32c645, on 21d4c73c) on every prover before H through 0.3.11; (b) republish H = tip + 86,400 by the fee-switch plan's rule. Decision due 16:00Z on 6 October; the coordinator, the proving agent and the 0.3.11 shipper hold the same line | open, dated: (a) or (b) by 16:00Z on 6 October |
| C5 (the same reviewer): the app kills its prover child on quit (`prover.rs` 266 to 272), so the `update-now` of this rollout aborts whichever shard each prover has in flight (up to 37 s each, no payout). Accepted as the cost of the restart; the count is read after the rollout from the provers' "stopped after" lines (section 8) so the coverage numbers measured across the restart are read with it | recorded at the rollout |
| `/api/resume` answered ok on the 0.3.9 app and never restarted the miners (PC 2's 5090 worker "off" with the card holding 1.7 GB from a job's resume at 21:25:11Z until its 0.3.10 restart, the iGPU miner too; the Counter ASIC coordinator, 21:4xZ). The class: a resume that reports success without a miner restart. The app should re-check the miner processes after a resume and report a failure | open: next cut |
| PC 2's prover after the 0.3.10 restart: every assigned shard fails at once with `CudaClientError: Connect(PermissionDenied)` (section 8), where 0.3.9's PC 2 proved shards from 17:36Z. The host, the pinned ids and the CUDA backend are detected as before; the connect that fails is the SP1 CUDA prover's client socket. The aggregation-cost agent's job `agg-cost-pc2-2` ran as root in WSL minutes before and its lines show no CUDA or socket change; the 0.3.10 app's prover path changed only to read the program ids (`--mode id`). Unexplained; handed to the coordinator and the Counter ASIC coordinator (PC 2's measurement agents) | open: PC 2 proves nothing until it is found; a restart of the WSL distro or of the CUDA prover service is the first thing to try |
| PC 37ba0461 (the US laptop) started the 0.3.10 install at 21:40:53Z, stopped its miners and node at 21:41:16Z and had not come back by 21:56Z: the per-user installer on the owner's machine, nothing to drive remotely | open: the console shows when it returns; its 0.3.10 line and worker start are read then |
| `dl/public/igneum-downloads.json` (the unsigned index the site's download page reads) alternates at the edge between the new bytes and the previous ones for over 20 minutes after the deploy (one fetch byte-identical at 21:52Z, the next three not): different edge nodes behind one hostname. The two signed manifests were byte-identical and verified from the first check. The ship's verify step counts it as a failure and refuses to post the console item, so the item was posted with `--from console` | open: the verify should accept the index after the signed manifests pass, or retry it for longer; the site serves the previous version's buttons from a stale edge until it settles |
| The PC-built node binaries embed no commit (the build inputs zip has no git dir, `build-info` falls back to the bare version): the console shows PC 1 and PC 2 as node `2.1.0` with no commit after 0.3.10, as the 0.3.6 PC build did. The build inputs manifest records the fork commit (2f88a82f and later) | open: a commit stamp through the build job (`jobbuild.rs`) is an app change, not tonight |