release-0.3.9 plan: the ship, the proving rollout, the fee switch (H 210,000, digest ab8847da) and the sweep; restart-hand-nodes.sh: grep -c instead of grep -q under pipefail, lines() never ends the script

This commit is contained in:
igneum-labs 2026-10-05 17:55:35 +00:00
parent 15e1563e0b
commit c90e477e44
2 changed files with 165 additions and 11 deletions

View file

@ -214,17 +214,105 @@ in the signer's usage before it writes anything (a 0.3.10 item).
|---|---|---|
| (1) provers off | Mac: already off (the app's `/api/state`: `enabled false, status off`, 15:46Z). PC 2: run job `prove-off-pc2-039` (the 0.3.8 PowerShell script: `POST <app.url>/api/prove {"on":false}`, the app state, the node's `igneum_getProvingStatus`), published 16:02:06Z (apps woken, stamp e933f990), started early so the drain overlaps the wait for the `fee-switch` branch | ran 16:02:27 to 16:02:38Z: `api/prove off: {"ok":true}`; PC 2's node pool `{entries 18, failed 0, pending 0, verified 178}`, paidShards 388, tip DAA 113,603 |
| (2) empty pool | A 30-s watch on the Mac app node's `igneum_getProvingStatus` that prints every change and every error line: 16:02:36Z `entries 18, pending 0, verified 178`, tip DAA 113,601; then one entry fewer every 30 to 60 s (16:03:07Z 16, 16:07:17Z 10, 16:11:48Z 5), no error line, no new record | OBSERVED empty 16:15:50Z: `entries 0, pending 0, failed 0, verified 178` (13 min 23 s after PC 2's prover went off; the 600-DAA record window at 0.965 blocks/s is 10.4 min) |
| (3) new host on every node | (pending) | |
| (4) `--mode id` | (pending) | |
| (5) provers on | (pending) | |
| (3) new host on every node | The Mac, PC 1, PC 2: by `update-now-039` (published 16:56:12Z through the main script, apps woken, stamp 029c1bb7), see 9. PC 2's WSL tools from the package: `fetch-prove-039` and `rebuild-prover-pc2-039` (7a) | |
| (4) `--mode id` | PC 2 (`/opt/igneum`, in the rebuild job, 16:54:5xZ): shard `0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a`, aggregator `0x474678f3...`, equal to the DMG's host (5d). Mac: the installed app after the update, see 9 | |
| (5) provers on | PC 2: run job `prove-on-pc2-039`; the Mac's prover stays off as before (it was off at 0.3.8 too) | (pending) |
## 8. The ship
### 7a. PC 2's WSL tools from the package
(pending)
| Job | Published | Result |
|---|---|---|
| `fetch-prove-039` (fetch, `--dir prove --extract --extract-dir igneum-prove-wsl2-039 --fresh`; the first add at 16:34:54Z never wrote the job, section 6) | 16:49:21Z | ran 16:49:48Z: the zip (1,530,932 bytes, sha256 ok) extracted into `%LOCALAPPDATA%\igneum\prove\igneum-prove-wsl2-039` |
| `rebuild-prover-pc2-039` (run, PowerShell, the 0.3.8 script with the folder names changed) | 16:54:09Z | ran 16:54:42 to 16:54:53Z as root in Ubuntu-24.04: rsync of `package/` into `~/igneum-prove` (`--delete`, target kept), every source re-stamped, `elf/` listed (the package manifest's shard id `0x2b1a81cb...`), `cargo clean -p igneum-prove-core -p igneum-prove-export -p igneum-prove-host`, `cargo build --release ... --features igneum-prove-host/cuda` (the three crates only, every dependency cached from the 0.3.8 build), installed into `/opt/igneum` (host 73,036,832 bytes 29cc476898c3ab2666110ef65f5b432ac243d1e87e565b57b28e4d140fcbfeee, export 3,602,008 bytes f8b7c19d...; `ldd`: 0 libs missing) |
| `--mode id` on PC 2 (`/opt/igneum/igneum-prove-host --mode id`, in the same job) | | `pinned guests: shard program id 0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a (2832504 bytes, sha256 0x150f4c05a2951fc5) aggregator id 0x474678f35f7545db28055d5e5bbc308231d84a5a072202087a2a8d5b09123896`: equal to the Mac DMG's |
## 9. The machines after the publish
## 8. The ship (16:33 to 16:54Z)
(pending)
`git push -u origin release-0.3.9` at 1269db8 (16:33:55Z), `gh workflow run windows.yml --ref release-0.3.9` -> run
37341810898 (green 16:39:45Z); the script commit 00c7b4f pushed 16:42Z and dispatched again -> run 37342911129
(started 16:42:42Z, green 16:48:10Z: parse checks; engine, window host, payload, installer, smoke run; the G13 inputs
step against the a24ab01a inputs of 5b). Two more merges of master followed (97b1a74, docs and site only; then the
site rebuilt as ea08798 and the plan as the next commit), so the ship state file got `sha` = 00c7b4f, what the
tool's own commit step would have recorded, and the run resumed with `--from ci`, as the 0.3.6 cut did.
```
node tools/ship-app.mjs 0.3.9 --node vendor/igneum-node-036 --branch release-0.3.9 --public \
--activation-height 84100 --deadline-note "proving v0" --notes "The prover mirrors the fee switch (new pinned guest, shard id 0x2b1a81cb...); node a24ab01a: igneum_exportSegments names the mergeset and every entry's block and body position; the devnet fee-switch runbook" \
--from ci
```
No `--dl-both`: the rotation of 15:25Z left one token folder (`dl-token.next` is gone; the tool would refuse it).
`--public` (coordinator, 16:32Z): the same version into `dl/public/` and the `/public/` aliases. Three false starts
in preflight (16:48:47Z, 16:50:25Z, 16:50:54Z): the pre-push hook's site build had left `site/index.html` and
`site/journey.json` modified after the push, then the plan file (tracked since the 97b1a74 merge) was modified; the
tool refuses any modified tracked file besides the six, which is right.
| Step | Result |
|---|---|
| preflight | ok: tree ea08798 + the plan commit, 0.3.9 in all 6, fork a24ab01a, gh igneum-labs, live inputs a24ab01a built 16:31:46Z |
| ci | already: run 37342911129 green |
| fetch | Igneum-Miner-Setup-0.3.9.exe 18.9 MB, igneum-windows-app.zip 26.4 MB |
| dmg | already (5e) |
| copy | the DMG into the downloads folder |
| manifest | 0.3.9 mac+windows, signed (key 8f186e37...), verified locally, and in `dl/public/` |
| deploy | one deploy, 16:51Z |
| verify | the three files and the manifest match the local copies; `dl/public/` and the `/public/` aliases serve the local bytes (try 8 of 12: the edge took about 70 s) |
| console | item #360 "Igneum Miner 0.3.9 shipped (mac+windows)"; exit 0 at 16:54:53Z, 211 s |
| File | sha256 | Size |
|---|---|---|
| Igneum-Miner-0.3.9.dmg | 5e57c5735dc99e44475c43fba8ecffadae9dbd3a150c32ef4858587a98256de3 | 41,333,873 |
| Igneum-Miner-Setup-0.3.9.exe | 1d62a4dbdcef768ebd6fac5e46b199fea4de8854f4cf4c905eee49a6ed9334ce | 19,836,912 |
| igneum-windows-app.zip | 22996a692128a97183c39b37b65275ad33eace4a7d89d83d8f097b315c062adc | 27,683,238 |
Manifest 0.3.9 published 2026-10-05T16:51:46Z.
The live manifests, read 16:55:13Z, compared field by field with the 0.3.8 manifest saved before the cut (section 4 of
this plan, read 15:38Z):
| Field | 0.3.8 (before) | 0.3.9 token folder | 0.3.9 `dl/public/` | 0.3.9 OLD folder |
|---|---|---|---|---|
| version, channel, min_supported_version | 0.3.8, devnet, 0.3.0 | 0.3.9, devnet, 0.3.0 | same | same |
| published_at | 2026-10-05T13:33:22Z | 2026-10-05T16:51:46Z | 2026-10-05T16:51:46Z | 2026-10-05T16:55:1xZ |
| consensus | override {difficulty_v2_activation_daa 33000, proving_v0_activation_daa 84100}, activation_height 84100, deadline_note "proving v0" | identical | identical | identical (carried over from the OLD folder's own 0.3.8 manifest) |
| fees_v1_activation_daa, tuning | absent | absent | absent | absent |
| mac | dmg 53c12b6e..., 41,247,419 | dmg 5e57c573..., 41,333,873, URL in its folder | same bytes, public URL | same bytes, URL in the OLD folder |
| windows | inno-setup 86cffd14..., 19,794,320 | inno-setup 1d62a4db..., 19,836,912, URL in its folder | same bytes, public URL | same bytes, URL in the OLD folder |
The OLD folder (what Sam's 0.3.5 Mac reads; `rotation-phase-2.md` 8b keeps it until 7 October): the scratchpad
`publish-old-folder.sh` copied the DMG and the installer from the token folder (byte-compared), ran
`publish-manifest.sh --dest <old> --base-url <old url> --no-deploy` with the same notes, activation height and deadline
note (the override carried over from the folder's own manifest), one deploy of the site by hand (16:55:18Z; `--deploy`
refuses `--dest`), then `--verify-only` for the OLD base URL: `version 0.3.9, byte-identical, signature OK` (16:55:44Z).
## 9. The machines after the publish (manifest live 16:51:46Z)
Baseline 15:35Z: Mac d937c69d app 0.3.8 node 2b6d23ef DAA 111,914 24.0 MH/s; PC 1 ae432dc7 0.3.8 DAA 111,876 110.2 MH/s;
PC 2 1ccfe586 0.3.8 DAA 111,895 122.3 MH/s; Sam's Mac 3a9bf309 silent 5 h (0.3.5); PC 37ba0461 not on the console's
card any more (silent since 13:52Z). A machine counts as updated only when its node reports a DAA score and the miner
a hash rate on 0.3.9.
| Machine | On 0.3.9 | Its log (`IGNEUM-APP version=0.3.9`) |
|---|---|---|
| PC 2 1ccfe586 (Windows) | `update-now-039` ran 16:56:42Z (30 s after the publish, woken); engine restart 16:56:51Z (unix 1791219412), 39 s after the job; node a24ab01a started 16:56:55Z, `node proof verifier reported: command` +6 s; console 17:30Z: node 2.1.0-a24ab01a, DAA 118,757, 120.6 MH/s | run `win-1ccfe586-20261005-165651` |
| Mac d937c69d | `update-now-039` ran 16:56:43Z; engine restart 16:57:10Z (unix 1791219431); node a24ab01a started (pid 58734), verifier `command` +10 s; console 17:30Z: node a24ab01a, DAA 118,710, 25.4 MH/s | run `mac-d937c69d-20261005-165710`; the node log `node-20261005-165710.log` |
| PC 1 ae432dc7 (Windows) | `update-now-039` ran 16:58:22Z (its wake came 100 s later than the others'); engine restart 16:58:34Z (unix 1791219515); node a24ab01a started (pid 13780), verifier `command` +6 s; a SECOND engine start at 17:26:31Z (run `win-ae432dc7-20261005-172631`, node pid 22280) whose cause is not this rollout (no job of this cut targets PC 1 after 16:56Z; another agent's job or a restart of its own; unverified); console 17:32Z: node a24ab01a, DAA 118,867, 116.8 MH/s | runs `win-ae432dc7-20261005-165834` and `-172631` |
| PC 37ba0461 (Windows, the US laptop) | not reached: silent since 13:52Z (on 0.3.7); the manifest and the job wait for it | |
| Sam's Mac 3a9bf309 | not reached: silent since 09:43Z (0.3.5); reads the OLD folder, which carries 0.3.9 since 16:55Z | |
The machines watch (`watch-machines.sh`, once a minute from 16:57Z) printed every machine on 0.3.9 with node
a24ab01a and a DAA score from its 17:00Z tick on, but its success test still named the old node commit (2b6d23ef), so
it reported TIMEOUT at 17:33Z instead of success at about 17:00Z: 25 minutes lost to a watcher that was not shown to
fire on a known-finished case first (the CLAUDE.md rule, again; the script is fixed to take the commit as an argument).
`--mode id` on the installed Mac app (`/Applications/Igneum Miner.app/Contents/Resources/bin/igneum-prove-host`, sha
6846d351..., the DMG's stripped copy): shard `0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a`,
aggregator `0x474678f3...`, equal to PC 2's `/opt/igneum` host (7a). The Mac app node's pool after its restart:
`{entries 0, failed 0, pending 0, verified 0}`, paidShards 388, tip DAA 119,097 (17:34Z).
Provers on: `prove-on-pc2-039` published 17:33:32Z, ran 17:35:55 to 17:36:16Z: `api/prove on: {"ok":true}`; the app's
state 20 s later: `backend cuda`, `enabled true`, `available true`, `current "block 77691 shard 0 (0 txs, 0 pgas, open)"`,
keys 10, "proving on the GPU".
## 10. The fee switch
@ -239,7 +327,63 @@ every app, wait for `Calibrated v1 fees from the override file: ... 210000` on e
object verbatim whenever it differs from `override.json` (same version or not) and the engine restarts the node at a
safe moment.
(pending: the publish itself)
### 10b. The publish (17:37 to 17:42Z)
Order kept: the proving rollout was complete (the first cross-verified shard under the new id, section 10c) before
anything consensus-bearing was published. The object:
`{"difficulty_v2_activation_daa":33000,"proving_v0_activation_daa":84100,"fees_v1_activation_daa":210000}`; H = 210,000;
tip DAA 119,097 at 17:34Z, so H - tip = 90,903 >= 86,400 at the publish (the 24-hour rule holds; H lands about 19:50Z
on 6 October at 0.965 blocks/s).
| Step | When | What |
|---|---|---|
| (a) the token folder and `dl/public/` | 17:37:35Z, died at once; again 17:38:34Z, written and deployed | `packaging/ota/publish-manifest.sh --version 0.3.9 --override '<object>' --activation-height 210000 --deadline-note "fees v1" --notes "Calibrated v1 fees from DAA score 210000 ..." --public --deploy` from the worktree. The first run printed only `failed printing to stdout: Broken pipe` and nothing was written (the signer's `embedded` output into `head -1` under `pipefail`, a race the ship tool's own run of the same script did not hit; the second run went through). The second run: mac and windows entries carried over from the folder's 0.3.9 manifest, signed (key 8f186e37...), `dl/public/` rewritten and signed, one deploy; exit 1 only because the public folder's `igneum-downloads.json` (1,317 bytes) was "NOT the local bytes yet" after 12 tries at the edge; the token manifest verified live (`--verify-only`, 17:41Z: byte-identical, signature OK) and the public manifest read live at 17:41:13Z carries the object |
| (b) the OLD folder | 17:37:35Z, verified 17:37:5xZ | `publish-old-folder.sh 0.3.9 "<notes>" --override '<object>' --activation-height 210000 --deadline-note "fees v1"`: written, deployed, `version 0.3.9, byte-identical, signature OK` (Sam's 0.3.5 Mac, if it wakes, takes 0.3.9 from this folder and then the object) |
| (c) `update-now` to every app | `update-now-fees-210000` 17:37:51Z (ran against the token manifest BEFORE (a) had landed: a no-op), then `update-now-fees-210000-b` 17:41:34Z (apps woken, stamp 3fa32fec) | the apps re-read the manifest, `ota.rs write_override` writes the object to `override.json` and the engine restarts the node at a safe moment |
The live manifests after (a), read 17:41:13Z: token folder and `dl/public/` both `0.3.9, published 2026-10-05T17:38:34Z,
consensus {activation_height 210000, deadline_note "fees v1", override {difficulty_v2_activation_daa 33000,
fees_v1_activation_daa 210000, proving_v0_activation_daa 84100}}`; the OLD folder the same object, published 17:37:35Z.
### 10c. The first cross-verified shard under the new id (before the publish)
PC 2's prover (key e809e396..., payout 0xCAfc6e74...) took block 77691 shard 0 at 17:35:5xZ; the Mac app node
(`node-20261005-165710.log`): `proof record for block 77691 shard 0 by e809e396... (peer): accepted, verify Command(".../igneum-prove-host")`
at 17:36:28.747Z, `proof of block 77691 shard 0 ...: verified in 0.2 s: RESULT verify: VERIFIED in 0.026 s` at
17:36:29.235Z, `chain block 77730: proof record for block 77691 shard 0 ... paid 1791503640000000000 wei to 0xCAfc6e74...`
at 17:36:31.460Z (1.7915 IGN). Proven on PC 2 under `0x2b1a81cb...`, verified on the Mac under the same id in 26 ms.
### 10d. The restarts, in the order they happened
| Node | Restarted with the object | Its lines |
|---|---|---|
| PC 2 1ccfe586 app node | 17:40:55Z (from the first update-now, 17:37:51Z: its app re-read the manifest after (a) had landed at 17:38:34Z) | `node-20261005-165651-r2.log` (digest job `digest-pcs-210000`, 17:43:08Z): `Refusing peer 192.168.68.64:26611 (node 1) and 188.245.5.161:26611 (the seed): consensus params digest mismatch, local ab8847da538dead1dc10e046dfaadab3c1c35928e3748810c4e050d4a886087a remote f10a4eab...`; `override.json` = the object |
| Mac d937c69d app node | 17:42:15Z (`update-now-fees-210000-b` ran 17:42:13Z) | `node-20261005-165710-r2.log`: `Calibrated v1 fees from the override file: ... from DAA score 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a`; `override.json` = the object; refusing node 1 and the seed until they moved |
| PC 1 ae432dc7 app node | between 17:46:06Z (its update-now-b) and 17:50Z: its card shows DAA 120,282 with a peer and 115 MH/s at 17:50Z, in step with the network, which a node on the old digest could not be (every peer would refuse it) | its digest line is OWED: `digest-pcs-210000` reported only from PC 2 and `digest-pc1-210000-b` (17:50:49Z) had not reported by 17:55Z (PC 1 took 100 s and 4 min to fetch the two update-now jobs: slow to wake today) |
| The seed | 17:48:27Z (`infra/devnet/restart-seed.sh '<object>'`, the a24ab01a cross-build 7e26e374... re-copied and checked, unit active, MainPID 119125) | journal: `Calibrated v1 fees ... 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a` |
| The observer | 17:48:53Z (pid 39882) | `observer-v4.out`: `Calibrated v1 fees ... 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a`; peers: the seed and 192.168.68.67 at once |
| Node 1 | 17:49:42Z (pid 40679, caffeinate 40681) | `node1.out`: the same three lines; peers at 17:49:42Z and 17:49:55Z |
Two script faults on the way, both fixed on this branch: (1) `infra/devnet/restart-hand-nodes.sh` refused the a24ab01a
binary (`strings | grep -q`: `-q` closes the pipe, `pipefail` turns `strings`' SIGPIPE into a failure); now `grep -c`
compared with 0, and `lines()` no longer ends the script when a node's lines are not in the log within 10 s; (2) the
scratchpad script used instead stopped after the observer for that second reason (`set -e` on an empty grep), so
node 1 was restarted by its pid by hand at 17:49:42Z with the same command line and the same file. Between 17:40:55Z and
17:49:42Z the network was split as the runbook predicts (app nodes on ab8847da, the three hand-run nodes on
f10a4eab), 8 min 47 s; node 1 was alone for the last 49 s.
### 10e. The digest sweep (17:55Z)
| Node | Digest | Read from |
|---|---|---|
| Mac app node d937c69d | ab8847da538dead1dc10e046dfaadab3c1c35928e3748810c4e050d4a886087a | its node log (above); app state: DAA 120,587, 6 peers, synced, `consensus_switch_daa` 210000 |
| PC 2 app node 1ccfe586 | ab8847da... | its node log through the digest job |
| PC 1 app node ae432dc7 | owed (the card's DAA and peer say the new digest; the line itself not yet read) | |
| The seed | ab8847da... | the journal through `restart-seed.sh` |
| The observer | ab8847da... | `/tmp/igneum-devnet/observer-v4.out` |
| Node 1 | ab8847da... | `/tmp/igneum-devnet/node1.out` |
Equal to the runbook's digest for the object (read on 2b6d23ef) and to this cut's reading on a24ab01a (5a).
## 10a. The fee-switch branch, as polled
@ -252,4 +396,13 @@ safe moment.
## 11. Open after the cut
(pending)
| Item | State |
|---|---|
| PC 1's digest line | owed: `digest-pc1-210000-b` (17:50:49Z) unreported at 17:55Z; read it with `node tools/jobs.mjs digest-pc1-210000-b` |
| The Mac's miner at 0.0 MH/s since the engine's own restart at 17:45:16Z (its hourly update slot, minute 57; `update check: 0.3.9 is current`) | the node is fine (DAA 120,587, 6 peers, synced); the miner's state after that restart not yet read; not caused by the switch (the node restart at 17:42:15Z was separate and mining was already 0.0 at the 17:47Z card) |
| PC 37ba0461 (0.3.7, silent since 13:52Z) and Sam's Mac (0.3.5, silent since 09:43Z) | take 0.3.9 and the object on their next check (the OLD folder carries both); until then each is refused by every peer from its next restart |
| `dl/public/igneum-downloads.json` | the publisher saw the previous bytes at the edge for its 12 tries at 17:39Z; at 17:42:38Z the live file was byte-identical to the local one (sha 3aa32e75...) |
| The 0.3.9 `publish-jobs.sh` through the worktree before the master merge | left the envelope stale once (16:02Z); every later publish went through the main checkout's script, which rewrote it |
| `make-package.sh` gate | skipped on instruction (`SKIP_GATE=1`); its native half ran on every fixture with the DMG's host (5d); PC 2's CUDA build proved and the Mac verified block 77691 within 40 s of provers-on, the stronger check |
| H = 210,000 at about 19:50Z on 6 October | step 8 of the runbook (budgets, base fees, the first v1 shard) is tomorrow's check |
| Carried from 0.3.8 section 11 | the PC 2 jobs-file signature refusal of 13:19Z (did not recur); the 0.3.6 items |

View file

@ -13,11 +13,12 @@ BIN="${IGNEUMD:-/Users/joshm/Projects/igneum/vendor/igneum-node/target-036/relea
COMMIT="${IGNEUMD_COMMIT:-a24ab01a}"
OV=/tmp/igneum-devnet/override-v3.json
echo "$OVJSON" | python3 -c 'import json,sys; o=json.load(sys.stdin); assert isinstance(o, dict) and "fees_v1_activation_daa" in o, "the object must carry fees_v1_activation_daa"; print("override:", json.dumps(o, sort_keys=True))'
strings "$BIN" | grep -q "$COMMIT" || { echo "$BIN is not the $COMMIT node (a 0.3.5 node refuses an override file with fees_v1_activation_daa)"; exit 1; }
# grep -c, not grep -q: -q closes the pipe early and pipefail turns strings' SIGPIPE into a refusal of a good binary (5 October 2026, 17:48Z)
[ "$(strings "$BIN" | grep -c "$COMMIT")" -gt 0 ] || { echo "$BIN is not the $COMMIT node (a 0.3.5 node refuses an override file with fees_v1_activation_daa)"; exit 1; }
cp -p "$OV" "$OV.prev-$(date -u +%Y%m%dT%H%M%SZ)" 2>/dev/null || true
printf '%s\n' "$OVJSON" > "$OV"
stop() { local p; p=$(pgrep -f "$1" | head -1 || true); if [ -n "$p" ]; then kill -INT "$p"; while kill -0 "$p" 2>/dev/null; do sleep 1; done; fi; }
lines() { sleep 10; grep -E "Calibrated v1 fees from the override file|Fees on igneum-devnet|Consensus params digest" "$1" | tail -3; }
lines() { sleep 10; grep -E "Calibrated v1 fees from the override file|Fees on igneum-devnet|Consensus params digest" "$1" | tail -3 || true; }
# the observer first
stop "igneumd --devnet.*observer-v4"
nohup "$BIN" --devnet --nodnsseed --disable-upnp --appdir=/tmp/igneum-devnet/observer-v4 --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 --listen=127.0.0.1:26641 \