igneum/docs/plans/release-0.3.13.md
igneum-labs 7eed16a29a Pre-public scrub, the text pass (7 October 2026, 19:5x UK): no founder name, personal login, earlier business or personal address in any tracked text file, and a gate check that keeps it so
The sweep (main's item 1): 199 tracked text files, 783 lines. The founder's full name, first name and possessive become "the founder" (sentence starts capitalised); the lowercase operating-system user name in WSL paths and commands becomes <user>; the second owner login becomes "the second owner login"; the three earlier businesses and the two other brands become "the other business", "the earlier entity", "the earlier business" and "another brand"; the Chrome profile rule names the igneum.network profile, not the profile's label. The standing commit login igneum-labs is not a founder term here: the fresh-repository step renames it in the history (docs/plans/history-rewrite.md, tools/repo/fresh-repo.sh).

The patterns never appear in plain text in the tree (a plaintext list would be the hit): tools/ci/founder-strings.b64 (perl regex, tab, a sample per row) is read by tools/ci/founder-strings-check.sh (every tracked text file, perl, known-failed first: the self-test plants each row's sample in a fixture and the hit must name the file), by tools/community/discord-hooks.mjs (the guard's founder and business rows; the test takes its fixtures from the samples) and by tools/repo/fresh-repo.sh (the business names of the rewrite rules). site/forbidden-strings.txt carries the same patterns as b64: lines, decoded case-insensitive by site/scrub.mjs and tools/ci/launch-gates-check.mjs (whose fixture now plants an encoded made-up name). The check runs in the gate's tree checks on every merge.

Not in this commit, by main's word: the 105 commit messages and 40 personal-identity commits that need the history rewrite (listed, not run), and the secrets found by gitleaks over the history (reported with owners).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 18:39:50 +00:00

29 KiB

Igneum Miner 0.3.13: node-only, the execution layer follows again and the finality route; prepared to the publish gate, 6 October 2026

Release engineer, from 13:05 UTC, on the coordinator's instruction: "prepare, so it ships the moment the fix lands: a release-0.3.13 branch, NODE-ONLY". Worktree /Users/joshm/Projects/igneum-wt-ship0313, branch release-0.3.13 from master 19edae0; the fork worktree vendor/igneum-node-0313, branch release-0.3.13-node, at 83089544 (the 0.3.12 node) until the two node fixes land on it; vendor/ symlinked to the main checkout's. The 0.3.12 recipe throughout; igneum-labs commits; times UTC.

1. Why, and what it carries

Since the publish-2 restarts of 0.3.12 (about 11:37Z) the execution layer is dead on every node: eth_blockNumber answers 0x0 and igneum_getProvingStatus reads active: false, paidShards 0, paidWei 0x0 (the observer at 13:05Z). The devnet's pruning point left genesis today, and the follower, which walks from genesis in memory, can no longer start; --archival does not help an existing datadir.

Change Where State
The execution layer follows again: the exec state persisted to the data dir every 5 min and at stop, resumed at start (--igneum-exec-snapshot=<path>[,<sha256>], igneum_exportExecSnapshot, the loud "exec not synced" status); the snapshot served and fetched over p2p (protocol 16, messages 76 and 77); the archival walk through the ghostdag store when the virtual-chain query refuses a tip below the retention root; exec_restart_number, exec_restart_hash and exec_restart_trust_daa in the override object (in the digest once set: without them a node on this build stays blocked, the bodies below 27,276 being gone on every hand) the proving agent's exec-sync-0313 05e93f0e (7 commits on 83089544) merged onto the route fix as release-0.3.13-node a9dfe78e (clean, 23 files). Measured by its agent on a copy of node 1's data dir: 27,276 header-only records, the EVM state restarted at chain block 27,276 and re-executed to the sink (130,272) in 93 s; paidShards 1,482, paidWei 1,825.70 IGN; the state persisted (114.8 MB) and resumed in 9 s
The finality route (the fleet's finding, 26 fresh nodes): a certificate for an index below this node's window is ignored (the seed re-locked index 2954 2,811 times in seven minutes and the echo filled every fresh peer's route); the IgneumFinality route takes a checkpoint burst (4,096); a full route drops the message and keeps the peer; votes are skipped during IBD fork branch fin-route-0313 5a339733 (the bench-log entry fin-route 05d0944, merged e6c939e) in: p2p 33, flows 19, finality 12 tests green; harness s7 PASS
The trust window off when exec_restart_trust_daa is never (05e93f0e read carrier_daa < trust with trust at u64::MAX, always true, so a node WITHOUT the field had the native-statement veto and the assignee check off and would have paid any carried shard record); startedFrom reads "genesis" when the follower executed genesis the proving agent's 78c7f961 (fe6756da inside), found by the igneum-exec test at proving.rs:1171 on the merged tree merged as release-0.3.13-node 544fc30f; igneum-exec 18 of 18
The Power Helper (the founder, 11:50Z: one administrator approval, ever: the first Power control action registers a per-user elevated scheduled task, no prompt after), the engine folder lock's ACL (user:(OI)(CI)F without /T), the verbatim settings copy for a measurement engine, the watchdog, no firewall prompt for a sweep engine, the jobrun follow_file, tools/ci/playbook-quit-check.sh (the 5 October rule as a gate; the two agg-cost scripts allow-listed under a dated note, db97665 drops them in the next cut with the aggregation-cost agent's --stop-miners change) ember-tune 32b2688 (07d5a72 + master 494c9c7 merged in + the dry-run-3 bench entry), on the coordinator's condition: the Ember agent's unelevated dry run 3 on PC 1 PASSED (ember-dryrun-pc1-3, 14:56:18 to 15:02:08Z, exit 0: the 5090 127.31 MH/s at 316.5 W, the 4070 28.68 MH/s at 102.7 W, nothing set, no prompt) merged 71f08d5 (07d5a72) then 4deea56 (32b2688); app tests 146 + 28 + 8, UI 26, relay 23, every CI check green including playbook-quit
A fresh node (the proving agent's rented 4090, 14:04Z): on the branch as first merged it executed genesis BEFORE IBD, and after IBD its bodies started at the pruning point, so the follower never proceeded and said nothing: every new 0.3.13 install would end with an empty EVM, silently. ecdccef3: nothing executes before consensus is synced (the sink within 10 minutes of the clock), the exec restart is retried every pass until consensus knows chain block 27,276, a walk that meets missing bodies below the retention root sets blocked and asks a peer for the snapshot; 8fe28de9: a flag's snapshot file whose tip consensus does not know yet is retried for 10 minutes the proving agent's ecdccef3 (8fe28de9 inside), the coordinator's "take it" 15:1xZ; its harness PASSED on it 15:09:48Z (12 checks, both halves) merged as release-0.3.13-node bb43e9a8 (2 files, no params or proto change); igneum-exec 18 of 18; the digests unchanged
The six version files 7c9b00f (--check: 0.3.13 in all 6)

A consensus change after all: the three exec-restart fields enter the digest once set, so this is the 0.3.12 two-publish shape (section 2), not the one carried-over publish first planned. The thirteen-field object (the packaged line 71cb8a4, publish 2, the hands' and the seed's files at step 2; exec_restart_trust_daa 200,000 pending the coordinator's word):

<the ten-field object of 0.3.12> + "exec_restart_number":27276,"exec_restart_hash":"bb45cf0dd2d7cc97ebfa5a2701527c09a8ede5d32de74efead9caa293b15688a","exec_restart_trust_daa":200000

PROTOCOL_VERSION 15 to 16: the handshake takes the lower version, a 0.3.12 and a 0.3.13 node peer during the window. The cut is no longer node-only: the Ember tip rides in the app (above), so the Windows app build reran with the node fix.

1a. The devnet's one-time state reset (the coordinator's question, answered plainly)

No verified snapshot at chain block 27,276 exists. The executor starts at chain block 27,276 (DAA 45,537, the pruning point of 11:40Z) from an EMPTY EVM state (daemon.rs: "Exec restart from the override file: the EVM state restarts empty at chain block {}"; 3dd9b2c9: "with header-only records below it") and executes forward from the stored bodies; exec_restart_hash bb45cf0d... is that chain block's hash (where, not a state root), and nothing is verified against a header's state root because there is no prior state to verify. Gone: every balance, contract and nonce from before chain block 27,276 (the coinbase credits of the chain's first 45,537 DAA, the txgen harness wallets' transfers from the relay tests, any contract state). Back, re-derived: coinbase credits from 27,276 on, every shard payout record (proving v0 began at DAA 84,100, above the restart, so the proving ledger re-derives whole) and the fee flows after it.

Reading Before 11:37Z (node 1) After the restart (the copy, 13:43Z)
igneum_getProvingStatus paidShards / paidWei 663 / 814.64 IGN at 00:3xZ; 1,261 / 1,573 IGN at 08:30Z 1,482 / 1,825.70 IGN (higher: it grows with the chain, nothing of it is lost)
PC 2's payout address 0xcafc6e74...516a not read (no balance reading before 11:37Z exists anywhere: the hub's intake carries status lines, not balances) 267,648 IGN (the rewards of chain blocks 27,276 to 130,272)
node 1's, PC 1's payout addresses not read re-derived from 27,276 on, as PC 2's
the first 45,537 DAA (chain blocks 1 to 27,275) about 124,600 IGN of producer rewards (the subsidy 3.17 IGN at genesis rising to 3.67 at DAA 45,537 on the launch ramp, one blue block a second, the producer share 80%) and about 31,100 IGN of pool escrow (the proving agent's computation from consensus/core/src/igneum.rs, approximate) gone; not attributable to addresses (the coinbase payloads with the IGNA payout addresses are in the pruned bodies, and the header's vote-key fallback names another address)
chain block 1 to 27,275 on the explorer bodies and state header-only; history below 27,276 is honest only as headers

The chain, the finality locks and the hash are untouched; the reset is of the execution layer's state, once. The founder approved the one-time reset with the go (15:20Z).

2. The order at the go (the coordinator relays it; nothing below runs before)

Runbook: the session scratchpad's r0313/rollout-0313.sh. The 0.3.12 shape: publish 1 the binary with the TEN-field object (digest 7bd98cc4... unchanged, no window), publish 2 the THIRTEEN-field object (a new digest, one window per side).

Step What Check
0 the baseline step_check_observer: eth_blockNumber 0x0 and igneum_getProvingStatus inactive on the observer today; node 1 read 814.64 IGN over 663 shards at 00:3xZ and 1,573 over 1,261 at 08:30Z (it grows with the chain) the numbers to beat after the switch: paidShards >= 1,482, paidWei >= 1,825 IGN
1a the hand nodes, the seed step_1a_hand_nodes, step_1a_seed: the 0.3.13 binary with the SAME ten-field file; step_mac_miners only if the Mac mines (paused on the founder's order since 07:10Z) each prints 7bd98cc4...; igneum_getExecStatus.blocked says why the exec layer waits (the three fields are not set yet)
1b publish 1 step_1b_ship (--from ci): consensus CARRIED OVER (the ten-field object), the DMG, the installer and zip from the Windows run, HiveOS with --public the live manifest 0.3.13, digest unchanged
1c update-now PC 1 FIRST (the founder at its screen for the Ember click; the coordinator's 15:19Z order), then the Mac (its node is node 1: the engine alone), then PC 2 each app's STATUS on 0.3.13, its node on 7bd98cc4; the coordinator told the second PC 1 shows 0.3.13 (the Ember elevated table run then takes PC 1 for about 45 minutes)
2a the hand nodes, the seed step_2b_hand_nodes, step_2b_seed: the thirteen-field file each prints the new digest; within about 2 minutes eth_blockNumber climbs to the sink, igneum_getExecStatus reads startedFrom "restart at chain block 27276" (then "snapshot" on every restart after), blocked null, and igneum_getProvingStatus reads active with paidShards >= 1,482 and paidWei >= 1,825 IGN
2b publish 2 step_2b_manifest: the thirteen-field object, --activation-height 198000, the note names the exec restart the live manifest carries the three fields
2c the switch jobs step_2b_update_now Mac (nothing to restart: node 1), then PC 2; PC 1's switch ONLY on the coordinator's "Ember closed" (a node restart under a step aborts the run) each PC's node restarts once (seconds), its [proving] lines resume within minutes, its digest the new one
3 the sweep every node on the new digest, exec climbing, proving active; the fleet's rented nodes take the thirteen-field object through their operator (the fleet agent) the per-machine times in section 4
3a a FRESH install the proving agent's rented 4090 joins from scratch on the branch build the minute I send "publish 1 done" (the snapshot path: a 0.3.13 hand serves it over protocol 16) and again after publish 2 (the restart path): the start time, IBD done, the minute eth_blockNumber reached the sink, startedFrom and blocked then the row in section 4: time to the executed tip

3. Builds and artefacts (to fill when the fork tip is set)

What Command Result
The fork's Mac node, a9dfe78e CARGO_TARGET_DIR=vendor/igneum-node/target-0313 cargo build --release -j 4 -p kaspad -p igneum-miner --features kaspad/igneum-pow from vendor/igneum-node-0313, under the lock (the target cloned from the route fix's) 14:51:39 to 14:55:3xZ: igneumd ef76ff5c1371317d783330e460ad4f6a1a8b3f2cdc55b228362ee64a227b30fa (41,686,528), igneum-miner b7926642... (8,763,888); igneumd/2.1.0-a9dfe78e
The seed's Linux node (glibc 2.36 target, zig) NODE_SRC=<abs fork> TARGET_DIR=vendor/igneum-node/target-0313-linux OUT_DIR=<scratch>/r0313/cross infra/cross/build-linux.sh under the lock 14:51:47 to 14:55:12Z (203 s): igneumd c7c696c5fa915350993b29378d3fceb252b88a1af880f6051472aef82f796777 (48,246,888), igneum-miner 77ab2e08... (9,860,400); handed to the fleet agent with the thirteen-field file at 14:56Z
The Windows node exes CARGO_TARGET_DIR=vendor/igneum-node/target-0313-win proto-cuda/windows-node/cross-build.sh <fork> 4 on the Mac (mingw) under the lock 14:51:54 to 14:55:1xZ: igneumd.exe f4e9ef8a83464535aa314e390682acb0ee0e23ceffea09815d546f5fd1ad90ae (52,518,912), igneum-miner.exe 574adb79... (11,039,232)
The inputs, the pin IGNEUM_WIN_RELEASE=<target-0313-win>/x86_64-pc-windows-gnu/release IGNEUM_NODE_SRC=vendor/igneum-node-0313 packaging/windows/push-inputs.sh, 14:56:51Z; the 0.3.11-verified workers 2b3b8c92.../edc4a75d... and the telemetry helper 8d679b52... unchanged payload-inputs.zip 2327e1da165afbc2194fc7cd1f1be67c509b6845f6d7c032f6878a00a878991d (65,393,804), signed, live; node-source.pin a9dfe78e committed as 0c4f90e (the CI commit)
The digests on the Mac node ef76ff5c... (ports 60995/60996, 22 s each, under run) the ten-field file: 7bd98cc4... (unchanged: publish 1 changes no handshake); the thirteen-field file: b18ed271f75dd46406d230f4156c37472127415a4c32c558bac662f6f840e61c with Exec restart from the override file: the EVM state restarts empty at chain block 27276 bb45cf0d...
The DMG NODE=<fork igneumd> MINER=<fork igneum-miner> PROVE_HOST/PROVE_EXPORT=<the 0.3.12 build, unchanged> packaging/mac/build-dmg.sh under the lock 14:58:18 to 14:58:5xZ: Igneum-Miner-0.3.13.dmg 90864092fb69a90c0bd90fbfba91f24f9663499252c86e453abe3fa5dedbe716 (41,879,478), engine 0.3.13, node a9dfe78e (41,462,352 inside), the prover host ce03ceb5..., igneum-bench from proto-metal/main.swift (unchanged), packaged-config carries the thirteen-field object, hdiutil checksum valid
The HiveOS package NODE_OUT=<scratch>/r0313/cross WORKERS_OUT=<the 0.3.11 Linux workers> VERSION=0.3.13 packaging/hive/make-hive-package.sh, then publish-public.sh --hive into dl/public (the 0.3.12 package removed; the ship's deploy carries it) 14:58:59Z: igneum-hive-0.3.13.tar.gz 41e0633b2677005fabd360ad80669221ef8f7ea4da0a4aa48e1b659df8718eec (24,632,169); the node inside is a9dfe78e
The node suites with the igneum-pow feature (the Mac, no fail-fast) CARGO_TARGET_DIR=vendor/igneum-node/target-0313 cargo test --release -j 4 --no-fail-fast -p kaspa-consensus -p kaspa-consensus-core -p igneum-exec -p kaspa-pow -p igneum-miner -p kaspa-p2p-flows -p kaspa-p2p-lib --features kaspa-consensus/igneum-pow,kaspa-pow/igneum-pow from the fork, under the lock, 14:58:26 to 15:00:1xZ igneum-miner 18 of 18, p2p-flows 33 of 33 (the IBD vote skip inside), p2p-lib 19 of 19 (the overflow-policy test), kaspa-pow 14 of 14, db_compat 7 of 7; kaspa-consensus 99 passed, 1 failed (the known M20 era test; the finality ban test green this run); consensus-core 107 passed, 1 failed (the known fast-time file duplicate-key test); igneum-exec 17 passed, 1 failed, NEW: proving::tests::assignment_follows_the_window_and_records_check_against_native_execution (proving.rs:1171: the test expects a record to be refused, the code after 05e93f0e checks it as assigned: true; the crate was 17 of 17 on 83089544): the proving agent's to resolve before the gate (a fix commit, or the test's expectation is the stale half)
The Windows run, first round windows.yml run 37483331039 on 0c4f90e (dispatched 14:57:37Z); ci 37483332527 both green by 15:06Z; its installer eab82086... and zip cef39414... carry the a9dfe78e node (the over-paying trust rule): SUPERSEDED, not shipped
The second round (node 544fc30f, app 4deea56) the Mac node rebuilt 15:02 to 15:03Z: igneumd dd5eeda5b92463e929d07479b2fc77b4ba75c7e9354ab48a96ed22a9637e0b4d (41,703,168); the Linux node 15:04Z: igneumd 2449d5fa3b16531b33068b75c3e5045de580119c9824053a8eaf58dfe6d8c484 (48,246,632), handed to the fleet agent in place of c7c696c5; the Windows node 15:04Z: igneumd.exe b06f08dac020f639d2dbeec20b60f48d7025c333e59eaa3fbbed037ea33b96b9 (52,518,912); the digests re-read on dd5eeda5: 7bd98cc4... (ten) and b18ed271... (thirteen), unchanged; the inputs pushed 15:05:16Z: payload-inputs.zip 3a99a86f6d4dcbe4c4cdd3cc11c13e8973b1af9f3c87647cd1a420885ea94819 (65,393,980), node-source.pin 544fc30f as 55ef102 (the CI commit)
The Windows run, second round windows.yml run 37484511273 on 55ef102; the DMG 3697306e... and HiveOS 6933c0c7... on 544fc30f SUPERSEDED by the third round (the fresh-joiner fix); the 544fc30f artefacts kept aside under the scratchpad
The third round (node bb43e9a8, app 4deea56) the Mac node 15:09 to 15:10Z: igneumd 487312aa31c85bde583b7cea220ba2dd76cd4c64913c224c58664ca12f4f4f23 (41,719,760), igneum-miner b7926642...; the Linux node 15:10Z: igneumd d6350586fe837b1f71696546628ec071b6accce2531aef776cf2b1e5487a8cdc (48,263,528), handed to the fleet agent in place of 2449d5fa (and c7c696c5 before it); the Windows node 15:1xZ: igneumd.exe c4441a3abed26465f2bddef6a60b237444219d4dcb6470694e19430d56d9f830 (52,527,616), igneum-miner.exe 574adb79...; the digests re-read on 487312aa: 7bd98cc4... (ten) and b18ed271... (thirteen); the inputs pushed 15:11:30Z: payload-inputs.zip 561878b8dcd91803ad3ff8055190e1201ba48975936bcbcdac17827f0fe7bb34 (65,398,104), node-source.pin bb43e9a8 as be344ff (the CI commit)
The Windows run windows.yml run 37485442000 on be344ff (dispatched 15:12:16Z); ci 37485442462 both GREEN 15:18:48Z: Igneum-Miner-Setup-0.3.13.exe 499a8ede6268426342ffd3ae0da2354f3a14522dda7ff7c41aa21ae3e169deab (45,396,595); igneum-windows-app.zip dc3116242fc55c22259e5eb0339a5cb9dfbd8a4e7deb918f01307f291e3d1b76 (65,664,639), its igneumd.exe c4441a3a... (the bb43e9a8 cross-build); fetched 15:18:5xZ, not deployed. The 0.3.13 CI verdict: ci 37485442462 on be344ff; the Windows build 37485442000 on be344ff
The DMG (third round) NODE=<fork igneumd 487312aa> MINER=... PROVE_HOST/PROVE_EXPORT=<the 0.3.12 build, unchanged> packaging/mac/build-dmg.sh under the lock 15:12:51 to 15:13:1xZ: Igneum-Miner-0.3.13.dmg 2d35a0160925ef5fcd6d85dc653deab328c20261bfac1e37a7c3e8f4c18baf48 (41,859,802), engine 0.3.13 (the Ember helper inside), node bb43e9a8, packaged-config with the thirteen-field object, hdiutil checksum valid
The HiveOS package (third round) make-hive-package.sh from the d6350586 Linux node and the 0.3.11 Linux workers, then publish-public.sh --hive into dl/public (the ship's deploy carries it) igneum-hive-0.3.13.tar.gz 65ea42600236c7024844d85fc401ef2c4650e671d92061c8db6415038bac9530 (24,634,543)

4. The rollout (the founder's go 15:20Z through the coordinator; two publishes)

Baseline 15:19:45Z: tip DAA about 196,900; the observer and node 1 were DOWN since 14:56:28Z (both SIGTERMed by a hand that was not mine, the same minute the Igneum Wallet app's own node started on this Mac, pid 81040 on 26620/26621/26800; the live stats stale 24 minutes); the seed on 83089544 at 7bd98cc4; the Mac 0.3.12 (paused on the founder's order), PC 2 0.3.12 at 115 MH/s, PC 1 0.3.12 with the founder at its screen.

Step Time Result
1a the observer, node 1 15:20:38Z (pid 46848), 15:20:51Z (pid 48213) igneumd/2.1.0-bb43e9a8 on the ten-field file, 7bd98cc4...; the exec layer reads blocked: exec not synced: the executor is at chain block 0 and the bodies below this node's retention root are gone, startedFrom genesis, as designed before the three fields
1a the seed 15:21:18Z (MainPID 140366) the same binary (d6350586...), the same digest
the fleet's first word 15:24Z "hands on 0.3.13" to the fleet agent (22 boxes on d6350586 with the ten-field file)
1b publish 1 the ship 15:22:01 to 15:23:45Z from 5ca4913 ci "already" (37485442000), fetch "already", dmg "already", copy ok, manifest 0.3.13 published 15:22:05Z with consensus CARRIED OVER (the ten-field object), deployed 15:22:40Z, HiveOS 65ea4260... served
1c update-now, PC 1 FIRST 15:23:56Z ran 15:24:42Z; engine restart 15:24:52Z (run win-ae432dc7-20261006-152452), "updated to Igneum Miner 0.3.13 from 0.3.12", per-user install, no prompt; cards "RTX 5090, RTX 4070, AMD integrated, RX 9070 XT"; mining 15:25:53Z; digest 7bd98cc4; the coordinator told 15:27Z, the Ember elevated table run then took PC 1
1c update-now, the Mac 15:24:39Z ran 15:25:11Z; engine restart 15:25:19Z (run mac-d937c69d-20261006-152519), 0.3.13, paused as ordered, node 1 six peers
1c update-now, PC 2 15:25:08Z ran 15:25:54Z; engine restart 15:26:06Z (run win-1ccfe586-20261006-152606), 0.3.13, worker ready 15:27:06Z, mining 15:27:07Z, digest 7bd98cc4
2a the observer, node 1 15:29:15Z (pid 63960), 15:29:29Z (pid 64106) the thirteen-field file: b18ed271..., Exec restart from the override file: the EVM state restarts empty at chain block 27276 bb45cf0d...
2a the seed 15:29:53Z (MainPID 140546) b18ed271...
2b publish 2 15:30:03Z the manifest with the thirteen-field object, activation 198000, "proving v1 fresh-record rule; exec restart at chain block 27276"; signed, verified, deployed
the exec checks, the observer (134,556 is the selected-chain height: chain blocks, one per selected-parent step, under the DAG's 151,606 blocks and the DAA 197,219; the two flat minutes were the node's own consensus re-sync after its restart, not the follower; the proving agent's reading) 15:33:23Z eth_blockNumber 134,556 (from 0 at 15:21Z), igneum_getExecStatus startedFrom "restart at chain block 27276", blocked null, igneum_getProvingStatus active, paidShards 1,482, paidWei 1825.699240038 IGN: the copy's numbers to the wei; the state persisted at 134,556 (118.7 MB, sha 0xe5194123...) at 15:35Z; 134,884 at 15:37:55Z, 134,909 at 15:39:47Z (the follower's rate after the restart: 1.2 then 0.2 chain blocks a second, under watch)
the fleet's second word 15:35Z "hands on b18ed271" to the fleet agent (STEP=file on 22 boxes; its per-box seconds-to-climbing follow)
2c switch, the Mac 15:35:42Z ran 15:36:12Z: "consensus override changed; the node restarts with it at a safe moment"; its node is node 1, already switched
2c switch, PC 2 15:36:08Z ran 15:37:02Z, node restarted 15:37:03Z (pid 27552) on b18ed271, "Exec restart ... shard records carried below DAA score 200000 are paid as carried"; mining again 15:38:38Z, 13.7 MH/s ramping at 15:39:08Z. Its prover then logged block 35012 shard 0: exporter: Error: segment 0: port state root 0x7e37a9fb... differs from the node's and the same for block 61972: the assignment window hands it blocks far below the tip whose carried records were made over the old state; the re-derived state at those heights has another root, so those shards cannot be proven (they pay as carried below the trust DAA, so nothing is lost but prover passes): the proving agent's ruling: REAL and not designed. The exporter (igneum-prove-export) rebuilds a fixture's pre-state by replaying the exported chain from genesis, crediting block rewards from the headers, so above the restart it holds 27,276 blocks of rewards the node's restarted state never had and its root differs from the node's record: every shard assigned above R fails on every prover (the fleet's boxes too) until the fix, so every segment and shard payout stops from publish 2 until 0.3.14 (the carried records below the trust DAA pay regardless). The fix (about an hour, its commits to follow): igneum_exportSegments carries the restart (number, hash) and the exporter starts its replay at R from the registry-only state; one RPC field on the node, the exporter side in the proving package; 0.3.14's first item
2c switch, PC 1 (last) 15:42:50Z, on the coordinator's "Ember closed" (its run ended 15:39:03Z) ran 15:43:36Z, node restarted 15:43:37Z (pid 3632) on b18ed271 with the exec restart line; the miner waiting at 15:43:54Z and mining again within the minute (the C43 class: a minute at 0 MH/s after a node restart); three cards
node 1's refusals 15:39Z 12 in the last 400 lines: the peers still on 7bd98cc4 (PC 1, the unswitched fleet boxes, the Igneum Wallet 0.1.4's bundled node on this Mac, which nobody updates: a wallet cut owes the thirteen-field object)
a FRESH install (the proving agent's rented 4090) "publish 1 done" sent 15:25Z, "publish 2 done" 15:36Z (its numbers pending)

4a. The incident after publish 2 (15:42 to 15:52Z): the activation inside the window, the deep reorg, the moved pruning point

Time What
15:42:57Z proving_v1_fresh_rule_daa 198,000 armed while PC 1 (its switch held for the Ember run) and the fleet's unswitched boxes still mined on the ten-field object: two sides for a minute, the losing side 229 blocks deep (the coordinator's reading); PC 1's switch job went 15:42:50Z, its node on b18ed271 at 15:43:37Z, 40 s after the arming
15:44:47Z the hands' exec layer: selected-chain reorg: 274 chain blocks removed, unwinding to height 134884 ... reorg deeper than the snapshot ring; replaying from genesis; genesis executed, then nothing; eth_blockNumber 0, startedFrom genesis, blocked null; the same on the seed and on every fleet box that had come back (seven of them had reached the tip through the restart path in 87 to 188 s each: hub 88 s, 3080 87 s, 4070-1 113 s, 4090-3 138 s, rig-4090x8 138 s, A5000 163 s, 3090-4 188 s)
15:48:00Z the observer and node 1 restarted by me (the hands script, the same file): the restart path REFUSED: sink ... is chain block 0; pruning point eb2a5d70... is chain block None (DAA 88763); retention root eb2a5d70... (DAA 88763), exec restart at chain block 27276 ... not yet possible (the selected-parent walk from the sink never met the queried block): the devnet's pruning point moved from bb45cf0d (DAA 45,537) to eb2a5d70 (DAA 88,763) since 11:40Z, the anchor is below it and off the walk, and the genesis replay had overwritten the good 119 MB snapshot (tip 135,138) with a 2,629-byte tip-0 one (exec-snapshot.prev.bin keeps the good one). The exec layer and proving are at 0 on every node until the fix
the fix the proving agent's, as 0.3.14 (the tooling refuses "0.3.13.1"): never replay from genesis on a pruned node (unwind through the persisted generations, else a peer's snapshot over protocol 16); keep executing from the persisted state when the pruning point passes the anchor (the anchor only for a node with nothing); never overwrite a good snapshot with a tip-0 one
node 1's follower found in the same hour: node 1's chain follower stopped at every start since 12:37Z because both hands bound the eth_ JSON-RPC on 26790 and node 1 lost ("Address already in use"), so the Mac app's node never executed; restart-hand-nodes.sh now gives node 1 26791 (this commit), live at the next hand restart

Rules from it (the coordinator's four and two more): the switch jobs go out before the height, no miner stays on the old side across an activation; an activation height is never set inside a rollout window (the floor of 10,800 exists for that); a deep reorg must not reset the exec layer (unwind through the persisted generations, else a peer's snapshot; never a genesis replay on a pruned node); the pruning point must not strand an anchor (a node keeps executing from its persisted state, the anchor is for a node with nothing); a hand node restart is announced before it runs (the 14:56:28Z SIGTERM of both hands by an unnamed hand cost 24 minutes of stale stats); the two hands never share a port.

The exec checks of section 2 are therefore RED at the close of this cut (every node at eth_blockNumber 0 since 15:44:47Z, startedFrom genesis or snapshot-at-0, paidShards 0; the chain, the finality locks and the hash untouched; the proving ledger below the trust DAA pays as carried once the exec layer is back); the cause is the pruning point past the anchor and the genesis replay over the snapshot; the fix is 0.3.14 (node-only, the proving agent's branch; one publish with the object carried over and a hand-node swap first if it is code-only, the exec check per node the gate; two publishes only if a field changes; no re-pin of the anchor). The coordinator's decision 15:53Z.

5. The one line for the founder

When he says go: the execution layer comes back on every node at its restart (the hands and the seed by hand, the Mac, PC 2 and PC 1 through two update-nows each, the node restarting once for the switch), with the state restarted empty at chain block 27,276 (6 October 11:40Z, DAA 45,537) and re-derived forward, so every IGN earned since then by mining and proving is back on the ledger (PC 2's address 267,648 IGN; 1,482 shards, 1,825.70 IGN of prover payouts) and the chain's first 45,537 DAA (about 124,600 IGN of mining rewards and 31,100 IGN of escrow, approximate) are gone from it and cannot be given back to addresses, once; the chain, the finality locks and the hash are untouched; a fresh node joining the devnet no longer loses the seed every checkpoint once the seed runs this.

6. Owed