150 lines
29 KiB
Markdown
150 lines
29 KiB
Markdown
# Igneum Miner 0.3.13: node-only, the execution layer follows again and the finality route; prepared to the publish gate, 6 October 2026
|
|
|
|
Release engineer, from 13:05 UTC, on the coordinator's instruction: "prepare, so it ships the moment the fix lands: a release-0.3.13 branch,
|
|
NODE-ONLY". Worktree `/Users/joshm/Projects/igneum-wt-ship0313`, branch `release-0.3.13` from master 19edae0; the fork worktree
|
|
`vendor/igneum-node-0313`, branch `release-0.3.13-node`, at 83089544 (the 0.3.12 node) until the two node fixes land on it; `vendor/`
|
|
symlinked to the main checkout's. The 0.3.12 recipe throughout; igneum-labs commits; times UTC.
|
|
|
|
## 1. Why, and what it carries
|
|
|
|
Since the publish-2 restarts of 0.3.12 (about 11:37Z) the execution layer is dead on every node: `eth_blockNumber` answers `0x0` and
|
|
`igneum_getProvingStatus` reads `active: false, paidShards 0, paidWei 0x0` (the observer at 13:05Z). The devnet's pruning point left genesis
|
|
today, and the follower, which walks from genesis in memory, can no longer start; `--archival` does not help an existing datadir.
|
|
|
|
| Change | Where | State |
|
|
|---|---|---|
|
|
| The execution layer follows again: the exec state persisted to the data dir every 5 min and at stop, resumed at start (`--igneum-exec-snapshot=<path>[,<sha256>]`, `igneum_exportExecSnapshot`, the loud "exec not synced" status); the snapshot served and fetched over p2p (protocol 16, messages 76 and 77); the archival walk through the ghostdag store when the virtual-chain query refuses a tip below the retention root; `exec_restart_number`, `exec_restart_hash` and `exec_restart_trust_daa` in the override object (in the digest once set: without them a node on this build stays blocked, the bodies below 27,276 being gone on every hand) | the proving agent's `exec-sync-0313` 05e93f0e (7 commits on 83089544) | merged onto the route fix as release-0.3.13-node **a9dfe78e** (clean, 23 files). Measured by its agent on a copy of node 1's data dir: 27,276 header-only records, the EVM state restarted at chain block 27,276 and re-executed to the sink (130,272) in 93 s; paidShards 1,482, paidWei 1,825.70 IGN; the state persisted (114.8 MB) and resumed in 9 s |
|
|
| The finality route (the fleet's finding, 26 fresh nodes): a certificate for an index below this node's window is ignored (the seed re-locked index 2954 2,811 times in seven minutes and the echo filled every fresh peer's route); the IgneumFinality route takes a checkpoint burst (4,096); a full route drops the message and keeps the peer; votes are skipped during IBD | fork branch `fin-route-0313` 5a339733 (the bench-log entry `fin-route` 05d0944, merged e6c939e) | in: p2p 33, flows 19, finality 12 tests green; harness s7 PASS |
|
|
| The trust window off when `exec_restart_trust_daa` is never (05e93f0e read `carrier_daa < trust` with trust at u64::MAX, always true, so a node WITHOUT the field had the native-statement veto and the assignee check off and would have paid any carried shard record); `startedFrom` reads "genesis" when the follower executed genesis | the proving agent's 78c7f961 (fe6756da inside), found by the igneum-exec test at `proving.rs:1171` on the merged tree | merged as release-0.3.13-node **544fc30f**; igneum-exec 18 of 18 |
|
|
| The Power Helper (the project lead, 11:50Z: one administrator approval, ever: the first Power control action registers a per-user elevated scheduled task, no prompt after), the engine folder lock's ACL (`user:(OI)(CI)F` without /T), the verbatim settings copy for a measurement engine, the watchdog, no firewall prompt for a sweep engine, the jobrun `follow_file`, `tools/ci/playbook-quit-check.sh` (the 5 October rule as a gate; the two agg-cost scripts allow-listed under a dated note, db97665 drops them in the next cut with the aggregation-cost agent's `--stop-miners` change) | `ember-tune` 32b2688 (07d5a72 + master 494c9c7 merged in + the dry-run-3 bench entry), on the coordinator's condition: the Ember agent's unelevated dry run 3 on PC 1 PASSED (ember-dryrun-pc1-3, 14:56:18 to 15:02:08Z, exit 0: the 5090 127.31 MH/s at 316.5 W, the 4070 28.68 MH/s at 102.7 W, nothing set, no prompt) | merged 71f08d5 (07d5a72) then **4deea56** (32b2688); app tests 146 + 28 + 8, UI 26, relay 23, every CI check green including playbook-quit |
|
|
| A fresh node (the proving agent's rented 4090, 14:04Z): on the branch as first merged it executed genesis BEFORE IBD, and after IBD its bodies started at the pruning point, so the follower never proceeded and said nothing: every new 0.3.13 install would end with an empty EVM, silently. ecdccef3: nothing executes before consensus is synced (the sink within 10 minutes of the clock), the exec restart is retried every pass until consensus knows chain block 27,276, a walk that meets missing bodies below the retention root sets `blocked` and asks a peer for the snapshot; 8fe28de9: a flag's snapshot file whose tip consensus does not know yet is retried for 10 minutes | the proving agent's ecdccef3 (8fe28de9 inside), the coordinator's "take it" 15:1xZ; its harness PASSED on it 15:09:48Z (12 checks, both halves) | merged as release-0.3.13-node **bb43e9a8** (2 files, no params or proto change); igneum-exec 18 of 18; the digests unchanged |
|
|
| The six version files | 7c9b00f (`--check`: 0.3.13 in all 6) | |
|
|
|
|
A consensus change after all: the three exec-restart fields enter the digest once set, so this is the 0.3.12 two-publish shape (section 2),
|
|
not the one carried-over publish first planned. The thirteen-field object (the packaged line 71cb8a4, publish 2, the hands' and the seed's
|
|
files at step 2; `exec_restart_trust_daa` 200,000 pending the coordinator's word):
|
|
|
|
```
|
|
<the ten-field object of 0.3.12> + "exec_restart_number":27276,"exec_restart_hash":"bb45cf0dd2d7cc97ebfa5a2701527c09a8ede5d32de74efead9caa293b15688a","exec_restart_trust_daa":200000
|
|
```
|
|
|
|
PROTOCOL_VERSION 15 to 16: the handshake takes the lower version, a 0.3.12 and a 0.3.13 node peer during the window. The cut is no longer
|
|
node-only: the Ember tip rides in the app (above), so the Windows app build reran with the node fix.
|
|
|
|
### 1a. The devnet's one-time state reset (the coordinator's question, answered plainly)
|
|
|
|
No verified snapshot at chain block 27,276 exists. The executor starts at chain block 27,276 (DAA 45,537, the pruning point of 11:40Z) from an
|
|
EMPTY EVM state (`daemon.rs`: "Exec restart from the override file: the EVM state restarts empty at chain block {}"; 3dd9b2c9: "with
|
|
header-only records below it") and executes forward from the stored bodies; `exec_restart_hash` bb45cf0d... is that chain block's hash (where,
|
|
not a state root), and nothing is verified against a header's state root because there is no prior state to verify. Gone: every balance,
|
|
contract and nonce from before chain block 27,276 (the coinbase credits of the chain's first 45,537 DAA, the txgen harness wallets' transfers
|
|
from the relay tests, any contract state). Back, re-derived: coinbase credits from 27,276 on, every shard payout record (proving v0 began at
|
|
DAA 84,100, above the restart, so the proving ledger re-derives whole) and the fee flows after it.
|
|
|
|
| Reading | Before 11:37Z (node 1) | After the restart (the copy, 13:43Z) |
|
|
|---|---|---|
|
|
| `igneum_getProvingStatus` paidShards / paidWei | 663 / 814.64 IGN at 00:3xZ; 1,261 / 1,573 IGN at 08:30Z | 1,482 / 1,825.70 IGN (higher: it grows with the chain, nothing of it is lost) |
|
|
| PC 2's payout address 0xcafc6e74...516a | not read (no balance reading before 11:37Z exists anywhere: the hub's intake carries status lines, not balances) | 267,648 IGN (the rewards of chain blocks 27,276 to 130,272) |
|
|
| node 1's, PC 1's payout addresses | not read | re-derived from 27,276 on, as PC 2's |
|
|
| the first 45,537 DAA (chain blocks 1 to 27,275) | about 124,600 IGN of producer rewards (the subsidy 3.17 IGN at genesis rising to 3.67 at DAA 45,537 on the launch ramp, one blue block a second, the producer share 80%) and about 31,100 IGN of pool escrow (the proving agent's computation from `consensus/core/src/igneum.rs`, approximate) | gone; not attributable to addresses (the coinbase payloads with the IGNA payout addresses are in the pruned bodies, and the header's vote-key fallback names another address) |
|
|
| chain block 1 to 27,275 on the explorer | bodies and state | header-only; history below 27,276 is honest only as headers |
|
|
|
|
The chain, the finality locks and the hash are untouched; the reset is of the execution layer's state, once. the project lead approved the one-time reset
|
|
with the go (15:20Z).
|
|
|
|
## 2. The order at the go (the coordinator relays it; nothing below runs before)
|
|
|
|
Runbook: the session scratchpad's `r0313/rollout-0313.sh`. The 0.3.12 shape: publish 1 the binary with the TEN-field object (digest
|
|
7bd98cc4... unchanged, no window), publish 2 the THIRTEEN-field object (a new digest, one window per side).
|
|
|
|
| Step | What | Check |
|
|
|---|---|---|
|
|
| 0 the baseline | `step_check_observer`: `eth_blockNumber` 0x0 and `igneum_getProvingStatus` inactive on the observer today; node 1 read 814.64 IGN over 663 shards at 00:3xZ and 1,573 over 1,261 at 08:30Z (it grows with the chain) | the numbers to beat after the switch: paidShards >= 1,482, paidWei >= 1,825 IGN |
|
|
| 1a the hand nodes, the seed | `step_1a_hand_nodes`, `step_1a_seed`: the 0.3.13 binary with the SAME ten-field file; `step_mac_miners` only if the Mac mines (paused on the project lead's order since 07:10Z) | each prints 7bd98cc4...; `igneum_getExecStatus.blocked` says why the exec layer waits (the three fields are not set yet) |
|
|
| 1b publish 1 | `step_1b_ship` (`--from ci`): consensus CARRIED OVER (the ten-field object), the DMG, the installer and zip from the Windows run, HiveOS with `--public` | the live manifest 0.3.13, digest unchanged |
|
|
| 1c update-now | PC 1 FIRST (the project lead at its screen for the Ember click; the coordinator's 15:19Z order), then the Mac (its node is node 1: the engine alone), then PC 2 | each app's STATUS on 0.3.13, its node on 7bd98cc4; the coordinator told the second PC 1 shows 0.3.13 (the Ember elevated table run then takes PC 1 for about 45 minutes) |
|
|
| 2a the hand nodes, the seed | `step_2b_hand_nodes`, `step_2b_seed`: the thirteen-field file | each prints the new digest; within about 2 minutes `eth_blockNumber` climbs to the sink, `igneum_getExecStatus` reads `startedFrom "restart at chain block 27276"` (then "snapshot" on every restart after), `blocked null`, and `igneum_getProvingStatus` reads active with paidShards >= 1,482 and paidWei >= 1,825 IGN |
|
|
| 2b publish 2 | `step_2b_manifest`: the thirteen-field object, `--activation-height 198000`, the note names the exec restart | the live manifest carries the three fields |
|
|
| 2c the switch jobs | `step_2b_update_now` Mac (nothing to restart: node 1), then PC 2; PC 1's switch ONLY on the coordinator's "Ember closed" (a node restart under a step aborts the run) | each PC's node restarts once (seconds), its `[proving]` lines resume within minutes, its digest the new one |
|
|
| 3 the sweep | every node on the new digest, exec climbing, proving active; the fleet's rented nodes take the thirteen-field object through their operator (the fleet agent) | the per-machine times in section 4 |
|
|
| 3a a FRESH install | the proving agent's rented 4090 joins from scratch on the branch build the minute I send "publish 1 done" (the snapshot path: a 0.3.13 hand serves it over protocol 16) and again after publish 2 (the restart path): the start time, IBD done, the minute `eth_blockNumber` reached the sink, `startedFrom` and `blocked` then | the row in section 4: time to the executed tip |
|
|
|
|
## 3. Builds and artefacts (to fill when the fork tip is set)
|
|
|
|
| What | Command | Result |
|
|
|---|---|---|
|
|
| The fork's Mac node, a9dfe78e | `CARGO_TARGET_DIR=vendor/igneum-node/target-0313 cargo build --release -j 4 -p kaspad -p igneum-miner --features kaspad/igneum-pow` from `vendor/igneum-node-0313`, under the lock (the target cloned from the route fix's) | 14:51:39 to 14:55:3xZ: igneumd **ef76ff5c1371317d783330e460ad4f6a1a8b3f2cdc55b228362ee64a227b30fa** (41,686,528), igneum-miner b7926642... (8,763,888); `igneumd/2.1.0-a9dfe78e` |
|
|
| The seed's Linux node (glibc 2.36 target, zig) | `NODE_SRC=<abs fork> TARGET_DIR=vendor/igneum-node/target-0313-linux OUT_DIR=<scratch>/r0313/cross infra/cross/build-linux.sh` under the lock | 14:51:47 to 14:55:12Z (203 s): igneumd **c7c696c5fa915350993b29378d3fceb252b88a1af880f6051472aef82f796777** (48,246,888), igneum-miner 77ab2e08... (9,860,400); handed to the fleet agent with the thirteen-field file at 14:56Z |
|
|
| The Windows node exes | `CARGO_TARGET_DIR=vendor/igneum-node/target-0313-win proto-cuda/windows-node/cross-build.sh <fork> 4` on the Mac (mingw) under the lock | 14:51:54 to 14:55:1xZ: igneumd.exe **f4e9ef8a83464535aa314e390682acb0ee0e23ceffea09815d546f5fd1ad90ae** (52,518,912), igneum-miner.exe 574adb79... (11,039,232) |
|
|
| The inputs, the pin | `IGNEUM_WIN_RELEASE=<target-0313-win>/x86_64-pc-windows-gnu/release IGNEUM_NODE_SRC=vendor/igneum-node-0313 packaging/windows/push-inputs.sh`, 14:56:51Z; the 0.3.11-verified workers 2b3b8c92.../edc4a75d... and the telemetry helper 8d679b52... unchanged | `payload-inputs.zip` 2327e1da165afbc2194fc7cd1f1be67c509b6845f6d7c032f6878a00a878991d (65,393,804), signed, live; `node-source.pin` a9dfe78e committed as **0c4f90e** (the CI commit) |
|
|
| The digests on the Mac node ef76ff5c... (ports 60995/60996, 22 s each, under `run`) | the ten-field file: **7bd98cc4...** (unchanged: publish 1 changes no handshake); the thirteen-field file: **b18ed271f75dd46406d230f4156c37472127415a4c32c558bac662f6f840e61c** with `Exec restart from the override file: the EVM state restarts empty at chain block 27276 bb45cf0d...` | |
|
|
| The DMG | `NODE=<fork igneumd> MINER=<fork igneum-miner> PROVE_HOST/PROVE_EXPORT=<the 0.3.12 build, unchanged> packaging/mac/build-dmg.sh` under the lock | 14:58:18 to 14:58:5xZ: `Igneum-Miner-0.3.13.dmg` **90864092fb69a90c0bd90fbfba91f24f9663499252c86e453abe3fa5dedbe716** (41,879,478), engine 0.3.13, node a9dfe78e (41,462,352 inside), the prover host ce03ceb5..., `igneum-bench` from `proto-metal/main.swift` (unchanged), `packaged-config` carries the thirteen-field object, hdiutil checksum valid |
|
|
| The HiveOS package | `NODE_OUT=<scratch>/r0313/cross WORKERS_OUT=<the 0.3.11 Linux workers> VERSION=0.3.13 packaging/hive/make-hive-package.sh`, then `publish-public.sh --hive` into `dl/public` (the 0.3.12 package removed; the ship's deploy carries it) | 14:58:59Z: `igneum-hive-0.3.13.tar.gz` **41e0633b2677005fabd360ad80669221ef8f7ea4da0a4aa48e1b659df8718eec** (24,632,169); the node inside is a9dfe78e |
|
|
| The node suites with the igneum-pow feature (the Mac, no fail-fast) | `CARGO_TARGET_DIR=vendor/igneum-node/target-0313 cargo test --release -j 4 --no-fail-fast -p kaspa-consensus -p kaspa-consensus-core -p igneum-exec -p kaspa-pow -p igneum-miner -p kaspa-p2p-flows -p kaspa-p2p-lib --features kaspa-consensus/igneum-pow,kaspa-pow/igneum-pow` from the fork, under the lock, 14:58:26 to 15:00:1xZ | igneum-miner 18 of 18, p2p-flows 33 of 33 (the IBD vote skip inside), p2p-lib 19 of 19 (the overflow-policy test), kaspa-pow 14 of 14, `db_compat` 7 of 7; kaspa-consensus 99 passed, 1 failed (the known M20 era test; the finality ban test green this run); consensus-core 107 passed, 1 failed (the known fast-time file duplicate-key test); igneum-exec 17 passed, **1 failed, NEW**: `proving::tests::assignment_follows_the_window_and_records_check_against_native_execution` (`proving.rs:1171`: the test expects a record to be refused, the code after 05e93f0e checks it as `assigned: true`; the crate was 17 of 17 on 83089544): the proving agent's to resolve before the gate (a fix commit, or the test's expectation is the stale half) |
|
|
| The Windows run, first round | `windows.yml` run 37483331039 on 0c4f90e (dispatched 14:57:37Z); `ci` 37483332527 | both green by 15:06Z; its installer eab82086... and zip cef39414... carry the a9dfe78e node (the over-paying trust rule): SUPERSEDED, not shipped |
|
|
| The second round (node 544fc30f, app 4deea56) | the Mac node rebuilt 15:02 to 15:03Z: igneumd **dd5eeda5b92463e929d07479b2fc77b4ba75c7e9354ab48a96ed22a9637e0b4d** (41,703,168); the Linux node 15:04Z: igneumd **2449d5fa3b16531b33068b75c3e5045de580119c9824053a8eaf58dfe6d8c484** (48,246,632), handed to the fleet agent in place of c7c696c5; the Windows node 15:04Z: igneumd.exe **b06f08dac020f639d2dbeec20b60f48d7025c333e59eaa3fbbed037ea33b96b9** (52,518,912); the digests re-read on dd5eeda5: 7bd98cc4... (ten) and b18ed271... (thirteen), unchanged; the inputs pushed 15:05:16Z: `payload-inputs.zip` 3a99a86f6d4dcbe4c4cdd3cc11c13e8973b1af9f3c87647cd1a420885ea94819 (65,393,980), `node-source.pin` 544fc30f as **55ef102** (the CI commit) | |
|
|
| The Windows run, second round | `windows.yml` run 37484511273 on 55ef102; the DMG 3697306e... and HiveOS 6933c0c7... on 544fc30f | SUPERSEDED by the third round (the fresh-joiner fix); the 544fc30f artefacts kept aside under the scratchpad |
|
|
| The third round (node bb43e9a8, app 4deea56) | the Mac node 15:09 to 15:10Z: igneumd **487312aa31c85bde583b7cea220ba2dd76cd4c64913c224c58664ca12f4f4f23** (41,719,760), igneum-miner b7926642...; the Linux node 15:10Z: igneumd **d6350586fe837b1f71696546628ec071b6accce2531aef776cf2b1e5487a8cdc** (48,263,528), handed to the fleet agent in place of 2449d5fa (and c7c696c5 before it); the Windows node 15:1xZ: igneumd.exe **c4441a3abed26465f2bddef6a60b237444219d4dcb6470694e19430d56d9f830** (52,527,616), igneum-miner.exe 574adb79...; the digests re-read on 487312aa: 7bd98cc4... (ten) and b18ed271... (thirteen); the inputs pushed 15:11:30Z: `payload-inputs.zip` 561878b8dcd91803ad3ff8055190e1201ba48975936bcbcdac17827f0fe7bb34 (65,398,104), `node-source.pin` bb43e9a8 as **be344ff** (the CI commit) | |
|
|
| The Windows run | `windows.yml` run 37485442000 on be344ff (dispatched 15:12:16Z); `ci` 37485442462 | both GREEN 15:18:48Z: `Igneum-Miner-Setup-0.3.13.exe` **499a8ede6268426342ffd3ae0da2354f3a14522dda7ff7c41aa21ae3e169deab** (45,396,595); `igneum-windows-app.zip` **dc3116242fc55c22259e5eb0339a5cb9dfbd8a4e7deb918f01307f291e3d1b76** (65,664,639), its igneumd.exe c4441a3a... (the bb43e9a8 cross-build); fetched 15:18:5xZ, not deployed. The 0.3.13 CI verdict: ci 37485442462 on be344ff; the Windows build 37485442000 on be344ff |
|
|
| The DMG (third round) | `NODE=<fork igneumd 487312aa> MINER=... PROVE_HOST/PROVE_EXPORT=<the 0.3.12 build, unchanged> packaging/mac/build-dmg.sh` under the lock | 15:12:51 to 15:13:1xZ: `Igneum-Miner-0.3.13.dmg` **2d35a0160925ef5fcd6d85dc653deab328c20261bfac1e37a7c3e8f4c18baf48** (41,859,802), engine 0.3.13 (the Ember helper inside), node bb43e9a8, `packaged-config` with the thirteen-field object, hdiutil checksum valid |
|
|
| The HiveOS package (third round) | `make-hive-package.sh` from the d6350586 Linux node and the 0.3.11 Linux workers, then `publish-public.sh --hive` into `dl/public` (the ship's deploy carries it) | `igneum-hive-0.3.13.tar.gz` **65ea42600236c7024844d85fc401ef2c4650e671d92061c8db6415038bac9530** (24,634,543) |
|
|
|
|
## 4. The rollout (the project lead's go 15:20Z through the coordinator; two publishes)
|
|
|
|
Baseline 15:19:45Z: tip DAA about 196,900; the observer and node 1 were DOWN since 14:56:28Z (both SIGTERMed by a hand that was not mine,
|
|
the same minute the Igneum Wallet app's own node started on this Mac, pid 81040 on 26620/26621/26800; the live stats stale 24 minutes);
|
|
the seed on 83089544 at 7bd98cc4; the Mac 0.3.12 (paused on the project lead's order), PC 2 0.3.12 at 115 MH/s, PC 1 0.3.12 with the project lead at its screen.
|
|
|
|
| Step | Time | Result |
|
|
|---|---|---|
|
|
| 1a the observer, node 1 | 15:20:38Z (pid 46848), 15:20:51Z (pid 48213) | `igneumd/2.1.0-bb43e9a8` on the ten-field file, 7bd98cc4...; the exec layer reads `blocked: exec not synced: the executor is at chain block 0 and the bodies below this node's retention root are gone`, `startedFrom genesis`, as designed before the three fields |
|
|
| 1a the seed | 15:21:18Z (MainPID 140366) | the same binary (d6350586...), the same digest |
|
|
| the fleet's first word | 15:24Z | "hands on 0.3.13" to the fleet agent (22 boxes on d6350586 with the ten-field file) |
|
|
| 1b publish 1 | the ship 15:22:01 to 15:23:45Z from 5ca4913 | ci "already" (37485442000), fetch "already", dmg "already", copy ok, manifest 0.3.13 published 15:22:05Z with `consensus` CARRIED OVER (the ten-field object), deployed 15:22:40Z, HiveOS 65ea4260... served |
|
|
| 1c update-now, PC 1 FIRST | 15:23:56Z | ran 15:24:42Z; engine restart 15:24:52Z (run `win-ae432dc7-20261006-152452`), "updated to Igneum Miner 0.3.13 from 0.3.12", per-user install, no prompt; cards "RTX 5090, RTX 4070, AMD integrated, RX 9070 XT"; mining 15:25:53Z; digest 7bd98cc4; the coordinator told 15:27Z, the Ember elevated table run then took PC 1 |
|
|
| 1c update-now, the Mac | 15:24:39Z | ran 15:25:11Z; engine restart 15:25:19Z (run `mac-d937c69d-20261006-152519`), 0.3.13, paused as ordered, node 1 six peers |
|
|
| 1c update-now, PC 2 | 15:25:08Z | ran 15:25:54Z; engine restart 15:26:06Z (run `win-1ccfe586-20261006-152606`), 0.3.13, worker ready 15:27:06Z, mining 15:27:07Z, digest 7bd98cc4 |
|
|
| 2a the observer, node 1 | 15:29:15Z (pid 63960), 15:29:29Z (pid 64106) | the thirteen-field file: **b18ed271...**, `Exec restart from the override file: the EVM state restarts empty at chain block 27276 bb45cf0d...` |
|
|
| 2a the seed | 15:29:53Z (MainPID 140546) | b18ed271... |
|
|
| 2b publish 2 | 15:30:03Z | the manifest with the thirteen-field object, activation 198000, "proving v1 fresh-record rule; exec restart at chain block 27276"; signed, verified, deployed |
|
|
| the exec checks, the observer (134,556 is the selected-chain height: chain blocks, one per selected-parent step, under the DAG's 151,606 blocks and the DAA 197,219; the two flat minutes were the node's own consensus re-sync after its restart, not the follower; the proving agent's reading) | 15:33:23Z | `eth_blockNumber` 134,556 (from 0 at 15:21Z), `igneum_getExecStatus` startedFrom "restart at chain block 27276", blocked null, `igneum_getProvingStatus` active, paidShards **1,482**, paidWei **1825.699240038 IGN**: the copy's numbers to the wei; the state persisted at 134,556 (118.7 MB, sha 0xe5194123...) at 15:35Z; 134,884 at 15:37:55Z, 134,909 at 15:39:47Z (the follower's rate after the restart: 1.2 then 0.2 chain blocks a second, under watch) |
|
|
| the fleet's second word | 15:35Z | "hands on b18ed271" to the fleet agent (STEP=file on 22 boxes; its per-box seconds-to-climbing follow) |
|
|
| 2c switch, the Mac | 15:35:42Z | ran 15:36:12Z: "consensus override changed; the node restarts with it at a safe moment"; its node is node 1, already switched |
|
|
| 2c switch, PC 2 | 15:36:08Z | ran 15:37:02Z, node restarted 15:37:03Z (pid 27552) on b18ed271, "Exec restart ... shard records carried below DAA score 200000 are paid as carried"; mining again 15:38:38Z, 13.7 MH/s ramping at 15:39:08Z. Its prover then logged `block 35012 shard 0: exporter: Error: segment 0: port state root 0x7e37a9fb... differs from the node's` and the same for block 61972: the assignment window hands it blocks far below the tip whose carried records were made over the old state; the re-derived state at those heights has another root, so those shards cannot be proven (they pay as carried below the trust DAA, so nothing is lost but prover passes): the proving agent's ruling: REAL and not designed. The exporter (igneum-prove-export) rebuilds a fixture's pre-state by replaying the exported chain from genesis, crediting block rewards from the headers, so above the restart it holds 27,276 blocks of rewards the node's restarted state never had and its root differs from the node's record: every shard assigned above R fails on every prover (the fleet's boxes too) until the fix, so every segment and shard payout stops from publish 2 until 0.3.14 (the carried records below the trust DAA pay regardless). The fix (about an hour, its commits to follow): `igneum_exportSegments` carries the restart (number, hash) and the exporter starts its replay at R from the registry-only state; one RPC field on the node, the exporter side in the proving package; 0.3.14's first item |
|
|
| 2c switch, PC 1 (last) | 15:42:50Z, on the coordinator's "Ember closed" (its run ended 15:39:03Z) | ran 15:43:36Z, node restarted 15:43:37Z (pid 3632) on b18ed271 with the exec restart line; the miner waiting at 15:43:54Z and mining again within the minute (the C43 class: a minute at 0 MH/s after a node restart); three cards |
|
|
| node 1's refusals | 15:39Z | 12 in the last 400 lines: the peers still on 7bd98cc4 (PC 1, the unswitched fleet boxes, the Igneum Wallet 0.1.4's bundled node on this Mac, which nobody updates: a wallet cut owes the thirteen-field object) |
|
|
| a FRESH install (the proving agent's rented 4090) | "publish 1 done" sent 15:25Z, "publish 2 done" 15:36Z | (its numbers pending) |
|
|
|
|
|
|
### 4a. The incident after publish 2 (15:42 to 15:52Z): the activation inside the window, the deep reorg, the moved pruning point
|
|
|
|
| Time | What |
|
|
|---|---|
|
|
| 15:42:57Z | `proving_v1_fresh_rule_daa` 198,000 armed while PC 1 (its switch held for the Ember run) and the fleet's unswitched boxes still mined on the ten-field object: two sides for a minute, the losing side 229 blocks deep (the coordinator's reading); PC 1's switch job went 15:42:50Z, its node on b18ed271 at 15:43:37Z, 40 s after the arming |
|
|
| 15:44:47Z | the hands' exec layer: `selected-chain reorg: 274 chain blocks removed, unwinding to height 134884 ... reorg deeper than the snapshot ring; replaying from genesis`; genesis executed, then nothing; `eth_blockNumber` 0, `startedFrom genesis`, `blocked null`; the same on the seed and on every fleet box that had come back (seven of them had reached the tip through the restart path in 87 to 188 s each: hub 88 s, 3080 87 s, 4070-1 113 s, 4090-3 138 s, rig-4090x8 138 s, A5000 163 s, 3090-4 188 s) |
|
|
| 15:48:00Z | the observer and node 1 restarted by me (the hands script, the same file): the restart path REFUSED: `sink ... is chain block 0; pruning point eb2a5d70... is chain block None (DAA 88763); retention root eb2a5d70... (DAA 88763)`, `exec restart at chain block 27276 ... not yet possible (the selected-parent walk from the sink never met the queried block)`: the devnet's pruning point moved from bb45cf0d (DAA 45,537) to eb2a5d70 (DAA 88,763) since 11:40Z, the anchor is below it and off the walk, and the genesis replay had overwritten the good 119 MB snapshot (tip 135,138) with a 2,629-byte tip-0 one (`exec-snapshot.prev.bin` keeps the good one). The exec layer and proving are at 0 on every node until the fix |
|
|
| the fix | the proving agent's, as 0.3.14 (the tooling refuses "0.3.13.1"): never replay from genesis on a pruned node (unwind through the persisted generations, else a peer's snapshot over protocol 16); keep executing from the persisted state when the pruning point passes the anchor (the anchor only for a node with nothing); never overwrite a good snapshot with a tip-0 one |
|
|
| node 1's follower | found in the same hour: node 1's `chain follower stopped` at every start since 12:37Z because both hands bound the eth_ JSON-RPC on 26790 and node 1 lost ("Address already in use"), so the Mac app's node never executed; `restart-hand-nodes.sh` now gives node 1 26791 (this commit), live at the next hand restart |
|
|
|
|
Rules from it (the coordinator's four and two more): the switch jobs go out before the height, no miner stays on the old side across an
|
|
activation; an activation height is never set inside a rollout window (the floor of 10,800 exists for that); a deep reorg must not reset the
|
|
exec layer (unwind through the persisted generations, else a peer's snapshot; never a genesis replay on a pruned node); the pruning point must
|
|
not strand an anchor (a node keeps executing from its persisted state, the anchor is for a node with nothing); a hand node restart is
|
|
announced before it runs (the 14:56:28Z SIGTERM of both hands by an unnamed hand cost 24 minutes of stale stats); the two hands never share a port.
|
|
|
|
The exec checks of section 2 are therefore RED at the close of this cut (every node at `eth_blockNumber` 0 since 15:44:47Z, `startedFrom`
|
|
genesis or snapshot-at-0, `paidShards` 0; the chain, the finality locks and the hash untouched; the proving ledger below the trust DAA pays
|
|
as carried once the exec layer is back); the cause is the pruning point past the anchor and the genesis replay over the snapshot; the fix is
|
|
0.3.14 (node-only, the proving agent's branch; one publish with the object carried over and a hand-node swap first if it is code-only, the
|
|
exec check per node the gate; two publishes only if a field changes; no re-pin of the anchor). The coordinator's decision 15:53Z.
|
|
|
|
## 5. The one line for the project lead
|
|
|
|
When he says go: the execution layer comes back on every node at its restart (the hands and the seed by hand, the Mac, PC 2 and PC 1
|
|
through two update-nows each, the node restarting once for the switch), with the state restarted empty at chain block 27,276 (6 October
|
|
11:40Z, DAA 45,537) and re-derived forward, so every IGN earned since then by mining and proving is back on the ledger (PC 2's address
|
|
267,648 IGN; 1,482 shards, 1,825.70 IGN of prover payouts) and the chain's first 45,537 DAA (about 124,600 IGN of mining rewards and 31,100
|
|
IGN of escrow, approximate) are gone from it and cannot be given back to addresses, once; the chain, the finality locks and the hash are untouched; a fresh node joining the devnet no longer loses the seed every
|
|
checkpoint once the seed runs this.
|
|
|
|
## 6. Owed
|