From 025cc19b767e8c79d31db2a5f6bf50d06b47a744 Mon Sep 17 00:00:00 2001 From: igneum-labs <337424239+igneum-labs@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:52:30 +0000 Subject: [PATCH] release 0.3.13: the incident after publish 2 (the activation inside the window, the deep reorg, the moved pruning point) and node 1's own eth port in restart-hand-nodes.sh Co-Authored-By: Claude Fable 5.1 --- docs/plans/release-0.3.13.md | 14 ++++++++++++++ infra/devnet/restart-hand-nodes.sh | 2 +- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/plans/release-0.3.13.md b/docs/plans/release-0.3.13.md index 8bf820c96..e09df33be 100644 --- a/docs/plans/release-0.3.13.md +++ b/docs/plans/release-0.3.13.md @@ -116,6 +116,20 @@ the seed on 83089544 at 7bd98cc4; the Mac 0.3.12 (paused on the project lead's o | a FRESH install (the proving agent's rented 4090) | "publish 1 done" sent 15:25Z, "publish 2 done" 15:36Z | (its numbers pending) | +### 4a. The incident after publish 2 (15:42 to 15:52Z): the activation inside the window, the deep reorg, the moved pruning point + +| Time | What | +|---|---| +| 15:42:57Z | `proving_v1_fresh_rule_daa` 198,000 armed while PC 1 (its switch held for the Ember run) and the fleet's unswitched boxes still mined on the ten-field object: two sides for a minute, the losing side 229 blocks deep (the coordinator's reading); PC 1's switch job went 15:42:50Z, its node on b18ed271 at 15:43:37Z, 40 s after the arming | +| 15:44:47Z | the hands' exec layer: `selected-chain reorg: 274 chain blocks removed, unwinding to height 134884 ... reorg deeper than the snapshot ring; replaying from genesis`; genesis executed, then nothing; `eth_blockNumber` 0, `startedFrom genesis`, `blocked null`; the same on the seed and on every fleet box that had come back (seven of them had reached the tip through the restart path in 87 to 188 s each: hub 88 s, 3080 87 s, 4070-1 113 s, 4090-3 138 s, rig-4090x8 138 s, A5000 163 s, 3090-4 188 s) | +| 15:48:00Z | the observer and node 1 restarted by me (the hands script, the same file): the restart path REFUSED: `sink ... is chain block 0; pruning point eb2a5d70... is chain block None (DAA 88763); retention root eb2a5d70... (DAA 88763)`, `exec restart at chain block 27276 ... not yet possible (the selected-parent walk from the sink never met the queried block)`: the devnet's pruning point moved from bb45cf0d (DAA 45,537) to eb2a5d70 (DAA 88,763) since 11:40Z, the anchor is below it and off the walk, and the genesis replay had overwritten the good 119 MB snapshot (tip 135,138) with a 2,629-byte tip-0 one (`exec-snapshot.prev.bin` keeps the good one). The exec layer and proving are at 0 on every node until the fix | +| the fix | the proving agent's, as 0.3.14 (the tooling refuses "0.3.13.1"): never replay from genesis on a pruned node (unwind through the persisted generations, else a peer's snapshot over protocol 16); keep executing from the persisted state when the pruning point passes the anchor (the anchor only for a node with nothing); never overwrite a good snapshot with a tip-0 one | +| node 1's follower | found in the same hour: node 1's `chain follower stopped` at every start since 12:37Z because both hands bound the eth_ JSON-RPC on 26790 and node 1 lost ("Address already in use"), so the Mac app's node never executed; `restart-hand-nodes.sh` now gives node 1 26791 (this commit), live at the next hand restart | + +Rules from it: a miner never stays on the old side across an activation (the switch jobs go out before the height; an activation height is +never set inside a rollout window, the floor of 10,800 exists for that); a hand node restart is announced before it runs (the 14:56:28Z +SIGTERM of both hands by an unnamed hand cost 24 minutes of stale stats); the two hands never share a port. + ## 5. The one line for the project lead When he says go: the execution layer comes back on every node at its restart (the hands and the seed by hand, the Mac, PC 2 and PC 1 diff --git a/infra/devnet/restart-hand-nodes.sh b/infra/devnet/restart-hand-nodes.sh index a499fcfab..0016213c4 100755 --- a/infra/devnet/restart-hand-nodes.sh +++ b/infra/devnet/restart-hand-nodes.sh @@ -26,7 +26,7 @@ nohup "$BIN" --devnet --nodnsseed --disable-upnp --appdir=/tmp/igneum-devnet/obs lines /tmp/igneum-devnet/observer-v4.out # node 1, under caffeinate as it runs today stop "igneumd --devnet.*appdir=/tmp/igneum-devnet/node1 " -nohup caffeinate -dims "$BIN" --devnet --nodnsseed --disable-upnp --enable-unsynced-mining --appdir=/tmp/igneum-devnet/node1 --rpclisten=0.0.0.0:26610 --listen=0.0.0.0:26611 \ +nohup caffeinate -dims "$BIN" --devnet --nodnsseed --disable-upnp --enable-unsynced-mining --appdir=/tmp/igneum-devnet/node1 --rpclisten=0.0.0.0:26610 --listen=0.0.0.0:26611 --evm-rpclisten=127.0.0.1:26791 \ --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 --override-params-file="$OV" --nologfiles --yes >> /tmp/igneum-devnet/node1.out 2>&1 & lines /tmp/igneum-devnet/node1.out echo "running now:"; ps -o pid=,lstart=,command= -p "$(pgrep -f 'igneumd --devnet' | tr '\n' ',' | sed 's/,$//')" | cut -c1-160