igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity

release 0.3.13: the incident after publish 2 (the activation inside the window, the deep reorg, the moved pruning point) and node 1's own eth port in restart-hand-nodes.sh

Browse source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-06 15:52:30 +00:00
parent 4a8d399a16
commit 2da469020a
2 changed files with 15 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
docs/plans/release-0.3.13.md
View file

@ -116,6 +116,20 @@ the seed on 83089544 at 7bd98cc4; the Mac 0.3.12 (paused on the project lead's o
| a FRESH install (the proving agent's rented 4090) | "publish 1 done" sent 15:25Z, "publish 2 done" 15:36Z | (its numbers pending) |
### 4a. The incident after publish 2 (15:42 to 15:52Z): the activation inside the window, the deep reorg, the moved pruning point
| Time | What |
|---|---|
| 15:42:57Z | `proving_v1_fresh_rule_daa` 198,000 armed while PC 1 (its switch held for the Ember run) and the fleet's unswitched boxes still mined on the ten-field object: two sides for a minute, the losing side 229 blocks deep (the coordinator's reading); PC 1's switch job went 15:42:50Z, its node on b18ed271 at 15:43:37Z, 40 s after the arming |
| 15:44:47Z | the hands' exec layer: `selected-chain reorg: 274 chain blocks removed, unwinding to height 134884 ... reorg deeper than the snapshot ring; replaying from genesis`; genesis executed, then nothing; `eth_blockNumber` 0, `startedFrom genesis`, `blocked null`; the same on the seed and on every fleet box that had come back (seven of them had reached the tip through the restart path in 87 to 188 s each: hub 88 s, 3080 87 s, 4070-1 113 s, 4090-3 138 s, rig-4090x8 138 s, A5000 163 s, 3090-4 188 s) |
| 15:48:00Z | the observer and node 1 restarted by me (the hands script, the same file): the restart path REFUSED: `sink ... is chain block 0; pruning point eb2a5d70... is chain block None (DAA 88763); retention root eb2a5d70... (DAA 88763)`, `exec restart at chain block 27276 ... not yet possible (the selected-parent walk from the sink never met the queried block)`: the devnet's pruning point moved from bb45cf0d (DAA 45,537) to eb2a5d70 (DAA 88,763) since 11:40Z, the anchor is below it and off the walk, and the genesis replay had overwritten the good 119 MB snapshot (tip 135,138) with a 2,629-byte tip-0 one (`exec-snapshot.prev.bin` keeps the good one). The exec layer and proving are at 0 on every node until the fix |
| the fix | the proving agent's, as 0.3.14 (the tooling refuses "0.3.13.1"): never replay from genesis on a pruned node (unwind through the persisted generations, else a peer's snapshot over protocol 16); keep executing from the persisted state when the pruning point passes the anchor (the anchor only for a node with nothing); never overwrite a good snapshot with a tip-0 one |
| node 1's follower | found in the same hour: node 1's `chain follower stopped` at every start since 12:37Z because both hands bound the eth_ JSON-RPC on 26790 and node 1 lost ("Address already in use"), so the Mac app's node never executed; `restart-hand-nodes.sh` now gives node 1 26791 (this commit), live at the next hand restart |
Rules from it: a miner never stays on the old side across an activation (the switch jobs go out before the height; an activation height is
never set inside a rollout window, the floor of 10,800 exists for that); a hand node restart is announced before it runs (the 14:56:28Z
SIGTERM of both hands by an unnamed hand cost 24 minutes of stale stats); the two hands never share a port.
## 5. The one line for the project lead
When he says go: the execution layer comes back on every node at its restart (the hands and the seed by hand, the Mac, PC 2 and PC 1

2
infra/devnet/restart-hand-nodes.sh
View file

@ -26,7 +26,7 @@ nohup "$BIN" --devnet --nodnsseed --disable-upnp --appdir=/tmp/igneum-devnet/obs
lines /tmp/igneum-devnet/observer-v4.out
# node 1, under caffeinate as it runs today
stop "igneumd --devnet.*appdir=/tmp/igneum-devnet/node1 "
nohup caffeinate -dims "$BIN" --devnet --nodnsseed --disable-upnp --enable-unsynced-mining --appdir=/tmp/igneum-devnet/node1 --rpclisten=0.0.0.0:26610 --listen=0.0.0.0:26611 \
nohup caffeinate -dims "$BIN" --devnet --nodnsseed --disable-upnp --enable-unsynced-mining --appdir=/tmp/igneum-devnet/node1 --rpclisten=0.0.0.0:26610 --listen=0.0.0.0:26611 --evm-rpclisten=127.0.0.1:26791 \
--addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 --override-params-file="$OV" --nologfiles --yes >> /tmp/igneum-devnet/node1.out 2>&1 &
lines /tmp/igneum-devnet/node1.out
echo "running now:"; ps -o pid=,lstart=,command= -p "$(pgrep -f 'igneumd --devnet' | tr '\n' ',' | sed 's/,$//')" | cut -c1-160