103 lines
9 KiB
Markdown
103 lines
9 KiB
Markdown
# Exec sync: the executor's state after the pruning point left genesis (6 October 2026)
|
|
|
|
## What happened
|
|
|
|
The devnet's pruning point left genesis on 6 October 2026 at about 11:40Z (chain block 27,276, DAA 45,537). Every
|
|
node pruned the block bodies and acceptance rows below it and kept the headers and ghostdag rows. The execution
|
|
layer's state lived only in memory and was rebuilt from genesis at every start; every node had restarted for
|
|
0.3.11 and 0.3.12. From 11:40Z every node read `eth_blockNumber` 0x0 and `igneum_getProvingStatus` active false:
|
|
no balances, no proving work list, no payouts, mining untouched. No hand ran `--archival`, so no data dir on the
|
|
network holds the bodies the state was built from, and the state cannot be rebuilt from genesis anywhere.
|
|
|
|
## The fix, fork branch `exec-sync-0313` (off the 0.3.12 node 83089544), node only
|
|
|
|
| Commit | What |
|
|
|---|---|
|
|
| 4f05e56a | the exec state persisted to the data dir every 5 minutes and at stop (two generations; resumed at start when its tip is still a chain block); `--igneum-exec-snapshot=<path>[,<sha256>]` and `igneum_exportExecSnapshot`; the loud status when the follower cannot execute (warn every minute, `igneum_getExecStatus.blocked`, the proving status's `execSync`) |
|
|
| 3bd31a20 | the snapshot served and fetched over p2p at protocol 16 (messages 76 and 77, 1 MiB chunks, the sha256 on every chunk); a blocked executor asks one peer every 20 s; `--igneum-exec-snapshot=peer` |
|
|
| efa6924d, 7cb712f5, 80674943 | the archival walk: the selected-parent chain read from the ghostdag store when the virtual-chain query refuses a tip below the retention root; the mergeset order rebuilt from ghostdag rows when the acceptance row is pruned |
|
|
| 3dd9b2c9 | the exec restart: `exec_restart_number` and `exec_restart_hash` in the override object (in the digest once set): header-only records below R, the EVM state fresh at R, executed from R's body on; the status reports the pruning point and the retention root with their chain block numbers |
|
|
| 05e93f0e | `exec_restart_trust_daa`: a shard record carried below it pays as carried, without the native-statement veto and the assignee check |
|
|
|
|
## Measured on a copy of node 1's data dir (the Mac, `tools/lock/with-lock.sh run`)
|
|
|
|
| Figure | Value |
|
|
|---|---|
|
|
| The pruning point and retention root, node 1 and the observer | chain block 27,276, hash bb45cf0dd2d7cc97ebfa5a2701527c09a8ede5d32de74efead9caa293b15688a, DAA 45,537; its body held on both |
|
|
| Bodies below it | gone ("cannot find full block" at chain block 1) |
|
|
| Replay from 27,276 to the sink (130,073, then 130,272) | 92 s and 93 s wall from the node's start, over 2,000 chain blocks a second |
|
|
| The before column (read on node 1 before 11:37Z) | paid shards 663 and 814.64 IGN at 00:3xZ, 1,261 and 1,573.13 IGN at 08:30Z; no balance reading was taken before 11:37Z |
|
|
| Without the trust rule | paidShards 0: every historical record vetoed (the records attest the original state roots) |
|
|
| With `exec_restart_trust_daa` 200,000 | paidShards 1,482, paidWei 1,825.70 IGN (node 1 read 1,261 and 1,573 at 08:30Z; the chain moved on) |
|
|
| PC 2's payout address at the tip | 267,648 IGN |
|
|
| The persisted file | 114,830,936 bytes, 130,273 records (1,200 full), 31 accounts |
|
|
| Resume after a restart | 9 s, no replay |
|
|
|
|
## What it means
|
|
|
|
| Tier | Before the fix | After the cut with the three fields |
|
|
|---|---|---|
|
|
| Every node, home miner, rig, pool | an empty EVM since 11:40Z: no balances, no proving, no payouts; mining fine | the chain's state from chain block 27,276 on, rebuilt in about 90 s at the switch; every restart after that resumes in seconds |
|
|
| Miners | rewards invisible | rewards since R back (PC 2: 267,648 IGN); the rewards of chain blocks 0 to 27,275 (the devnet's first 12 hours) lost: from the schedule (`consensus/core/src/igneum.rs`: 31.69 IGN a second base, the launch ramp from 10% at genesis to 100% at 30 days, the producer share 80%; one blue block per DAA second at 1 bps, approximate) about 124,600 IGN of producer rewards and about 31,100 IGN of pool escrow over DAA 0 to 45,536, 3.17 IGN a block at genesis rising to 3.67 at DAA 45,537; the per-address split is unrecoverable (the IGNA payout addresses were in the pruned coinbase payloads); the project lead's call on seeding |
|
|
| Provers | payouts invisible | every carried record since proving v0 paid as carried (1,482 shards); new proofs verify against the new state from the trust DAA on |
|
|
| Joiners today | an empty EVM for ever | IBD gives the pruning point 27,276 and the bodies from it, so a fresh node replays the same 90 s with no snapshot; once the pruning point moves past R the p2p snapshot (protocol 16) starts it |
|
|
|
|
## A fresh node today (the rented RTX 4090 box, 6 October 2026, 14:04Z on)
|
|
|
|
The exec-sync node built with the `igneum-pow` feature (a node without it refuses the headers proof: "epoch seed
|
|
header ... fails its own proof of work", the stub engine; the Linux cross-build sets the feature, a hand build must),
|
|
a fresh data dir, the live override object, the seed as the peer.
|
|
|
|
| Phase | What happened |
|
|
|---|---|
|
|
| IBD with the headers proof | failed 10 times from 14:04Z to 14:45Z, "peer connection is closed" from the seed and the hub about 25 s after the proof arrived, each right after "incoming route capacity for message type IgneumFinality has been reached" for that peer; the 11th attempt went through at 14:45Z |
|
|
| Bodies from the pruning point | 5 to 10 blocks a second (1% at 14:46Z, 2% at 14:47Z of the blocks since 11:40Z) |
|
|
|
|
Consequence: a home miner installing today waits about an hour before the node is synced, most of it in IBD retries
|
|
the finality relay causes, not in execution. The fix is in the p2p layer (the finality flow's route capacity during
|
|
IBD, or no finality messages to a peer still in IBD): the consensus engineer's, not this branch's. The executed tip
|
|
and the replay time follow the sync.
|
|
|
|
## The second incident, 15:43Z, and 0.3.13.1 / 0.3.14 (fork 1f59c5d0, app 9c2a148)
|
|
|
|
What forked at DAA 198,000 (15:42:57Z): the fresh-record rule's crossing with PC 1 and the unswitched fleet boxes on the
|
|
old ten-field object (digest 7bd98cc4) against the hands on the thirteen-field one (b18ed271); the digest refusal
|
|
split the chain for about a minute and blue work resolved it 274 chain blocks deep. The executor's deep-reorg rule
|
|
replayed from genesis on every pruned node, reset the EVM to 0 and persisted a tip-0 file over the good one; the
|
|
pruning point had moved to eb2a5d70 (DAA 88,763), so the restart anchor at 27,276 was dead too. The hands came back
|
|
on the 13:45Z export (`--igneum-exec-snapshot`, the shipper, 16:11Z to 16:18Z).
|
|
|
|
The rules now (all in fork 1f59c5d0, measured below):
|
|
|
|
| Rule | Before | Now |
|
|
|---|---|---|
|
|
| Reorg deeper than the ring | genesis replay | the newest persisted generation at or below the fork, else blocked and a peer asked; genesis only with the whole history |
|
|
| Ring | 64 states | 2,048; a snapshot load seeds it |
|
|
| Persist | any state | never tip 0; never a lower tip over a file still on the chain |
|
|
| Flag file missing, unreadable, mispinned | ran as if unset | blocked loudly, retried every 2 s |
|
|
| p2p snapshot | any newer | never tip 0, below the restart, at or below the own tip, or while running |
|
|
| Pin at the restart block | none | `exec_restart_state_root` (14th field, in the digest); a differing state never runs |
|
|
| Exporter input | replay from 0 or the restart | `preState`, the account dump after block n-1 from the ring |
|
|
|
|
Measured (Mac, `tools/lock/with-lock.sh run`, 6 October 2026): `tools/exec-sync/reorg.mjs` 18/18 in 263.6 s (a
|
|
300-block reorg unwound from the ring in 134.6 s, and from the persisted generation after a restart in 261.6 s,
|
|
re-executing from the fork block 69; both reach B's state root and balance; nothing from genesis);
|
|
`tools/exec-sync/net.mjs` 15/15 in 104.7 s (the loud flag, the account-dump cut); `cargo test -p igneum-exec` 19.
|
|
|
|
The pin's value: block 27,276 (bb45cf0d...) post-state root 0xed27bb2d6128daf50493ed5539a73bb93f9d7c63ad7f70a7cbb46ba81c9e164e,
|
|
the same on a restart-path node and a snapshot-recovered node (node 1 read at 26791; the fleet at 16:5xZ). The
|
|
record at 27,275 is header-only (root 0), so the pre-state at the restart is the registry-only state the rule names.
|
|
|
|
Consequence per tier: a node restarted from the recovery file proves again with the 0.3.14 app, since the dump needs
|
|
the block within the last 2,048 chain blocks (about 34 minutes at 1 block/s); a fresh joiner of any tier (8 to 32 GB,
|
|
rig, pool, Apple) needs a peer's snapshot (protocol 16) or the file, because no node holds bodies from 27,276; the
|
|
fourteenth field changes the digest, so every miner switches before the hands (the publish-order rule).
|
|
|
|
Harness note: A's lone miner casts no finality vote in the reorg harness; with one key at 83% of A's window its votes
|
|
certified A's own branch and the finality rule refused B's chain for good (by design; the live reorg happened on an
|
|
uncertified minute).
|
|
|
|
## Open
|
|
|
|
The fresh-join time on a rented box (in progress); the trust DAA's value (at or above the switch); the first 12
|
|
hours' rewards (the project lead); the p2p snapshot's fast-time harness case; `--archival` on at least one hand from now on.
|