diff --git a/docs/design/snapshot-sync.md b/docs/design/snapshot-sync.md new file mode 100644 index 000000000..ba465cf5a --- /dev/null +++ b/docs/design/snapshot-sync.md @@ -0,0 +1,43 @@ +# Snapshot sync (node 2.0.3.1): a joiner starts from a signed state snapshot served by the seeds + +The node lane, 9 October 2026, 11:4x UK, on main's ruling of 11:1x UK: the fresh-sync memory is a public blocker. Measured this morning on a quiet box with the three hubs as the only peers (build-2, RSS every 30 s): a fresh 2.0.2 node (5d53a591) reads 29 GB at DAA 43,816 and a fresh 2.0.3 node (27f54124) 34 GB at the tip, DAA 53,670, in twenty minutes — about 0.65 GB per 1,000 blocks, linear, before any inbound traffic (a hand on build-1 with 93 inbound island peers read 62 to 86 GB on the same sync). Ten reset boxes with 16 to 38 GB container limits were OOM-killed mid-sync between 09:42 and 10:27 UK (ln-node-03 and ln-node-09 at 16 GB, lc-3070-01 at 18 GB, lp-4090-03/17/22/23/24/30 at 28 GB, lp-4090-08 at 38 GB; `oom_kill` 1 to 2 each; /Users/joshm/igneum-fleet/reset-decision.txt under the 10:35 FAULT line); the box that survived had 123 GB. The Mac entry serves 2.0.2 to machines with 16 and 32 GB. + +What the memory is NOT: the executor's persisted state. The exec snapshot of the same synced node is 56 MB (`evm/exec-snapshot.bin`, 1,902 proof-record files beside it), the consensus store 263 MB. What it IS, by the on-disk shape (`evm/proof-records` 4.8 GB on the 2.0.3 node, `evm/proofs` and the records 7.1 GB on the 2.0.2 one, growing with the chain) and by the curve's linearity in executed blocks: the records a fresh sync holds while it executes from genesis — the proof records and segment records of every block it carried, kept in memory for the proving window's checks and the RPC's lookups, never dropped behind the executed tip because a node that executes from genesis has no attested point below which they are safe to drop. A heap profile of a fresh 27f54124 sync to DAA 10,000 on build-2 names the holders by function this afternoon; this design does not wait for it, because the fix is the same whatever holds them: a joiner that does not execute from genesis never holds them. + +## 1. The snapshot: content and format + +The executor already has the artefact and most of the path (`igneum/exec/src/snapshot.rs`, 6 October 2026): `ExecSnapshot` version 1 — `chain_id`, `genesis_hash`, `tip_number`, `tip_hash`, `state_root` at the tip, `written_at_unix`, every chain block record (`records`, full for the last `FULL_RECORDS` blocks), the EVM state (`db`: accounts, storage, code, the BLOCKHASH ring), the fee and epoch state, the paid shards and segments, and the class v5 epoch state streams (`epoch_streams`, the IGSD1 streams of the current and next epoch, so a joiner holds them without executing them); the consensus digest stamp (0.3.25+). The daemon flag `--igneum-exec-snapshot=[,]` starts from a file when the data dir has none; `--igneum-exec-snapshot=peer` makes the follower wait for a peer's snapshot instead of replaying from genesis (`service.rs` 994–1005), served over the P2P flow `IgneumRequestExecSnapshot` → `IgneumExecSnapshotChunk` (`protocol/flows/src/v10/execsync.rs` 28–69: the serving side answers with the executor's persisted snapshot in chunks). The load (`service.rs` 590–680 `load_snapshot`) refuses a wrong chain id, a wrong genesis, a wrong or missing digest stamp, a mid-epoch resume without the tip's capture, and a state root that the rebuilt accounts do not reproduce (`check_roots`); then `pin_check` refuses a snapshot whose record at the restart block carries another root than this network's. + +Snapshot sync adds two things to that artefact, not a new one: (a) the snapshot is cut AT A LOCKED BLOCK — the seed's executor persists it when its tip is the finality checkpoint the seed last locked (`latest_locked_hash` of `igneum_getFinalityCheckpoints`, `rpc/core/src/model/finality.rs` 297–311), so the snapshot's `tip_hash` is a block every honest node holds as final; (b) a MANIFEST beside it: `snapshot.json` = { chain_id, digest, tip_number, tip_hash, tip_daa, state_root, locked_index, size, sha256, written_at } signed by the seed's vote key (the same key that signs its finality votes; the signature over the manifest's canonical bytes under a new domain `igneum-snapshot-v1`), so a joiner knows which seed vouches for it and can weigh that key in the finality table it will read. + +## 2. Who signs it, with what key + +The seeds (build-1's seed and hand, the three lp hubs after their upgrade) — each signs its own snapshot with its vote key; the manifest carries the key and the signature. A joiner accepts a snapshot whose signing key holds weight in the finality table at the snapshot's locked index (read from the certificate of that index: `RpcCheckpoint` in the same response), or — until the hubs run 2.0.3.1 — a key named in the packaged seed list (the release manifest's `snapshot_signers`, the shipper's). No new key, no central key: a snapshot's trust is the signer's vote weight plus the verification in section 3; a bad snapshot from a signing seed is an equivocation-class fault against that key, recorded the way a bad vote is. + +## 3. How a joiner verifies it + +In order, each a refusal with its line: (1) the manifest's signature verifies under the named key, and the key is in the finality table at `locked_index` or in the packaged signer list; (2) the file's sha256 equals the manifest's; (3) the snapshot's chain id, genesis and digest are this node's (the load's own checks); (4) THE LOCK: the joiner syncs headers first as it does today (headers-first IBD; the headers' chain is PoW-verified and the finality certificates carried in blocks are verified as they arrive), reads its own `latest_locked_hash` at `locked_index` from the certificate it verified, and refuses a snapshot whose `tip_hash` is not that locked block or an ancestor of it on its selected chain (`is_chain_ancestor_of`); (5) THE ROOT: the Kaspa header carries no exec state root (design 2.2: the root is an output of execution), so the root is checked three ways: the rebuilt accounts reproduce `state_root` (today's `check_roots`), the record at the tip carries that root (today's `pin_check`), and the first verified segment record attesting the snapshot's block (the proving pool's rows, `post_root`) agrees — the third is the proof-carrying check and the one a lying signer cannot pass once the chain's provers have attested the block. A joiner refusing at any step falls back to today's path (genesis execution) with the refusal named, never to a silent wall. + +## 4. The executor's start and persist-and-drop + +Start: the follower installs the snapshot as today (`load_snapshot`: the records, the db, the epoch streams, the ring's first entry), then executes forward from `tip_number + 1` as the bodies arrive — the IBD's body sync starts at the snapshot's tip, not genesis (the `highest_known_syncer_chain_hash` negotiation already lands at a block the joiner holds; the body sync's low bound becomes `max(negotiated, snapshot tip)`), so the joiner downloads headers for the whole chain (small) and bodies only above the snapshot. Persist-and-drop, the bound that makes the sync fit: every proof record and segment record below `executed_tip − proving_window` (the window the pool checks, `weight_window` DAA) is persisted to the record store (`c90212e3`'s `ProofRecordStore` under the evm dir, which already holds them on disk) and dropped from memory; the store answers the RPC's and the body rule's lookups from disk. The in-memory set is then bounded by the window, not the chain: at today's window (7,200 DAA) about 7,200 × 0.65 MB ≈ 4.7 GB at most, under 16 GB with the consensus side (263 MB on disk, its caches the box's), the EVM state (tens of MB) and the executor's ring. Today's `FULL_RECORDS = u64::MAX` (every record full in memory) is the constant that changes: full for the window, thin below, dropped below the snapshot's tip (the snapshot is the attestation below which nothing is replayed). + +## 5. The serving path + +Two paths, both existing in shape: (a) P2P, as today's `IgneumRequestExecSnapshot` chunks from any connected peer that holds a snapshot — extended to carry the manifest first so the joiner runs section 3's steps (1)–(3) before it takes the chunks; (b) HTTPS from the seeds' download host (the shipper's `dl.igneum.network`), the file and `snapshot.json` beside it, the same manifest — the path a fresh install takes before it has peers, pointed at by the release manifest's `snapshot_url`. The seed cuts a new snapshot at every lock it holds past a fixed stride (every 3,600 blocks, the epoch) and keeps the last two; a joiner takes the newest whose lock it has verified. + +## 6. The 16 GB fit, stated + +A joiner from the snapshot holds: the headers (about 300 bytes a block in memory during IBD, 16 MB at 54,000 blocks), the bodies above the snapshot (streamed, executed, thinned), the EVM state (tens of MB), the window's records (≤ 4.7 GB at today's rate), the consensus store's caches and the allocator's slack — a fresh install on a 16 GB machine syncs from the snapshot to the moving tip without the 0.65 GB-per-1,000-blocks growth, and a 2.0.2 node on 16 GB today walls at about DAA 8,000. Until 2.0.3.1 ships, /download and /miner state 64 GB for a fresh sync (the site lane, by 11:30 UK), or the recipe's snapshot by hand. + +## 7. The known-failed tests, first + +- `a_joiner_refuses_a_snapshot_whose_tip_is_not_under_its_verified_lock`: a snapshot whose `tip_hash` is a block off the joiner's selected chain at `locked_index` (an island's tip) is refused with the lock's line; the same snapshot with the locked block's hash is accepted. Fails today (no lock check exists; a wrong-chain snapshot is refused only by `pin_check` after the records are installed). +- `a_snapshot_manifest_signed_by_a_key_outside_the_table_is_refused`: a manifest signed by a key with no weight at `locked_index` and not in the packaged list is refused before any byte of the file is read. Fails today (no manifest). +- `records_below_the_window_leave_memory`: a follower that executed 20,000 blocks holds in memory only the records inside `executed_tip − weight_window`; the store answers a lookup below it from disk. Fails today (`FULL_RECORDS = u64::MAX`). +- `the_body_sync_starts_at_the_snapshot_tip`: a joiner with a snapshot at block 46,800 requests bodies from 46,801, never from genesis. Fails today (the body sync's low bound is the negotiation's alone). +- The canary (not a unit test): a fresh install on a 16 GB host (the fleet lane's rented Vast host at the 16 GB class, container `memory.max` read back at the handover, 13:15 UK) joining from build-1's seed's served snapshot to the moving tip and mining five minutes — the measured peak RSS against 16 GB, by 15:00 UK; the ten OOM-killed boxes above are its known-failed shape. + +## 8. What ships when + +2.0.3.1 on the node line after 27f54124: the lock cut and the manifest (the seed side), the four tests and their code (the joiner side: the lock check in `load_snapshot`'s caller, the manifest check in the P2P and HTTPS fetch, `FULL_RECORDS` as a window and the persist-and-drop in the follower, the body sync's low bound), the release manifest's `snapshot_url` and `snapshot_signers` (the shipper). The first served snapshot from build-1's seed by hand (the shipper's script on the seed's cleanly stopped datadir: the file, the sidecar, the manifest signed by the seed's key) this afternoon for the canary; the P2P manifest and the HTTPS path on the node line by tomorrow's cut. Nothing here moves the consensus digest: the snapshot is the executor's and the flows', the headers' chain stays the proof of the chain.