diff --git a/docs/bench-log.md b/docs/bench-log.md index 234a43a22..1476091a5 100644 --- a/docs/bench-log.md +++ b/docs/bench-log.md @@ -1884,3 +1884,18 @@ Measurement (`/Users/joshm/Projects/igneum/tools/lock/with-lock.sh measure bash | v1 shard | cpu (no feature, `target/cpu`) | 20 | 10,049,917; 11; 0.31 | 585.8 (587.71) | 11; 11,534,336; 1,093,455; 390,964 | 10,583,965,696 (10.58) | 9,005.0 | 223,882 (222,668) | 0.010 | 0.21 / 0.009 to 0.012 / 32.9 MB, three times | yes | Reading. (1) The same statement costs RISC Zero 10,049,917 user cycles against SP1's 4,717,439 (bench-log 5 October, the S_p curve), 2.13x, with 10.7% more in paging and reserved cycles at po2 20. (2) On this CPU the v1 shard takes 574 s at po2 20; SP1's CPU prover on the same Mac under the same lock this morning took 71.7 s for the same shard (the proof-system watch, `bench/proof-systems/rows/20261006-1146-MacBook-Pro-sp1.json`, another agent's row): 8.0x. (3) The succinct receipt is 223,882 bytes against SP1's 1,272,897-byte compressed proof, 5.7x smaller, and the same size at every po2 and for the empty shard. (4) The verify is 9 to 10 ms in process; the whole verify mode is 0.21 s, of which 0.18 to 0.19 s is `compute_image_id` over the 3.67 MB program binary at start (the SP1 host's verify mode ran 0.26 s on the watch's row). (5) Memory against the segment limit: 8.3 to 8.4 GB of RSS at po2 19 and for any one-segment shard, 10.7 GB at po2 20, 20.4 GB at po2 21; the limit is the dial, and po2 19 costs 22% more time (24 segments instead of 11). (6) The empty shard is one 2^19 segment and still 47.9 s: the per-receipt floor of segment prove plus lift to a succinct receipt on this CPU. (7) `/usr/bin/time` reports 15.7 of the 18 cores busy over the po2 20 run (9,035 s user in 575 s wall). (8) The ratio `metal` build to CPU build on the v1 shard at po2 20 is 574.4 s to 585.8 s, 0.98: the same code, run-to-run noise, as the sources say. (9) Nothing here is a GPU number: PC 2 (CUDA) measures the GPU memory and time; the Mac's Metal number does not exist in 3.0.6. + +### 6 October 2026, 13:17Z to 13:45Z, the exec restart on a copy of node 1's data dir (Mac, `tools/lock/with-lock.sh run`) + +`cp -c -R /tmp/igneum-devnet/node1 /tmp/igneum-devnet/node1-copy`, the exec-sync node (branch exec-sync-0313) on ports 29990+ with node 1's override file plus `exec_restart_number` 27276, `exec_restart_hash` bb45cf0d..., `exec_restart_trust_daa` 200000 (the script `node1-copy-test.sh`, every figure a RESULT line): + +| Figure | Value | +|---|---| +| Chain facts | sink chain block 130,073 (later 130,272); pruning point = retention root = chain block 27,276, DAA 45,537, body held; bodies below gone | +| Replay from 27,276 to the sink | 92 s (run 4, no trust rule), 93 s (run 5, trust rule) wall from the node's start | +| Paid shards without the trust rule | 0 (every record vetoed by the native-statement check) | +| Paid shards with the trust rule | 1,482, 1,825.699 IGN | +| PC 2's payout balance at the tip | 265,359.6 IGN (run 4), 267,648.4 IGN (run 5) | +| Persisted file | 114,375,262 B then 114,830,936 B; 31 accounts; 1,200 full records | +| Resume after a restart | 8 s and 9 s to answer, startedFrom "snapshot", no replay | + diff --git a/docs/plans/exec-sync.md b/docs/plans/exec-sync.md new file mode 100644 index 000000000..a1c7decdf --- /dev/null +++ b/docs/plans/exec-sync.md @@ -0,0 +1,47 @@ +# Exec sync: the executor's state after the pruning point left genesis (6 October 2026) + +## What happened + +The devnet's pruning point left genesis on 6 October 2026 at about 11:40Z (chain block 27,276, DAA 45,537). Every +node pruned the block bodies and acceptance rows below it and kept the headers and ghostdag rows. The execution +layer's state lived only in memory and was rebuilt from genesis at every start; every node had restarted for +0.3.11 and 0.3.12. From 11:40Z every node read `eth_blockNumber` 0x0 and `igneum_getProvingStatus` active false: +no balances, no proving work list, no payouts, mining untouched. No hand ran `--archival`, so no data dir on the +network holds the bodies the state was built from, and the state cannot be rebuilt from genesis anywhere. + +## The fix, fork branch `exec-sync-0313` (off the 0.3.12 node 83089544), node only + +| Commit | What | +|---|---| +| 4f05e56a | the exec state persisted to the data dir every 5 minutes and at stop (two generations; resumed at start when its tip is still a chain block); `--igneum-exec-snapshot=[,]` and `igneum_exportExecSnapshot`; the loud status when the follower cannot execute (warn every minute, `igneum_getExecStatus.blocked`, the proving status's `execSync`) | +| 3bd31a20 | the snapshot served and fetched over p2p at protocol 16 (messages 76 and 77, 1 MiB chunks, the sha256 on every chunk); a blocked executor asks one peer every 20 s; `--igneum-exec-snapshot=peer` | +| efa6924d, 7cb712f5, 80674943 | the archival walk: the selected-parent chain read from the ghostdag store when the virtual-chain query refuses a tip below the retention root; the mergeset order rebuilt from ghostdag rows when the acceptance row is pruned | +| 3dd9b2c9 | the exec restart: `exec_restart_number` and `exec_restart_hash` in the override object (in the digest once set): header-only records below R, the EVM state fresh at R, executed from R's body on; the status reports the pruning point and the retention root with their chain block numbers | +| 05e93f0e | `exec_restart_trust_daa`: a shard record carried below it pays as carried, without the native-statement veto and the assignee check | + +## Measured on a copy of node 1's data dir (the Mac, `tools/lock/with-lock.sh run`) + +| Figure | Value | +|---|---| +| The pruning point and retention root, node 1 and the observer | chain block 27,276, hash bb45cf0dd2d7cc97ebfa5a2701527c09a8ede5d32de74efead9caa293b15688a, DAA 45,537; its body held on both | +| Bodies below it | gone ("cannot find full block" at chain block 1) | +| Replay from 27,276 to the sink (130,073, then 130,272) | 92 s and 93 s wall from the node's start, over 2,000 chain blocks a second | +| Without the trust rule | paidShards 0: every historical record vetoed (the records attest the original state roots) | +| With `exec_restart_trust_daa` 200,000 | paidShards 1,482, paidWei 1,825.70 IGN (node 1 read 1,261 and 1,573 at 08:30Z; the chain moved on) | +| PC 2's payout address at the tip | 267,648 IGN | +| The persisted file | 114,830,936 bytes, 130,273 records (1,200 full), 31 accounts | +| Resume after a restart | 9 s, no replay | + +## What it means + +| Tier | Before the fix | After the cut with the three fields | +|---|---|---| +| Every node, home miner, rig, pool | an empty EVM since 11:40Z: no balances, no proving, no payouts; mining fine | the chain's state from chain block 27,276 on, rebuilt in about 90 s at the switch; every restart after that resumes in seconds | +| Miners | rewards invisible | rewards since R back (PC 2: 267,648 IGN); the rewards of chain blocks 0 to 27,275 (the devnet's first 12 hours) lost, the project lead's call on seeding them | +| Provers | payouts invisible | every carried record since proving v0 paid as carried (1,482 shards); new proofs verify against the new state from the trust DAA on | +| Joiners today | an empty EVM for ever | IBD gives the pruning point 27,276 and the bodies from it, so a fresh node replays the same 90 s with no snapshot; once the pruning point moves past R the p2p snapshot (protocol 16) starts it | + +## Open + +The fresh-join time on a rented box (in progress); the trust DAA's value (at or above the switch); the first 12 +hours' rewards (the project lead); the p2p snapshot's fast-time harness case; `--archival` on at least one hand from now on. diff --git a/tools/exec-sync/net.mjs b/tools/exec-sync/net.mjs new file mode 100644 index 000000000..ff253eeb6 --- /dev/null +++ b/tools/exec-sync/net.mjs @@ -0,0 +1,137 @@ +#!/usr/bin/env node +// Exec sync harness (6 October 2026): the executor's persisted state and the snapshot file, on a private fast-time +// simnet (ports 29970+, suffix 957; never the live devnet). Cases, each asserted: +// 1. known-finished: node A mines past N chain blocks; `igneum_exportExecSnapshot` writes a file; its tip, state +// root and sha256 are reported; A is stopped (the state is persisted at stop) and started again: it resumes from +// the data dir's snapshot (startedFrom "snapshot", snapshotTip = the persisted tip) and keeps executing +// 2. a fresh node B with --igneum-exec-snapshot=, and no bodies of its own beyond what it syncs: +// startedFrom "snapshot", snapshotTip = the file's tip, eth_blockNumber reaches A's tip, every miner balance +// and the state root at the file's tip equal A's +// 3. known-failed: a fresh node C with the file and a WRONG pin ignores the file (startedFrom "genesis" here, since +// a simnet peer still serves every body) and says why in its log +// 4. a fresh node D with the file and NO consensus data of its own but the file's tip unknown: refused with +// "unknown to consensus" (the file's tip is from another chain: a second simnet) +// Usage: node tools/exec-sync/net.mjs [--blocks 60] +// IGNEUM_EXEC_BIN= (default vendor/igneum-node/target-exec-sync/release) +import { spawn, spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync, rmSync, writeFileSync, openSync } from 'node:fs'; +import { createHash } from 'node:crypto'; + +const ROOT = new URL('../../', import.meta.url).pathname; +const REL = process.env.IGNEUM_EXEC_BIN || `${ROOT}vendor/igneum-node/target-exec-sync/release`; +const IGNEUMD = `${REL}/igneumd`; +const MINER = `${REL}/igneum-miner`; +const TMP = '/tmp/igneum-exec-sync'; +const BASE = 29970, SUFFIX = 957; +const flag = (name, d) => { const i = process.argv.indexOf(name); return i >= 0 ? Number(process.argv[i + 1]) : d; }; +const BLOCKS = flag('--blocks', 60); +const FILE = `${ROOT}infra/fast-time/override-60x.json`; +for (const b of [IGNEUMD, MINER]) if (!existsSync(b)) { console.error(`missing ${b}`); process.exit(2); } +rmSync(TMP, { recursive: true, force: true }); mkdirSync(TMP, { recursive: true }); +const override = `${TMP}/override.json`; +// the exec-sync node is cut from 0.3.12 and knows no proof-system-2 field: drop it as text (JSON.parse would turn +// the u64::MAX "never" values into floats the node refuses) +writeFileSync(override, readFileSync(FILE, 'utf8').replace(/\s*"proving_v2_activation_daa":\s*\d+,?/, '')); +const t0 = Date.now(); +const log = (m) => console.log(`${new Date().toISOString().slice(11, 23)} t=${((Date.now() - t0) / 1000).toFixed(1)}s ${m}`); +const sleep = (ms) => new Promise(r => setTimeout(r, ms)); +const hexn = (h) => Number(BigInt(h)); +const started = []; + +class Node { + constructor(i, connect = [], extra = [], suffix = SUFFIX) { + this.i = i; this.grpcPort = BASE + i * 10; this.p2pPort = BASE + i * 10 + 1; this.jsonPort = BASE + i * 10 + 2; this.evmPort = BASE + i * 10 + 3; + this.connect = connect; this.extra = extra; this.suffix = suffix; this.dir = `${TMP}/n${i}`; this.logFile = `${this.dir}/node.log`; + } + get evm() { return `http://127.0.0.1:${this.evmPort}`; } + async start() { + mkdirSync(this.dir, { recursive: true }); + const a = ['--devnet', `--devnet-suffix=${this.suffix}`, '--nodnsseed', '--disable-upnp', '--nologfiles', '--enable-unsynced-mining', '--utxoindex', '--unsaferpc', + `--appdir=${this.dir}`, `--rpclisten=127.0.0.1:${this.grpcPort}`, `--rpclisten-json=127.0.0.1:${this.jsonPort}`, `--evm-rpclisten=127.0.0.1:${this.evmPort}`, + `--listen=127.0.0.1:${this.p2pPort}`, `--override-params-file=${override}`, '--loglevel=info', '--yes', ...this.connect.map(c => `--addpeer=${c}`), ...this.extra]; + const out = openSync(this.logFile, 'a'); + this.proc = spawn(IGNEUMD, a, { stdio: ['ignore', out, out], env: { ...process.env, IGNEUM_PROOF_VERIFY: 'trust' } }); + started.push(this.proc); + for (let k = 0; k < 120; k++) { try { await this.eth('eth_chainId'); return this; } catch { await sleep(500); } } + throw new Error(`node ${this.i} did not answer on ${this.evm}`); + } + async stop() { try { this.proc.kill('SIGINT'); } catch { } for (let k = 0; k < 60; k++) { if (this.proc.exitCode !== null) return; await sleep(500); } try { this.proc.kill('SIGKILL'); } catch { } } + async eth(method, params = []) { + const r = await fetch(this.evm, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ jsonrpc: '2.0', id: 1, method, params }) }); + const j = await r.json(); if (j.error) throw new Error(`${method}: ${JSON.stringify(j.error)}`); return j.result; + } + logText() { try { return readFileSync(this.logFile, 'utf8'); } catch { return ''; } } +} +function mine(node, label) { + const out = openSync(`${TMP}/miner-${label}.log`, 'a'); + const p = spawn(MINER, ['mine', `127.0.0.1:${node.grpcPort}`, '--label', label, '--payout', '0x4343434343434343434343434343434343434343', '--threads', '1'], { stdio: ['ignore', out, out] }); + started.push(p); return p; +} +const checks = []; +const check = (name, ok, detail) => { checks.push({ name, ok }); log(`${ok ? 'PASS' : 'FAIL'} ${name}${detail ? ': ' + JSON.stringify(detail).slice(0, 300) : ''}`); if (!ok) throw new Error(`check failed: ${name}`); }; +async function waitTip(n, want, secs = 600) { for (let k = 0; k < secs * 2; k++) { const tip = hexn(await n.eth('eth_blockNumber')); if (tip >= want) return tip; await sleep(500); } throw new Error(`node ${n.i} did not reach ${want}`); } +function sha256(path) { return '0x' + createHash('sha256').update(readFileSync(path)).digest('hex'); } +async function main() { + const a = await new Node(0).start(); + log(`node A up on ${a.evm}`); + const miner = mine(a, 'exec-sync-a'); + await waitTip(a, BLOCKS, 900); + const tipA = hexn(await a.eth('eth_blockNumber')); + const st0 = await a.eth('igneum_getExecStatus'); + check('A executes from genesis', st0.startedFrom === 'genesis' && st0.blocked === null && hexn(st0.executedTip) >= BLOCKS, st0); + // case 1: the export, the stop, the resume + const file = `${TMP}/snapshot.bin`; + const ex = await a.eth('igneum_exportExecSnapshot', [file]); + const sha = sha256(file); + check('the export writes the file with its tip, root and sha256', existsSync(file) && ex.sha256 === sha && hexn(ex.tip) >= BLOCKS && ex.records === hexn(ex.tip) + 1, { tip: ex.tip, bytes: ex.bytes, sha256: ex.sha256, accounts: ex.accounts }); + const blockAtTip = await a.eth('eth_getBlockByNumber', [ex.tip, false]); + check('the export\'s state root is the tip block\'s', blockAtTip.stateRoot === ex.stateRoot, { root: ex.stateRoot }); + try { miner.kill('SIGINT'); } catch { } + await a.stop(); + const persistedLine = a.logText().split('\n').find(l => l.includes('exec state persisted at stop')); + check('A persisted its state at stop', !!persistedLine, persistedLine && persistedLine.slice(-120)); + await a.start(); + const resumed = a.logText().split('\n').filter(l => l.includes('exec sync: resumed from')).pop(); + const st1 = await a.eth('igneum_getExecStatus'); + check('A resumed from the data dir\'s snapshot', st1.startedFrom === 'snapshot' && hexn(st1.snapshotTip) >= BLOCKS && !!resumed, { startedFrom: st1.startedFrom, snapshotTip: st1.snapshotTip, line: resumed && resumed.slice(-160) }); + const miner2 = mine(a, 'exec-sync-a2'); + await waitTip(a, hexn(st1.snapshotTip) + 5, 300); + check('A keeps executing after the resume', hexn(await a.eth('eth_blockNumber')) > hexn(st1.snapshotTip)); + // case 2: a fresh node from the file with the right pin + const b = await new Node(1, [`127.0.0.1:${a.p2pPort}`], [`--igneum-exec-snapshot=${file},${sha}`]).start(); + const tipNow = hexn(await a.eth('eth_blockNumber')); + await waitTip(b, tipNow, 600); + const stB = await b.eth('igneum_getExecStatus'); + check('B started from the file', stB.startedFrom === 'snapshot' && hexn(stB.snapshotTip) === hexn(ex.tip) && stB.snapshotSha256 === sha, { startedFrom: stB.startedFrom, snapshotTip: stB.snapshotTip }); + const blkB = await b.eth('eth_getBlockByNumber', [ex.tip, false]); + check('B\'s state root at the file\'s tip equals A\'s', blkB.stateRoot === ex.stateRoot, { b: blkB.stateRoot }); + const payout = '0x4343434343434343434343434343434343434343'; + const h = '0x' + tipNow.toString(16); + const [balA, balB] = await Promise.all([a.eth('eth_getBalance', [payout, h]), b.eth('eth_getBalance', [payout, h])]); + check('the miner\'s balance at the same height equals on A and B', balA === balB && BigInt(balA) > 0n, { balA, balB, height: tipNow }); + const [rootA, rootB] = await Promise.all([a.eth('eth_getBlockByNumber', [h, false]), b.eth('eth_getBlockByNumber', [h, false])]); + check('the state root at the same height equals on A and B', rootA.stateRoot === rootB.stateRoot, { height: tipNow }); + // case 3: known-failed, the wrong pin + const wrong = '0x' + 'ab'.repeat(32); + const c = await new Node(2, [`127.0.0.1:${a.p2pPort}`], [`--igneum-exec-snapshot=${file},${wrong}`]).start(); + await sleep(3000); + const stC = await c.eth('igneum_getExecStatus'); + const cline = c.logText().split('\n').find(l => l.includes('is not the pinned')); + check('known-failed: C refuses the file under a wrong pin and says so', stC.startedFrom !== 'snapshot' && !!cline, { startedFrom: stC.startedFrom, line: cline && cline.slice(-140) }); + // case 4: known-failed, a file from another chain + const z = await new Node(3, [], [], 958).start(); + const zm = mine(z, 'exec-sync-z'); + await waitTip(z, 12, 300); + const fileZ = `${TMP}/snapshot-z.bin`; + const exZ = await z.eth('igneum_exportExecSnapshot', [fileZ]); + try { zm.kill('SIGINT'); } catch { } + const d = await new Node(4, [`127.0.0.1:${a.p2pPort}`], [`--igneum-exec-snapshot=${fileZ},${exZ.sha256}`]).start(); + await sleep(3000); + const stD = await d.eth('igneum_getExecStatus'); + const dline = d.logText().split('\n').find(l => l.includes('unknown to consensus') || l.includes('not on this node') || l.includes('is not this network')); + check('known-failed: D refuses a snapshot from another chain and says why', stD.startedFrom !== 'snapshot' && !!dline, { startedFrom: stD.startedFrom, line: dline && dline.slice(-160) }); + try { miner2.kill('SIGINT'); } catch { } + log(`RESULT exec sync harness: PASSED (${checks.length} checks) in ${((Date.now() - t0) / 1000).toFixed(1)} s; snapshot ${ex.bytes} bytes at tip ${hexn(ex.tip)}, ${ex.accounts} accounts`); +} +main().then(() => cleanup(0)).catch(e => { log(`FAILED: ${e.message}`); cleanup(1); }); +function cleanup(code) { for (const p of started.reverse()) { try { p.kill('SIGINT'); } catch { } } setTimeout(() => { for (const p of started) { try { p.kill('SIGKILL'); } catch { } } process.exit(code); }, 3000); }