Prover: the 5 October post-root assertion explained (stale build, empty-segment plan), fixture 58927, fixture test, source stamp

The fact: PC 2's exporter failed blocks 58,752 to 58,843 (and the task's 58,584 to 58,984) at
shard.rs:175, "shard 0 post-root from the witness", while three empty shards were later proven, verified and
paid. Every failing block is empty: one blue block, one reward, the pool credit, no transactions, no payouts
(igneum_exportSegments 0x0..0xe738 from the Mac node, fork 2b6d23ef). The proven block 59,507 has the same
shape and the same miner as the failing 58,927. So the difference is not block content.

The rule that differed: the planner's post-root of an empty segment. The exporter on PC 2 was a stale build
whose core predated commit 476063b (the assertion sits at line 175 there and at 179 since). That planner
returned root_at(end) for a segment with no transactions, which is the pre-root; the statement applied the
rewards and the pool credit, as the node does (vendor/igneum-node-036/igneum/exec/src/executor.rs,
execute_segment) and as spec 7.7 item 8 says. Left = the root after the rewards (the node's), right = the root
before them; PC 2's export log for 58,752 shows exactly that pair. Reproduced here: master's core with that one
rule put back fires the same assertion on 58,927 with left 0x7886b9cf (the node's root) and right 0xea9db302
(the pre-root). Master's core as it is reproduces 58,927 and 59,192 with the node's roots, the host's native
mode matches the fixture, and the SP1 executor runs shard 0 to post-root 0x7886b9cf.

So the prover core needs no rule change: the fix is commit 476063b, which PC 2 received with the 10:49 and
10:52 UTC rebuilds (job-rebuild-prover-pc2-037 and 037b), after which its proofs were paid. What this commit
adds is the regression and the guard for the class:

- proving/fixtures/block-58927-empty-reward.json: the failing shape cut from the devnet (33 KB).
- proving/igneum-prove/export/tests/fixtures.rs: every fixture in proving/fixtures reproduces (block
  statement, plan, every shard statement from its witness, the chain of roots and links), and the empty
  segment's shard ends at the root after the rewards, never the pre-root. With the pre-476063b rule put back
  the test fails. The test lives in the export crate so the core's manifest, part of the guest build, stays
  untouched.
- export/build.rs and host/build.rs stamp each binary with a hash of the native sources it was built from,
  printed on the first line of every run, so a stale build names itself in the log instead of in a line
  number (the stale-build class of 4 and 5 October).
- docs/bench-log.md: the row under the first paid proofs.

The guest is unchanged: built in one directory, this branch and master give byte-identical loadable segments
for the shard program and the aggregator, so no prover needs a rebuild for this commit. Noted on the way and
left open: the same sources built in three directories on this Mac gave two different guest ELFs, so the
program id is not yet a pure function of the sources on a native build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-05 11:38:59 +00:00
parent 41ee3748b8
commit cbada9f838
10 changed files with 1371 additions and 1 deletions

View file

@ -1373,5 +1373,6 @@ Proving v0 activated at DAA 84,100 (manifest `consensus.override`, every node re
| Paid shards by 10:53 UTC (Mac node `igneum_getProvingStatus`) | 3 shards, 3.370437410 IGN in total, pool balance 68,601.72 IGN |
| Pool at that moment | 4 entries: 3 pending, 1 failed verification, 0 verified-and-waiting |
| Non-empty shards | not yet: the exporter's post-root assertion fires on blocks with content (58,584 to 58,984 on 5 October); investigation open |
| The assertion, explained (12:30 UTC, branch prover-match) | Not block content. Every block it fired on is empty (PC 2's export logs: 58,752 to 58,843 hit the assertion; 58,584 to 58,740 hit the backslash path of 6d51e53), one reward plus the pool credit, no transactions, no payouts. PC 2's exporter was a stale build: the panic names shard.rs:175, the line before commit 1251f0a moved the assert to 179, and that core's planner gave an empty segment the pre-root as its post-root while the statement applied the rewards (left = the node's root after the rewards, right = the root before them, as the log shows for 58,752). The core at master reproduces 58,927 and 59,192 with the node's roots, and the same shape (59,507: one reward to the same miner) was proven and paid after the 10:49 and 10:52 UTC rebuilds on PC 2. Branch prover-match: fixture `block-58927-empty-reward.json`, `export/tests/fixtures.rs` (every fixture reproduces; an empty segment ends at the root after the rewards), and a source stamp on the first line of the exporter and the host so a stale binary names itself. The guest is untouched: built in one directory, master and the branch give byte-identical loadable segments for the shard program and the aggregator (shard program id 0x1ec8b941 at master in that directory). Noted on the way: the same sources built in three directories on this Mac gave two different guest ELFs (text segment c173b3de in the main checkout and in a fresh worktree, 830f7433 in the branch's worktree, shard program id 0x366e2aca there), so the program id is not yet a pure function of the sources on a native build; SP1's docker build is the reproducible path and is not in use. Open item. |
Commands: `curl -X POST http://127.0.0.1:26800 -d '{"jsonrpc":"2.0","id":1,"method":"igneum_getProvingStatus","params":[]}'` on the Mac; `node tools/logs.mjs` for PC 2's prover lines (`prover: block N shard 0 assigned to win-1ccfe586-1-1: export, cut, prove (CUDA), sign, submit`).

File diff suppressed because it is too large Load diff

View file

@ -2796,6 +2796,7 @@ dependencies = [
"hex",
"igneum-prove-core",
"serde_json",
"sha2 0.10.9",
]
[[package]]

View file

@ -12,3 +12,8 @@ serde_json.workspace = true
hex.workspace = true
anyhow.workspace = true
bincode.workspace = true
[build-dependencies]
# the source stamp (build.rs); sha2 0.10 is already in the lock through the host
sha2 = "0.10"
hex.workspace = true

View file

@ -0,0 +1,30 @@
//! Stamps the binary with a hash of the sources it was built from (`IGNEUM_PROVE_SOURCES`), printed on the
//! exporter's first line. The stale-build class (5 October 2026: PC 2's exporter carried a core from before
//! commit 1251f0a because cargo judged the copied sources older than its cache) is then visible in the first
//! line of every export log instead of in a line number. Reproduce from a tree with
//! `cat $(ls core/src/*.rs export/src/*.rs | sort) | shasum -a 256 | cut -c1-16` in proving/igneum-prove.
use sha2::{Digest, Sha256};
use std::path::Path;
fn main() {
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("..");
let mut files: Vec<String> = Vec::new();
for dir in ["core/src", "export/src"] {
for entry in std::fs::read_dir(root.join(dir)).expect("source dir") {
let name = entry.expect("entry").file_name().to_string_lossy().into_owned();
if name.ends_with(".rs") {
files.push(format!("{dir}/{name}"));
}
}
}
files.sort();
let mut h = Sha256::new();
for f in &files {
let path = root.join(f);
println!("cargo:rerun-if-changed={}", path.display());
h.update(std::fs::read(&path).expect("read source"));
}
println!("cargo:rerun-if-changed=build.rs");
println!("cargo:rustc-env=IGNEUM_PROVE_SOURCES={}", &hex::encode(h.finalize())[..16]);
}

View file

@ -99,6 +99,9 @@ fn main() -> Result<()> {
let source = args.iter().position(|a| a == "--source").and_then(|i| args.get(i + 1)).cloned().unwrap_or_else(|| format!("{} (igneum_exportSegments), block {}", args[1], want));
let budget: u64 = args.iter().position(|a| a == "--budget").and_then(|i| args.get(i + 1)).map(|b| b.parse()).transpose()?.unwrap_or(SHARD_PROVING_GAS_BUDGET);
// The sources this binary was built from (export/build.rs), so a stale build names itself in every export log.
println!("igneum-prove-export sources {} (core/src and export/src)", env!("IGNEUM_PROVE_SOURCES"));
let segments = export["segments"].as_array().context("segments")?.clone();
let registry_code = bytes(&export["registryCode"])?;
let mut db = IgneumDb::new();

View file

@ -0,0 +1,128 @@
//! Every fixture in `proving/fixtures` reproduces natively: the whole-block statement gives the node's state
//! root, the plan cuts the same shards, and every shard statement from its witness ends at the plan's post-root.
//! A core whose rules drift from the node's, or whose planner drifts from the statement, fails here before any
//! binary is packaged.
//!
//! `block-58927-empty-reward.json` is the regression for 5 October 2026: PC 2's exporter, a stale build whose
//! core predated commit 1251f0a, failed every empty devnet block with "shard 0 post-root from the witness",
//! left = the node's root after the rewards, right = the root before them. The planner of that core gave an
//! empty segment the pre-root as its post-root (`root_at(end)` with no transactions); the statement, correctly,
//! applied the rewards and the pool credit. The rule (spec 7.7 item 8, design 1.1): an empty segment is one
//! shard whose statement applies the rewards and the payouts, so its post-root is the segment's root after them,
//! never the pre-root.
use alloy_primitives::{Address, B256};
use igneum_prove_core::config::PROVING_POOL_ADDRESS;
use igneum_prove_core::executor::load_pre_state;
use igneum_prove_core::shard::{build_shards, shard_statement};
use igneum_prove_core::Fixture;
use std::path::PathBuf;
fn fixtures_dir() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../fixtures")
}
// This test lives in the export crate, not in igneum-prove-core, so the core's manifest (part of the guest build)
// stays untouched: the export crate already has serde_json and is never part of the guest build. Checked on
// 5 October 2026: built in one directory, this branch and master give byte-identical guest ELF segments.
fn load(name: &str) -> Fixture {
let path = fixtures_dir().join(name);
let text = std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("read {}: {e}", path.display()));
let f: Fixture = serde_json::from_str(&text).unwrap_or_else(|e| panic!("parse {}: {e}", path.display()));
assert_eq!(f.format, igneum_prove_core::fixture::FORMAT, "{name}: fixture format");
f
}
/// Runs the exporter's own check on one fixture: block statement against the expected values, the cut against
/// the plan, every shard's witness statement against the plan's roots and links.
fn check(name: &str) {
let f = load(name);
let budget = f.plan.shard_budget;
// The exporter builds the plan with the zero address as the prover; the roots do not depend on it.
let (outcome, pre_root, shards) = build_shards(&f.block, budget, Address::ZERO);
let e = &f.expected;
assert_eq!(pre_root, e.pre_state_root, "{name}: pre-state root");
assert_eq!(outcome.state_root, e.post_state_root, "{name}: post-state root");
assert_eq!(e.post_state_root, e.node_state_root, "{name}: the exporter's root is the node's");
assert_eq!(outcome.receipts_root, e.receipts_root, "{name}: receipts root");
assert_eq!(outcome.tx_commitment, e.tx_commitment, "{name}: tx commitment");
assert_eq!((outcome.gas_used, outcome.pgas_used), (e.gas_used, e.pgas_used), "{name}: gas and pgas");
assert_eq!((outcome.executed.len(), outcome.skipped.len()), (e.executed as usize, e.skipped as usize), "{name}: executed and skipped");
assert_eq!(shards.len(), f.plan.shards.len(), "{name}: shard count");
for (s, x) in shards.iter().zip(&f.plan.shards) {
let o = &s.output;
let tag = format!("{name} shard {}", x.index);
assert_eq!(s.spec.index, x.index, "{tag}: index");
assert_eq!((s.spec.tx_start, s.spec.tx_end), (x.tx_start, x.tx_end), "{tag}: range");
assert_eq!(s.spec.over_budget, x.over_budget, "{tag}: over budget");
assert_eq!(o.pre_root, x.pre_root, "{tag}: pre-root");
assert_eq!(o.post_root, x.post_root, "{tag}: post-root");
assert_eq!(o.receipts_root, x.receipts_root, "{tag}: receipts root");
assert_eq!((o.link_in, o.link_out), (x.link_in, x.link_out), "{tag}: links");
assert_eq!((o.gas_used, o.pgas_used), (x.gas_used, x.pgas_used), "{tag}: gas and pgas");
assert_eq!((o.executed, o.skipped), (x.executed, x.skipped), "{tag}: executed and skipped");
// The statement recomputed from the shard input alone (what the guest does) is the same statement.
assert_eq!(shard_statement(&s.input), *o, "{tag}: statement from the input");
}
// The shards chain and end at the block's root.
assert_eq!(shards.first().map(|s| s.output.pre_root), Some(e.pre_state_root), "{name}: first shard starts at the pre-root");
assert_eq!(shards.last().map(|s| s.output.post_root), Some(e.post_state_root), "{name}: last shard ends at the post-root");
for w in shards.windows(2) {
assert_eq!(w[0].output.post_root, w[1].output.pre_root, "{name}: shards chain on roots");
assert_eq!(w[0].output.link_out, w[1].output.link_in, "{name}: shards chain on links");
}
}
#[test]
fn every_fixture_reproduces() {
let mut names: Vec<String> = std::fs::read_dir(fixtures_dir())
.expect("proving/fixtures")
.filter_map(|e| e.ok())
.map(|e| e.file_name().to_string_lossy().into_owned())
.filter(|n| n.ends_with(".json"))
.collect();
names.sort();
assert!(names.len() >= 7, "fixtures present: {names:?}");
for n in &names {
check(n);
}
}
#[test]
fn an_empty_segment_shard_ends_at_the_root_after_the_rewards() {
let name = "block-58927-empty-reward.json";
let f = load(name);
let b = &f.block;
assert!(b.blocks.iter().all(|x| x.txs.is_empty()), "the block carries no transactions");
assert_eq!(b.rewards.len(), 1, "one blue block, one reward");
assert!(b.payouts.is_empty(), "no payouts in this segment");
assert!(!b.proving_pool_credit.is_zero(), "the pool credit is part of the statement");
let (outcome, pre_root, shards) = build_shards(b, f.plan.shard_budget, Address::ZERO);
assert_eq!(shards.len(), 1, "an empty segment is one shard");
let s = &shards[0];
assert_eq!((s.spec.tx_start, s.spec.tx_end), (0, 0));
assert_eq!(s.output.pre_root, pre_root);
assert_ne!(s.output.post_root, pre_root, "the rewards move the root: the post-root is never the pre-root");
assert_eq!(s.output.post_root, f.expected.node_state_root, "the witness statement ends at the node's root");
assert_eq!(s.spec.post_root, f.expected.node_state_root, "the plan ends the empty shard at the node's root, not the pre-root");
assert_eq!(outcome.state_root, f.expected.node_state_root);
// The rule itself, by hand: rewards to each blue miner, the 20% credit to the pool escrow, then the payouts
// (none here), before any transaction; the root after that is the shard's post-root.
let mut db = load_pre_state(b);
assert_eq!(db.state_root(), pre_root);
for (miner, wei) in &b.rewards {
db.add_balance(*miner, *wei);
}
db.add_balance(PROVING_POOL_ADDRESS, b.proving_pool_credit);
for (to, wei) in &b.payouts {
db.sub_balance(PROVING_POOL_ADDRESS, *wei);
db.add_balance(*to, *wei);
}
assert_eq!(db.state_root(), s.output.post_root, "rewards, pool credit and payouts, nothing else");
// The pre-root PC 2 printed on the right of its assertion is what a planner without the fix would return.
let wrong: B256 = pre_root;
assert_ne!(s.spec.post_root, wrong);
}

View file

@ -22,6 +22,9 @@ tokio = { version = "1", features = ["rt-multi-thread", "time"] }
[build-dependencies]
sp1-build.workspace = true
# the source stamp (build.rs)
sha2 = "0.10"
hex.workspace = true
[features]
default = []

View file

@ -1,4 +1,32 @@
//! Builds the two SP1 guests and stamps the host with a hash of the native sources it was built from
//! (`IGNEUM_PROVE_SOURCES`, printed in the host's first line; the guests have their own identity, the shard
//! program's verifying key). Same purpose and recipe as export/build.rs (the stale-build class of 5 October
//! 2026): `cat $(ls core/src/*.rs host/src/*.rs | sort) | shasum -a 256 | cut -c1-16` in proving/igneum-prove.
use sha2::{Digest, Sha256};
use std::path::Path;
fn main() {
sp1_build::build_program("../program");
sp1_build::build_program("../aggregator");
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("..");
let mut files: Vec<String> = Vec::new();
for dir in ["core/src", "host/src"] {
for entry in std::fs::read_dir(root.join(dir)).expect("source dir") {
let name = entry.expect("entry").file_name().to_string_lossy().into_owned();
if name.ends_with(".rs") {
files.push(format!("{dir}/{name}"));
}
}
}
files.sort();
let mut h = Sha256::new();
for f in &files {
let path = root.join(f);
println!("cargo:rerun-if-changed={}", path.display());
h.update(std::fs::read(&path).expect("read source"));
}
println!("cargo:rerun-if-changed=build.rs");
println!("cargo:rustc-env=IGNEUM_PROVE_SOURCES={}", &hex::encode(h.finalize())[..16]);
}

View file

@ -82,7 +82,8 @@ fn run() -> Result<()> {
let txs: usize = block.blocks.iter().map(|b| b.txs.len()).sum();
let prover_kind = std::env::var("SP1_PROVER").unwrap_or_else(|_| "cpu".into());
println!(
"fixture {path}: chain {} block {} ({}), {txs} transactions in {} including blocks, {} accounts in the pre-state, plan {} shard(s) at S_p = {} pgas{}; SP1_PROVER={prover_kind}; prover payout {prover}; {}",
"igneum-prove-host sources {}: fixture {path}: chain {} block {} ({}), {txs} transactions in {} including blocks, {} accounts in the pre-state, plan {} shard(s) at S_p = {} pgas{}; SP1_PROVER={prover_kind}; prover payout {prover}; {}",
env!("IGNEUM_PROVE_SOURCES"),
block.chain_id,
block.env.number,
block.env.hash,