Merge program-id: pinned shard and aggregator guests (elf/ + manifest), fast verify with the pinned key, CI check

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

# Conflicts:
#	proving/igneum-prove/host/build.rs
This commit is contained in:
igneum-labs 2026-10-05 13:00:54 +00:00
commit 9cd5fe4645
19 changed files with 497 additions and 23 deletions

View file

@ -63,6 +63,8 @@ jobs:
run: bash tools/ci/identity-check.sh
- name: copied sources are re-stamped before a build
run: bash tools/ci/copied-sources-check.sh
- name: pinned guest programs match their manifest and are built only by pin-guests.sh
run: bash tools/ci/pinned-guests-check.sh
- name: relay unit tests (parsers, secret compare, the wake endpoint)
run: node --test relay/test/parse.test.mjs relay/test/auth.test.mjs relay/test/wake.test.mjs
- name: miner app notice strip (ordering, keys, wording, timers)

View file

@ -1376,3 +1376,28 @@ Proving v0 activated at DAA 84,100 (manifest `consensus.override`, every node re
| The assertion, explained (12:30 UTC, branch prover-match) | Not block content. Every block it fired on is empty (PC 2's export logs: 58,752 to 58,843 hit the assertion; 58,584 to 58,740 hit the backslash path of 6d51e53), one reward plus the pool credit, no transactions, no payouts. PC 2's exporter was a stale build: the panic names shard.rs:175, the line before commit 1251f0a moved the assert to 179, and that core's planner gave an empty segment the pre-root as its post-root while the statement applied the rewards (left = the node's root after the rewards, right = the root before them, as the log shows for 58,752). The core at master reproduces 58,927 and 59,192 with the node's roots, and the same shape (59,507: one reward to the same miner) was proven and paid after the 10:49 and 10:52 UTC rebuilds on PC 2. Branch prover-match: fixture `block-58927-empty-reward.json`, `export/tests/fixtures.rs` (every fixture reproduces; an empty segment ends at the root after the rewards), and a source stamp on the first line of the exporter and the host so a stale binary names itself. The guest is untouched: built in one directory, master and the branch give byte-identical loadable segments for the shard program and the aggregator (shard program id 0x1ec8b941 at master in that directory). Noted on the way: the same sources built in three directories on this Mac gave two different guest ELFs (text segment c173b3de in the main checkout and in a fresh worktree, 830f7433 in the branch's worktree, shard program id 0x366e2aca there), so the program id is not yet a pure function of the sources on a native build; SP1's docker build is the reproducible path and is not in use. Open item. |
Commands: `curl -X POST http://127.0.0.1:26800 -d '{"jsonrpc":"2.0","id":1,"method":"igneum_getProvingStatus","params":[]}'` on the Mac; `node tools/logs.mjs` for PC 2's prover lines (`prover: block N shard 0 assigned to win-1ccfe586-1-1: export, cut, prove (CUDA), sign, submit`).
## 5 October 2026, the program id split: why the Mac rejected PC 2's proofs, and the verifier at 114 s
Machine: Apple M5 Max under the live devnet node, the Metal miner and two other agents' builds (every number here is wall time under that load, taken through the measure lock). Code: `proving/igneum-prove` on branch `program-id`, SP1 6.8.1, circuit v6.1.0.
| Host | Built | Shard program id | Source |
|---|---|---|---|
| Mac, shipped (`Igneum Miner.app/Contents/Resources/bin/igneum-prove-host`, app 0.3.7) | 5 Oct 10:48, release tree on the Mac | `0x0559759b3d8740b26ceceb2c56054b89194878ab691b018d7dd2f8af2f2242dd` | its own `--mode verify` setup line |
| PC 2, WSL2 CUDA (`/opt/igneum/igneum-prove-host`) | 4 Oct 19:00Z, package sources | `0x05db1aca65f8ae9d585c7bd178a832d92a67275857f21c0d484a58c06dba61a3` | app log `run-20261004-r3-shards` (`node tools/logs.mjs job-collect-pc2-applog-paid-1ccfe586`); confirmed by the `sp1_vk_digest` inside its proof of block 59507 shard 0 (below) |
| Mac, fresh worktree `igneum-wt-programid` | 5 Oct 12:07Z, same sources as the shipped host | `0x0dfade071ffc05a50be5f7e6640fb12638bac0ea63697ec252863f55658be16a` | `igneum-prove-pin` |
Three builds of the same guest sources, three ids. Cause: `host/build.rs` compiled the guest with `sp1_build::build_program` on whatever machine built the host, and the guest ELF depends on where it is built. Shown by `strings` on the two Mac ELFs: 946 anonymous symbol names differ, and the crate hash of `igneum_prove_core` is `Csl6o96CsXEfN_` in the main checkout against `Cs5Jl7brLd39a_` in the worktree (cargo's `-C metadata` for a path crate includes the checkout path, and rustc's symbol names carry it); the ELFs also embed `/Users/joshm/.cargo/registry/...` panic-location strings, which differ again on Linux. A different ELF is a different verifying key, so every verifier rejects every other machine's proof ("sp1 vk hash mismatch" inside SP1's `verify_compressed`), and the node log showed it as a bare `NOT VERIFIED` after 114 s to 138 s. Over the same window the Mac's pool read 16 entries, 9 failed, 0 verified, 22 shards paid (included by PC 2's own node).
Fix: the guests are pinned build artefacts (`proving/igneum-prove/elf/`: both ELFs, both verifying keys, `manifest.json` with SHA-256 hashes and ids), embedded by the host and checked at every start; `--mode verify` runs on SP1's light verifier with the pinned key, no prover client and no key setup; the verify line prints the id the proof was made with next to ours. Pinned set: shard `0x0dfade07...be16a`, aggregator `0x135e67e7...6c62`.
| Verify of PC 2's proof of block 59507 shard 0 (1,272,897 bytes) on the Mac | Setup | Verify | Verdict |
|---|---|---|---|
| Before: shipped host, `ProverClient::from_env` + two key setups | 125.82 s | 0.383 s | NOT VERIFIED, no reason given |
| Before, as the node saw it (blocks 59373 and 59402) | 138.6 s and 114.4 s in all | 0.409 s and 0.104 s | NOT VERIFIED |
| After: pinned key, light verifier (`program-id` host, same proof) | 2.085 s | 0.002 s (refused on the program id before any field arithmetic) | NOT VERIFIED, `program id 0x05db1aca...61a3 IS NOT OURS 0x0dfade07...be16a`; 2.35 s wall, exit 3 |
| After, known-good case: block 56 shard 0 proven with the pinned ELF on this Mac (`--mode compressed`, 558,137 cycles, prove 1,066 s under load 113), verified against its real statement | 1.323 s | 0.108 s | VERIFIED, `program id ... (ours)`; 1.80 s wall, exit 0 |
Before: 127.0 s wall per proof on the Mac (the node saw 114 s to 139 s). After: 1.8 s to 2.4 s wall, under the 2 s target for the verify call itself; the remaining 1.3 s to 2.1 s is SP1's light verifier construction plus paging a 58 MB binary under load, and would shrink in a long-lived verifier process. Unit tests (`cargo test -p igneum-prove-host --bin igneum-prove-host`): the embedded files hash to the manifest, the embedded keys derive the manifest's ids, a changed file is refused; the ignored test re-runs SP1's setup on the embedded ELFs and gets the pinned ids. `tools/ci/pinned-guests-check.sh` was shown failing on an empty `elf/` and passing on the pinned one.
What every machine must do: the pinned shard id `0x0dfade07...be16a` differs from every id now running (Mac `0x0559759b...`, PC 2 `0x05db1aca...`), so this is a guest change for the whole devnet, and proofs in flight at the switch are rejected by a verifier that has moved. Rollout order (proving/README.md, "Pinned guest programs"): provers off on every machine; wait until `igneum_getProvingStatus` shows an empty pool on every node; install the host built from this `elf/` on every node (Mac DMG; PCs through `igneum-prove-wsl2.zip`, whose package carries `elf/`, so the WSL build embeds the same files); confirm `igneum-prove-host --mode id` prints the same shard id everywhere; provers back on. From then on a differing id is impossible without a change to the committed `elf/`.

View file

@ -43,6 +43,7 @@ Implemented on the fork branch `proving` (`vendor/igneum-node-proving`, from dev
| Proof pool with the external verifier (`igneum-prove-host --mode verify`), trust mode for test networks, template section of verified records, payout at the carrying segment, reorg unwinding | Implemented; pool unit-tested; verifier and payout exercised on the test network | `igneum/exec/src/proving.rs`, `service.rs` |
| RPCs `igneum_getShardPlan`, `igneum_getProofRecords`, `igneum_submitProofRecord`, `igneum_getAssignedShards`, `igneum_getProvingStatus`; `igneum_exportSegments` carries payouts | Implemented | `igneum/exec/src/rpc.rs` |
| Payouts in the shard statement (fixture, `ShardInput`, `execute_range`, the guest), the empty-segment plan fix, host modes `compressed` (execute + compressed, statement and proof hash in the results) and `verify` | Implemented; guest rebuilt (new shard vk) | `proving/igneum-prove` |
| Pinned guest programs (5 October 2026): the two guest ELFs and their verifying keys committed under `elf/` with a manifest of hashes and program ids, embedded by the host and checked at every start; `--mode verify` on SP1's light verifier with the pinned key (no prover client, no key setup); `--mode id`; `igneum-prove-pin` and `pin-guests.sh` to re-pin; CI check `tools/ci/pinned-guests-check.sh` | Implemented after the Mac (shard program id `0x0559759b...`) rejected every proof of PC 2 (`0x05db1aca...`): the same guest sources built on two machines gave two ELFs. Not yet rolled out: every prover and verifier moves together (proving/README.md, "Pinned guest programs") | `proving/igneum-prove/elf/`, `host/src/pinned.rs`, `host/src/bin/pin.rs` |
| `igneum-miner vmine` (voting producer for PoW-less test networks, with an EVM payout address), `sign-record`, `key-hash` | Implemented | `igneum/miner/src/proving.rs` |
| The app's prover service: `prove` setting (default off), the loop (work list, export, cut, prove, sign, submit, open-shard fallback), the Proving tile (assigned, proving, submitted, paid), WSL2 detection and Set up on Windows, CPU on macOS | Implemented; run end to end inside the engine on the Mac with the packaged binaries against the 3-node test network (bench-log, 4 October 2026 afternoon, "the app's prover loop"): proving, submitted, paid in 71 s for the smallest shard; the Windows/WSL2 path has not run | `app/igneum-app/src/prover.rs` |
| Packaging: the Mac DMG carries the host and the exporter; the Windows payload carries the Linux x86_64 host and exporter (cargo-zigbuild, glibc 2.36, sp1 `cuda` feature, 11 min 18 s on the Mac) under `wsl2/bin/` with the WSL2 scripts and the fixtures | Implemented; dry-run staged; no payload cut (0.3.3 is the coordinator's) | `packaging/mac/build-dmg.sh`, `packaging/windows/make-payload.sh` |

View file

@ -106,6 +106,7 @@ if [ -f "$PROVE_HOST" ] && [ -f "$PROVE_EXPORT" ]; then
cp "$PROVE_HOST" "$APP/Contents/Resources/bin/igneum-prove-host"
cp "$PROVE_EXPORT" "$APP/Contents/Resources/bin/igneum-prove-export"
echo "prover: igneum-prove-host and igneum-prove-export from $(dirname "$PROVE_HOST")"
"$PROVE_HOST" --mode id || { echo "the prover host's pinned guests do not pass their manifest check"; exit 1; }
else echo "warning: no $PROVE_HOST / $PROVE_EXPORT (cd proving/igneum-prove && cargo build --release -p igneum-prove-host -p igneum-prove-export); the Proving tile will say the prover is missing"; fi
write_packaged_config "$APP/Contents/Resources/igneum-app.json"
chmod 755 "$APP/Contents/MacOS/"* "$APP/Contents/Resources/bin/"*

View file

@ -6,7 +6,17 @@ Igneum proving: v0 (3 October 2026, one SP1 proof per block) and the devnet v4 s
- `fixtures/`: `block-338-shard1`, `block-341-shards2`, `block-344-shards4` (one, two and four shards at `S_p` = 7.5 M pgas, from the private simnet of `tools/prove-fixtures`), `block-78-increment` and `block-56-transfers` (the v0 blocks, one shard each), `block-56-transfers-3shards` (a test cut at 200 pgas for the CPU multi-shard check).
- `windows-wsl2/`: SETUP-PROVER.bat, PROVE-SHARD.bat (the shard at `S_p` and the two- and four-shard blocks on the GPU), PROVE-BLOCK.bat (the small block) and the Linux scripts for the project lead's PC; `make-package.sh` builds the zip.
Build and run on a machine with the SP1 toolchain (`curl -L https://sp1up.succinct.xyz | bash && sp1up`):
## Pinned guest programs (5 October 2026)
The two SP1 guests are build artefacts committed under `igneum-prove/elf/`: `igneum-prove-program.elf` and `igneum-prove-aggregator.elf`, their verifying keys (`.vk`, bincode) and `manifest.json` (SHA-256 of every file, the program ids, the SP1 crate and circuit versions, when and where they were pinned). The host embeds these files (`host/src/pinned.rs`), never a guest it compiled itself, checks every hash against the manifest at each start, refuses in the prove modes when SP1's key setup does not derive the manifest's program id, and in `--mode verify` uses the pinned key with SP1's light verifier (no prover client, no key generation). `igneum-prove-host --mode id` prints the pinned ids with no setup.
Why: on 5 October 2026 the Mac's host (shard program id `0x0559759b...`) rejected every proof from PC 2's host (`0x05db1aca...`). Both were built from the same guest sources; `host/build.rs` compiled the guest on each machine and the two toolchains produced different ELFs, so a node only ever included its own prover's records. The node's verifier also spent 114 s to 138 s per proof in the prover client and key setups before a 0.1 s to 0.4 s verify.
- A normal host build compiles nothing for the zkVM and needs no Succinct toolchain.
- Changing a guest: `igneum-prove/pin-guests.sh` (builds both guests with `IGNEUM_BUILD_GUESTS=1`, runs `igneum-prove-pin`, rebuilds the host, runs the unit tests). Commit `elf/` with the change. `tools/ci/pinned-guests-check.sh` fails CI when `elf/` and its manifest disagree or when any other script builds a guest.
- A new pin is a new program id: every prover and verifier must move together, and proofs made with the old id are rejected by a verifier with the new one (the verify line then says `program id 0x... IS NOT OURS`). Rollout order: (1) stop the provers (the app's prove setting off on every machine); (2) wait until every record in flight is either included or expired (`igneum_getProvingStatus` shows an empty pool on every node); (3) install the new host on every node (DMG on the Mac, `igneum-prove-wsl2.zip` and `SETUP-PROVER.bat` on the PCs: the package carries `elf/`, so the PC build embeds the same files) and restart the nodes; (4) confirm `--mode id` prints the same shard program id on every machine; (5) turn the provers back on.
Build and run (the Succinct toolchain, `curl -L https://sp1up.succinct.xyz | bash && sp1up`, is needed only to re-pin the guests):
```
cd proving/igneum-prove

View file

@ -2815,6 +2815,9 @@ dependencies = [
"serde_json",
"sha2 0.10.9",
"sp1-build",
"sp1-hypercube",
"sp1-primitives",
"sp1-recursion-executor",
"sp1-sdk",
"tokio",
]

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View file

@ -0,0 +1,23 @@
{
"aggregator": {
"elf": "igneum-prove-aggregator.elf",
"elf_bytes": 320264,
"elf_sha256": "0x788a67ec86972e7e27e181aeece28b320a13237cdeebdb8794527c8d6f6278f7",
"program_id": "0x135e67e742fbbcc8303676765266f25a6520ab36299820280125541051fb6c62",
"vk": "igneum-prove-aggregator.vk",
"vk_sha256": "0xc208a42bd50fdd9c959ff9af55de666b8130d103763c8e8a349f5a3feee8d339"
},
"format": "igneum-prove-elf-manifest-v1",
"pinned_at": "2026-10-05T12:07:56Z",
"pinned_on": "Darwin MacBook-Pro.local 25.6.0 Darwin Kernel Version 25.6.0: Fri Jul 31 19:19:08 PDT 2026; root:xnu-12377.161.14~5/RELEASE_ARM64_T6050 arm64",
"shard": {
"elf": "igneum-prove-program.elf",
"elf_bytes": 2825296,
"elf_sha256": "0x65901e2dd6cc5762c5a989cffc199d4621828225fcf1513e44d002660f036a1d",
"program_id": "0x0dfade071ffc05a50be5f7e6640fb12638bac0ea63697ec252863f55658be16a",
"vk": "igneum-prove-program.vk",
"vk_sha256": "0x262f5b4be7d17ec26c6aa37c438aa774a7037249202f8401d027ef73e3612e4c"
},
"sp1_circuit_version": "v6.1.0",
"sp1_crate_version": "6.8.1"
}

View file

@ -19,6 +19,19 @@ anyhow.workspace = true
hex.workspace = true
sha2 = "0.10"
tokio = { version = "1", features = ["rt-multi-thread", "time"] }
# To read the program id a compressed proof was made with (the sp1_vk_digest in its recursion public values).
sp1-recursion-executor = "=6.8.1"
sp1-hypercube = "=6.8.1"
sp1-primitives = "=6.8.1"
[[bin]]
name = "igneum-prove-host"
path = "src/main.rs"
# Does not embed the pinned files, so it builds before elf/ exists.
[[bin]]
name = "igneum-prove-pin"
path = "src/bin/pin.rs"
[build-dependencies]
sp1-build.workspace = true

View file

@ -1,14 +1,22 @@
//! Builds the two SP1 guests and stamps the host with a hash of the native sources it was built from
//! (`IGNEUM_PROVE_SOURCES`, printed in the host's first line; the guests have their own identity, the shard
//! program's verifying key). Same purpose and recipe as export/build.rs (the stale-build class of 5 October
//! 2026): `cat $(ls core/src/*.rs host/src/*.rs | sort) | shasum -a 256 | cut -c1-16` in proving/igneum-prove.
//! Stamps the host with a hash of the native sources it was built from (`IGNEUM_PROVE_SOURCES`, printed in the
//! host's first line; the stale-build class of 5 October 2026): `cat $(ls core/src/*.rs host/src/*.rs | sort) |
//! shasum -a 256 | cut -c1-16` in proving/igneum-prove. Same recipe as export/build.rs.
//!
//! The guests are pinned build artefacts (host/src/pinned.rs, elf/manifest.json), so a normal host build compiles
//! nothing for the zkVM and needs no Succinct toolchain. `IGNEUM_BUILD_GUESTS=1` builds both guests with sp1-build
//! (into target/elf-compilation/...), for `igneum-prove-pin` to turn into the next pinned set; see
//! proving/igneum-prove/pin-guests.sh. Building the guest on every machine is what gave the Mac and PC 2 different
//! program ids on 5 October 2026.
use sha2::{Digest, Sha256};
use std::path::Path;
fn main() {
sp1_build::build_program("../program");
sp1_build::build_program("../aggregator");
println!("cargo:rerun-if-env-changed=IGNEUM_BUILD_GUESTS");
if std::env::var("IGNEUM_BUILD_GUESTS").map(|v| v == "1").unwrap_or(false) {
sp1_build::build_program("../program");
sp1_build::build_program("../aggregator");
}
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("..");
let mut files: Vec<String> = Vec::new();

View file

@ -0,0 +1,88 @@
//! igneum-prove-pin: turns a fresh guest build into the pinned artefacts the host embeds (host/src/pinned.rs).
//!
//! Usage: igneum-prove-pin [--from <dir with igneum-prove-program and igneum-prove-aggregator>] [--out <elf dir>]
//! default --from: target/elf-compilation/riscv64im-succinct-zkvm-elf/release (what `IGNEUM_BUILD_GUESTS=1
//! cargo build` leaves behind), default --out: proving/igneum-prove/elf.
//!
//! Reads both ELFs, runs SP1's key setup on each (the light node: no prover), writes the ELFs, the bincode
//! verifying keys and manifest.json with SHA-256 hashes and program ids, then prints the ids. This binary does
//! not include the pinned files, so it builds before `elf/` exists (the bootstrap) and after it changes.
use anyhow::{Context, Result};
use sha2::{Digest, Sha256};
use sp1_sdk::blocking::{LightProver, Prover};
use sp1_sdk::{Elf, HashableKey, ProvingKey};
use std::path::{Path, PathBuf};
fn sha256_hex(bytes: &[u8]) -> String {
format!("0x{}", hex::encode(Sha256::digest(bytes)))
}
fn program_id(words: &[u32; 8]) -> String {
let mut b = [0u8; 32];
for (i, w) in words.iter().enumerate() {
b[i * 4..i * 4 + 4].copy_from_slice(&w.to_be_bytes());
}
format!("0x{}", hex::encode(b))
}
fn now_utc() -> String {
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let (h, m, s) = ((secs / 3600) % 24, (secs / 60) % 60, secs % 60);
let z = (secs / 86400) as i64 + 719468;
let era = z.div_euclid(146097);
let doe = z - era * 146097;
let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365;
let y = yoe + era * 400;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let d = doy - (153 * mp + 2) / 5 + 1;
let mo = if mp < 10 { mp + 3 } else { mp - 9 };
let y = if mo <= 2 { y + 1 } else { y };
format!("{y:04}-{mo:02}-{d:02}T{h:02}:{m:02}:{s:02}Z")
}
fn pin_one(light: &LightProver, from: &Path, out: &Path, name: &str) -> Result<serde_json::Value> {
let src = from.join(name);
let elf = std::fs::read(&src).with_context(|| format!("read {} (build the guests first: IGNEUM_BUILD_GUESTS=1 cargo build -p igneum-prove-host --bin igneum-prove-pin)", src.display()))?;
let pk = light.setup(Elf::Dynamic(elf.clone().into())).map_err(|e| anyhow::anyhow!("setup of {name}: {e}"))?;
let vk = pk.verifying_key();
let vk_bytes = bincode::serialize(vk)?;
let id = program_id(&vk.hash_u32());
std::fs::write(out.join(format!("{name}.elf")), &elf)?;
std::fs::write(out.join(format!("{name}.vk")), &vk_bytes)?;
println!("RESULT pin {name}: {} bytes, sha256 {}, program id {id}", elf.len(), sha256_hex(&elf));
Ok(serde_json::json!({
"elf": format!("{name}.elf"),
"elf_sha256": sha256_hex(&elf),
"elf_bytes": elf.len(),
"vk": format!("{name}.vk"),
"vk_sha256": sha256_hex(&vk_bytes),
"program_id": id,
}))
}
fn main() -> Result<()> {
let args: Vec<String> = std::env::args().collect();
let arg = |name: &str| args.iter().position(|a| a == name).and_then(|i| args.get(i + 1)).cloned();
let here = PathBuf::from(env!("CARGO_MANIFEST_DIR")).parent().map(|p| p.to_path_buf()).unwrap_or_default();
let from = arg("--from").map(PathBuf::from).unwrap_or_else(|| here.join("target/elf-compilation/riscv64im-succinct-zkvm-elf/release"));
let out = arg("--out").map(PathBuf::from).unwrap_or_else(|| here.join("elf"));
std::fs::create_dir_all(&out)?;
let host = std::process::Command::new("uname").arg("-a").output().ok().map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()).unwrap_or_else(|| "unknown".into());
let light = LightProver::new();
let shard = pin_one(&light, &from, &out, "igneum-prove-program")?;
let aggregator = pin_one(&light, &from, &out, "igneum-prove-aggregator")?;
let manifest = serde_json::json!({
"format": "igneum-prove-elf-manifest-v1",
"sp1_crate_version": "6.8.1",
"sp1_circuit_version": sp1_sdk::SP1_CIRCUIT_VERSION,
"pinned_at": now_utc(),
"pinned_on": host,
"shard": shard,
"aggregator": aggregator,
});
std::fs::write(out.join("manifest.json"), format!("{}\n", serde_json::to_string_pretty(&manifest)?))?;
println!("RESULT pin: manifest written to {}; rebuild the host so it embeds these files", out.join("manifest.json").display());
Ok(())
}

View file

@ -13,7 +13,14 @@
//! Every stage prints a `STAGE ... start` line and one `RESULT` line with a UTC timestamp, so a silent gap
//! between stages is visible (ledger P20). The host holds a Tokio runtime for the whole run and drops the
//! proof system inside it, so the CUDA client's destructor finds a runtime (ledger P20).
//!
//! The guests are pinned (5 October 2026, `pinned.rs`): the host embeds the ELFs and verifying keys committed
//! under `elf/`, checks their hashes against `elf/manifest.json` at every start, and in the prove modes refuses
//! to continue when SP1's key setup does not derive the manifest's program id. `--mode id` prints the pinned
//! ids with no setup. `--mode verify` uses SP1's light verifier and the pinned verifying key: no prover client,
//! no key generation (the 114 s to 138 s the Mac's node spent per proof on 5 October).
mod pinned;
mod proof_system;
use alloy_primitives::{Address, B256};
@ -23,12 +30,8 @@ use igneum_prove_core::shard::{build_shards, shard_statement, BuiltShard, ShardI
use igneum_prove_core::Fixture;
use proof_system::{ProofSystem, SegmentClaim, ShardWitness, Sp1ProofSystem, Sp1ShardProof, StubProofSystem};
use sha2::Digest;
use sp1_sdk::{include_elf, Elf};
use std::time::{Duration, Instant};
const SHARD_ELF: Elf = include_elf!("igneum-prove-program");
const AGG_ELF: Elf = include_elf!("igneum-prove-aggregator");
fn now() -> String {
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let (h, m, s) = ((secs / 3600) % 24, (secs / 60) % 60, secs % 60);
@ -66,11 +69,16 @@ fn run() -> Result<()> {
let args: Vec<String> = std::env::args().collect();
let arg = |name: &str| args.iter().position(|a| a == name).and_then(|i| args.get(i + 1)).cloned();
let mode = arg("--mode").unwrap_or_else(|| "all".into());
let pinned = pinned::Pinned::load()?;
if mode == "id" {
println!("RESULT id: {}", pinned.describe());
return Ok(());
}
if mode == "verify" {
// proving v0 (spec 7.7): the node's proof pool verifies a submitted shard proof off the consensus path
return run_verify(&arg("--proof").context("--proof <file>")?, &arg("--statement").context("--statement 0x<keccak of the public values>")?);
return run_verify(&pinned, &arg("--proof").context("--proof <file>")?, &arg("--statement").context("--statement 0x<keccak of the public values>")?);
}
let path = args.get(1).filter(|a| !a.starts_with("--")).context("usage: igneum-prove-host <fixture.json> [--mode native|execute|shard|compressed|block|all] [--shard N] [--prover 0x..] [--out results.json]; igneum-prove-host --mode verify --proof <file> --statement 0x..")?;
let path = args.get(1).filter(|a| !a.starts_with("--")).context("usage: igneum-prove-host <fixture.json> [--mode native|execute|shard|compressed|block|all] [--shard N] [--prover 0x..] [--out results.json]; igneum-prove-host --mode verify --proof <file> --statement 0x..; igneum-prove-host --mode id")?;
let shard_index: usize = arg("--shard").map(|s| s.parse()).transpose()?.unwrap_or(0);
let prover: Address = arg("--prover").map(|s| s.parse()).transpose()?.unwrap_or_else(|| Address::from_slice(&[0x19; 20]));
let out_path = arg("--out");
@ -193,10 +201,14 @@ fn run() -> Result<()> {
// 4. SP1, version 1 behind the trait: two programs.
stage("setup");
let t = Instant::now();
let sp1 = Sp1ProofSystem::from_env(SHARD_ELF, AGG_ELF)?;
let sp1 = Sp1ProofSystem::from_env(pinned.shard_elf(), pinned.agg_elf())?;
let setup_s = t.elapsed().as_secs_f64();
let secs = |k: &str| sp1.last_timing(k).map(|d| d.as_secs_f64()).unwrap_or(0.0);
println!("RESULT setup: {:.2} s (prover client {:.2} s, shard keys {:.2} s, aggregator keys {:.2} s), ProofSystem v{} shard program id {} aggregator id {} at {}", setup_s, secs("client"), secs("setup-shard"), secs("setup-aggregator"), Sp1ProofSystem::VERSION, sp1.program_id(), sp1.aggregator_id(), now());
if sp1.program_id() != pinned.shard_id || sp1.aggregator_id() != pinned.agg_id {
bail!("SP1's key setup derived shard program id {} and aggregator id {} from the embedded guests, the pinned manifest says {} and {}: this host would make proofs no other node accepts (re-pin with proving/igneum-prove/pin-guests.sh)", sp1.program_id(), sp1.aggregator_id(), pinned.shard_id, pinned.agg_id);
}
println!("RESULT pinned: setup matches the manifest ({})", pinned.describe());
results.insert("client_seconds".into(), secs("client").into());
results.insert("setup_seconds".into(), setup_s.into());
results.insert("shard_program_id".into(), sp1.program_id().to_string().into());
@ -409,28 +421,38 @@ fn run_compressed(sp1: &Sp1ProofSystem, shards: &[BuiltShard], index: usize, out
Ok(())
}
/// `--mode verify --proof <file> --statement 0x..`: loads the SP1 shard verifying key, verifies the compressed
/// proof against it and checks that the keccak of its public values is the statement. Exit 0 = verified,
/// `--mode verify --proof <file> --statement 0x..`: verifies the compressed proof against the PINNED shard
/// verifying key with SP1's light verifier (no prover client, no key generation) and checks that the keccak of
/// its public values is the statement. Prints the program id the proof was made with next to ours, so a proof
/// from a host with another guest build is rejected with the reason in the node log. Exit 0 = verified,
/// 3 = not verified (what the node's proof pool reads).
fn run_verify(proof_path: &str, statement: &str) -> Result<()> {
fn run_verify(pinned: &pinned::Pinned, proof_path: &str, statement: &str) -> Result<()> {
use sp1_sdk::blocking::{LightProver, Prover};
let bytes = std::fs::read(proof_path).with_context(|| format!("read {proof_path}"))?;
let want: B256 = statement.parse().context("statement is not 32 bytes of hex")?;
stage("setup");
let t = Instant::now();
let sp1 = Sp1ProofSystem::from_env(SHARD_ELF, AGG_ELF)?;
println!("RESULT setup: {:.2} s, shard program id {} at {}", t.elapsed().as_secs_f64(), sp1.program_id(), now());
let verifier = LightProver::new();
println!("RESULT setup: {:.3} s (light verifier, pinned key), shard program id {} at {}", t.elapsed().as_secs_f64(), pinned.shard_id, now());
stage("verify");
let t = Instant::now();
let proof: sp1_sdk::SP1ProofWithPublicValues = bincode::deserialize(&bytes).context("the file is not a bincode SP1 proof")?;
let got = alloy_primitives::keccak256(proof.public_values.as_slice());
let output = ShardOutput::from_bytes(proof.public_values.as_slice());
let ok = sp1.verify_shard_bytes(&proof) && got == want && output.is_some();
let claimed = pinned::claimed_program_id(&proof);
let same_program = claimed == Some(pinned.shard_id);
let crypto_ok = same_program && verifier.verify(&proof, &pinned.shard_vk, None).is_ok();
let ok = crypto_ok && got == want && output.is_some();
let dt = t.elapsed().as_secs_f64();
let program = match claimed {
Some(c) if same_program => format!("program id {c} (ours)"),
Some(c) => format!("program id {c} IS NOT OURS {} (the prover runs another guest build)", pinned.shard_id),
None => "not a compressed proof".to_string(),
};
match &output {
Some(o) => println!("RESULT verify: {} in {dt:.3} s; block {} shard {} prover {} statement {got} (want {want}) proof {} bytes at {}", if ok { "VERIFIED" } else { "NOT VERIFIED" }, o.number, o.shard_index, o.prover, bytes.len(), now()),
None => println!("RESULT verify: NOT VERIFIED in {dt:.3} s; public values are not a shard statement at {}", now()),
Some(o) => println!("RESULT verify: {} in {dt:.3} s; block {} shard {} prover {} statement {got} (want {want}) {program} proof {} bytes at {}", if ok { "VERIFIED" } else { "NOT VERIFIED" }, o.number, o.shard_index, o.prover, bytes.len(), now()),
None => println!("RESULT verify: NOT VERIFIED in {dt:.3} s; public values are not a shard statement; {program} at {}", now()),
}
drop(sp1);
if ok {
Ok(())
} else {

View file

@ -0,0 +1,219 @@
//! The pinned guest programs (5 October 2026). The two SP1 guests (`igneum-prove-program`, the shard program, and
//! `igneum-prove-aggregator`) are build artefacts committed under `proving/igneum-prove/elf/` with their verifying
//! keys and a manifest of SHA-256 hashes and program ids. The host embeds those files, never a guest it built
//! itself, so every machine that builds the host carries the same program id.
//!
//! Why: on 5 October 2026 the Mac's host (shard program id `0x0559759b...`) rejected every proof from PC 2's
//! host (`0x05db1aca...`). Both were built from the same guest sources; `sp1_build::build_program` in `build.rs`
//! compiled the guest on each machine and the two toolchains produced different ELFs, hence different verifying
//! keys. A network whose verifiers disagree about the program id only ever includes its own prover's records.
//!
//! `igneum-prove-pin` (src/bin/pin.rs) regenerates the files from a fresh guest build (`IGNEUM_BUILD_GUESTS=1`),
//! see `proving/igneum-prove/pin-guests.sh`. `check()` refuses to run when an embedded file does not hash to the
//! manifest, and the prove modes refuse when SP1's key setup does not derive the manifest's program id.
use alloy_primitives::B256;
use anyhow::{bail, Context, Result};
use igneum_prove_core::agg;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use sp1_sdk::{Elf, HashableKey, SP1Proof, SP1ProofWithPublicValues, SP1VerifyingKey};
pub const MANIFEST_FORMAT: &str = "igneum-prove-elf-manifest-v1";
pub const SHARD_ELF_BYTES: &[u8] = include_bytes!("../../elf/igneum-prove-program.elf");
pub const SHARD_VK_BYTES: &[u8] = include_bytes!("../../elf/igneum-prove-program.vk");
pub const AGG_ELF_BYTES: &[u8] = include_bytes!("../../elf/igneum-prove-aggregator.elf");
pub const AGG_VK_BYTES: &[u8] = include_bytes!("../../elf/igneum-prove-aggregator.vk");
pub const MANIFEST_JSON: &str = include_str!("../../elf/manifest.json");
/// One pinned program: file names, hashes and the program id (`agg::vk_bytes` of the verifying key's `hash_u32`).
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
pub struct PinnedProgram {
pub elf: String,
pub elf_sha256: String,
pub elf_bytes: u64,
pub vk: String,
pub vk_sha256: String,
pub program_id: String,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Manifest {
pub format: String,
pub sp1_crate_version: String,
pub sp1_circuit_version: String,
pub pinned_at: String,
pub pinned_on: String,
pub shard: PinnedProgram,
pub aggregator: PinnedProgram,
}
impl Manifest {
pub fn parse(json: &str) -> Result<Self> {
let m: Manifest = serde_json::from_str(json).context("elf/manifest.json does not parse")?;
if m.format != MANIFEST_FORMAT {
bail!("elf/manifest.json format {} is not {MANIFEST_FORMAT}", m.format);
}
Ok(m)
}
pub fn embedded() -> Result<Self> {
Self::parse(MANIFEST_JSON)
}
}
pub fn sha256_hex(bytes: &[u8]) -> String {
format!("0x{}", hex::encode(Sha256::digest(bytes)))
}
pub fn program_id_of(vk: &SP1VerifyingKey) -> B256 {
agg::vk_bytes(&vk.hash_u32())
}
/// The checked, embedded set: the manifest, both ELFs as `Elf::Static` and both verifying keys.
pub struct Pinned {
pub manifest: Manifest,
pub shard_vk: SP1VerifyingKey,
/// For the aggregated record's verifier (proving v0 next step); checked against the manifest today.
#[allow(dead_code)]
pub agg_vk: SP1VerifyingKey,
pub shard_id: B256,
pub agg_id: B256,
}
impl Pinned {
/// Hashes every embedded file against the manifest and derives the ids from the embedded keys (no SP1 setup).
/// Errors name the file that differs, so a host built against a half-updated `elf/` says so at once.
pub fn load() -> Result<Self> {
let manifest = Manifest::embedded()?;
check_file("shard ELF", SHARD_ELF_BYTES, &manifest.shard.elf_sha256)?;
check_file("shard verifying key", SHARD_VK_BYTES, &manifest.shard.vk_sha256)?;
check_file("aggregator ELF", AGG_ELF_BYTES, &manifest.aggregator.elf_sha256)?;
check_file("aggregator verifying key", AGG_VK_BYTES, &manifest.aggregator.vk_sha256)?;
let shard_vk: SP1VerifyingKey = bincode::deserialize(SHARD_VK_BYTES).context("the embedded shard verifying key does not deserialize")?;
let agg_vk: SP1VerifyingKey = bincode::deserialize(AGG_VK_BYTES).context("the embedded aggregator verifying key does not deserialize")?;
let shard_id = program_id_of(&shard_vk);
let agg_id = program_id_of(&agg_vk);
check_id("shard", shard_id, &manifest.shard.program_id)?;
check_id("aggregator", agg_id, &manifest.aggregator.program_id)?;
Ok(Self { manifest, shard_vk, agg_vk, shard_id, agg_id })
}
pub fn shard_elf(&self) -> Elf {
Elf::Static(SHARD_ELF_BYTES)
}
pub fn agg_elf(&self) -> Elf {
Elf::Static(AGG_ELF_BYTES)
}
/// One line for logs: both ids and the manifest's provenance.
pub fn describe(&self) -> String {
format!(
"pinned guests: shard program id {} ({} bytes, sha256 {}) aggregator id {} ({} bytes), pinned {} on {}, SP1 {} circuit {}",
self.shard_id,
SHARD_ELF_BYTES.len(),
&self.manifest.shard.elf_sha256[..18],
self.agg_id,
AGG_ELF_BYTES.len(),
self.manifest.pinned_at,
self.manifest.pinned_on,
self.manifest.sp1_crate_version,
self.manifest.sp1_circuit_version
)
}
}
fn check_file(what: &str, bytes: &[u8], want: &str) -> Result<()> {
let got = sha256_hex(bytes);
if !got.eq_ignore_ascii_case(want) {
bail!("the embedded {what} hashes to {got}, the manifest says {want}: elf/ was changed without re-pinning (run proving/igneum-prove/pin-guests.sh)");
}
Ok(())
}
fn check_id(what: &str, got: B256, want: &str) -> Result<()> {
let want: B256 = want.parse().with_context(|| format!("the manifest's {what} program id is not 32 bytes of hex"))?;
if got != want {
bail!("the embedded {what} verifying key derives program id {got}, the manifest says {want}");
}
Ok(())
}
/// The program id a compressed proof was made with: the `sp1_vk_digest` words in its recursion public values,
/// which the verifier compares with the verifying key it holds. Printed next to our own id, so a rejection
/// caused by a different guest build says so in the node log instead of a bare NOT VERIFIED.
pub fn claimed_program_id(proof: &SP1ProofWithPublicValues) -> Option<B256> {
use sp1_hypercube::PrimeField32;
use sp1_recursion_executor::RecursionPublicValues;
use std::borrow::Borrow;
let SP1Proof::Compressed(p) = &proof.proof else { return None };
if p.proof.public_values.len() != sp1_hypercube::PROOF_MAX_NUM_PVS {
return None;
}
let pv: &RecursionPublicValues<sp1_primitives::SP1Field> = p.proof.public_values.as_slice().borrow();
let words: [u32; 8] = pv.sp1_vk_digest.map(|x| x.as_canonical_u32());
Some(agg::vk_bytes(&words))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn manifest_parses_and_names_both_programs() {
let m = Manifest::embedded().unwrap();
assert_eq!(m.format, MANIFEST_FORMAT);
assert_eq!(m.sp1_crate_version, "6.8.1");
assert_eq!(m.shard.elf, "igneum-prove-program.elf");
assert_eq!(m.aggregator.elf, "igneum-prove-aggregator.elf");
assert!(m.shard.program_id.starts_with("0x") && m.shard.program_id.len() == 66, "{}", m.shard.program_id);
assert_ne!(m.shard.program_id, m.aggregator.program_id);
}
#[test]
fn embedded_files_hash_to_the_manifest() {
let m = Manifest::embedded().unwrap();
assert_eq!(sha256_hex(SHARD_ELF_BYTES), m.shard.elf_sha256);
assert_eq!(SHARD_ELF_BYTES.len() as u64, m.shard.elf_bytes);
assert_eq!(sha256_hex(SHARD_VK_BYTES), m.shard.vk_sha256);
assert_eq!(sha256_hex(AGG_ELF_BYTES), m.aggregator.elf_sha256);
assert_eq!(AGG_ELF_BYTES.len() as u64, m.aggregator.elf_bytes);
assert_eq!(sha256_hex(AGG_VK_BYTES), m.aggregator.vk_sha256);
}
#[test]
fn embedded_keys_derive_the_manifest_ids() {
let p = Pinned::load().unwrap();
assert_eq!(p.shard_id.to_string(), p.manifest.shard.program_id);
assert_eq!(p.agg_id.to_string(), p.manifest.aggregator.program_id);
}
#[test]
fn a_changed_file_is_refused() {
let mut bytes = SHARD_ELF_BYTES.to_vec();
bytes[100] ^= 0x01;
let err = check_file("shard ELF", &bytes, &Manifest::embedded().unwrap().shard.elf_sha256).unwrap_err().to_string();
assert!(err.contains("re-pinning"), "{err}");
let err = check_id("shard", B256::ZERO, &Manifest::embedded().unwrap().shard.program_id).unwrap_err().to_string();
assert!(err.contains("derives program id"), "{err}");
}
#[test]
fn sha256_hex_is_lower_case_with_prefix() {
assert_eq!(sha256_hex(b""), "0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855");
}
/// SP1's own key setup over the embedded shard ELF reproduces the pinned id (about 15 s; `cargo test -- --ignored`).
#[test]
#[ignore]
fn sp1_setup_of_the_embedded_elf_reproduces_the_pinned_id() {
use sp1_sdk::blocking::{LightProver, Prover};
use sp1_sdk::ProvingKey;
let p = Pinned::load().unwrap();
let light = LightProver::new();
let pk = light.setup(p.shard_elf()).unwrap();
assert_eq!(program_id_of(pk.verifying_key()), p.shard_id);
let pk = light.setup(p.agg_elf()).unwrap();
assert_eq!(program_id_of(pk.verifying_key()), p.agg_id);
}
}

View file

@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Re-pins the guest programs (host/src/pinned.rs): builds both SP1 guests from the sources here, derives their
# verifying keys and program ids, writes proving/igneum-prove/elf/, rebuilds the host so it embeds them, and runs
# the host's unit tests. Every prover and verifier must then move to a host built from the new elf/ together
# (proving/README.md, "Pinned guest programs"). Needs the Succinct toolchain (cargo prove); run on the Mac through
# the build lock. Usage: proving/igneum-prove/pin-guests.sh
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
LOCK="$HERE/../../tools/lock/with-lock.sh"; [ -x "$LOCK" ] || LOCK=/Users/joshm/Projects/igneum/tools/lock/with-lock.sh
export PATH="$HOME/.cargo/bin:$HOME/.sp1/bin:$PATH"
cd "$HERE"
echo "1/4 building the guests and the pin tool"
"$LOCK" build env IGNEUM_BUILD_GUESTS=1 nice -n 19 cargo build --release -j 4 -p igneum-prove-host --bin igneum-prove-pin
echo "2/4 pinning"
./target/release/igneum-prove-pin
echo "3/4 rebuilding the host with the pinned files embedded"
"$LOCK" build env nice -n 19 cargo build --release -j 4 -p igneum-prove-host --bin igneum-prove-host
./target/release/igneum-prove-host --mode id
echo "4/4 unit tests"
"$LOCK" build env nice -n 19 cargo test --release -j 4 -p igneum-prove-host --bin igneum-prove-host
echo "pinned: commit proving/igneum-prove/elf/ with the host change; every prover and verifier moves together"

View file

@ -39,6 +39,7 @@ if [ "${SKIP_GATE:-0}" != "1" ]; then
if ! "$ROOT/tools/lock/with-lock.sh" measure "$H" "$ROOT/proving/fixtures/block-338-shard1.json" --mode execute --shard 0 >>"$GATE_LOG" 2>&1; then
echo "GATE FAILED: the guest did not execute the shard fixture (the exact failure the PC hit on 4 October); see $GATE_LOG"; exit 1
fi
"$H" --mode id | tee -a "$GATE_LOG" # the pinned program ids this package carries (the PC's build embeds the same elf/ files)
echo "gate: passed ($(grep -c '^RESULT' "$GATE_LOG") RESULT lines)"
fi
rm -f "$OUT"

37
tools/ci/pinned-guests-check.sh Executable file
View file

@ -0,0 +1,37 @@
#!/usr/bin/env bash
# The divergent-guest class (5 October 2026): the host used to compile the SP1 guests on whatever machine built it
# (`sp1_build::build_program` in host/build.rs), and the Mac's and PC 2's toolchains produced different ELFs, so
# the two nodes had different shard program ids and neither accepted the other's proofs. Rule: the guests are
# pinned under proving/igneum-prove/elf/ with a manifest of hashes and ids, the host embeds them, and only
# proving/igneum-prove/pin-guests.sh builds a guest. This check fails CI when (a) a committed elf/ file does not
# hash to the manifest, or (b) any script other than pin-guests.sh sets IGNEUM_BUILD_GUESTS, or calls
# `cargo prove build`, or builds a guest crate (-p igneum-prove-program / igneum-prove-aggregator) directly.
set -euo pipefail
cd "$(dirname "$0")/../.."
fail=0
E=proving/igneum-prove/elf
for f in manifest.json igneum-prove-program.elf igneum-prove-program.vk igneum-prove-aggregator.elf igneum-prove-aggregator.vk; do
[ -f "$E/$f" ] || { echo "pinned-guests: $E/$f is missing"; fail=1; }
done
if [ "$fail" = 0 ]; then
while IFS= read -r line; do
file="${line%% *}"; want="${line#* }"
got="0x$(shasum -a 256 "$E/$file" | cut -c1-64)"
if [ "$got" != "$want" ]; then echo "pinned-guests: $E/$file hashes to $got, the manifest says $want (run proving/igneum-prove/pin-guests.sh)"; fail=1; fi
done < <(python3 -c '
import json,sys
m=json.load(open(sys.argv[1]))
assert m["format"]=="igneum-prove-elf-manifest-v1", m["format"]
for k in ("shard","aggregator"):
print(m[k]["elf"], m[k]["elf_sha256"]); print(m[k]["vk"], m[k]["vk_sha256"])
' "$E/manifest.json")
fi
ALLOW='^(proving/igneum-prove/pin-guests\.sh|proving/igneum-prove/host/build\.rs|proving/igneum-prove/host/src/bin/pin\.rs|tools/ci/pinned-guests-check\.sh)$'
while IFS= read -r f; do
[[ "$f" =~ $ALLOW ]] && continue
if grep -qE 'IGNEUM_BUILD_GUESTS=1|cargo prove build|-p igneum-prove-(program|aggregator)\b' "$f"; then
echo "pinned-guests: $f builds a guest outside pin-guests.sh (the host embeds the pinned elf/ files; never build a guest elsewhere)"; fail=1
fi
done < <(git ls-files 'packaging/**' 'proving/**' 'infra/**' 'tools/**' 'relay/playbooks/**' 'app/igneum-app/src/**' '.github/**' | grep -E '\.(sh|ps1|mjs|rs|yml|bat)$')
[ "$fail" = 0 ] && echo "pinned-guests: elf/ matches its manifest and no script builds a guest outside pin-guests.sh"
exit $fail