igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity

The acceptance layer: the test harness map, its page, and the recorder that writes runs into the registry

Browse source 
tools/ci/test-map.json maps every automated case of the Test and Acceptance Standard's registry to the cell that runs it (the seven crate suites of the release matrix, the pre-push gate, the freeze check, the fast-time harness, the finality simulator, the P01 vectors and parser-malformed harnesses as they land), each cell with its command, box class and fixtures (F0 onward), each case with what the cell proves and what remains; 31 of 109 automated cases map tonight, 78 read NOT RUN with the reason. tools/ci/test-map-doc.mjs writes docs/plans/igneum-2.0-test-harness-map.md from the JSON (--check in the gate refuses a stale page). tools/ci/test-record.mjs --check refuses an unmapped automated case without a reason or an unknown id; --record <batch> writes run_status, run_id, evidence_path, updated and the evidence record (cell, manifest sha, coverage) to the mapped cases only, leaves every accept text byte-identical (the self-test diffs them), gives an unmapped automated case NOT RUN with its reason and a manual case nothing. Rule: a mapped cell's green writes RUNNING; PASS only when coverage is full and the evidence file exists; never PASS by inference.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-08 17:35:45 +00:00
parent afddf22292
commit 21cec35edf
8 changed files with 4635 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

218
docs/plans/igneum-2.0-test-harness-map.md Normal file
View file

@ -0,0 +1,218 @@
# Igneum 2.0 test harness map
Generated from tools/ci/test-map.json by tools/ci/test-map-doc.mjs; edit the JSON, never this page. Registry: docs/plans/igneum-2.0-test-registry.json (128 cases).
Rule: a case maps to a cell only where the cell's tests visibly answer it; coverage names what the cell proves and what remains; a mapped cell's green writes RUNNING, PASS only when coverage is full and the evidence file exists; an automated case with no cell reads NOT RUN with its reason, never PASS by inference.
## Cells (the harness or suite, what it runs, where, the fixtures it needs)
### suite:pow
- Command: `tools/build-remote.sh --priority gate -- test --release (from igneum-pow/)`
- Box class: suite (gate priority)
- Fixtures: F0, F1
- Cases:
- POW-02 Validate generated programs and index folding: partial: program derivation, ds55 geometry, the v6 fold, the packs and the spec read-back tests; the independent re-implementation and the index-fold census are the hash lane's harnesses
- POW-08 Keep rejected mechanisms out of the shipped claim: partial: the freeze list and the generator version pin; the public-claim scrub is the site gate's
### suite:app
- Command: `tools/build-remote.sh --priority gate -- test --release (from app/igneum-app/)`
- Box class: suite (gate priority)
- Fixtures: F0, F1
- Cases:
- UX-02 Make pause, stop and safe tuning reliable: partial: the engine state machine's pause, stop and knob tests; the operator study is UX-01's
- UX-03 Show net earnings and compatibility honestly: partial: the card and earnings rendering tests; the net-earnings model against real bills is COM/ECO evidence
- UX-07 Expose actionable failures and safe updates: partial: the refused-update and manifest tests; the update-return lane's install classes are its own rows
- VER-08 Keep every user-facing state truthful: partial: the app's own state-word tests; the wallet and light client rows are VER-01 to VER-03
### suite:core
- Command: `tools/build-remote.sh --priority gate -- test --release -p kaspa-consensus-core (from the node fork)`
- Box class: suite (gate priority)
- Fixtures: F0, F1
- Cases:
- FIN-01 Agree on ordering, work and executed state: partial: the finality field and checkpoint tests on the object; ordering under load is the consensus suite and the sim
- INC-01 Reconcile issuance, fees, burns and recipients: partial: emission, fee and payout arithmetic tests; the reconciliation over a live chain is the economics lane's
- ROT-06 Activate datasets without hidden exclusions: partial: the dataset activation and digest tests; a live activation is the fast-time harness
### suite:consensus
- Command: `tools/build-remote.sh --priority gate -- test --release -p kaspa-consensus (from the node fork)`
- Box class: suite (gate priority)
- Fixtures: F0, F1
- Cases:
- FIN-01 Agree on ordering, work and executed state: partial: the finality processes' agreement tests
- FIN-03 Cross authority expiry in a long partition: partial: rule v4's anchored table and authority expiry tests (bee41b5e)
- FIN-04 Stop signing while mining continues: partial: the frozen table without its time expiry (rule v4)
- FIN-05 Authenticate voter-set changes and pooled keys: partial: key succession and another scheme's vote refused
- FIN-07 Recover deterministically after reconnection and crash: partial: majority-continuity recovery (rule v4)
- ROT-05 Continue or pause correctly when finality stops: partial: the finality-stopped pause tests
- ZKP-01 Reject missing and invalid proofs: partial: a proof-less or wrong-statement block refused by consensus
### suite:exec
- Command: `tools/build-remote.sh --priority gate -- test --release -p igneum-exec (from the node fork)`
- Box class: suite (gate priority)
- Fixtures: F0, F1
- Cases:
- ZKP-01 Reject missing and invalid proofs: partial: a real proof verifies, a wrong statement and garbage are refused (nativeverify)
- ZKP-03 Bind network, epoch, job and state roots: partial: a snapshot under another digest refused, the epoch streams bound to the digest
- OPS-04 Recover nodes from crash and storage damage: partial: snapshot install, replay and genesis recapture after a crash
- EVM-02 Preserve transaction binding and replay protection: partial: the replay tests on the day stream; the signature and chain-id binding vectors are an EVM harness still to map
### suite:miner
- Command: `tools/build-remote.sh --priority gate -- test --release -p igneum-miner (from the node fork)`
- Box class: suite (gate priority)
- Fixtures: F0, F1
- Cases:
- UX-06 Verify actual miner-selected work templates: partial: the miner's own template selection tests
- UX-04 Pay small operators without hidden custody: partial: payout label and share accounting tests; custody is the pool lane's row
- POW-01 Match independent execution across every backend: partial: the pack re-check seam against the pack's 96 vectors (recheck_pack); the million-vector campaign is harness:p01-vectors
### suite:p2p-flows
- Command: `tools/build-remote.sh --priority gate -- test --release -p kaspa-p2p-flows (from the node fork)`
- Box class: suite (gate priority)
- Fixtures: F0, F1
- Cases:
- OPS-06 Contain malicious network and API traffic: partial: malformed relay messages refused; traffic containment under load is an OPS harness
- ZKP-06 Verify aggregation coverage and completeness: partial: proof relay coverage tests; aggregation completeness is the proving lane's test set
### gate:pre-push
- Command: `tools/ci/pre-push.sh (every landing; the heavy checks ship to a box)`
- Box class: gate
- Fixtures: F1
- Cases:
- GOV-05 Prove the test oracle detects broken behaviour: partial: every gate check carries a self-test that fires on a known failure and passes a known success (77 checks); the suites' own oracles are their mutation rows
- GOV-06 Enforce scope and optional-feature discipline: partial: the launch-gates, forbidden-strings and scope checks on the served site
- UX-08 Publish competitive accessible software: partial: the site gate (overlaps, lateral scroll, the phone menu, previews) on every served page
### check:freeze
- Command: `consensus/pow/build.rs rule 19 at every kaspa-pow build; packaging/pow-freeze.txt and packaging/d1-freeze.txt`
- Box class: build
- Fixtures: F0
- Cases:
- GOV-01 Freeze the release and its claims: partial: the linked igneum-pow tree's fingerprint must match a listed freeze or the build fails, the binary prints which; the manifest (igneum_getManifest) names the object digest; the signed F0 manifest is the node lane's row tonight
### harness:fast-time-60x
- Command: `the v5 fast-time harness on the devnet object at 60x (infra/fast-time/override-60x.json; the node lane's run shape)`
- Box class: harness (lease pool, never build-1)
- Fixtures: F0, F3
- Cases:
- ROT-01 Agree across every hourly boundary: partial: hourly boundaries crossed at 60x with every node agreeing
- ROT-02 Cross weekly and family boundaries together: partial: the weekly and family boundary crossings at 60x
### harness:finality-sim
- Command: `igneum/harness-sim (the finality lane's scenarios: partition, split populations, long partition)`
- Box class: harness
- Fixtures: F0, F4
- Cases:
- FIN-02 Attack finality with split honest populations: partial: split honest populations in the simulator; the real-node run is the fault network F4
- FIN-08 Combine boundaries, faults and adversarial scheduling: partial: the combined boundary and fault scenarios in the simulator; model checking is the formal review
### harness:p01-vectors
- Command: `igneum-miner --recheck-vectors 1000000 --device <cuda|opencl|metal> --pack <dir> --out <evidence.json> (building; the fleet's TAS pods)`
- Box class: pods (one backend each)
- Fixtures: F0, F2
- Cases:
- POW-01 Match independent execution across every backend: the million-vector campaign per backend as P01 is written; PASS only when every supported backend reads a million vectors with zero disagreement
### harness:parser-malformed
- Command: `cargo test --release -p <crate> --test parser_malformed (building; igneum-exec, rpc-core, igneum-miner)`
- Box class: suite
- Fixtures: F5
- Cases:
- POW-06 Bound verifier work and malformed-input cost: partial: ten thousand seeded malformed cases per parser refused without panic in bounded time; the verifier work bound is the proving lane's measurement
## Automated cases with no harness in the matrix (NOT RUN, the reason)
- GOV-02 Approve thresholds before results: the approval is recorded in the registry's approval field; the automated half (thresholds frozen before any run_status) is the gate rule landing by 21:00
- GOV-04 Preserve raw and negative evidence: the evidence vault F9 (raw and negative evidence preserved) is the gate rule landing by 21:00: a PASS must carry its evidence file
- GOV-08 Invalidate stale evidence and control public status: the stale-evidence rule (evidence older than the manifest sha reads NOT RUN) is the gate rule landing by 21:00
- GPU-01 Cover the declared commodity population: the hardware lab F2 population rows are the fleet and floor lanes' benches, not in the matrix
- GPU-02 Reproduce Ember clock-lock savings: the Ember clock-lock rows are the hash lane's PC 1 bench
- GPU-03 Measure the real 64-register GPU cost: the 64-register cost rows are the floor lane's rented-card bench
- GPU-04 Find the memory-clock operating ladder: the memory-clock ladder is the hash lane's PC bench
- GPU-05 Test dataset fit and support-horizon costs: dataset fit rows are the floor lane's
- GPU-06 Measure accepted work under ordinary connectivity: accepted work under ordinary connectivity needs the fault network F4
- GPU-07 Survive sustained thermal and power operation: the sustained thermal run is a 24-hour bench on the PCs
- POW-05 Prevent amortised cheap winning attempts: amortised cheap winning attempts are the attack lanes' grind and era harnesses (tools/attack/f7-era, f9-grind), not in the release matrix; their rows come from those lanes
- ADV-04 Measure profitable selective participation: selective participation needs the economic model F7 and a live chain window
- ADV-06 Separate process advantage from specialisation: process-advantage separation is the adversary lanes' chip study
- ADV-07 Evaluate lifetime without forced obsolescence: lifetime rows are the adversary lanes' models
- ROT-03 Test miner-voted bring-forward governance: miner-voted bring-forward needs a vote harness on the fault network F4
- ROT-04 Resist seed selection and faster evaluators: seed-selection resistance is the census harness (the class v6 invention lane), not yet in the matrix
- ROT-07 Ablate redundant rotation layers: layer ablation is a research harness, not a release suite
- ROT-08 Pass the no-new-rules counterfactual: the no-new-rules counterfactual is a research harness
- ECO-01 Reconcile complete cost per accepted work: the economic model F7 is the economics lane's; no automated harness in the matrix
- ECO-02 Separate existing-owner and new-entrant viability: F7
- ECO-03 Let the specialist keep its sunk development: F7
- ECO-04 Model entry, exit and difficulty response: F7
- ECO-05 Stress success, contraction and cheap electricity: F7
- ECO-06 Fund security and proving as issuance falls: F7
- ECO-07 Price memory growth and honest-card displacement: F7
- EVM-01 Match the selected EVM semantics: no EVM conformance-vector harness is mapped tonight; the exec suite does not run the reference test vectors
- EVM-03 Test two-dimensional fees and proving limits: the two-dimensional fee tests need the proving-limit harness
- EVM-04 Exercise block context and randomness assumptions: block context and randomness vectors are an EVM harness still to map
- EVM-05 Run representative contract integration journeys: contract journeys need the DEX lane's integration set on Devnet 4
- EVM-06 Validate wallets, RPC and indexers: wallet, RPC and indexer validation is the explorer and wallet lanes' rows
- EVM-07 Handle execution denial-of-service workloads: execution DoS workloads need the workload catalogue F3
- EVM-08 Verify controlled execution and verifier upgrades: verifier upgrade tests need the enforced-proving lane's upgrade scenario
- ZKP-02 Bind program, verifier and security parameters: program and verifier binding vectors are the proving lane's test set, not yet in the matrix
- ZKP-04 Prevent reward and payout substitution: reward substitution tests are the proving lane's
- ZKP-05 Make proof payment idempotent across races: proof payment idempotence under races needs the capacity harness
- ZKP-08 Preserve authority and audit all acceptance paths: acceptance-path audit needs the authority test set
- CAP-01 Reproduce the historical consumer-shard result: the consumer-shard reproduction is the fleet lane's pods
- CAP-02 Prove on the actual mining configuration: proving on the mining configuration is the fleet lane's
- CAP-03 Measure the entire request-to-payment path: the request-to-payment path is the proving fleet's measurement
- CAP-04 Sustain meaningful load without queue growth: sustained load is the proving fleet's
- CAP-05 Overload and recover without false acceptance: overload and recovery is the proving fleet's
- CAP-06 Calibrate assignment windows to paid completion: assignment windows are the proving fleet's
- CAP-07 Reassign work when inputs or providers disappear: reassignment is the proving fleet's
- CAP-08 Deliver customer-verifiable output at scale: customer-verifiable output is the reference apps plus the fleet
- INC-02 Keep revenue streams and claims separate: revenue separation is the economics lane's model
- INC-03 Let a modified client choose the most profitable task: the modified-client task choice needs an adversarial client harness
- INC-04 Survive external-demand spikes and token declines: demand spikes are the economic model
- INC-05 Contain job reservation and identity-splitting abuse: reservation abuse needs the capacity harness
- INC-06 Test difficulty and timestamp manipulation: difficulty and timestamp manipulation needs the fault network F4
- INC-07 Resist self-dealing fees and fake proving demand: self-dealing fees need the economic model and a live window
- INC-08 Quantify provider and supplier failure concentration: failure concentration is a live-window measurement
- FIN-06 Analyse old-key compromise and long-range histories: old-key compromise analysis is the finality lane's long-range scenario, not yet in the sim
- VER-01 Authenticate light-client bootstrap: light-client bootstrap is the reference apps lane (/light); no automated harness mapped tonight
- VER-02 Verify evolving authority and execution statements: the evolving-authority statements are the reference apps lane's
- VER-03 Prove successful payment rather than inclusion: payment proof (/receipt) is the reference apps lane's
- VER-05 Reconstruct required state without founder storage: state reconstruction without founder storage is the OPS no-founder exercise
- VER-06 Detect withholding, corruption and stale data: withholding and corruption detection needs the fault network F4
- VER-07 Protect wallet keys, signing and recovery: wallet key protection is the wallet lane's security row
- OPS-01 Run the complete no-founder exercise: the no-founder exercise is an operations run, not a suite
- OPS-02 Diversify bootstrap and resist peer isolation: bootstrap diversity needs the fault network F4
- OPS-03 Separate update distribution from consensus authority: update distribution separation is the update-return lane's row
- OPS-05 Isolate untrusted proving workloads: proving workload isolation is the fleet's pod row
- OPS-07 Detect failures with usable evidence and runbooks: runbook detection is an operations run
- OPS-08 Repeat independent operation across releases: repeat independent operation is a cross-release observation
- UX-05 Keep voting keys with the miner through pooling: voting keys through pooling is the pool lane's row
- COM-01 Deliver a genuine contracted proof pilot: commercial evidence, no automated harness
- COM-02 Establish independent repeat purchasing: commercial evidence
- COM-03 Demonstrate service and operator margins: commercial evidence
- COM-04 Meet the customer service guarantee: commercial evidence
- COM-05 Compare against the buyer's real alternative: commercial evidence
- COM-06 Retain buyers after the pilot and subsidy period: commercial evidence
- COM-07 Fund maintenance without assumed appreciation: commercial evidence
- LEAD-02 Demonstrate comparable operator advantages: leadership comparison, an observation window
- LEAD-03 Substantiate the specialist-coexistence claim: observation window
- LEAD-04 Observe ordinary-operator retention and margins: observation window
- LEAD-05 Measure control and dependency concentration: observation window
- LEAD-06 Complete the reliability observation window: observation window
- LEAD-08 Keep leadership claims valid after release: observation window
## Count
109 automated cases: 32 mapped to a cell, 78 NOT RUN with a reason.

3998
docs/plans/igneum-2.0-test-registry.json Normal file
View file

File diff suppressed because it is too large Load diff

1
tools/ci/README.md
View file

@ -10,6 +10,7 @@
| the red watcher fires on cancelled and timed-out runs too (`ci-red.yml`, `red-watch.mjs`) | The watcher's `if` missing any of failure, cancelled, timed_out, or the conclusion not handed to the record step (the self-test reads the workflow file); the line names the kind: CI red, CI cancelled, CI timed out. | 7 October 2026 |
| gh's active account is the stored Igneum entry (`gh-account-check.sh`, in Igneum's own gh directory `~/.config/gh-igneum` through `gh-env.sh`, never the founder's) | A push or a landing from this Mac while Igneum's gh directory names any other account as active, or none (the refusal names the one step: the founder or main stores the Igneum token there with `GH_CONFIG_DIR=~/.config/gh-igneum gh auth login --with-token`; no lane does); skipped with a line while `github-suspended` stands. RULE: no lane switches gh accounts on this Mac, ever; the second owner's login belongs to other projects and must never touch Igneum; the stored entry's name is in ~/.config/igneum/gh-user, never in the repository. | 7 October 2026, 21:41 UK: a lane switched gh to the other login during the suspension; nobody could say which |
| the acceptance layer (`test-record.mjs`, `test-map.json`, `test-map-doc.mjs`; the founder's Test and Acceptance Standard, docs/plans/igneum-2.0-test-registry.json) | An automated case of the registry with no cell in the map and no NOT RUN reason; a map naming an unknown case; a stale harness-map page (generated from the JSON); the recorder's self-test: a run batch writes run_status, run_id, evidence_path, updated and the evidence record to the mapped cases only, never an accept text, and a case with no harness reads NOT RUN with its reason, never PASS by inference | 8 Oct 2026 |
| rule 26: a landing that touches a site/ or docs/ path another lane landed since the branch point carries master's copy (`rule26-no-revert.sh`, called by `merge-to-master.sh` before the merge) | A branch whose copy of such a path lacks lines master added after the merge base, or that deletes the path; refused with the path named and the fix (merge master on the branch, or name the path in the landing message as intentional). Class: 16b3e140 and 6fc57381 put the reference-apps lane's receipt.html and then all of site/lc back to older copies | 8 Oct 2026 |
| rule 24: a landing that touches a .rs, Cargo.toml, Cargo.lock, build.rs or .cargo/config checks and tests its crates on the box first (`rule24-crate-gate.sh`, called by `merge-to-master.sh` before the merge and by the hook on a push of master or release-*) | A touched crate that does not `cargo check` or whose suite is red on the box at gate priority; a file under no crate (the root .cargo/config) runs the core pair igneum-pow and app/igneum-app; a vendor crate is named and left to its lane; docs/ and site/ alone (docs-only-check.sh) run nothing; any other diff takes the full gate alone. Class: master's igneum-pow stopped compiling at 17:07 UK under landings that never built it | 8 Oct 2026 |
| no landing on the public host while the marker stands (`pre-push.sh` `forgejo_master_frozen`, `merge-to-master.sh` `forgejo_master_refusal`); the hook binds every remote rule to the remote's URL | A push of master to git.igneum.network, or a `--remote` naming it, while `tools/ci/github-suspended` stands: its master is a rewritten copy replaced at cut-over, so the landing would be lost (a branch pushed there for safekeeping passes). Before the fix the hook matched the remote NAME, so `git push origin master` bound neither the GitHub refusal nor the CI rule; the self-test now drives the hook by name through a fixture repo. Also: `gate-manifest-check.sh` and four other pipefail checks no longer pipe a file-sized producer into `grep -q` (GNU sed took SIGPIPE on an early match and the check read it as a missing run line on the Linux runners and boxes); `mirror_master` fast-forwards every box with a build-server file after a landing on ANY remote (before, only a GitHub landing fanned out, so a box landing left build-3 and build-4 at a tip 23 hours old), as a `--no-verify` copy of the master the gate already passed (a stale mirror had re-run the full gate for six minutes per box) | 8 Oct 2026 |

3
tools/ci/checks.txt
View file

@ -69,6 +69,9 @@ harness summaries never carry a raw 64-hex key (the writer's own redaction and c
docs-only pushes skip the compile-or-compute CI jobs (the changes job's classifier)
rule 24: a landing that touches a .rs, Cargo.toml, build.rs or .cargo file checks and tests its crates on the box first; docs and site alone skip it (self-test)
rule 26: a landing that touches a site/ or docs/ path another lane landed since the branch point carries master's copy, never a replace (self-test)
the acceptance recorder: a run batch writes the registry's live fields and never an accept text; an unmapped automated case reads NOT RUN with its reason (self-test)
the test map: every automated case of the registry maps to a cell or carries a NOT RUN reason; the map's ids exist
the harness map page is generated from tools/ci/test-map.json and current
the public ledger (docs/ledger-public.md) is what docs/fud-ledger.md generates: one row per item, no commit ids, times or team names (self-test first)
the ledger page reads both entry heading forms (M1 and AP-F8-1) so no in-house pass row is dropped from /ledger (known-failed first)
every workflow job carries timeout-minutes (site 15, changes 10, pow 60, sims 45; the hung-job class of 7 October 2026)

3
tools/ci/pre-push.sh
View file

@ -168,6 +168,9 @@ tree_checks() {
run "docs-only pushes skip the compile-or-compute CI jobs (the changes job's classifier)" bash tools/ci/docs-only-check.sh --self-test
run "rule 24: a landing that touches a .rs, Cargo.toml, build.rs or .cargo file checks and tests its crates on the box first; docs and site alone skip it (self-test)" bash tools/ci/rule24-crate-gate.sh --self-test
run "rule 26: a landing that touches a site/ or docs/ path another lane landed since the branch point carries master's copy, never a replace (self-test)" bash tools/ci/rule26-no-revert.sh --self-test
run "the acceptance recorder: a run batch writes the registry's live fields and never an accept text; an unmapped automated case reads NOT RUN with its reason (self-test)" node tools/ci/test-record.mjs --self-test
run "the test map: every automated case of the registry maps to a cell or carries a NOT RUN reason; the map's ids exist" node tools/ci/test-record.mjs --check
run "the harness map page is generated from tools/ci/test-map.json and current" node tools/ci/test-map-doc.mjs --check
run "the public ledger (docs/ledger-public.md) is what docs/fud-ledger.md generates: one row per item, no commit ids, times or team names (self-test first)" bash -c 'node tools/ledger/export-public.mjs --self-test && node tools/ledger/export-public.mjs --check'
run "the ledger page reads both entry heading forms (M1 and AP-F8-1) so no in-house pass row is dropped from /ledger (known-failed first)" node tools/ledger-page.mjs --self-test
run "every workflow job carries timeout-minutes (site 15, changes 10, pow 60, sims 45; the hung-job class of 7 October 2026)" bash tools/ci/workflow-timeouts-check.sh --self-test

20
tools/ci/test-map-doc.mjs Normal file
View file

@ -0,0 +1,20 @@
#!/usr/bin/env node
// docs/plans/igneum-2.0-test-harness-map.md is generated from tools/ci/test-map.json (the single source); --check refuses a stale page.
import fs from 'node:fs'; import path from 'node:path';
const ROOT = path.resolve(path.dirname(new URL(import.meta.url).pathname), '..', '..');
const map = JSON.parse(fs.readFileSync(path.join(ROOT, 'tools/ci/test-map.json'), 'utf8'));
const reg = JSON.parse(fs.readFileSync(path.join(ROOT, map.registry), 'utf8'));
const cases = (reg.suites || []).flatMap((s) => s.tests || []); const byId = new Map(cases.map((c) => [c.id, c]));
let md = `# ${map.title}\n\nGenerated from tools/ci/test-map.json by tools/ci/test-map-doc.mjs; edit the JSON, never this page. Registry: ${map.registry} (${cases.length} cases).\n\nRule: ${map.rule}.\n\n## Cells (the harness or suite, what it runs, where, the fixtures it needs)\n\n`;
for (const [cell, m] of Object.entries(map.cells)) {
md += `### ${cell}\n\n- Command: \`${m.command}\`\n- Box class: ${m.box_class}\n- Fixtures: ${m.fixtures.join(', ') || 'none'}\n- Cases:\n`;
for (const id of m.cases) md += ` - ${id} ${byId.get(id)?.title ?? ''}: ${m.coverage?.[id] ?? 'full'}\n`;
md += '\n';
}
md += `## Automated cases with no harness in the matrix (NOT RUN, the reason)\n\n`;
for (const [id, why] of Object.entries(map.not_run)) md += `- ${id} ${byId.get(id)?.title ?? ''}: ${why}\n`;
const auto = cases.filter((c) => /automated/i.test(String(c.method))).length; const mapped = new Set(Object.values(map.cells).flatMap((m) => m.cases)).size;
md += `\n## Count\n\n${auto} automated cases: ${mapped} mapped to a cell, ${Object.keys(map.not_run).length} NOT RUN with a reason.\n`;
const out = path.join(ROOT, 'docs/plans/igneum-2.0-test-harness-map.md');
if (process.argv.includes('--check')) { const cur = fs.existsSync(out) ? fs.readFileSync(out, 'utf8') : ''; if (cur !== md) { console.error('test-map-doc: the harness map page is stale; run node tools/ci/test-map-doc.mjs and commit it'); process.exit(1); } console.log('test-map-doc: the harness map page matches the JSON'); process.exit(0); }
fs.writeFileSync(out, md); console.log(`test-map-doc: wrote ${out}`);

310
tools/ci/test-map.json Normal file
View file

@ -0,0 +1,310 @@
{
"title": "Igneum 2.0 test harness map",
"registry": "docs/plans/igneum-2.0-test-registry.json",
"rule": "a case maps to a cell only where the cell's tests visibly answer it; coverage names what the cell proves and what remains; a mapped cell's green writes RUNNING, PASS only when coverage is full and the evidence file exists; an automated case with no cell reads NOT RUN with its reason, never PASS by inference",
"cells": {
"suite:pow": {
"command": "tools/build-remote.sh --priority gate -- test --release (from igneum-pow/)",
"box_class": "suite (gate priority)",
"fixtures": [
"F0",
"F1"
],
"cases": [
"POW-02",
"POW-08"
],
"coverage": {
"POW-02": "partial: program derivation, ds55 geometry, the v6 fold, the packs and the spec read-back tests; the independent re-implementation and the index-fold census are the hash lane's harnesses",
"POW-08": "partial: the freeze list and the generator version pin; the public-claim scrub is the site gate's"
}
},
"suite:app": {
"command": "tools/build-remote.sh --priority gate -- test --release (from app/igneum-app/)",
"box_class": "suite (gate priority)",
"fixtures": [
"F0",
"F1"
],
"cases": [
"UX-02",
"UX-03",
"UX-07",
"VER-08"
],
"coverage": {
"UX-02": "partial: the engine state machine's pause, stop and knob tests; the operator study is UX-01's",
"UX-03": "partial: the card and earnings rendering tests; the net-earnings model against real bills is COM/ECO evidence",
"UX-07": "partial: the refused-update and manifest tests; the update-return lane's install classes are its own rows",
"VER-08": "partial: the app's own state-word tests; the wallet and light client rows are VER-01 to VER-03"
}
},
"suite:core": {
"command": "tools/build-remote.sh --priority gate -- test --release -p kaspa-consensus-core (from the node fork)",
"box_class": "suite (gate priority)",
"fixtures": [
"F0",
"F1"
],
"cases": [
"FIN-01",
"INC-01",
"ROT-06"
],
"coverage": {
"FIN-01": "partial: the finality field and checkpoint tests on the object; ordering under load is the consensus suite and the sim",
"INC-01": "partial: emission, fee and payout arithmetic tests; the reconciliation over a live chain is the economics lane's",
"ROT-06": "partial: the dataset activation and digest tests; a live activation is the fast-time harness"
}
},
"suite:consensus": {
"command": "tools/build-remote.sh --priority gate -- test --release -p kaspa-consensus (from the node fork)",
"box_class": "suite (gate priority)",
"fixtures": [
"F0",
"F1"
],
"cases": [
"FIN-01",
"FIN-03",
"FIN-04",
"FIN-05",
"FIN-07",
"ROT-05",
"ZKP-01"
],
"coverage": {
"FIN-01": "partial: the finality processes' agreement tests",
"FIN-03": "partial: rule v4's anchored table and authority expiry tests (bee41b5e)",
"FIN-04": "partial: the frozen table without its time expiry (rule v4)",
"FIN-05": "partial: key succession and another scheme's vote refused",
"FIN-07": "partial: majority-continuity recovery (rule v4)",
"ROT-05": "partial: the finality-stopped pause tests",
"ZKP-01": "partial: a proof-less or wrong-statement block refused by consensus"
}
},
"suite:exec": {
"command": "tools/build-remote.sh --priority gate -- test --release -p igneum-exec (from the node fork)",
"box_class": "suite (gate priority)",
"fixtures": [
"F0",
"F1"
],
"cases": [
"ZKP-01",
"ZKP-03",
"OPS-04",
"EVM-02"
],
"coverage": {
"ZKP-01": "partial: a real proof verifies, a wrong statement and garbage are refused (nativeverify)",
"ZKP-03": "partial: a snapshot under another digest refused, the epoch streams bound to the digest",
"OPS-04": "partial: snapshot install, replay and genesis recapture after a crash",
"EVM-02": "partial: the replay tests on the day stream; the signature and chain-id binding vectors are an EVM harness still to map"
}
},
"suite:miner": {
"command": "tools/build-remote.sh --priority gate -- test --release -p igneum-miner (from the node fork)",
"box_class": "suite (gate priority)",
"fixtures": [
"F0",
"F1"
],
"cases": [
"UX-06",
"UX-04",
"POW-01"
],
"coverage": {
"UX-06": "partial: the miner's own template selection tests",
"UX-04": "partial: payout label and share accounting tests; custody is the pool lane's row",
"POW-01": "partial: the pack re-check seam against the pack's 96 vectors (recheck_pack); the million-vector campaign is harness:p01-vectors"
}
},
"suite:p2p-flows": {
"command": "tools/build-remote.sh --priority gate -- test --release -p kaspa-p2p-flows (from the node fork)",
"box_class": "suite (gate priority)",
"fixtures": [
"F0",
"F1"
],
"cases": [
"OPS-06",
"ZKP-06"
],
"coverage": {
"OPS-06": "partial: malformed relay messages refused; traffic containment under load is an OPS harness",
"ZKP-06": "partial: proof relay coverage tests; aggregation completeness is the proving lane's test set"
}
},
"gate:pre-push": {
"command": "tools/ci/pre-push.sh (every landing; the heavy checks ship to a box)",
"box_class": "gate",
"fixtures": [
"F1"
],
"cases": [
"GOV-05",
"GOV-06",
"UX-08"
],
"coverage": {
"GOV-05": "partial: every gate check carries a self-test that fires on a known failure and passes a known success (77 checks); the suites' own oracles are their mutation rows",
"GOV-06": "partial: the launch-gates, forbidden-strings and scope checks on the served site",
"UX-08": "partial: the site gate (overlaps, lateral scroll, the phone menu, previews) on every served page"
}
},
"check:freeze": {
"command": "consensus/pow/build.rs rule 19 at every kaspa-pow build; packaging/pow-freeze.txt and packaging/d1-freeze.txt",
"box_class": "build",
"fixtures": [
"F0"
],
"cases": [
"GOV-01"
],
"coverage": {
"GOV-01": "partial: the linked igneum-pow tree's fingerprint must match a listed freeze or the build fails, the binary prints which; the manifest (igneum_getManifest) names the object digest; the signed F0 manifest is the node lane's row tonight"
}
},
"harness:fast-time-60x": {
"command": "the v5 fast-time harness on the devnet object at 60x (infra/fast-time/override-60x.json; the node lane's run shape)",
"box_class": "harness (lease pool, never build-1)",
"fixtures": [
"F0",
"F3"
],
"cases": [
"ROT-01",
"ROT-02"
],
"coverage": {
"ROT-01": "partial: hourly boundaries crossed at 60x with every node agreeing",
"ROT-02": "partial: the weekly and family boundary crossings at 60x"
}
},
"harness:finality-sim": {
"command": "igneum/harness-sim (the finality lane's scenarios: partition, split populations, long partition)",
"box_class": "harness",
"fixtures": [
"F0",
"F4"
],
"cases": [
"FIN-02",
"FIN-08"
],
"coverage": {
"FIN-02": "partial: split honest populations in the simulator; the real-node run is the fault network F4",
"FIN-08": "partial: the combined boundary and fault scenarios in the simulator; model checking is the formal review"
}
},
"harness:p01-vectors": {
"command": "igneum-miner --recheck-vectors 1000000 --device <cuda|opencl|metal> --pack <dir> --out <evidence.json> (building; the fleet's TAS pods)",
"box_class": "pods (one backend each)",
"fixtures": [
"F0",
"F2"
],
"cases": [
"POW-01"
],
"coverage": {
"POW-01": "the million-vector campaign per backend as P01 is written; PASS only when every supported backend reads a million vectors with zero disagreement"
}
},
"harness:parser-malformed": {
"command": "cargo test --release -p <crate> --test parser_malformed (building; igneum-exec, rpc-core, igneum-miner)",
"box_class": "suite",
"fixtures": [
"F5"
],
"cases": [
"POW-06"
],
"coverage": {
"POW-06": "partial: ten thousand seeded malformed cases per parser refused without panic in bounded time; the verifier work bound is the proving lane's measurement"
}
}
},
"not_run": {
"GOV-02": "the approval is recorded in the registry's approval field; the automated half (thresholds frozen before any run_status) is the gate rule landing by 21:00",
"GOV-04": "the evidence vault F9 (raw and negative evidence preserved) is the gate rule landing by 21:00: a PASS must carry its evidence file",
"GOV-08": "the stale-evidence rule (evidence older than the manifest sha reads NOT RUN) is the gate rule landing by 21:00",
"GPU-01": "the hardware lab F2 population rows are the fleet and floor lanes' benches, not in the matrix",
"GPU-02": "the Ember clock-lock rows are the hash lane's PC 1 bench",
"GPU-03": "the 64-register cost rows are the floor lane's rented-card bench",
"GPU-04": "the memory-clock ladder is the hash lane's PC bench",
"GPU-05": "dataset fit rows are the floor lane's",
"GPU-06": "accepted work under ordinary connectivity needs the fault network F4",
"GPU-07": "the sustained thermal run is a 24-hour bench on the PCs",
"POW-05": "amortised cheap winning attempts are the attack lanes' grind and era harnesses (tools/attack/f7-era, f9-grind), not in the release matrix; their rows come from those lanes",
"ADV-04": "selective participation needs the economic model F7 and a live chain window",
"ADV-06": "process-advantage separation is the adversary lanes' chip study",
"ADV-07": "lifetime rows are the adversary lanes' models",
"ROT-03": "miner-voted bring-forward needs a vote harness on the fault network F4",
"ROT-04": "seed-selection resistance is the census harness (the class v6 invention lane), not yet in the matrix",
"ROT-07": "layer ablation is a research harness, not a release suite",
"ROT-08": "the no-new-rules counterfactual is a research harness",
"ECO-01": "the economic model F7 is the economics lane's; no automated harness in the matrix",
"ECO-02": "F7",
"ECO-03": "F7",
"ECO-04": "F7",
"ECO-05": "F7",
"ECO-06": "F7",
"ECO-07": "F7",
"EVM-01": "no EVM conformance-vector harness is mapped tonight; the exec suite does not run the reference test vectors",
"EVM-03": "the two-dimensional fee tests need the proving-limit harness",
"EVM-04": "block context and randomness vectors are an EVM harness still to map",
"EVM-05": "contract journeys need the DEX lane's integration set on Devnet 4",
"EVM-06": "wallet, RPC and indexer validation is the explorer and wallet lanes' rows",
"EVM-07": "execution DoS workloads need the workload catalogue F3",
"EVM-08": "verifier upgrade tests need the enforced-proving lane's upgrade scenario",
"ZKP-02": "program and verifier binding vectors are the proving lane's test set, not yet in the matrix",
"ZKP-04": "reward substitution tests are the proving lane's",
"ZKP-05": "proof payment idempotence under races needs the capacity harness",
"ZKP-08": "acceptance-path audit needs the authority test set",
"CAP-01": "the consumer-shard reproduction is the fleet lane's pods",
"CAP-02": "proving on the mining configuration is the fleet lane's",
"CAP-03": "the request-to-payment path is the proving fleet's measurement",
"CAP-04": "sustained load is the proving fleet's",
"CAP-05": "overload and recovery is the proving fleet's",
"CAP-06": "assignment windows are the proving fleet's",
"CAP-07": "reassignment is the proving fleet's",
"CAP-08": "customer-verifiable output is the reference apps plus the fleet",
"INC-02": "revenue separation is the economics lane's model",
"INC-03": "the modified-client task choice needs an adversarial client harness",
"INC-04": "demand spikes are the economic model",
"INC-05": "reservation abuse needs the capacity harness",
"INC-06": "difficulty and timestamp manipulation needs the fault network F4",
"INC-07": "self-dealing fees need the economic model and a live window",
"INC-08": "failure concentration is a live-window measurement",
"FIN-06": "old-key compromise analysis is the finality lane's long-range scenario, not yet in the sim",
"VER-01": "light-client bootstrap is the reference apps lane (/light); no automated harness mapped tonight",
"VER-02": "the evolving-authority statements are the reference apps lane's",
"VER-03": "payment proof (/receipt) is the reference apps lane's",
"VER-05": "state reconstruction without founder storage is the OPS no-founder exercise",
"VER-06": "withholding and corruption detection needs the fault network F4",
"VER-07": "wallet key protection is the wallet lane's security row",
"OPS-01": "the no-founder exercise is an operations run, not a suite",
"OPS-02": "bootstrap diversity needs the fault network F4",
"OPS-03": "update distribution separation is the update-return lane's row",
"OPS-05": "proving workload isolation is the fleet's pod row",
"OPS-07": "runbook detection is an operations run",
"OPS-08": "repeat independent operation is a cross-release observation",
"UX-05": "voting keys through pooling is the pool lane's row",
"COM-01": "commercial evidence, no automated harness",
"COM-02": "commercial evidence",
"COM-03": "commercial evidence",
"COM-04": "commercial evidence",
"COM-05": "commercial evidence",
"COM-06": "commercial evidence",
"COM-07": "commercial evidence",
"LEAD-02": "leadership comparison, an observation window",
"LEAD-03": "observation window",
"LEAD-04": "observation window",
"LEAD-05": "observation window",
"LEAD-06": "observation window",
"LEAD-08": "observation window"
}
}

82
tools/ci/test-record.mjs Normal file
View file

@ -0,0 +1,82 @@
#!/usr/bin/env node
// The acceptance layer's recorder (the founder's Test and Acceptance Standard, 8 October 2026; main through the coordinator, 18:2x UK).
// The registry docs/plans/igneum-2.0-test-registry.json holds the cases; tools/ci/test-map.json maps every case whose method includes
// "Automated" to the harness or suite that answers it (the command, the box class, the fixture ids it needs). A run batch writes
// status, run id, evidence path and the pinned manifest sha back into the registry for the cases it covered; a case's accept text is
// never edited (the self-test diffs every accept field before and after); a case with no harness reads NOT RUN with the reason,
// never PASS by inference.
//
// node tools/ci/test-record.mjs --check every Automated case has a map entry or a NOT RUN reason; the map's ids exist
// node tools/ci/test-record.mjs --record <batch.json> batch: {run_id, manifest_sha, evidence_dir, cells:[{cell, status, evidence}]}
// each cell's case ids come from the map; the registry gains status/run/evidence/manifest
// node tools/ci/test-record.mjs --cases <cell> the case ids a matrix cell answers (for the matrix scripts' column)
// node tools/ci/test-record.mjs --self-test
import fs from 'node:fs'; import path from 'node:path';
const args = process.argv.slice(2); const arg = (n) => { const i = args.indexOf(n); return i >= 0 ? args[i + 1] : undefined; };
const ROOT = process.env.TEST_RECORD_ROOT || path.resolve(path.dirname(new URL(import.meta.url).pathname), '..', '..');
const REG = process.env.TEST_REGISTRY || path.join(ROOT, 'docs/plans/igneum-2.0-test-registry.json');
const MAP = process.env.TEST_MAP || path.join(ROOT, 'tools/ci/test-map.json');
const ACCEPT_KEYS = ['accept', 'acceptance', 'accept_text', 'criteria', 'expected']; // whichever the registry uses; all are frozen
const load = (p) => JSON.parse(fs.readFileSync(p, 'utf8'));
const casesOf = (reg) => Array.isArray(reg) ? reg : (reg.cases || reg.tests || (reg.suites || []).flatMap((s) => s.tests || s.cases || []));
const isAutomated = (c) => /automated/i.test(String(c.method ?? c.methods ?? ''));
const idOf = (c) => String(c.id ?? c.case ?? c.case_id);
function check(reg, map) {
const cases = casesOf(reg); const ids = new Set(cases.map(idOf)); const out = []; let bad = 0;
for (const [cell, m] of Object.entries(map.cells || {})) for (const id of m.cases || []) if (!ids.has(id)) { out.push(`map: cell ${cell} names an unknown case ${id}`); bad++; }
const mapped = new Set(Object.values(map.cells || {}).flatMap((m) => m.cases || []));
const notRun = map.not_run || {};
for (const c of cases) {
if (!isAutomated(c)) continue; const id = idOf(c);
if (!mapped.has(id) && !notRun[id]) { out.push(`case ${id} (Automated) has no harness in the map and no NOT RUN reason`); bad++; }
}
for (const [cell, m] of Object.entries(map.cells || {})) { if (!m.command) { out.push(`map: cell ${cell} has no command`); bad++; } if (!m.box_class) { out.push(`map: cell ${cell} has no box class`); bad++; } if (!Array.isArray(m.fixtures)) { out.push(`map: cell ${cell} has no fixtures list (F0 onward, or empty)`); bad++; } }
return { bad, lines: out, automated: cases.filter(isAutomated).length, mapped: [...mapped].length, notRun: Object.keys(notRun).length };
}
function record(reg, map, batch) {
const cases = casesOf(reg); const byId = new Map(cases.map((c) => [idOf(c), c])); const touched = [];
const now = new Date().toISOString();
for (const cell of batch.cells || []) {
const m = (map.cells || {})[cell.cell]; if (!m) throw new Error(`batch names a cell not in the map: ${cell.cell}`);
for (const id of m.cases || []) {
const c = byId.get(id); if (!c) throw new Error(`map names an unknown case ${id}`);
c.run_status = cell.status; c.run_id = batch.run_id; c.evidence_path = cell.evidence || batch.evidence_dir; c.updated = now;
c.evidence_record = { cell: cell.cell, manifest_sha: batch.manifest_sha, coverage: m.coverage || {}, at: now };
touched.push(id);
}
}
for (const [id, reason] of Object.entries(map.not_run || {})) { const c = byId.get(id); if (c && (!c.run_status || c.run_status === 'NOT RUN')) { c.run_status = 'NOT RUN'; c.evidence_record = { reason, at: now }; c.updated = now; } }
return touched;
}
const acceptSnapshot = (reg) => JSON.stringify(casesOf(reg).map((c) => { const o = { id: idOf(c) }; for (const k of ACCEPT_KEYS) if (k in c) o[k] = c[k]; return o; }));
if (args.includes('--self-test')) {
const d = fs.mkdtempSync('/tmp/test-record-'); let fails = 0;
const reg = { cases: [
{ id: 'C1', method: 'Automated', accept: 'one million vectors agree' }, { id: 'C2', method: 'Automated', accept: 'parser refuses ten thousand malformed' },
{ id: 'C3', method: 'Manual review', accept: 'the reviewer signs' }, { id: 'C4', method: 'Automated; manual', accept: 'frozen verify time holds' } ] };
const map = { cells: { 'pow': { command: 'cargo test --release', box_class: 'suite', fixtures: ['F0'], cases: ['C1'] } }, not_run: { C2: 'no malformed-case corpus yet', C4: 'the hostile-load rig is not built' } };
fs.writeFileSync(`${d}/reg.json`, JSON.stringify(reg)); fs.writeFileSync(`${d}/map.json`, JSON.stringify(map));
let r = check(reg, map); if (r.bad !== 0) { console.log(`self-test failed: a complete map was refused: ${r.lines.join('; ')}`); fails = 1; }
const map2 = { cells: { pow: { ...map.cells.pow, cases: ['C1', 'C9'] } }, not_run: {} };
r = check(reg, map2); if (!(r.bad >= 2 && r.lines.some((l) => l.includes('unknown case C9')) && r.lines.some((l) => l.includes('C2') && l.includes('no harness')))) { console.log(`self-test failed: an unknown id or an unmapped Automated case was not refused: ${r.lines.join('; ')}`); fails = 1; }
const before = acceptSnapshot(reg); const touched = record(reg, map, { run_id: 'r1', manifest_sha: 'abc', evidence_dir: '/e', cells: [{ cell: 'pow', status: 'PASS', evidence: '/e/pow.log' }] });
if (acceptSnapshot(reg) !== before) { console.log('self-test failed: a record changed an accept text'); fails = 1; }
if (!(touched.length === 1 && reg.cases[0].run_status === 'PASS' && reg.cases[0].run_id === 'r1' && reg.cases[0].evidence_record.manifest_sha === 'abc' && reg.cases[0].evidence_path === '/e/pow.log' && reg.cases[0].updated)) { console.log(`self-test failed: the run was not written to the mapped case's live fields: ${JSON.stringify(reg.cases[0])}`); fails = 1; }
if (!(reg.cases[1].run_status === 'NOT RUN' && /corpus/.test(reg.cases[1].evidence_record.reason))) { console.log('self-test failed: an unmapped Automated case did not read NOT RUN with its reason'); fails = 1; }
if (reg.cases[2].run_status) { console.log('self-test failed: a manual case was given a run status'); fails = 1; }
const regS = { suites: [{ code: 'X', tests: [{ id: 'X-1', method: 'Automated', accept: 'a' }] }] }; if (casesOf(regS).length !== 1) { console.log('self-test failed: the suites/tests registry shape was not read'); fails = 1; }
let threw = false; try { record(reg, map, { run_id: 'r2', manifest_sha: 'x', cells: [{ cell: 'ghost', status: 'PASS' }] }); } catch { threw = true; }
if (!threw) { console.log('self-test failed: a batch naming a cell not in the map was accepted'); fails = 1; }
fs.rmSync(d, { recursive: true, force: true });
if (!fails) console.log('self-test passed: a complete map checks; an unknown case id and an unmapped Automated case are refused; a run batch writes run_status, run_id, evidence_path, updated and the evidence record to the mapped cases only, leaves every accept text byte-identical, gives an unmapped Automated case NOT RUN with its reason and a manual case nothing; a batch naming an unknown cell is refused');
process.exit(fails);
}
const reg = load(REG); const map = load(MAP);
if (args.includes('--check')) { const r = check(reg, map); for (const l of r.lines) console.error(`test-record: ${l}`); console.log(`test-record: ${r.automated} Automated cases, ${r.mapped} mapped to cells, ${r.notRun} NOT RUN with a reason${r.bad ? `, ${r.bad} problems` : ''}`); process.exit(r.bad ? 1 : 0); }
if (arg('--cases')) { console.log(((map.cells || {})[arg('--cases')]?.cases || []).join(',')); process.exit(0); }
if (arg('--record')) {
const batch = load(arg('--record')); const before = acceptSnapshot(reg); const touched = record(reg, map, batch);
if (acceptSnapshot(reg) !== before) { console.error('test-record: REFUSED: the record would change an accept text'); process.exit(1); }
fs.writeFileSync(REG, JSON.stringify(reg, null, 2) + '\n'); console.log(`test-record: run ${batch.run_id} (manifest ${String(batch.manifest_sha).slice(0, 8)}) written to ${touched.length} cases: ${touched.join(', ')}`); process.exit(0);
}
console.error('usage: test-record.mjs --check | --record <batch.json> | --cases <cell> | --self-test'); process.exit(2);