From 1f47e636aeb05967aaa1a24133637049bd3dbab6 Mon Sep 17 00:00:00 2001 From: igneum-labs <337424239+igneum-labs@users.noreply.github.com> Date: Mon, 5 Oct 2026 00:38:33 +0000 Subject: [PATCH] Harness: port and data-dir overrides, per-load RSS deltas, cache-build counts; bench-log entry for ledger M30 tools/harness: IGNEUM_HARNESS_BASE_PORT and IGNEUM_HARNESS_TMP move the test network's ports and data directory so two agents can run it at once; the u64 sentinel round-trip in overrideParams is fixed with the BigInt reviver from tools/finality-attacks (ledger F25); s6 records rss_start, rss_delta and cache_builds per load and reports per-load growth (the old row subtracted one baseline taken before all three loads, which is how the mempool flood was read as +270 MB); s7 counts "PoW cache built" lines beside every RSS sample; --live-only skips the s7 simulator part. docs/bench-log.md: the 4 October 2026 (night) entry: the floods' growth was one 256 MiB PoW cache per epoch roll (the engine kept a cache per (epoch, day) pair, KEEP 4), measured before and after the fork fix (fork branch fud-memory, 796f758d): submit load +263/+257 MB with 1/1 builds before, +9/+2 MB with 0/0 after; block flood 302 to 1,085 MB with 3/3 builds before, 300 to 315 MB with 0/0 after. Result JSON under docs/benchmarks/memory-floods-2026-10-04/. Co-Authored-By: Claude Fable 5.1 --- docs/bench-log.md | 35 ++ .../after-pass1-s6-exhaustion.json | 413 ++++++++++++++++++ .../after-pass1-s7-flood.json | 163 +++++++ .../before-s6-exhaustion.json | 408 +++++++++++++++++ .../before-s7-flood.json | 163 +++++++ tools/harness/README.md | 5 +- tools/harness/lib/net.mjs | 22 +- tools/harness/run.mjs | 15 +- tools/harness/scenarios/s6-exhaustion.mjs | 16 +- tools/harness/scenarios/s7-flood.mjs | 17 +- 10 files changed, 1234 insertions(+), 23 deletions(-) create mode 100644 docs/benchmarks/memory-floods-2026-10-04/after-pass1-s6-exhaustion.json create mode 100644 docs/benchmarks/memory-floods-2026-10-04/after-pass1-s7-flood.json create mode 100644 docs/benchmarks/memory-floods-2026-10-04/before-s6-exhaustion.json create mode 100644 docs/benchmarks/memory-floods-2026-10-04/before-s7-flood.json diff --git a/docs/bench-log.md b/docs/bench-log.md index ef53c322a..c271bd624 100644 --- a/docs/bench-log.md +++ b/docs/bench-log.md @@ -1095,3 +1095,38 @@ the project lead, 4 October 2026 evening: "we need to fix these serious issues b | split70, v3: 4/2 keys, the 4 side at 70% of weight (shares 0.175 x 4 against 0.15 x 2), 150 s | the 4 side locks during the split, the 2 side does not; 0 conflicts | 4 side: 4 new locks, the first 30 s after the cut; 2 side: 0; heal: all three at 17; 0 conflicting certificates; 0 disagreeing indices | PASS (6B at 70/30; exactly 4/6 is a knife edge under both rules, simulator row above) | **What remains uncertain.** (1) The v3 split's hold was observed over the last 24 s of a 150-s split (the control crossed at 126 s); a longer split under the 240-s expiry (say 200 s) would show more held checkpoints, and the "held by the frozen table" line is logged at debug, which the runs did not enable. (2) No cloud rehearsal: the 12-node Hetzner network was destroyed at 15:30 UTC, so the 95% target is shown on three nodes with emulated 300-ms links and on the cloud logs' arithmetic, not on the cloud topology itself; the rollout plan names the re-creation and the partition experiment to run first. (3) The frozen reference is the node's own highest lock on the chain, not the certificate carried in C_i's past, so two honest nodes can test one checkpoint against tables 30 s apart; in a connected network those tables differ by a minute of blocks, under a partition both are pre-split, and no run showed a disagreement, but it is a property argued, not proved. (4) The price: a sudden departure of a third or more now pauses finality for a full window (30 days on mainnet) instead of 1.4 to 10 days; a gradual one costs nothing. the project lead asked for the pause over the fork; the number is stated in spec 3.7 item 2. (5) The fold clock is in memory: a restarted node folds from `daa(C_i) + depth`, a few seconds late at worst. (6) Binaries, all from `finality-fixes` 6aa69a45, hashes and checks in the rollout plan's section 2: Mac native `fe982a1d...` (verified running), Linux `7c100fc2...` (cargo-zigbuild, 34 min, not run on a Linux host), Windows `cc1d1001...` (mingw, 12 min 28 s, the v2 exe's DLL set, cannot run here); the Windows payload inputs were staged with `push-inputs.sh --no-deploy` into a scratch folder and NOT deployed (plan 7a). + +## 4 October 2026 (night), ledger M30: the block and transaction floods grew the node by 256 MiB per epoch roll, fixed by sharing the PoW cache across the epochs of a day (memory engineer) + +Machine: Apple M5 Max, 64 GB, load averages 126 to 146 for the whole session (ten or more agents building and running at once). Every count here (blocks accepted, cache builds, RSS before and after) is valid under that load; every latency is an upper bound and is not a number. Private test network of two igneumd on 127.0.0.1 ports 29500+ (node A on 29500/29501/29502, node B on 29510/29511/29512), data under `/tmp/igneum-fud-mem/{baseline,after,after2}`, the 60x fast-time profile (`infra/fast-time/override-60x.json`, `skip_proof_of_work` on, `pow_epoch_blocks` 60, `pow_day_ms` 1,440,000) exactly as the red team ran it. The live devnet and other agents' ports were not touched. Every run went through `tools/lock/with-lock.sh run`, every build and the unit tests through `with-lock.sh build`. + +What the red team saw (`docs/review/redteam-2026-10-04.md` rows 5 and 8, ledger M30): on the 0.3.4 build the s6 submit flood grew RSS by +269 MB, the mempool flood by +270 MB, and the s7 block flood took both nodes from 302 to 1,082 MB. The s6 figures were cumulative from one baseline taken before all three loads (`rss_peak - rss_baseline` in `s6-exhaustion.mjs`), so the mempool flood's "+270 MB" was the submit flood's growth carried forward; its own cost was 1 MB. The red team's guess (execution-layer records, rejected transactions retained) did not hold: the mempool flood retains nothing measurable. + +Cause, measured: every RSS step is one `PoW cache built` line in the node log (`consensus/src/pipeline/header_processor/pre_ghostdag_validation.rs:136`), 256 MiB each. The lottery engine (`consensus/pow/src/igneum.rs`, `IgneumEngine::epoch_for_impl`) keyed its resident entries by `(epoch seed, day)` and built a full `igneum_pow::Epoch` (program plus the 256 MiB ChaCha12 cache) per entry, keeping `KEEP = 4` of them, although the cache is a function of the day seed alone (`igneum_pow::Epoch::from_seed_bytes`: `seed_words_from_bytes(day_bytes)`; the epoch seed only feeds the program). On the 60x profile an epoch is 60 DAA, so the honest chain rolls an epoch every minute and the 50x block flood every 10 to 20 s; each roll cost a cache build and 256 MiB until the fourth entry, then evictions. The engine runs under `skip_proof_of_work` too (`check_pow_and_calc_block_level` always calls it and forces the pass afterwards), so the harness exercised it. The 3 October harness run (this file, "2026-10-03, consensus attack harness") was on the plain devnet profile (3,600-DAA epochs), where no roll falls inside a 60 s flood, which is why it grew 11 to 33 MB; the profile change and the build change were conflated in M30. On the live devnet the same rule means 256 MiB per hour until 1 GiB resident, and a 50x fast miner reaches that in minutes. + +Fix (node fork branch `fud-memory`, from `finality-fixes` 6aa69a45): the engine now holds the 256 MiB caches keyed by day (`IgneumEngine::KEEP_DAYS = 3`: the chain's current day, the next day, one slot for a late block of the previous day or an off-day header; the live pair is never evicted while another day's cache exists; bound 3 x 256 MiB = 768 MiB resident plus `MAX_INFLIGHT_BUILDS = 2` x 256 MiB while builds run) and the programs keyed by `(epoch seed, day)` (`IgneumEngine::KEEP = 8`, a few KB each, LRU). An epoch roll on the same day generates a program and builds no cache; a `BuildReport` (the p2p off-day strike and the "PoW cache built" line) is produced only for a cache build. `EpochRef { program, dataset: Arc }` replaces `Arc` for the node and the miner and computes the identical hash (`interpret_warp_init` on `block_init_words`, as `Epoch::pow_bound` does); the unit test `epoch_rolls_share_the_day_cache` checks the engine's pow against a standalone `igneum_pow::Epoch` for two epochs of one day. Programs whose day cache was evicted are dropped with it, so no entry pins a cache. No consensus rule changed: the hash, the seeds and the acceptance are as before. Miner cost: the GPU prepare path (`--prepare-packs`) exports a pack per epoch roll from a standalone epoch (its own transient fill), because `igneum_pow::emit::export_pack` wants the crate's own `Epoch` and `DatasetSource` is not shareable without a change to the `igneum-pow` crate. + +Commands (run from `igneum-wt-fud-memory`; the "before" binary is the shipping `vendor/igneum-node/target-finality/release/igneumd`, the "after" binary is `vendor/igneum-node-fud-mem/target/release/igneumd` from the commit below): + +``` +IGNEUMD= IGNEUM_HARNESS_BASE_PORT=29500 IGNEUM_HARNESS_TMP=/tmp/igneum-fud-mem/ \ + tools/lock/with-lock.sh run node tools/harness/run.mjs s6 s7 --quick --fast-time --live-only --no-bench-log +cd vendor/igneum-node-fud-mem && tools/lock/with-lock.sh build nice -n 19 cargo build --release -j 4 -p kaspad -p igneum-miner --features kaspad/igneum-pow +cd vendor/igneum-node-fud-mem && tools/lock/with-lock.sh build nice -n 19 cargo test --release -j 4 -p kaspa-pow --features kaspa-pow/igneum-pow -p igneum-exec +``` + +RSS per load, node A / node B, MB (start of the load to its peak; "builds" = `PoW cache built` lines during the load). s6 loads are 30 s each in `--quick`; s7 is the vmine miner at 50x for 60 s. + +| Load | Before: start to peak A / B | Before: builds A / B | After (pass 1, build without the insert guard): start to peak A / B | After: builds A / B | +|---|---|---|---|---| +| s6 warm-up baseline (RSS before any load) | 303 / 304 | 1 / 1 (startup) | 304 / 304 | 1 / 1 (startup) | +| s6 template flood, 500/s, 14,995 and 14,993 sent, all answered | 304 to 309 / 304 to 309 (+5 / +5) | 0 / 0 | 304 to 310 / 304 to 308 (+6 / +4) | 0 / 0 | +| s6 submit flood, 50/s, 1,499 known-block submits, all answered; the 60-DAA epoch rolled during it | 309 to 572 / 309 to 566 (+263 / +257) | 1 / 1 | 310 to 319 / 308 to 310 (+9 / +2) | 0 / 0 | +| s6 mempool flood, 500/s, 14,993 and 14,999 unknown-outpoint transactions, all rejected | 572 to 573 / 566 to 567 (+1 / +1) | 0 / 0 | 319 to 320 / 310 to 310 (+1 / +0) | 0 / 0 | +| s7 block flood, vmine at 50x for 60 s: 198 and 203 blocks accepted (3.3 and 3.4 per s under load), epochs 0 to 3 rolled | 302 to 1,085 / 303 to 1,083 (+783 / +780); steps at 10 s 569, 20 s 827, 40 s 1,084 | 3 / 3 | 300 to 315 / 302 to 315 (+15 / +13) | 0 / 0 | + +Honest template p95 (upper bounds, load over 100): before 130.8 / 86.7 / 104.7 ms per s6 load against a baseline of 84.7, s7 91.5 against 92.8; after 101.7 / 66.1 / 84.4 against 89.5, s7 75.4 against 69.4. Both nodes alive and on one sink at the end of every run except the before-run s6 (sinks differed at the instant of the check, 120 against 121 blocks, under load 136; the after runs agreed). The red team's harness criterion (baseline + 512 MB) still passes before and after; the 50 MB-per-load target holds after. + +Harness changes (this repo): `IGNEUM_HARNESS_BASE_PORT` and `IGNEUM_HARNESS_TMP` (ports and data directory, so two agents can run the harness at once), the u64 sentinel round-trip fixed with the BigInt reviver from `tools/finality-attacks/lib/net.mjs` (ledger F25), s6 records `rss_start`, `rss_delta` and `cache_builds` per load and its row reports per-load growth, s7 records `cache_builds` beside every RSS sample, and `--live-only` skips the s7 simulator part. Result JSON: `docs/benchmarks/memory-floods-2026-10-04/{before,after-pass1}-{s6-exhaustion,s7-flood}.json`. Pass 1 ran the build of the engine change before its last three-line guard (`insert_program` skips a program whose day cache was evicted during its generation; the guard cannot fire in these single-day runs); the committed fork build (796f758d) is the one the unit tests ran on. + +Not covered tonight: the execution layer's `ExecState.records` (`igneum/exec/src/service.rs:31`, pushed per chain block, never truncated) is a slow growth, not a flood effect: 197 chain blocks cost under 1 MB in these runs, and a record on an empty devnet is roughly 1 to 2 KB (approximate, from the struct), so about 100 to 170 MB per day at 1 block/s; bounding it needs a window at least as long as the proving sortition window (`proving.rs:200`) plus the RPC's by-number history, which is a design choice, not a cache. The snapshot ring (`SNAPSHOT_RING = 64` full `IgneumDb` clones) is bounded in count but scales with the state size. The finality store trims votes, checkpoints, certificates and locks every index (`processes/finality.rs:531`); its `keys` and `stripped` maps grow with distinct vote keys (about 150 bytes per key, approximate). The proof pool keeps `RECORD_WINDOW_CHAIN_BLOCKS` of entries. None of these moved in these floods. diff --git a/docs/benchmarks/memory-floods-2026-10-04/after-pass1-s6-exhaustion.json b/docs/benchmarks/memory-floods-2026-10-04/after-pass1-s6-exhaustion.json new file mode 100644 index 000000000..7819733dd --- /dev/null +++ b/docs/benchmarks/memory-floods-2026-10-04/after-pass1-s6-exhaustion.json @@ -0,0 +1,413 @@ +{ + "rows": [ + { + "scenario": "6 resource exhaustion (50x template, submit and mempool floods from one peer)", + "criterion": "honest template p95 < 200 ms and both nodes under baseline RSS + 512 MB, alive, one sink", + "result": "honest template p95 worst 101.7 ms across loads (baseline 89.5 ms); template 500ps 500/s, submit 50ps 50/s, mempool 500ps 500/s; RSS growth per load template +6MB (0/0 cache builds), submit +9MB (0/0 cache builds), mempool +1MB (0/0 cache builds), cumulative +16MB over baseline 304/304; alive true; same sink true", + "pass": true + } + ], + "data": { + "baseline_template_ms": { + "a": { + "n": 75, + "p50": 6.5, + "p95": 89.6, + "p99": 96.4, + "max": 96.4, + "mean": 26.8 + }, + "b": { + "n": 76, + "p50": 10.1, + "p95": 89.5, + "p99": 107.5, + "max": 107.5, + "mean": 25.2 + } + }, + "rss_baseline": { + "a": 304, + "b": 304 + }, + "loads": { + "template_flood_500ps": { + "requests_sent": 14993, + "accepted": 14993, + "rejected_or_error": 0, + "rate_per_s": 500, + "request_latency_ms": { + "n": 14993, + "p50": 23.9, + "p95": 168.2, + "p99": 670.8, + "max": 807.1, + "mean": 54.4 + }, + "honest_template_ms": { + "n": 170, + "p50": 3.6, + "p95": 101.7, + "p99": 698.7, + "max": 700.7, + "mean": 44.8 + }, + "attacked_node_template_ms": { + "n": 167, + "p50": 4.3, + "p95": 93.7, + "p99": 698.4, + "max": 701, + "mean": 43.5 + }, + "rss_series": [ + { + "t_s": 2.673, + "a": 305, + "b": 304 + }, + { + "t_s": 4.69, + "a": 306, + "b": 305 + }, + { + "t_s": 6.706, + "a": 306, + "b": 305 + }, + { + "t_s": 8.707, + "a": 306, + "b": 305 + }, + { + "t_s": 10.71, + "a": 306, + "b": 306 + }, + { + "t_s": 12.723, + "a": 307, + "b": 306 + }, + { + "t_s": 14.727, + "a": 307, + "b": 306 + }, + { + "t_s": 16.728, + "a": 308, + "b": 306 + }, + { + "t_s": 18.748, + "a": 308, + "b": 307 + }, + { + "t_s": 20.749, + "a": 308, + "b": 307 + }, + { + "t_s": 22.752, + "a": 308, + "b": 307 + }, + { + "t_s": 24.753, + "a": 308, + "b": 307 + }, + { + "t_s": 26.754, + "a": 309, + "b": 307 + }, + { + "t_s": 28.754, + "a": 310, + "b": 308 + } + ], + "rss_start": { + "a": 304, + "b": 304 + }, + "rss_peak": { + "a": 310, + "b": 308 + }, + "both_alive": true, + "rss_delta": { + "a": 6, + "b": 4 + }, + "cache_builds": { + "a": 0, + "b": 0 + } + }, + "submit_flood_50ps": { + "requests_sent": 1499, + "accepted": 1499, + "rejected_or_error": 0, + "rate_per_s": 50, + "request_latency_ms": { + "n": 1499, + "p50": 387.7, + "p95": 1435.5, + "p99": 1887.7, + "max": 2166.4, + "mean": 481.3 + }, + "honest_template_ms": { + "n": 175, + "p50": 5, + "p95": 66.1, + "p99": 91.4, + "max": 94.1, + "mean": 19.9 + }, + "attacked_node_template_ms": { + "n": 174, + "p50": 6.2, + "p95": 67.4, + "p99": 90.2, + "max": 91.5, + "mean": 21.3 + }, + "rss_series": [ + { + "t_s": 2.49, + "a": 312, + "b": 309 + }, + { + "t_s": 4.49, + "a": 312, + "b": 309 + }, + { + "t_s": 6.491, + "a": 314, + "b": 309 + }, + { + "t_s": 8.493, + "a": 315, + "b": 309 + }, + { + "t_s": 10.497, + "a": 315, + "b": 309 + }, + { + "t_s": 12.505, + "a": 316, + "b": 309 + }, + { + "t_s": 14.513, + "a": 316, + "b": 309 + }, + { + "t_s": 16.524, + "a": 317, + "b": 309 + }, + { + "t_s": 18.525, + "a": 318, + "b": 309 + }, + { + "t_s": 20.525, + "a": 318, + "b": 309 + }, + { + "t_s": 22.525, + "a": 318, + "b": 309 + }, + { + "t_s": 24.544, + "a": 318, + "b": 309 + }, + { + "t_s": 26.544, + "a": 318, + "b": 310 + }, + { + "t_s": 28.544, + "a": 319, + "b": 310 + }, + { + "t_s": 30.545, + "a": 319, + "b": 310 + } + ], + "rss_start": { + "a": 310, + "b": 308 + }, + "rss_peak": { + "a": 319, + "b": 310 + }, + "both_alive": true, + "rss_delta": { + "a": 9, + "b": 2 + }, + "cache_builds": { + "a": 0, + "b": 0 + } + }, + "mempool_flood_500ps": { + "requests_sent": 14999, + "accepted": 0, + "rejected_or_error": 14999, + "rate_per_s": 500, + "request_latency_ms": { + "n": 14999, + "p50": 6.9, + "p95": 30.6, + "p99": 101.6, + "max": 823, + "mean": 13.2 + }, + "honest_template_ms": { + "n": 171, + "p50": 6.2, + "p95": 84.4, + "p99": 557, + "max": 665.4, + "mean": 32 + }, + "attacked_node_template_ms": { + "n": 171, + "p50": 7.4, + "p95": 74.2, + "p99": 103, + "max": 488.7, + "mean": 25.2 + }, + "rss_series": [ + { + "t_s": 2.649, + "a": 319, + "b": 310 + }, + { + "t_s": 4.65, + "a": 319, + "b": 310 + }, + { + "t_s": 6.65, + "a": 319, + "b": 310 + }, + { + "t_s": 8.649, + "a": 319, + "b": 310 + }, + { + "t_s": 10.65, + "a": 319, + "b": 310 + }, + { + "t_s": 12.662, + "a": 319, + "b": 310 + }, + { + "t_s": 14.662, + "a": 319, + "b": 310 + }, + { + "t_s": 16.662, + "a": 320, + "b": 310 + }, + { + "t_s": 18.663, + "a": 320, + "b": 310 + }, + { + "t_s": 20.736, + "a": 320, + "b": 310 + }, + { + "t_s": 22.748, + "a": 320, + "b": 310 + }, + { + "t_s": 24.749, + "a": 320, + "b": 310 + }, + { + "t_s": 26.75, + "a": 320, + "b": 310 + }, + { + "t_s": 28.749, + "a": 320, + "b": 310 + } + ], + "rss_start": { + "a": 319, + "b": 310 + }, + "rss_peak": { + "a": 320, + "b": 310 + }, + "both_alive": true, + "rss_delta": { + "a": 1, + "b": 0 + }, + "cache_builds": { + "a": 0, + "b": 0 + } + } + }, + "recovery_template_ms": { + "n": 210, + "p50": 6.2, + "p95": 82.5, + "p99": 488.6, + "max": 665.4, + "mean": 29.9 + }, + "final": { + "blocks_a": 121, + "blocks_b": 121, + "same_sink": true + }, + "alive": true, + "mem_bound_mb": 512 + } +} \ No newline at end of file diff --git a/docs/benchmarks/memory-floods-2026-10-04/after-pass1-s7-flood.json b/docs/benchmarks/memory-floods-2026-10-04/after-pass1-s7-flood.json new file mode 100644 index 000000000..807740080 --- /dev/null +++ b/docs/benchmarks/memory-floods-2026-10-04/after-pass1-s7-flood.json @@ -0,0 +1,163 @@ +{ + "rows": [ + { + "scenario": "7 fast-miner flood, controller trajectory (sim)", + "criterion": "trajectory recorded", + "result": "skipped (--live-only)", + "pass": null + }, + { + "scenario": "7 fast-miner flood, live (50 blocks/s from one peer)", + "criterion": "node stays responsive: honest template p95 < 200 ms, both nodes alive, same sink", + "result": "flood accepted 203 blocks in 60 s (3.4/s); honest template p50/p95/max 12.4/75.4/895.1 ms under flood (baseline 7/69.4/107.3); rss a 300->315 MB, b 302->315 MB (cache builds 0/0); alive true; same sink true", + "pass": true + } + ], + "data": { + "live": { + "baseline_template_ms": { + "n": 79, + "p50": 7, + "p95": 69.4, + "p99": 107.3, + "max": 107.3, + "mean": 24.1 + }, + "under_flood_template_ms": { + "n": 465, + "p50": 12.4, + "p95": 75.4, + "p99": 123.8, + "max": 895.1, + "mean": 27 + }, + "after_flood_template_ms": { + "n": 39, + "p50": 1.7, + "p95": 88.3, + "p99": 100.2, + "max": 100.2, + "mean": 23.1 + }, + "samples": [ + { + "t_s": 10, + "blocks_a": 110, + "blocks_b": 110, + "difficulty_a": 616132003.4293169, + "sink_same": true, + "rss_a": 312, + "rss_b": 311, + "cache_builds_a": 0, + "cache_builds_b": 0, + "flood_accepted": 100, + "flood_rejected": 0, + "honest_accepted": 10 + }, + { + "t_s": 20, + "blocks_a": 151, + "blocks_b": 150, + "difficulty_a": 1634372941.381798, + "sink_same": false, + "rss_a": 313, + "rss_b": 313, + "cache_builds_a": 0, + "cache_builds_b": 0, + "flood_accepted": 139, + "flood_rejected": 0, + "honest_accepted": 12 + }, + { + "t_s": 30, + "blocks_a": 176, + "blocks_b": 176, + "difficulty_a": 3422770765.9742208, + "sink_same": true, + "rss_a": 314, + "rss_b": 313, + "cache_builds_a": 0, + "cache_builds_b": 0, + "flood_accepted": 164, + "flood_rejected": 0, + "honest_accepted": 12 + }, + { + "t_s": 40, + "blocks_a": 197, + "blocks_b": 196, + "difficulty_a": 5026862033.766903, + "sink_same": false, + "rss_a": 314, + "rss_b": 314, + "cache_builds_a": 0, + "cache_builds_b": 0, + "flood_accepted": 184, + "flood_rejected": 0, + "honest_accepted": 13 + }, + { + "t_s": 50, + "blocks_a": 207, + "blocks_b": 207, + "difficulty_a": 6515322825.578815, + "sink_same": true, + "rss_a": 315, + "rss_b": 314, + "cache_builds_a": 0, + "cache_builds_b": 0, + "flood_accepted": 194, + "flood_rejected": 0, + "honest_accepted": 13 + }, + { + "t_s": 60, + "blocks_a": 216, + "blocks_b": 216, + "difficulty_a": 6949902898.525323, + "sink_same": true, + "rss_a": 315, + "rss_b": 315, + "cache_builds_a": 0, + "cache_builds_b": 0, + "flood_accepted": 203, + "flood_rejected": 0, + "honest_accepted": 13 + } + ], + "rss_before": { + "a": 300, + "b": 302 + }, + "rss_peak": { + "a": 315, + "b": 315 + }, + "cache_builds": { + "a": 0, + "b": 0, + "before_flood": { + "a": 1, + "b": 1 + } + }, + "flood": { + "accepted": 203, + "rejected": 0, + "errors": 0 + }, + "honest": { + "accepted": 13, + "rejected": 0 + }, + "final": { + "blocks_a": 216, + "blocks_b": 216, + "same_sink": true, + "difficulty_a": 6949902898.525323, + "ratio": null + }, + "alive": true + } + } +} \ No newline at end of file diff --git a/docs/benchmarks/memory-floods-2026-10-04/before-s6-exhaustion.json b/docs/benchmarks/memory-floods-2026-10-04/before-s6-exhaustion.json new file mode 100644 index 000000000..5e4e88133 --- /dev/null +++ b/docs/benchmarks/memory-floods-2026-10-04/before-s6-exhaustion.json @@ -0,0 +1,408 @@ +{ + "rows": [ + { + "scenario": "6 resource exhaustion (50x template, submit and mempool floods from one peer)", + "criterion": "honest template p95 < 200 ms and both nodes under baseline RSS + 512 MB, alive, one sink", + "result": "honest template p95 worst 130.8 ms across loads (baseline 84.7 ms); template 500ps 500/s, submit 50ps 50/s, mempool 500ps 500/s; RSS growth per load template +5MB (0/0 cache builds), submit +263MB (1/1 cache builds), mempool +1MB (0/0 cache builds), cumulative +270MB over baseline 303/304; alive true; same sink false", + "pass": false + } + ], + "data": { + "baseline_template_ms": { + "a": { + "n": 77, + "p50": 3.8, + "p95": 81.5, + "p99": 84.5, + "max": 84.5, + "mean": 25.6 + }, + "b": { + "n": 76, + "p50": 5.5, + "p95": 84.7, + "p99": 104.2, + "max": 104.2, + "mean": 27.1 + } + }, + "rss_baseline": { + "a": 303, + "b": 304 + }, + "loads": { + "template_flood_500ps": { + "requests_sent": 14995, + "accepted": 14995, + "rejected_or_error": 0, + "rate_per_s": 500, + "request_latency_ms": { + "n": 14995, + "p50": 36.7, + "p95": 439.3, + "p99": 1215.9, + "max": 1501.2, + "mean": 109.1 + }, + "honest_template_ms": { + "n": 159, + "p50": 11, + "p95": 130.8, + "p99": 754, + "max": 858.2, + "mean": 47 + }, + "attacked_node_template_ms": { + "n": 160, + "p50": 15.7, + "p95": 130.3, + "p99": 760.6, + "max": 858.1, + "mean": 50.2 + }, + "rss_series": [ + { + "t_s": 2.695, + "a": 305, + "b": 306 + }, + { + "t_s": 4.705, + "a": 306, + "b": 306 + }, + { + "t_s": 6.706, + "a": 307, + "b": 307 + }, + { + "t_s": 8.706, + "a": 307, + "b": 307 + }, + { + "t_s": 10.706, + "a": 307, + "b": 307 + }, + { + "t_s": 12.708, + "a": 307, + "b": 307 + }, + { + "t_s": 14.714, + "a": 308, + "b": 308 + }, + { + "t_s": 16.723, + "a": 308, + "b": 308 + }, + { + "t_s": 18.724, + "a": 308, + "b": 308 + }, + { + "t_s": 20.735, + "a": 309, + "b": 308 + }, + { + "t_s": 22.734, + "a": 309, + "b": 308 + }, + { + "t_s": 24.736, + "a": 309, + "b": 308 + }, + { + "t_s": 26.736, + "a": 309, + "b": 308 + }, + { + "t_s": 28.736, + "a": 309, + "b": 309 + } + ], + "rss_start": { + "a": 304, + "b": 304 + }, + "rss_peak": { + "a": 309, + "b": 309 + }, + "both_alive": true, + "rss_delta": { + "a": 5, + "b": 5 + }, + "cache_builds": { + "a": 0, + "b": 0 + } + }, + "submit_flood_50ps": { + "requests_sent": 1499, + "accepted": 1499, + "rejected_or_error": 0, + "rate_per_s": 50, + "request_latency_ms": { + "n": 1499, + "p50": 401.9, + "p95": 1120.6, + "p99": 1405.9, + "max": 1610.1, + "mean": 453.6 + }, + "honest_template_ms": { + "n": 166, + "p50": 5.9, + "p95": 86.7, + "p99": 600.8, + "max": 689.4, + "mean": 36.7 + }, + "attacked_node_template_ms": { + "n": 166, + "p50": 6.6, + "p95": 84.9, + "p99": 689.3, + "max": 743.1, + "mean": 38.1 + }, + "rss_series": [ + { + "t_s": 2.743, + "a": 310, + "b": 309 + }, + { + "t_s": 4.752, + "a": 311, + "b": 309 + }, + { + "t_s": 6.802, + "a": 311, + "b": 565 + }, + { + "t_s": 8.804, + "a": 311, + "b": 565 + }, + { + "t_s": 10.804, + "a": 568, + "b": 565 + }, + { + "t_s": 12.805, + "a": 568, + "b": 565 + }, + { + "t_s": 14.812, + "a": 569, + "b": 566 + }, + { + "t_s": 16.817, + "a": 569, + "b": 566 + }, + { + "t_s": 18.823, + "a": 569, + "b": 566 + }, + { + "t_s": 20.823, + "a": 570, + "b": 566 + }, + { + "t_s": 22.829, + "a": 571, + "b": 566 + }, + { + "t_s": 24.829, + "a": 571, + "b": 566 + }, + { + "t_s": 26.829, + "a": 571, + "b": 566 + }, + { + "t_s": 28.845, + "a": 572, + "b": 566 + } + ], + "rss_start": { + "a": 309, + "b": 309 + }, + "rss_peak": { + "a": 572, + "b": 566 + }, + "both_alive": true, + "rss_delta": { + "a": 263, + "b": 257 + }, + "cache_builds": { + "a": 1, + "b": 1 + } + }, + "mempool_flood_500ps": { + "requests_sent": 14993, + "accepted": 0, + "rejected_or_error": 14993, + "rate_per_s": 500, + "request_latency_ms": { + "n": 14993, + "p50": 7.1, + "p95": 49.9, + "p99": 126.6, + "max": 815.9, + "mean": 15.8 + }, + "honest_template_ms": { + "n": 164, + "p50": 7.9, + "p95": 104.7, + "p99": 735.5, + "max": 815.8, + "mean": 44.5 + }, + "attacked_node_template_ms": { + "n": 166, + "p50": 6.8, + "p95": 90.8, + "p99": 745.9, + "max": 815.9, + "mean": 37.3 + }, + "rss_series": [ + { + "t_s": 2.533, + "a": 572, + "b": 566 + }, + { + "t_s": 4.534, + "a": 572, + "b": 566 + }, + { + "t_s": 6.537, + "a": 572, + "b": 566 + }, + { + "t_s": 8.546, + "a": 572, + "b": 566 + }, + { + "t_s": 10.546, + "a": 572, + "b": 566 + }, + { + "t_s": 12.547, + "a": 572, + "b": 566 + }, + { + "t_s": 14.548, + "a": 572, + "b": 566 + }, + { + "t_s": 16.549, + "a": 573, + "b": 566 + }, + { + "t_s": 18.548, + "a": 573, + "b": 566 + }, + { + "t_s": 20.559, + "a": 573, + "b": 567 + }, + { + "t_s": 22.57, + "a": 573, + "b": 567 + }, + { + "t_s": 24.57, + "a": 573, + "b": 567 + }, + { + "t_s": 26.57, + "a": 573, + "b": 567 + }, + { + "t_s": 28.57, + "a": 573, + "b": 567 + } + ], + "rss_start": { + "a": 572, + "b": 566 + }, + "rss_peak": { + "a": 573, + "b": 567 + }, + "both_alive": true, + "rss_delta": { + "a": 1, + "b": 1 + }, + "cache_builds": { + "a": 0, + "b": 0 + } + } + }, + "recovery_template_ms": { + "n": 203, + "p50": 7.9, + "p95": 96.1, + "p99": 731.6, + "max": 815.8, + "mean": 40.4 + }, + "final": { + "blocks_a": 120, + "blocks_b": 121, + "same_sink": false + }, + "alive": true, + "mem_bound_mb": 512 + } +} \ No newline at end of file diff --git a/docs/benchmarks/memory-floods-2026-10-04/before-s7-flood.json b/docs/benchmarks/memory-floods-2026-10-04/before-s7-flood.json new file mode 100644 index 000000000..301e5dacd --- /dev/null +++ b/docs/benchmarks/memory-floods-2026-10-04/before-s7-flood.json @@ -0,0 +1,163 @@ +{ + "rows": [ + { + "scenario": "7 fast-miner flood, controller trajectory (sim)", + "criterion": "trajectory recorded", + "result": "skipped (--live-only)", + "pass": null + }, + { + "scenario": "7 fast-miner flood, live (50 blocks/s from one peer)", + "criterion": "node stays responsive: honest template p95 < 200 ms, both nodes alive, same sink", + "result": "flood accepted 198 blocks in 60 s (3.3/s); honest template p50/p95/max 7.7/91.5/1454.3 ms under flood (baseline 23.5/92.8/188.8); rss a 302->1085 MB, b 303->1083 MB (cache builds 3/3); alive true; same sink true", + "pass": true + } + ], + "data": { + "live": { + "baseline_template_ms": { + "n": 76, + "p50": 23.5, + "p95": 92.8, + "p99": 188.8, + "max": 188.8, + "mean": 27.7 + }, + "under_flood_template_ms": { + "n": 458, + "p50": 7.7, + "p95": 91.5, + "p99": 116.7, + "max": 1454.3, + "mean": 31.9 + }, + "after_flood_template_ms": { + "n": 39, + "p50": 7.9, + "p95": 87.6, + "p99": 91.2, + "max": 91.2, + "mean": 24.8 + }, + "samples": [ + { + "t_s": 10, + "blocks_a": 94, + "blocks_b": 93, + "difficulty_a": 457438921.53559726, + "sink_same": false, + "rss_a": 569, + "rss_b": 567, + "cache_builds_a": 1, + "cache_builds_b": 1, + "flood_accepted": 84, + "flood_rejected": 0, + "honest_accepted": 10 + }, + { + "t_s": 20, + "blocks_a": 137, + "blocks_b": 137, + "difficulty_a": 1287295920.1241035, + "sink_same": true, + "rss_a": 827, + "rss_b": 825, + "cache_builds_a": 2, + "cache_builds_b": 2, + "flood_accepted": 126, + "flood_rejected": 0, + "honest_accepted": 11 + }, + { + "t_s": 30, + "blocks_a": 167, + "blocks_b": 167, + "difficulty_a": 3125416130.4758606, + "sink_same": true, + "rss_a": 828, + "rss_b": 826, + "cache_builds_a": 2, + "cache_builds_b": 2, + "flood_accepted": 155, + "flood_rejected": 0, + "honest_accepted": 12 + }, + { + "t_s": 40, + "blocks_a": 183, + "blocks_b": 180, + "difficulty_a": 4327152934.829311, + "sink_same": false, + "rss_a": 1084, + "rss_b": 1082, + "cache_builds_a": 3, + "cache_builds_b": 3, + "flood_accepted": 169, + "flood_rejected": 0, + "honest_accepted": 14 + }, + { + "t_s": 50, + "blocks_a": 196, + "blocks_b": 196, + "difficulty_a": 4435640076.843163, + "sink_same": true, + "rss_a": 1084, + "rss_b": 1083, + "cache_builds_a": 3, + "cache_builds_b": 3, + "flood_accepted": 181, + "flood_rejected": 0, + "honest_accepted": 15 + }, + { + "t_s": 60, + "blocks_a": 213, + "blocks_b": 213, + "difficulty_a": 6255943304.461032, + "sink_same": true, + "rss_a": 1085, + "rss_b": 1083, + "cache_builds_a": 3, + "cache_builds_b": 3, + "flood_accepted": 198, + "flood_rejected": 0, + "honest_accepted": 15 + } + ], + "rss_before": { + "a": 302, + "b": 303 + }, + "rss_peak": { + "a": 1085, + "b": 1083 + }, + "cache_builds": { + "a": 3, + "b": 3, + "before_flood": { + "a": 1, + "b": 1 + } + }, + "flood": { + "accepted": 198, + "rejected": 0, + "errors": 0 + }, + "honest": { + "accepted": 16, + "rejected": 0 + }, + "final": { + "blocks_a": 214, + "blocks_b": 214, + "same_sink": true, + "difficulty_a": 5868605790.480026, + "ratio": null + }, + "alive": true + } + } +} \ No newline at end of file diff --git a/tools/harness/README.md b/tools/harness/README.md index b492a6d5d..02240e546 100644 --- a/tools/harness/README.md +++ b/tools/harness/README.md @@ -22,7 +22,9 @@ the harness exercises the ordering layer, not a weakened copy of it. The test network uses `127.0.0.1` ports 27200 and up and data under `/tmp/igneum-harness`. It never touches the live devnet (gRPC 26610, P2P 26611, observer 26640/26641/28640), the PC node at 192.168.68.67, or any port other -agents use (up to 27199). Loopback peers are never gossiped (`components/addressmanager/src/lib.rs`), so no link +agents use (up to 27199). `IGNEUM_HARNESS_BASE_PORT=29500 IGNEUM_HARNESS_TMP=/tmp/my-harness` moves the ports (node +`i` takes base + 10i, proxies base + 900 + i) and the data and results directories, so two agents can run the +harness at once (4 October 2026). Loopback peers are never gossiped (`components/addressmanager/src/lib.rs`), so no link forms that a scenario did not ask for. Everything the harness starts is stopped at the end, including on SIGINT. ## Build @@ -48,6 +50,7 @@ node tools/harness/run.mjs # the full catalogue, priority order 5 node tools/harness/run.mjs s5 s2 --quick # named scenarios, short durations node tools/harness/run.mjs --no-bench-log # do not append to docs/bench-log.md node tools/harness/scenarios/s1-withhold.mjs --quick # one scenario on its own +node tools/harness/run.mjs s7 --quick --live-only # s7 Part B only (the vmine flood against real nodes; no simulator binary needed) node tools/harness/run.mjs s3 s4 --fast-time # the 60x fast-time profile (infra/fast-time): merge depth 60 s, so the # partition and eclipse cuts are 10, 30 and 62 s instead of 600, 1,800 and # 3,700; nodes and the simulator come from vendor/igneum-node/target-integration diff --git a/tools/harness/lib/net.mjs b/tools/harness/lib/net.mjs index f0317bc39..e905793a9 100644 --- a/tools/harness/lib/net.mjs +++ b/tools/harness/lib/net.mjs @@ -1,5 +1,7 @@ // Private test network of igneumd processes on 127.0.0.1, ports 27200 and up, data under /tmp/igneum-harness. // Nothing here touches the live devnet (26610/26611, 26640/26641, 28640) or any port below 27200. +// IGNEUM_HARNESS_BASE_PORT and IGNEUM_HARNESS_TMP move the ports and the data directory, so two agents can run the +// harness at the same time (4 October 2026: 29500+ and a private directory for the memory-flood re-run). // // Topology is explicit: a node with `connect: [...]` dials only those addresses and accepts no inbound // connections (kaspad/src/daemon.rs: connect_peers sets outbound target and inbound limit to 0); a node without @@ -24,8 +26,16 @@ export const TARGET = process.env.IGNEUM_HARNESS_TARGET || (FAST_TIME ? `${ROOT} export const IGNEUMD = process.env.IGNEUMD || `${TARGET}/igneumd`; export const PROBE = process.env.IGNEUM_P2P_PROBE || `${TARGET}/igneum-p2p-probe`; export const SIM = process.env.IGNEUM_HARNESS_SIM || `${TARGET}/igneum-harness-sim`; -export const TMP = '/tmp/igneum-harness'; -export const BASE_PORT = 27200; +export const TMP = process.env.IGNEUM_HARNESS_TMP || '/tmp/igneum-harness'; +export const BASE_PORT = parseInt(process.env.IGNEUM_HARNESS_BASE_PORT || '27200', 10); +if (!Number.isInteger(BASE_PORT) || BASE_PORT < 27200 || BASE_PORT > 64000) throw new Error(`IGNEUM_HARNESS_BASE_PORT ${process.env.IGNEUM_HARNESS_BASE_PORT} is not a port in 27200..64000`); + +// u64::MAX ("never" for the height switches) is not a JavaScript number: keep it as a BigInt through the merge and +// write it back as the integer literal the node's parser wants (ledger F25; the same reviver as +// tools/finality-attacks/lib/net.mjs). +const bigReviver = (k, v, ctx) => (typeof v === 'number' && !Number.isSafeInteger(v) && ctx?.source ? BigInt(ctx.source) : v); +const bigStringify = (o) => JSON.stringify(o, (k, v) => (typeof v === 'bigint' ? `BIGINT:${v}` : v)).replace(/"BIGINT:(\d+)"/g, '$1'); +export function readParamsFile(file) { return JSON.parse(readFileSync(file, 'utf8'), bigReviver); } const started = []; // everything to stop at exit @@ -38,8 +48,8 @@ export function overrideParams(extra = {}) { // skip_proof_of_work: the harness miner never hashes; every other rule (timestamps, DAA, GHOSTDAG, merge // depth, mass, coinbase) runs unchanged. Devnet parameters otherwise (1 BPS, k 18, merge depth 3,600), or the // 60x fast-time profile under --fast-time (merge depth 60, finality window 120 DAA, 60-block epochs). - const base = FAST_TIME ? JSON.parse(readFileSync(FAST_TIME_FILE, 'utf8')) : {}; - writeFileSync(file, JSON.stringify({ ...base, skip_proof_of_work: true, ...extra })); + const base = FAST_TIME ? readParamsFile(FAST_TIME_FILE) : {}; + writeFileSync(file, bigStringify({ ...base, skip_proof_of_work: true, ...extra })); return file; } @@ -47,8 +57,8 @@ export function overrideParams(extra = {}) { // devnet, 60 under --fast-time). A scenario cut "beyond merge depth" is mergeDepth + 100 s on the devnet and // mergeDepth + 2 s at 60x. export function clockParams() { - const o = FAST_TIME ? JSON.parse(readFileSync(FAST_TIME_FILE, 'utf8')) : {}; - const mergeDepth = o.blockrate?.merge_depth ?? 3600; + const o = FAST_TIME ? readParamsFile(FAST_TIME_FILE) : {}; + const mergeDepth = Number(o.blockrate?.merge_depth ?? 3600); return { fastTime: FAST_TIME, mergeDepth, scale: 3600 / mergeDepth }; } diff --git a/tools/harness/run.mjs b/tools/harness/run.mjs index 014a898ec..5d0948074 100644 --- a/tools/harness/run.mjs +++ b/tools/harness/run.mjs @@ -3,7 +3,9 @@ // 127.0.0.1 ports 27200+ and /tmp/igneum-harness, writes a results table per run to docs/bench-log.md, and leaves // the test network stopped. The live devnet (26610/26611, 26640/26641, 28640) and the PC node are never touched. // -// node tools/harness/run.mjs [scenario ...] [--quick] [--no-bench-log] [--fast-time] +// node tools/harness/run.mjs [scenario ...] [--quick] [--no-bench-log] [--fast-time] [--live-only] +// --live-only skips the simulator part of s7 (the vmine flood against real nodes needs only igneumd). +// IGNEUM_HARNESS_BASE_PORT and IGNEUM_HARNESS_TMP move the ports and the data directory (lib/net.mjs). // scenarios: s5 s2 s1 s3 s6 s4 s7 (default: priority order 5,2,1,3,6,4,7) // --quick runs shorter block counts and durations for a smoke run. // --fast-time runs the network and the simulator on infra/fast-time/override-60x.json (every clock-like consensus @@ -12,7 +14,7 @@ // // See tools/harness/README.md. -import { stopAll, assertBinaries, FAST_TIME, TARGET } from './lib/net.mjs'; +import { stopAll, assertBinaries, FAST_TIME, TARGET, BASE_PORT, TMP } from './lib/net.mjs'; import { benchLogEntry, appendBenchLog } from './lib/report.mjs'; import { stubRows } from './scenarios/stubs.mjs'; import { execSync } from 'node:child_process'; @@ -34,7 +36,7 @@ function machineLine() { try { mem = (parseInt(execSync('sysctl -n hw.memsize').toString().trim(), 10) / 2 ** 30).toFixed(0) + ' GB'; } catch { } let load = ''; try { load = execSync('uptime').toString().match(/load averages?: ([\d. ]+)/)?.[1] || ''; } catch { } const profile = FAST_TIME ? 'skip_proof_of_work devnet on the 60x fast-time profile, infra/fast-time/override-60x.json' : 'skip_proof_of_work devnet'; - return `Machine: ${cpu}, ${mem}, load ${load.trim()}. Private test network of igneumd (release, ${profile}) on 127.0.0.1 ports 27200+, data /tmp/igneum-harness; the live devnet and the PC node were not touched. Harness: tools/harness/, binaries ${TARGET}.`; + return `Machine: ${cpu}, ${mem}, load ${load.trim()}. Private test network of igneumd (release, ${profile}) on 127.0.0.1 ports ${BASE_PORT}+, data ${TMP}; the live devnet and the PC node were not touched. Harness: tools/harness/, binaries ${TARGET}.`; } // Assemble one bench-log entry from result JSONs already written under /tmp/igneum-harness/results, without @@ -59,6 +61,7 @@ async function main() { const quick = args.includes('--quick'); const noBench = args.includes('--no-bench-log'); const assemble = args.includes('--assemble'); + const liveOnly = args.includes('--live-only'); if (assemble) { const allRows = await assembleFromResults(); @@ -70,7 +73,7 @@ async function main() { machine: machineLine(), rows: allRows, notes: [ - 'Full JSON per scenario under /tmp/igneum-harness/results and /tmp/igneum-harness/sim. The simulator (igneum/harness-sim in the fork worktree) runs real consensus code in virtual time with PoW skipped, as rusty-kaspa simpa does; the live scenarios (5, 6, 7 Part B) drive real igneumd processes over wRPC and the fork\'s own p2p (igneum/p2p-probe).', + `Full JSON per scenario under ${TMP}/results and ${TMP}/sim. The simulator (igneum/harness-sim in the fork worktree) runs real consensus code in virtual time with PoW skipped, as rusty-kaspa simpa does; the live scenarios (5, 6, 7 Part B) drive real igneumd processes over wRPC and the fork's own p2p (igneum/p2p-probe).`, 'Finality and difficulty-controller scenarios are stubs here: their criteria are written and they run against those branches once merged into the harness worktree (see tools/harness/scenarios/stubs.mjs).', ], }); @@ -89,7 +92,7 @@ async function main() { console.log(`\n==== scenario ${key}${quick ? ' (quick)' : ''}${FAST_TIME ? ' (fast-time 60x)' : ''} ====`); try { const mod = await SCENARIOS[key](); - const { rows } = await mod.run({ quick }); + const { rows } = await mod.run({ quick, liveOnly }); allRows.push(...rows); } catch (e) { console.error(`scenario ${key} threw: ${e.stack || e}`); @@ -110,7 +113,7 @@ async function main() { machine: machineLine(), rows: allRows, notes: [ - 'Full JSON per scenario under /tmp/igneum-harness/results and /tmp/igneum-harness/sim. The simulator (igneum/harness-sim in the fork worktree) runs real consensus code in virtual time with PoW skipped, as rusty-kaspa simpa does; the live scenarios (5, 6, 7 Part B) drive real igneumd processes over wRPC and the fork\'s own p2p (igneum/p2p-probe).', + `Full JSON per scenario under ${TMP}/results and ${TMP}/sim. The simulator (igneum/harness-sim in the fork worktree) runs real consensus code in virtual time with PoW skipped, as rusty-kaspa simpa does; the live scenarios (5, 6, 7 Part B) drive real igneumd processes over wRPC and the fork's own p2p (igneum/p2p-probe).`, 'Finality and difficulty-controller scenarios are stubs here: their criteria are written and they run against those branches once merged into the harness worktree (see tools/harness/scenarios/stubs.mjs).', ], }); diff --git a/tools/harness/scenarios/s6-exhaustion.mjs b/tools/harness/scenarios/s6-exhaustion.mjs index f7f081dfb..2d67311f4 100644 --- a/tools/harness/scenarios/s6-exhaustion.mjs +++ b/tools/harness/scenarios/s6-exhaustion.mjs @@ -5,7 +5,9 @@ // mempool: submitTransaction at 500/s of transactions spending unknown outputs (rejected one by one; funded // transactions need a wallet key, not done here). // Criterion: the honest peer's template latency p95 stays under 200 ms and both nodes stay under their memory bound -// (baseline RSS + 512 MB), alive, on one sink. Numbers are recorded per load. +// (baseline RSS + 512 MB), alive, on one sink. Numbers are recorded per load: `rss_start` and `rss_delta` are the +// load's own growth (RSS at its start to its peak); `rss_peak` minus `rss_baseline` is cumulative since the warm-up. +// `cache_builds` counts the node's "PoW cache built" log lines during the load (ledger M30: each is 256 MiB). import { Node, stopAll, dagInfo, log, sleep, assertBinaries } from '../lib/net.mjs'; import { Rpc } from '../lib/rpc.mjs'; @@ -34,6 +36,8 @@ export async function run({ quick = false } = {}) { async function load(name, perSec, fire, maxInflight = 200) { probeB.samples = []; probeA.samples = []; const t0 = Date.now(); let sent = 0, ok = 0, err = 0; const lat = []; let inflight = 0; + const rssStart = { a: a.rssMb(), b: b.rssMb() }; + const builds0 = { a: a.grepLog(/PoW cache built/).length, b: b.grepLog(/PoW cache built/).length }; const rssSeries = []; const tick = setInterval(() => rssSeries.push({ t_s: (Date.now() - t0) / 1000, a: a.rssMb(), b: b.rssMb() }), 2000); while (Date.now() - t0 < loadSecs * 1000) { @@ -51,11 +55,13 @@ export async function run({ quick = false } = {}) { const r = { requests_sent: sent, accepted: ok, rejected_or_error: err, rate_per_s: +(sent / loadSecs).toFixed(0), request_latency_ms: summarize(lat), honest_template_ms: bStats, attacked_node_template_ms: aStats, - rss_series: rssSeries, rss_peak: { a: Math.max(rss0.a, ...rssSeries.map(x => x.a)), b: Math.max(rss0.b, ...rssSeries.map(x => x.b)) }, + rss_series: rssSeries, rss_start: rssStart, rss_peak: { a: Math.max(rss0.a, ...rssSeries.map(x => x.a)), b: Math.max(rss0.b, ...rssSeries.map(x => x.b)) }, both_alive: a.alive() && b.alive(), }; + r.rss_delta = { a: r.rss_peak.a - rssStart.a, b: r.rss_peak.b - rssStart.b }; + r.cache_builds = { a: a.grepLog(/PoW cache built/).length - builds0.a, b: b.grepLog(/PoW cache built/).length - builds0.b }; results[name] = r; - log(`s6 ${name}: ${r.rate_per_s}/s, honest p95 ${bStats.p95} ms (base ${baseB.p95}), rss a ${r.rss_peak.a} b ${r.rss_peak.b}, alive ${r.both_alive}`); + log(`s6 ${name}: ${r.rate_per_s}/s, honest p95 ${bStats.p95} ms (base ${baseB.p95}), rss a ${rssStart.a}->${r.rss_peak.a} b ${rssStart.b}->${r.rss_peak.b} MB (+${r.rss_delta.a}/+${r.rss_delta.b}), cache builds ${r.cache_builds.a}/${r.cache_builds.b}, alive ${r.both_alive}`); return r; } @@ -83,11 +89,11 @@ export async function run({ quick = false } = {}) { const data = { baseline_template_ms: { a: baseA, b: baseB }, rss_baseline: rss0, loads: results, recovery_template_ms: recovery, final: { blocks_a: ia.blockCount, blocks_b: ib.blockCount, same_sink: sameSink }, alive, mem_bound_mb: MEM_BOUND_MB }; const worst = Math.max(...Object.values(results).map(r => r.honest_template_ms.p95)); - const peakRss = Object.entries(results).map(([k, r]) => `${k.split('_')[0]} +${Math.max(r.rss_peak.a - rss0.a, r.rss_peak.b - rss0.b)}MB`).join(', '); + const peakRss = Object.entries(results).map(([k, r]) => `${k.split('_')[0]} +${Math.max(r.rss_delta.a, r.rss_delta.b)}MB (${r.cache_builds.a}/${r.cache_builds.b} cache builds)`).join(', '); const row = { scenario: '6 resource exhaustion (50x template, submit and mempool floods from one peer)', criterion: 'honest template p95 < 200 ms and both nodes under baseline RSS + 512 MB, alive, one sink', - result: `honest template p95 worst ${worst} ms across loads (baseline ${baseB.p95} ms); ${Object.entries(results).map(([k, r]) => k.replace('_flood', '').replace('_', ' ') + ' ' + r.rate_per_s + '/s').join(', ')}; RSS growth ${peakRss}; alive ${alive}; same sink ${sameSink}`, + result: `honest template p95 worst ${worst} ms across loads (baseline ${baseB.p95} ms); ${Object.entries(results).map(([k, r]) => k.replace('_flood', '').replace('_', ' ') + ' ' + r.rate_per_s + '/s').join(', ')}; RSS growth per load ${peakRss}, cumulative +${Math.max(...Object.values(results).map(r => Math.max(r.rss_peak.a - rss0.a, r.rss_peak.b - rss0.b)))}MB over baseline ${rss0.a}/${rss0.b}; alive ${alive}; same sink ${sameSink}`, pass: alive && latencyOk && underBound && sameSink, }; saveResult('s6-exhaustion', { rows: [row], data }); diff --git a/tools/harness/scenarios/s7-flood.mjs b/tools/harness/scenarios/s7-flood.mjs index 871398d60..47d450852 100644 --- a/tools/harness/scenarios/s7-flood.mjs +++ b/tools/harness/scenarios/s7-flood.mjs @@ -11,11 +11,14 @@ import { Miner, LatencyProbe, difficultyRatio } from '../lib/miner.mjs'; import { saveResult } from '../lib/report.mjs'; import { runSim } from '../lib/sim.mjs'; -export async function run({ quick = false } = {}) { +export async function run({ quick = false, liveOnly = false } = {}) { assertBinaries(); const rows = []; const data = {}; // ---------- Part A: simulator ---------- const leave = quick ? 1200 : 1800, secs = quick ? 2400 : 4800; + if (liveOnly) { + rows.push({ scenario: '7 fast-miner flood, controller trajectory (sim)', criterion: 'trajectory recorded', result: 'skipped (--live-only)', pass: null }); + } else { const r = runSim('s7-flood', ['--scenario', 'flood', '--join-at', '600', '--leave-at', String(leave), '--flood-mult', '50', '--secs', String(secs), '--sample-secs', '30', '--seed', '77']); const g = r.genesis_time_ms; const traj = r.samples.map((s, i, a) => ({ t_s: (s.t - g) / 1000, bits: s.bits[0], ratio: +s.difficulty_ratio[0].toFixed(3), blocks: s.block_counts[0], rate: i ? +((s.block_counts[0] - a[i - 1].block_counts[0]) / ((s.t - a[i - 1].t) / 1000)).toFixed(2) : 0, daa: s.daa_scores[0] })); @@ -26,6 +29,7 @@ export async function run({ quick = false } = {}) { const settledAfterLeave = afterLeave.find(t => Math.abs(t.rate - 1) < 0.25 && afterLeave.slice(afterLeave.indexOf(t), afterLeave.indexOf(t) + 4).every(x => Math.abs(x.rate - 1) < 0.25)); data.sim = { trajectory: traj, peak_rate_bps: peakRate, peak_difficulty_ratio: peakRatio, trough_rate_after_leave: troughRate, settled_after_join_s: settled ? settled.t_s - 600 : null, settled_after_leave_s: settledAfterLeave ? settledAfterLeave.t_s - leave : null, counts: r.counts, wall_s: r.wall_s }; rows.push({ scenario: '7 fast-miner flood, controller trajectory (sim, Kaspa sampled DAA on HEAD)', criterion: 'trajectory recorded for the difficulty branch (bits, blocks per second, settle times)', result: `50x joins at 600 s: peak ${peakRate} blocks/s, difficulty x${peakRatio.toFixed(1)}, within 25% of 1 BPS after ${data.sim.settled_after_join_s ?? 'never'} s; leaves at ${leave} s: trough ${troughRate} blocks/s, back within 25% after ${data.sim.settled_after_leave_s ?? 'never'} s`, pass: true }); + } // ---------- Part B: live responsiveness ---------- const a = await new Node(0, { name: 's7a' }).start(); @@ -39,7 +43,10 @@ export async function run({ quick = false } = {}) { const flood = new Miner({ node: a, share: 1, label: 'flood-s7', rateMult: 50 }); await flood.start(); const floodSecs = quick ? 60 : 180; const samples = []; - for (let i = 0; i < floodSecs / 10; i++) { await sleep(10000); const ia = await dagInfo(a); const ib = await dagInfo(b); samples.push({ t_s: (i + 1) * 10, blocks_a: ia.blockCount, blocks_b: ib.blockCount, difficulty_a: ia.difficulty, sink_same: ia.sink === ib.sink, rss_a: a.rssMb(), rss_b: b.rssMb(), flood_accepted: flood.accepted, flood_rejected: flood.rejected, honest_accepted: honest.accepted }); log(`s7 live ${(i + 1) * 10}s: a ${ia.blockCount} b ${ib.blockCount} same=${ia.sink === ib.sink} flood ${flood.accepted}/${flood.rejected} honest ${honest.accepted}`); } + // Ledger M30: the node's "PoW cache built" lines (256 MiB each) are counted beside every RSS sample + const builds = (n) => n.grepLog(/PoW cache built/).length; + const builds0 = { a: builds(a), b: builds(b) }; + for (let i = 0; i < floodSecs / 10; i++) { await sleep(10000); const ia = await dagInfo(a); const ib = await dagInfo(b); samples.push({ t_s: (i + 1) * 10, blocks_a: ia.blockCount, blocks_b: ib.blockCount, difficulty_a: ia.difficulty, sink_same: ia.sink === ib.sink, rss_a: a.rssMb(), rss_b: b.rssMb(), cache_builds_a: builds(a) - builds0.a, cache_builds_b: builds(b) - builds0.b, flood_accepted: flood.accepted, flood_rejected: flood.rejected, honest_accepted: honest.accepted }); log(`s7 live ${(i + 1) * 10}s: a ${ia.blockCount} b ${ib.blockCount} same=${ia.sink === ib.sink} flood ${flood.accepted}/${flood.rejected} honest ${honest.accepted} rss ${a.rssMb()}/${b.rssMb()} MB cache builds ${builds(a) - builds0.a}/${builds(b) - builds0.b}`); } flood.stop(); const under = probe.stats(); probe.samples = []; await sleep(5000); @@ -47,14 +54,14 @@ export async function run({ quick = false } = {}) { const alive = a.alive() && b.alive(); const ia = await dagInfo(a), ib = await dagInfo(b); await stopAll(); - data.live = { baseline_template_ms: base, under_flood_template_ms: under, after_flood_template_ms: after, samples, rss_before: rss0, rss_peak: { a: Math.max(...samples.map(s => s.rss_a)), b: Math.max(...samples.map(s => s.rss_b)) }, flood: { accepted: flood.accepted, rejected: flood.rejected, errors: flood.errors }, honest: { accepted: honest.accepted, rejected: honest.rejected }, final: { blocks_a: ia.blockCount, blocks_b: ib.blockCount, same_sink: ia.sink === ib.sink, difficulty_a: ia.difficulty, ratio: difficultyRatio(ia.difficulty ? 0 : 0) }, alive }; - rows.push({ scenario: '7 fast-miner flood, live (50 blocks/s from one peer)', criterion: 'node stays responsive: honest template p95 < 200 ms, both nodes alive, same sink', result: `flood accepted ${flood.accepted} blocks in ${floodSecs} s (${(flood.accepted / floodSecs).toFixed(1)}/s); honest template p50/p95/max ${under.p50}/${under.p95}/${under.max} ms under flood (baseline ${base.p50}/${base.p95}/${base.max}); rss a ${rss0.a}->${data.live.rss_peak.a} MB, b ${rss0.b}->${data.live.rss_peak.b} MB; alive ${alive}; same sink ${ia.sink === ib.sink}`, pass: alive && under.p95 < 200 && ia.sink === ib.sink }); + data.live = { baseline_template_ms: base, under_flood_template_ms: under, after_flood_template_ms: after, samples, rss_before: rss0, rss_peak: { a: Math.max(...samples.map(s => s.rss_a)), b: Math.max(...samples.map(s => s.rss_b)) }, cache_builds: { a: builds(a) - builds0.a, b: builds(b) - builds0.b, before_flood: builds0 }, flood: { accepted: flood.accepted, rejected: flood.rejected, errors: flood.errors }, honest: { accepted: honest.accepted, rejected: honest.rejected }, final: { blocks_a: ia.blockCount, blocks_b: ib.blockCount, same_sink: ia.sink === ib.sink, difficulty_a: ia.difficulty, ratio: difficultyRatio(ia.difficulty ? 0 : 0) }, alive }; + rows.push({ scenario: '7 fast-miner flood, live (50 blocks/s from one peer)', criterion: 'node stays responsive: honest template p95 < 200 ms, both nodes alive, same sink', result: `flood accepted ${flood.accepted} blocks in ${floodSecs} s (${(flood.accepted / floodSecs).toFixed(1)}/s); honest template p50/p95/max ${under.p50}/${under.p95}/${under.max} ms under flood (baseline ${base.p50}/${base.p95}/${base.max}); rss a ${rss0.a}->${data.live.rss_peak.a} MB, b ${rss0.b}->${data.live.rss_peak.b} MB (cache builds ${data.live.cache_builds.a}/${data.live.cache_builds.b}); alive ${alive}; same sink ${ia.sink === ib.sink}`, pass: alive && under.p95 < 200 && ia.sink === ib.sink }); saveResult('s7-flood', { rows, data }); return { rows, data }; } if (import.meta.url === `file://${process.argv[1]}`) { - const r = await run({ quick: process.argv.includes('--quick') }); + const r = await run({ quick: process.argv.includes('--quick'), liveOnly: process.argv.includes('--live-only') }); console.log(JSON.stringify(r.rows, null, 2)); process.exit(0); }