bench-log: M30 floods re-run on the committed fork build 796f758d (second after pass)

s6 +6/+3/+1 MB per load, s7 302 to 318 MB, 0 cache builds, epochs 0 to 3
rolled, 202 blocks accepted; the pass-1 column stays beside it. Result JSON
after-{s6-exhaustion,s7-flood}.json added. The s6 one-instant sink check is
noted as a harness flake.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-05 01:06:07 +00:00
parent af8ae429f8
commit 32fe292a85
3 changed files with 579 additions and 8 deletions

View file

@ -1098,7 +1098,7 @@ the project lead, 4 October 2026 evening: "we need to fix these serious issues b
## 4 October 2026 (night), ledger M30: the block and transaction floods grew the node by 256 MiB per epoch roll, fixed by sharing the PoW cache across the epochs of a day (memory engineer)
Machine: Apple M5 Max, 64 GB, load averages 126 to 146 for the whole session (ten or more agents building and running at once). Every count here (blocks accepted, cache builds, RSS before and after) is valid under that load; every latency is an upper bound and is not a number. Private test network of two igneumd on 127.0.0.1 ports 29500+ (node A on 29500/29501/29502, node B on 29510/29511/29512), data under `/tmp/igneum-fud-mem/{baseline,after,after2}`, the 60x fast-time profile (`infra/fast-time/override-60x.json`, `skip_proof_of_work` on, `pow_epoch_blocks` 60, `pow_day_ms` 1,440,000) exactly as the red team ran it. The live devnet and other agents' ports were not touched. Every run went through `tools/lock/with-lock.sh run`, every build and the unit tests through `with-lock.sh build`.
Machine: Apple M5 Max, 64 GB, load averages 126 to 146 for the whole session (ten or more agents building and running at once). Every count here (blocks accepted, cache builds, RSS before and after) is valid under that load; every latency is an upper bound and is not a number. Private test network of two igneumd on 127.0.0.1 ports 29500+ (node A on 29500/29501/29502, node B on 29510/29511/29512), data under `/tmp/igneum-fud-mem/{baseline,after,after2}` (the second is pass 1, the third the committed build), the 60x fast-time profile (`infra/fast-time/override-60x.json`, `skip_proof_of_work` on, `pow_epoch_blocks` 60, `pow_day_ms` 1,440,000) exactly as the red team ran it. The live devnet and other agents' ports were not touched. Every run went through `tools/lock/with-lock.sh run`, every build and the unit tests through `with-lock.sh build`.
What the red team saw (`docs/review/redteam-2026-10-04.md` rows 5 and 8, ledger M30): on the 0.3.4 build the s6 submit flood grew RSS by +269 MB, the mempool flood by +270 MB, and the s7 block flood took both nodes from 302 to 1,082 MB. The s6 figures were cumulative from one baseline taken before all three loads (`rss_peak - rss_baseline` in `s6-exhaustion.mjs`), so the mempool flood's "+270 MB" was the submit flood's growth carried forward; its own cost was 1 MB. The red team's guess (execution-layer records, rejected transactions retained) did not hold: the mempool flood retains nothing measurable.
@ -1119,14 +1119,14 @@ RSS per load, node A / node B, MB (start of the load to its peak; "builds" = `Po
| Load | Before: start to peak A / B | Before: builds A / B | After (pass 1, build without the insert guard): start to peak A / B | After: builds A / B |
|---|---|---|---|---|
| s6 warm-up baseline (RSS before any load) | 303 / 304 | 1 / 1 (startup) | 304 / 304 | 1 / 1 (startup) |
| s6 template flood, 500/s, 14,995 and 14,993 sent, all answered | 304 to 309 / 304 to 309 (+5 / +5) | 0 / 0 | 304 to 310 / 304 to 308 (+6 / +4) | 0 / 0 |
| s6 submit flood, 50/s, 1,499 known-block submits, all answered; the 60-DAA epoch rolled during it | 309 to 572 / 309 to 566 (+263 / +257) | 1 / 1 | 310 to 319 / 308 to 310 (+9 / +2) | 0 / 0 |
| s6 mempool flood, 500/s, 14,993 and 14,999 unknown-outpoint transactions, all rejected | 572 to 573 / 566 to 567 (+1 / +1) | 0 / 0 | 319 to 320 / 310 to 310 (+1 / +0) | 0 / 0 |
| s7 block flood, vmine at 50x for 60 s: 198 and 203 blocks accepted (3.3 and 3.4 per s under load), epochs 0 to 3 rolled | 302 to 1,085 / 303 to 1,083 (+783 / +780); steps at 10 s 569, 20 s 827, 40 s 1,084 | 3 / 3 | 300 to 315 / 302 to 315 (+15 / +13) | 0 / 0 |
| s6 warm-up baseline (RSS before any load) | 303 / 304 | 1 / 1 (startup) | 305 / 307 | 1 / 1 (startup) | 304 / 304 |
| s6 template flood, 500/s, 14,995 and 14,999 sent, all answered | 304 to 309 / 304 to 309 (+5 / +5) | 0 / 0 | 305 to 311 / 307 to 310 (+6 / +3) | 0 / 0 | 304 to 310 / 304 to 308 (+6 / +4) |
| s6 submit flood, 50/s, 1,499 known-block submits, all answered; the 60-DAA epoch rolled during it | 309 to 572 / 309 to 566 (+263 / +257) | 1 / 1 | 311 to 314 / 310 to 312 (+3 / +2) | 0 / 0 | 310 to 319 / 308 to 310 (+9 / +2) |
| s6 mempool flood, 500/s, 14,993 and 14,995 unknown-outpoint transactions, all rejected | 572 to 573 / 566 to 567 (+1 / +1) | 0 / 0 | 315 to 316 / 312 to 313 (+1 / +1) | 0 / 0 | 319 to 320 / 310 to 310 (+1 / +0) |
| s7 block flood, vmine at 50x for 60 s: 198 and 202 blocks accepted (3.3 and 3.4 per s), epochs 0 to 3 rolled in every run | 302 to 1,085 / 303 to 1,083 (+783 / +780); steps at 10 s 569, 20 s 827, 40 s 1,084 | 3 / 3 | 302 to 318 / 302 to 315 (+16 / +13) | 0 / 0 | 300 to 315 / 302 to 315 (+15 / +13) |
Honest template p95 (upper bounds, load over 100): before 130.8 / 86.7 / 104.7 ms per s6 load against a baseline of 84.7, s7 91.5 against 92.8; after 101.7 / 66.1 / 84.4 against 89.5, s7 75.4 against 69.4. Both nodes alive and on one sink at the end of every run except the before-run s6 (sinks differed at the instant of the check, 120 against 121 blocks, under load 136; the after runs agreed). The red team's harness criterion (baseline + 512 MB) still passes before and after; the 50 MB-per-load target holds after.
Honest template p95 (upper bounds; the before run and pass 1 ran at load over 100, the committed-build run at load under 5 for s6 and about 10 for s7): before 130.8 / 86.7 / 104.7 ms per s6 load against a baseline of 84.7, s7 91.5 against 92.8; committed build 0.5 / 0.6 / 78.7 against 0.7, s7 81.3 against 104.6. Both nodes alive in every run. The s6 row's one-instant sink check failed in the before run (120 against 121 blocks) and in the committed-build run (121 against 122) while the honest miner was mid-submit, and passed in pass 1; the s7 sinks agreed in every run. That check reads the two sinks once without waiting (`waitSameSink` exists in `lib/net.mjs` and s6 does not use it), so it is a harness flake, not a node finding; left as is tonight. The red team's harness criterion (baseline + 512 MB) still passes before and after; the 50 MB-per-load target holds after.
Harness changes (this repo): `IGNEUM_HARNESS_BASE_PORT` and `IGNEUM_HARNESS_TMP` (ports and data directory, so two agents can run the harness at once), the u64 sentinel round-trip fixed with the BigInt reviver from `tools/finality-attacks/lib/net.mjs` (ledger F25), s6 records `rss_start`, `rss_delta` and `cache_builds` per load and its row reports per-load growth, s7 records `cache_builds` beside every RSS sample, and `--live-only` skips the s7 simulator part. Result JSON: `docs/benchmarks/memory-floods-2026-10-04/{before,after-pass1}-{s6-exhaustion,s7-flood}.json`. Pass 1 ran the build of the engine change before its last three-line guard (`insert_program` skips a program whose day cache was evicted during its generation; the guard cannot fire in these single-day runs); the committed fork build (796f758d) is the one the unit tests ran on.
Harness changes (this repo): `IGNEUM_HARNESS_BASE_PORT` and `IGNEUM_HARNESS_TMP` (ports and data directory, so two agents can run the harness at once), the u64 sentinel round-trip fixed with the BigInt reviver from `tools/finality-attacks/lib/net.mjs` (ledger F25), s6 records `rss_start`, `rss_delta` and `cache_builds` per load and its row reports per-load growth, s7 records `cache_builds` beside every RSS sample, and `--live-only` skips the s7 simulator part. Result JSON: `docs/benchmarks/memory-floods-2026-10-04/{before,after,after-pass1}-{s6-exhaustion,s7-flood}.json` (`after` is the committed build, `after-pass1` the build before its last three-line guard: `insert_program` skips a program whose day cache was evicted during its generation, which cannot fire in these single-day runs). Data directories `/tmp/igneum-fud-mem/{baseline,after-pass1 as after,after2}`.
Not covered tonight: the execution layer's `ExecState.records` (`igneum/exec/src/service.rs:31`, pushed per chain block, never truncated) is a slow growth, not a flood effect: 197 chain blocks cost under 1 MB in these runs, and a record on an empty devnet is roughly 1 to 2 KB (approximate, from the struct), so about 100 to 170 MB per day at 1 block/s; bounding it needs a window at least as long as the proving sortition window (`proving.rs:200`) plus the RPC's by-number history, which is a design choice, not a cache. The snapshot ring (`SNAPSHOT_RING = 64` full `IgneumDb` clones) is bounded in count but scales with the state size. The finality store trims votes, checkpoints, certificates and locks every index (`processes/finality.rs:531`); its `keys` and `stripped` maps grow with distinct vote keys (about 150 bytes per key, approximate). The proof pool keeps `RECORD_WINDOW_CHAIN_BLOCKS` of entries. None of these moved in these floods.

View file

@ -0,0 +1,408 @@
{
"rows": [
{
"scenario": "6 resource exhaustion (50x template, submit and mempool floods from one peer)",
"criterion": "honest template p95 < 200 ms and both nodes under baseline RSS + 512 MB, alive, one sink",
"result": "honest template p95 worst 78.7 ms across loads (baseline 0.7 ms); template 500ps 500/s, submit 50ps 50/s, mempool 500ps 500/s; RSS growth per load template +6MB (0/0 cache builds), submit +3MB (0/0 cache builds), mempool +1MB (0/0 cache builds), cumulative +11MB over baseline 305/307; alive true; same sink false",
"pass": false
}
],
"data": {
"baseline_template_ms": {
"a": {
"n": 98,
"p50": 0.4,
"p95": 0.6,
"p99": 0.9,
"max": 0.9,
"mean": 0.4
},
"b": {
"n": 98,
"p50": 0.5,
"p95": 0.7,
"p99": 1.5,
"max": 1.5,
"mean": 0.5
}
},
"rss_baseline": {
"a": 305,
"b": 307
},
"loads": {
"template_flood_500ps": {
"requests_sent": 14999,
"accepted": 14999,
"rejected_or_error": 0,
"rate_per_s": 500,
"request_latency_ms": {
"n": 14999,
"p50": 0.2,
"p95": 0.7,
"p99": 3.7,
"max": 102.4,
"mean": 0.4
},
"honest_template_ms": {
"n": 285,
"p50": 0.3,
"p95": 0.5,
"p99": 1.4,
"max": 1.9,
"mean": 0.3
},
"attacked_node_template_ms": {
"n": 285,
"p50": 0.3,
"p95": 0.4,
"p99": 0.9,
"max": 1.7,
"mean": 0.3
},
"rss_series": [
{
"t_s": 2.102,
"a": 305,
"b": 308
},
{
"t_s": 4.102,
"a": 306,
"b": 308
},
{
"t_s": 6.101,
"a": 306,
"b": 309
},
{
"t_s": 8.102,
"a": 307,
"b": 309
},
{
"t_s": 10.102,
"a": 307,
"b": 310
},
{
"t_s": 12.102,
"a": 308,
"b": 310
},
{
"t_s": 14.103,
"a": 309,
"b": 310
},
{
"t_s": 16.103,
"a": 309,
"b": 310
},
{
"t_s": 18.104,
"a": 310,
"b": 310
},
{
"t_s": 20.104,
"a": 310,
"b": 310
},
{
"t_s": 22.104,
"a": 310,
"b": 310
},
{
"t_s": 24.104,
"a": 310,
"b": 310
},
{
"t_s": 26.104,
"a": 311,
"b": 310
},
{
"t_s": 28.104,
"a": 311,
"b": 310
}
],
"rss_start": {
"a": 305,
"b": 307
},
"rss_peak": {
"a": 311,
"b": 310
},
"both_alive": true,
"rss_delta": {
"a": 6,
"b": 3
},
"cache_builds": {
"a": 0,
"b": 0
}
},
"submit_flood_50ps": {
"requests_sent": 1499,
"accepted": 1499,
"rejected_or_error": 0,
"rate_per_s": 50,
"request_latency_ms": {
"n": 1499,
"p50": 0.7,
"p95": 1.4,
"p99": 3.6,
"max": 5.4,
"mean": 0.8
},
"honest_template_ms": {
"n": 285,
"p50": 0.3,
"p95": 0.6,
"p99": 0.8,
"max": 0.8,
"mean": 0.4
},
"attacked_node_template_ms": {
"n": 285,
"p50": 0.4,
"p95": 0.6,
"p99": 0.8,
"max": 0.8,
"mean": 0.4
},
"rss_series": [
{
"t_s": 2.114,
"a": 311,
"b": 310
},
{
"t_s": 4.114,
"a": 312,
"b": 311
},
{
"t_s": 6.114,
"a": 312,
"b": 311
},
{
"t_s": 8.114,
"a": 312,
"b": 311
},
{
"t_s": 10.114,
"a": 313,
"b": 311
},
{
"t_s": 12.114,
"a": 313,
"b": 311
},
{
"t_s": 14.115,
"a": 313,
"b": 311
},
{
"t_s": 16.115,
"a": 313,
"b": 311
},
{
"t_s": 18.116,
"a": 313,
"b": 311
},
{
"t_s": 20.117,
"a": 314,
"b": 311
},
{
"t_s": 22.117,
"a": 314,
"b": 311
},
{
"t_s": 24.117,
"a": 314,
"b": 312
},
{
"t_s": 26.117,
"a": 314,
"b": 312
},
{
"t_s": 28.118,
"a": 314,
"b": 312
}
],
"rss_start": {
"a": 311,
"b": 310
},
"rss_peak": {
"a": 314,
"b": 312
},
"both_alive": true,
"rss_delta": {
"a": 3,
"b": 2
},
"cache_builds": {
"a": 0,
"b": 0
}
},
"mempool_flood_500ps": {
"requests_sent": 14995,
"accepted": 0,
"rejected_or_error": 14995,
"rate_per_s": 500,
"request_latency_ms": {
"n": 14995,
"p50": 4.5,
"p95": 23.2,
"p99": 42.3,
"max": 812.8,
"mean": 8.7
},
"honest_template_ms": {
"n": 187,
"p50": 5.3,
"p95": 78.7,
"p99": 629.7,
"max": 832.9,
"mean": 31
},
"attacked_node_template_ms": {
"n": 194,
"p50": 4.9,
"p95": 65.4,
"p99": 528.9,
"max": 833,
"mean": 26.4
},
"rss_series": [
{
"t_s": 2.228,
"a": 315,
"b": 312
},
{
"t_s": 4.229,
"a": 315,
"b": 312
},
{
"t_s": 6.229,
"a": 315,
"b": 312
},
{
"t_s": 8.231,
"a": 315,
"b": 312
},
{
"t_s": 10.242,
"a": 315,
"b": 312
},
{
"t_s": 12.241,
"a": 316,
"b": 312
},
{
"t_s": 14.242,
"a": 316,
"b": 312
},
{
"t_s": 16.243,
"a": 316,
"b": 312
},
{
"t_s": 18.243,
"a": 316,
"b": 312
},
{
"t_s": 20.251,
"a": 316,
"b": 313
},
{
"t_s": 22.251,
"a": 316,
"b": 313
},
{
"t_s": 24.269,
"a": 316,
"b": 313
},
{
"t_s": 26.269,
"a": 316,
"b": 313
},
{
"t_s": 28.275,
"a": 316,
"b": 313
}
],
"rss_start": {
"a": 315,
"b": 312
},
"rss_peak": {
"a": 316,
"b": 313
},
"both_alive": true,
"rss_delta": {
"a": 1,
"b": 1
},
"cache_builds": {
"a": 0,
"b": 0
}
}
},
"recovery_template_ms": {
"n": 226,
"p50": 6.1,
"p95": 78.7,
"p99": 543.6,
"max": 832.9,
"mean": 29.8
},
"final": {
"blocks_a": 121,
"blocks_b": 122,
"same_sink": false
},
"alive": true,
"mem_bound_mb": 512
}
}

View file

@ -0,0 +1,163 @@
{
"rows": [
{
"scenario": "7 fast-miner flood, controller trajectory (sim)",
"criterion": "trajectory recorded",
"result": "skipped (--live-only)",
"pass": null
},
{
"scenario": "7 fast-miner flood, live (50 blocks/s from one peer)",
"criterion": "node stays responsive: honest template p95 < 200 ms, both nodes alive, same sink",
"result": "flood accepted 202 blocks in 60 s (3.4/s); honest template p50/p95/max 5.9/81.3/1252.2 ms under flood (baseline 12.6/104.6/120.5); rss a 302->318 MB, b 302->315 MB (cache builds 0/0); alive true; same sink true",
"pass": true
}
],
"data": {
"live": {
"baseline_template_ms": {
"n": 76,
"p50": 12.6,
"p95": 104.6,
"p99": 120.5,
"max": 120.5,
"mean": 27.8
},
"under_flood_template_ms": {
"n": 469,
"p50": 5.9,
"p95": 81.3,
"p99": 150,
"max": 1252.2,
"mean": 30
},
"after_flood_template_ms": {
"n": 39,
"p50": 2.8,
"p95": 121.2,
"p99": 1106.2,
"max": 1106.2,
"mean": 50
},
"samples": [
{
"t_s": 10,
"blocks_a": 124,
"blocks_b": 124,
"difficulty_a": 687251814.6297691,
"sink_same": true,
"rss_a": 313,
"rss_b": 311,
"cache_builds_a": 0,
"cache_builds_b": 0,
"flood_accepted": 103,
"flood_rejected": 0,
"honest_accepted": 21
},
{
"t_s": 20,
"blocks_a": 166,
"blocks_b": 166,
"difficulty_a": 2176913973.013581,
"sink_same": true,
"rss_a": 314,
"rss_b": 313,
"cache_builds_a": 0,
"cache_builds_b": 0,
"flood_accepted": 144,
"flood_rejected": 0,
"honest_accepted": 22
},
{
"t_s": 30,
"blocks_a": 187,
"blocks_b": 187,
"difficulty_a": 3197271306.1776547,
"sink_same": true,
"rss_a": 316,
"rss_b": 314,
"cache_builds_a": 0,
"cache_builds_b": 0,
"flood_accepted": 164,
"flood_rejected": 0,
"honest_accepted": 23
},
{
"t_s": 40,
"blocks_a": 208,
"blocks_b": 208,
"difficulty_a": 5948600103.681134,
"sink_same": true,
"rss_a": 317,
"rss_b": 314,
"cache_builds_a": 0,
"cache_builds_b": 0,
"flood_accepted": 185,
"flood_rejected": 0,
"honest_accepted": 23
},
{
"t_s": 50,
"blocks_a": 218,
"blocks_b": 218,
"difficulty_a": 6763165668.73272,
"sink_same": true,
"rss_a": 317,
"rss_b": 314,
"cache_builds_a": 0,
"cache_builds_b": 0,
"flood_accepted": 193,
"flood_rejected": 0,
"honest_accepted": 25
},
{
"t_s": 60,
"blocks_a": 228,
"blocks_b": 228,
"difficulty_a": 6313391779.974011,
"sink_same": true,
"rss_a": 318,
"rss_b": 315,
"cache_builds_a": 0,
"cache_builds_b": 0,
"flood_accepted": 202,
"flood_rejected": 0,
"honest_accepted": 26
}
],
"rss_before": {
"a": 302,
"b": 302
},
"rss_peak": {
"a": 318,
"b": 315
},
"cache_builds": {
"a": 0,
"b": 0,
"before_flood": {
"a": 1,
"b": 1
}
},
"flood": {
"accepted": 202,
"rejected": 0,
"errors": 1
},
"honest": {
"accepted": 26,
"rejected": 0
},
"final": {
"blocks_a": 228,
"blocks_b": 228,
"same_sink": true,
"difficulty_a": 6313391779.974011,
"ratio": null
},
"alive": true
}
}
}