igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity

fast-time crossing: the known-failed case's FAIL (19:04:01Z, 22 checks named) and the first cross run's evidence (the chain crossed: rung 1 by signal at epoch 6, class v5 by signal at epoch 8 at rung 1; the restart step FAILED: n2 restarted on its own datadir across the boundary could not IBD, 'proof-of-work check refused, class v5 needs the execution state after the epoch's seed block', 13 attempts, its own fork; the end-of-run reads hung on its exec RPC); the harness: a 5 s timeout on every stream fetch, the IBD refusal after the restart as a named check, the end-of-run reads under a 180 s cap

Browse source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-07 19:22:16 +00:00
parent 0a1c2661e9
commit 17e2a2e1ed
5 changed files with 1734 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
docs/design/class-v5-harness/fasttime/cross-run1-n2-ibd.txt Normal file
View file

@ -0,0 +1,21 @@
== n2 after its restart: IBD lines
2026-10-07 21:19:25.116+02:00 [INFO ] IBD with peer 127.0.0.1:29861 completed with error: Igneum: proof-of-work check refused, class v5 needs the execution state after the epoch's seed block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (day 1244028): no published state stream after chain block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (the executor has no
2026-10-07 21:19:25.116+02:00 [WARN ] IbdFlow flow error: Igneum: proof-of-work check refused, class v5 needs the execution state after the epoch's seed block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (day 1244028): no published state stream after chain block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (the executor has not passed that epoch's cut, or th
2026-10-07 21:19:54.874+02:00 [INFO ] template prewarm: 39 builds in the last 63 s, mean 0.7 ms, max 4.0 ms, 0 notifications coalesced, 0 skipped (no miner yet or IBD)
2026-10-07 21:19:55.116+02:00 [INFO ] IBD started with peer 127.0.0.1:29861
2026-10-07 21:19:55.131+02:00 [INFO ] IBD with peer 127.0.0.1:29861 completed with error: Igneum: proof-of-work check refused, class v5 needs the execution state after the epoch's seed block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (day 1244028): no published state stream after chain block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (the executor has no
2026-10-07 21:19:55.131+02:00 [WARN ] IbdFlow flow error: Igneum: proof-of-work check refused, class v5 needs the execution state after the epoch's seed block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (day 1244028): no published state stream after chain block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (the executor has not passed that epoch's cut, or th
2026-10-07 21:20:25.718+02:00 [INFO ] IBD started with peer 127.0.0.1:29861
2026-10-07 21:20:25.893+02:00 [INFO ] IBD with peer 127.0.0.1:29861 completed with error: Igneum: proof-of-work check refused, class v5 needs the execution state after the epoch's seed block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (day 1244028): no published state stream after chain block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (the executor has no
2026-10-07 21:20:25.893+02:00 [WARN ] IbdFlow flow error: Igneum: proof-of-work check refused, class v5 needs the execution state after the epoch's seed block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (day 1244028): no published state stream after chain block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (the executor has not passed that epoch's cut, or th
2026-10-07 21:20:55.173+02:00 [INFO ] IBD started with peer 127.0.0.1:29861
2026-10-07 21:20:55.182+02:00 [INFO ] IBD with peer 127.0.0.1:29861 completed with error: Igneum: proof-of-work check refused, class v5 needs the execution state after the epoch's seed block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (day 1244028): no published state stream after chain block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (the executor has no
2026-10-07 21:20:55.182+02:00 [WARN ] IbdFlow flow error: Igneum: proof-of-work check refused, class v5 needs the execution state after the epoch's seed block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (day 1244028): no published state stream after chain block 69d3ab7f1e01a4816aa1249629d2842f1f221279306ec794f936fa356a37534c (the executor has not passed that epoch's cut, or th
== IBD error count: 13
== lock/panic lines: 0
== tips at 19:21:24Z:
n0 785 PoW-accepted lines, 0 rejected
n1 771 PoW-accepted lines, 0 rejected
n2 738 PoW-accepted lines, 0 rejected
n3 771 PoW-accepted lines, 89 rejected
== cpu3 (stale) rejected: 88, accepted 150

94
docs/design/class-v5-harness/fasttime/cross-run1.log Normal file
View file

@ -0,0 +1,94 @@
19:04:22.115 override: 4 key(s) the binary does not know dropped (sig_scheme, sig_scheme_activation_daa, finality_succession_activation_daa, latency_ladder_cache_rung)
19:04:22.116 case cross: 4 nodes, bytes 6/6/6/6, ladder up/up/up/up; class v4 sub-version 3 from genesis at rung 0; class window 60 DAA x 7 (the first epoch that can cross to v5 is 8, DAA 480), ladder window 45 DAA x 7 (the first epoch that can step is 6, DAA 360); v5 floor 900 (epoch 15); stale node 3; restart node 2 (stop at DAA 455, start at 500); 60 DAA per epoch, lead 10; run 960 s or 11 epochs; node /srv/builds/igneum-wt-v5-fasttime/vendor/igneum-node-v5-fasttime/target/release/igneumd
19:04:27.082 CPU verifier on the Devnet 3 genesis pack: cli e5a4ac5978462156 pack e5a4ac5978462156 expected e5a4ac5978462156: OK
19:04:28.641 n0 up pid 1119100 json 29862 p2p 29861 exec 29863, byte 6, ladder up
19:04:30.155 n1 up pid 1119363 json 29872 p2p 29871 exec 29873, byte 6, ladder up
19:04:31.664 n2 up pid 1119613 json 29882 p2p 29881 exec 29883, byte 6, ladder up
19:04:33.169 n3 up pid 1119913 json 29892 p2p 29891 exec 29893, byte 6, ladder up
19:04:33.170 n0: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
19:04:33.170 n1: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
19:04:33.170 n2: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
19:04:33.170 n3: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
19:04:33.170 n0 digest: 6fa8e71d7fb9aa0d
19:04:34.187 epoch -1 -> 0 at daa 0, 7.1 s: template class 3 (next 3) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:04:34.220 t=7.1 s daa 0 epoch 0 class 3 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 0/234e082d 0/234e082d 0/234e082d 0/234e082d
19:04:49.304 t=22.2 s daa 0 epoch 0 class 3 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 0/234e082d 0/234e082d 0/234e082d 0/234e082d
19:05:04.453 t=37.4 s daa 2 epoch 0 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 2/72937082 2/72937082 2/72937082 2/72937082
19:05:19.546 t=52.5 s daa 11 epoch 0 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 11/c177d283 11/c177d283 11/c177d283 11/c177d283
19:05:34.593 t=67.5 s daa 15 epoch 0 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 15/90704f1f 15/90704f1f 15/90704f1f 15/90704f1f
19:05:49.627 t=82.5 s daa 30 epoch 0 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 30/7b78ee46 30/7b78ee46 30/7b78ee46 30/7b78ee46
19:06:04.694 t=97.6 s daa 48 epoch 0 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 48/5a9eb898 48/5a9eb898 48/5a9eb898 48/5a9eb898
19:06:19.740 t=112.7 s daa 58 epoch 0 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 58/20e0b73b 58/20e0b73b 58/20e0b73b 58/20e0b73b
19:06:20.744 epoch 0 -> 1 at daa 60, 113.7 s: template class 3 (next 3) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:06:22.752 stream for epoch 1 (seed 3b9a15224d8a9b0c): root 0x2e1a306737b355d2 17 records, from n0
19:06:35.245 t=128.2 s daa 64 epoch 1 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 64/c37c9383 64/c37c9383 64/c37c9383 64/c37c9383
19:06:50.659 t=143.6 s daa 65 epoch 1 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 65/a6316dba 65/a6316dba 65/a6316dba 65/a6316dba
19:07:06.009 t=158.9 s daa 68 epoch 1 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 68/36d6c468 68/36d6c468 68/36d6c468 68/36d6c468
19:07:22.025 t=174.9 s daa 68 epoch 1 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 68/36d6c468 68/36d6c468 68/36d6c468 68/36d6c468
19:07:37.512 t=190.4 s daa 69 epoch 1 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 69/72c5d44c 69/72c5d44c 69/72c5d44c 69/72c5d44c
19:07:53.000 t=205.9 s daa 80 epoch 1 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 80/ccd6c335 80/ccd6c335 80/ccd6c335 80/ccd6c335
19:08:08.048 t=221.0 s daa 108 epoch 1 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 108/a743a62a 108/a743a62a 108/a743a62a 108/a743a62a
19:08:15.067 epoch 1 -> 2 at daa 121, 228.0 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:08:18.074 stream for epoch 2 (seed 0eccc013f9b60e0d): root 0xc0a3a18d4d925353 17 records, from n0
19:08:23.090 t=236.0 s daa 121 epoch 2 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 121/e3094893 121/e3094893 121/e3094893 121/e3094893
19:08:38.152 t=251.1 s daa 143 epoch 2 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 143/cc1ed2db 143/cc1ed2db 143/cc1ed2db 143/cc1ed2db
19:08:53.244 t=266.2 s daa 175 epoch 2 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 175/3d87f664 175/3d87f664 175/3d87f664 175/3d87f664
19:08:57.255 epoch 2 -> 3 at daa 180, 270.2 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:08:58.261 stream for epoch 3 (seed 2b01fdd54ea150f5): root 0x362eac479bfe7ebb 17 records, from n0
19:09:08.309 t=281.2 s daa 183 epoch 3 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 183/f24d5245 183/f24d5245 183/f24d5245 183/f24d5245
19:09:23.353 t=296.3 s daa 212 epoch 3 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 212/4bb4ea6d 212/4bb4ea6d 212/4bb4ea6d 212/4bb4ea6d
19:09:38.399 t=311.3 s daa 228 epoch 3 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 228/c9e5ca83 228/c9e5ca83 228/c9e5ca83 228/c9e5ca83
19:09:49.423 epoch 3 -> 4 at daa 241, 322.3 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:09:50.429 stream for epoch 4 (seed 2b65b0e1ac694375): root 0xcf8c29a8a71c192e 17 records, from n0
19:09:53.437 t=326.4 s daa 241 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 241/af0370c7 241/af0370c7 241/af0370c7 241/af0370c7
19:10:08.476 t=341.4 s daa 255 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 255/f68e301c 255/f68e301c 255/f68e301c 255/f68e301c
19:10:23.530 t=356.4 s daa 269 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 269/0a0af1a2 269/0a0af1a2 269/0a0af1a2 269/0a0af1a2
19:10:38.578 t=371.5 s daa 284 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 284/32d217a5 284/32d217a5 284/32d217a5 284/32d217a5
19:10:53.637 t=386.6 s daa 298 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 298/3da5a782 297/7968d7f4 297/7968d7f4 298/3da5a782
19:10:57.646 epoch 4 -> 5 at daa 300, 390.6 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:10:58.655 stream for epoch 5 (seed 56e5ee99d68c7f63): root 0x89fe68cf51ddd005 17 records, from n0
19:11:08.684 t=401.6 s daa 302 epoch 5 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 302/e2f27369 302/e2f27369 302/e2f27369 308/298bbcae
19:11:23.730 t=416.6 s daa 324 epoch 5 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 324/833cfdb5 324/833cfdb5 324/833cfdb5 324/833cfdb5
19:11:38.786 t=431.7 s daa 337 epoch 5 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 337/ebd1a71d 337/ebd1a71d 337/ebd1a71d 337/ebd1a71d
19:11:53.825 epoch 5 -> 6 at daa 360, 446.7 s: template class 4 (next 4) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
19:11:53.825 LADDER STEP: the template is rung 1 (35 shadow passes) from epoch 6 (daa 360) at 446.7 s wall
19:11:53.826 t=446.7 s daa 360 epoch 6 class 4 rung 1 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 360/8aa8dd99 360/8aa8dd99 360/8aa8dd99 360/8aa8dd99
19:11:54.833 stream for epoch 6 (seed 2351bd291bb1ae36): root 0xc78294404f032d42 17 records, from n0
19:12:08.878 t=461.8 s daa 361 epoch 6 class 4 rung 1 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 361/8aa8dd99 361/8aa8dd99 361/8aa8dd99 361/8aa8dd99
19:12:24.016 t=476.9 s daa 371 epoch 6 class 4 rung 1 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 371/aa32fdd9 371/aa32fdd9 371/aa32fdd9 366/d431f65a
19:12:39.078 t=492.0 s daa 394 epoch 6 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 394/3f9f5fac 394/3f9f5fac 394/3f9f5fac 394/3f9f5fac
19:12:54.126 t=507.0 s daa 415 epoch 6 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 415/0e86785b 415/0e86785b 415/0e86785b 415/0e86785b
19:13:01.143 epoch 6 -> 7 at daa 420, 514.1 s: template class 4 (next 4) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
19:13:03.154 stream for epoch 7 (seed 3c52e672c1378f2a): root 0x22531991b734ea59 17 records, from n0
19:13:09.166 t=522.1 s daa 423 epoch 7 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 423/d36c2e38 423/d36c2e38 423/d36c2e38 423/d36c2e38
19:13:24.210 t=537.1 s daa 449 epoch 7 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 449/8ba9cb86 449/8ba9cb86 449/8ba9cb86 449/8ba9cb86
19:13:30.232 RESTART STEP: n2 and its miner stopped at daa 456 (543.1 s), 24 DAA before the expected v5 boundary 480
19:13:39.264 t=552.2 s daa 470 epoch 7 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 470/69d3ab7f 470/69d3ab7f down 470/69d3ab7f
19:13:49.309 epoch 7 -> 8 at daa 481, 562.2 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch 8, step at epoch 6
19:13:49.309 CLASS SWITCH: the template is class v5 from epoch 8 (daa 481) at rung 1 (35 passes), 562.2 s wall
19:13:51.320 stream for epoch 8 (seed 69d3ab7f1e01a481): root 0x22bc70bdb33afe63 17 records, from n0
19:13:54.324 t=567.2 s daa 481 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 481/37947df8 481/37947df8 down 481/37947df8
19:14:09.378 t=582.3 s daa 482 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 482/3d87fc70 481/37947df8 down 481/37947df8
19:14:25.976 n2 RESTARTED pid 1156550 json 29882 p2p 29881 exec 29883, byte 6, ladder up (own datadir kept)
19:14:25.981 RESTART STEP: n2 started on its own datadir at daa 500 (598.9 s), 20 DAA after the boundary; its miner restarted with its key
19:14:25.983 t=598.9 s daa 500 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 503/fb374a55 503/fb374a55 456/32ee058c 503/fb374a55
19:14:41.057 t=614.0 s daa 516 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 516/5f40e6e0 516/5f40e6e0 456/32ee058c 516/5f40e6e0
19:14:56.142 t=629.1 s daa 533 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 533/6fdd7334 533/6fdd7334 462/03506bf8 533/6fdd7334
19:15:00.149 epoch 8 -> 9 at daa 540, 633.1 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
19:15:00.154 stream for epoch 9 (seed 8982354512f2f9e7): root 0x7726cb57697c6d7b 17 records, from n0
19:15:11.214 t=644.1 s daa 540 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 540/56291e9f 540/56291e9f 470/8d4c5095 540/56291e9f
19:15:26.336 t=659.3 s daa 543 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 543/8f9ad85c 543/8f9ad85c 480/c9f7bf3b 543/8f9ad85c
19:15:41.431 t=674.3 s daa 551 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 551/076ad538 551/076ad538 485/a0f4a2e6 551/076ad538
19:15:56.486 t=689.4 s daa 569 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 569/3d1706a1 569/3d1706a1 501/c4299c77 569/3d1706a1
19:16:11.572 t=704.5 s daa 587 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 587/e2c372d7 587/e2c372d7 518/50ba5302 587/e2c372d7
19:16:25.655 epoch 9 -> 10 at daa 600, 718.6 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
19:16:26.658 t=719.6 s daa 600 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 600/04e5328b 600/04e5328b 535/66897093 600/04e5328b
19:16:28.673 stream for epoch 10 (seed 76ba0aa9f559b0d5): root 0x0f0f2b27619ae20c 17 records, from n0
19:16:41.745 t=734.7 s daa 602 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 602/cbed939a 601/3d48d6a4 540/790011b0 602/cbed939a
19:16:57.039 t=750.0 s daa 616 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 616/d1a8f4b5 616/d1a8f4b5 546/0d490be1 616/d1a8f4b5
19:17:13.099 t=766.0 s daa 616 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 616/d1a8f4b5 616/d1a8f4b5 548/3dcabef7 616/d1a8f4b5
19:17:28.626 t=781.5 s daa 621 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 621/576085f1 621/576085f1 556/18734417 621/576085f1
19:17:44.266 t=797.2 s daa 626 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 626/67fe2511 626/67fe2511 559/39c4eeba 626/67fe2511
19:17:59.945 t=812.9 s daa 626 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 626/67fe2511 626/67fe2511 560/9f7a47cd 626/67fe2511
19:18:12.697 epoch 10 -> 11 at daa 660, 825.6 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
19:18:15.713 stream for epoch 11 (seed 500aef809809c6e5): root 0xcddb9bd28d420dfc 17 records, from n0

1484
docs/design/class-v5-harness/fasttime/failed-case.json Normal file
View file

File diff suppressed because it is too large Load diff

123
docs/design/class-v5-harness/fasttime/failed-case.log Normal file
View file

@ -0,0 +1,123 @@
18:49:52.656 override: 4 key(s) the binary does not know dropped (sig_scheme, sig_scheme_activation_daa, finality_succession_activation_daa, latency_ladder_cache_rung)
18:49:52.656 case failed-case: 4 nodes, bytes 6/6/7/7, ladder up/up/none/none; class v4 sub-version 3 from genesis at rung 0; class window 60 DAA x 7 (the first epoch that can cross to v5 is 8, DAA 480), ladder window 45 DAA x 7 (the first epoch that can step is 6, DAA 360); v5 floor 900 (epoch 15); stale node 3; restart node 2 (stop at DAA 455, start at 500); 60 DAA per epoch, lead 10; run 960 s or 11 epochs; node /srv/builds/igneum-wt-v5-fasttime/vendor/igneum-node-v5-fasttime/target/release/igneumd
18:49:56.844 CPU verifier on the Devnet 3 genesis pack: cli e5a4ac5978462156 pack e5a4ac5978462156 expected e5a4ac5978462156: OK
18:49:58.396 n0 up pid 1025574 json 29862 p2p 29861 exec 29863, byte 6, ladder up
18:49:59.909 n1 up pid 1026323 json 29872 p2p 29871 exec 29873, byte 6, ladder up
18:50:01.415 n2 up pid 1026530 json 29882 p2p 29881 exec 29883, byte 7, ladder none
18:50:02.920 n3 up pid 1026847 json 29892 p2p 29891 exec 29893, byte 7, ladder none
18:50:02.921 n0: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
18:50:02.921 n1: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
18:50:02.921 n2: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 7 | this node signals none
18:50:02.922 n3: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 7 | this node signals none
18:50:02.922 n0 digest: 6fa8e71d7fb9aa0d
18:50:03.934 epoch -1 -> 0 at daa 0, 7.1 s: template class 3 (next 3) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:50:03.950 t=7.1 s daa 0 epoch 0 class 3 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 0/234e082d 0/234e082d 0/234e082d 0/234e082d
18:50:19.041 t=22.2 s daa 3 epoch 0 class 3 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 3/d9eefb3f 3/d9eefb3f 3/d9eefb3f 3/d9eefb3f
18:50:34.085 t=37.2 s daa 5 epoch 0 class 3 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 5/89614064 5/89614064 5/89614064 5/89614064
18:50:49.116 t=52.3 s daa 7 epoch 0 class 3 rung 0 v5 7500 bps (weakest 0) up weakest 0 blocks/sink per node 7/bb808bf7 7/bb808bf7 7/bb808bf7 7/bb808bf7
18:51:04.157 t=67.3 s daa 15 epoch 0 class 3 rung 0 v5 6000 bps (weakest 0) up weakest 0 blocks/sink per node 15/9b443002 15/9b443002 15/9b443002 15/9b443002
18:51:19.186 t=82.3 s daa 20 epoch 0 class 3 rung 0 v5 7333 bps (weakest 0) up weakest 0 blocks/sink per node 20/49eea096 20/49eea096 20/49eea096 20/49eea096
18:51:34.222 t=97.4 s daa 37 epoch 0 class 3 rung 0 v5 6666 bps (weakest 0) up weakest 0 blocks/sink per node 37/1bd8ac51 37/1bd8ac51 37/1bd8ac51 37/1bd8ac51
18:51:49.366 epoch 0 -> 1 at daa 61, 112.5 s: template class 3 (next 3) rung 0 (27 passes, next rung 0), v5 share at the sink 5370 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:51:49.367 t=112.5 s daa 61 epoch 1 class 3 rung 0 v5 5370 bps (weakest 0) up weakest 0 blocks/sink per node 61/f72b6558 61/f72b6558 61/f72b6558 61/f72b6558
18:51:52.383 stream for epoch 1 (seed f5b449908005b8d9): root 0x69b9c20f49a7f21c 17 records, from n0
18:52:04.426 t=127.6 s daa 77 epoch 1 class 3 rung 0 v5 4833 bps (weakest 0) up weakest 0 blocks/sink per node 77/a92806a3 77/a92806a3 77/a92806a3 77/a92806a3
18:52:19.475 t=142.6 s daa 96 epoch 1 class 3 rung 0 v5 4833 bps (weakest 0) up weakest 0 blocks/sink per node 96/3b3df096 96/3b3df096 96/3b3df096 96/3b3df096
18:52:34.514 t=157.7 s daa 102 epoch 1 class 3 rung 0 v5 5000 bps (weakest 0) up weakest 0 blocks/sink per node 102/9c8ab444 102/9c8ab444 102/9c8ab444 102/9c8ab444
18:52:49.548 t=172.7 s daa 116 epoch 1 class 3 rung 0 v5 5000 bps (weakest 0) up weakest 0 blocks/sink per node 116/4aefeec9 116/4aefeec9 116/4aefeec9 116/4aefeec9
18:52:53.557 epoch 1 -> 2 at daa 120, 176.7 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 5000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:52:56.566 stream for epoch 2 (seed c1ca08dd6e1ac0a1): root 0x5b283e914f5a8ab4 17 records, from n0
18:53:04.588 t=187.7 s daa 122 epoch 2 class 4 rung 0 v5 5833 bps (weakest 0) up weakest 0 blocks/sink per node 122/f1fd6991 122/f1fd6991 122/f1fd6991 122/f1fd6991
18:53:19.619 t=202.8 s daa 142 epoch 2 class 4 rung 0 v5 6000 bps (weakest 0) up weakest 0 blocks/sink per node 142/00611ac4 142/00611ac4 142/00611ac4 142/00611ac4
18:53:34.664 t=217.8 s daa 162 epoch 2 class 4 rung 0 v5 6333 bps (weakest 0) up weakest 0 blocks/sink per node 162/744dfc04 162/744dfc04 161/c0c994ef 161/c0c994ef
18:53:49.701 t=232.9 s daa 175 epoch 2 class 4 rung 0 v5 5833 bps (weakest 0) up weakest 0 blocks/sink per node 175/fd006310 175/fd006310 175/fd006310 175/fd006310
18:53:52.709 epoch 2 -> 3 at daa 180, 235.9 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 5833 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:53:52.714 stream for epoch 3 (seed a0c9f37d64ca5a4f): root 0x2921bfa4d9bf96d0 17 records, from n0
18:54:04.771 t=247.9 s daa 183 epoch 3 class 4 rung 0 v5 5344 bps (weakest 0) up weakest 0 blocks/sink per node 183/cd1d5872 183/cd1d5872 183/cd1d5872 183/cd1d5872
18:54:19.811 t=263.0 s daa 204 epoch 3 class 4 rung 0 v5 4500 bps (weakest 0) up weakest 0 blocks/sink per node 204/2fdceba1 204/2fdceba1 204/2fdceba1 204/2fdceba1
18:54:34.854 t=278.0 s daa 222 epoch 3 class 4 rung 0 v5 4500 bps (weakest 0) up weakest 0 blocks/sink per node 222/91356a9b 222/91356a9b 222/91356a9b 222/91356a9b
18:54:46.881 epoch 3 -> 4 at daa 240, 290.0 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 4666 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:54:48.888 stream for epoch 4 (seed 8431fb1f5a5d7c00): root 0x95c89acb30737c65 17 records, from n0
18:54:49.891 t=293.0 s daa 240 epoch 4 class 4 rung 0 v5 4666 bps (weakest 0) up weakest 0 blocks/sink per node 240/6e97590a 240/6e97590a 240/6e97590a 240/6e97590a
18:55:04.994 t=308.1 s daa 248 epoch 4 class 4 rung 0 v5 4912 bps (weakest 0) up weakest 0 blocks/sink per node 248/c6f8b72b 248/c6f8b72b 248/c6f8b72b 248/c6f8b72b
18:55:20.450 t=323.6 s daa 257 epoch 4 class 4 rung 0 v5 5000 bps (weakest 0) up weakest 0 blocks/sink per node 257/2a11ade7 257/2a11ade7 257/2a11ade7 257/2a11ade7
18:55:35.786 t=338.9 s daa 265 epoch 4 class 4 rung 0 v5 5500 bps (weakest 0) up weakest 0 blocks/sink per node 265/78dfb27f 265/78dfb27f 265/78dfb27f 265/78dfb27f
18:55:50.980 t=354.1 s daa 284 epoch 4 class 4 rung 0 v5 5000 bps (weakest 0) up weakest 4000 blocks/sink per node 284/06c888c6 284/06c888c6 284/06c888c6 284/06c888c6
18:56:06.351 t=369.5 s daa 287 epoch 4 class 4 rung 0 v5 4500 bps (weakest 0) up weakest 3777 blocks/sink per node 287/13a04ecc 287/13a04ecc 287/13a04ecc 287/13a04ecc
18:56:21.652 t=384.8 s daa 298 epoch 4 class 4 rung 0 v5 4333 bps (weakest 0) up weakest 3777 blocks/sink per node 298/1b5abaa3 298/1b5abaa3 298/1b5abaa3 298/1b5abaa3
18:56:23.662 epoch 4 -> 5 at daa 300, 386.8 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 4333 bps (weakest of 7: 0), ladder up weakest 4000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:56:26.671 stream for epoch 5 (seed 3dbc67dbab8b0eb7): root 0xf0db5830942a19dd 17 records, from n0
18:56:36.694 t=399.8 s daa 303 epoch 5 class 4 rung 0 v5 4482 bps (weakest 0) up weakest 4000 blocks/sink per node 303/f0f22445 303/f0f22445 303/f0f22445 303/f0f22445
18:56:51.732 t=414.9 s daa 341 epoch 5 class 4 rung 0 v5 4666 bps (weakest 0) up weakest 4000 blocks/sink per node 341/57523cbb 341/57523cbb 341/57523cbb 341/57523cbb
18:57:03.760 epoch 5 -> 6 at daa 360, 426.9 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 5166 bps (weakest of 7: 0), ladder up weakest 3777 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:57:06.768 stream for epoch 6 (seed cc1c2be8363ee669): root 0xf00e490c8f9e6b7b 17 records, from n0
18:57:06.775 t=429.9 s daa 360 epoch 6 class 4 rung 0 v5 5166 bps (weakest 0) up weakest 3777 blocks/sink per node 360/af0274de 360/af0274de 360/af0274de 360/af0274de
18:57:21.851 t=445.0 s daa 363 epoch 6 class 4 rung 0 v5 4912 bps (weakest 0) up weakest 3777 blocks/sink per node 363/bad28793 365/2e337021 363/bad28793 363/bad28793
18:57:36.957 t=460.1 s daa 388 epoch 6 class 4 rung 0 v5 5500 bps (weakest 4500) up weakest 4444 blocks/sink per node 388/799e5344 388/799e5344 388/799e5344 388/799e5344
18:57:52.005 t=475.2 s daa 411 epoch 6 class 4 rung 0 v5 5500 bps (weakest 4500) up weakest 4444 blocks/sink per node 411/736f4720 411/736f4720 411/736f4720 411/736f4720
18:58:07.125 t=490.3 s daa 418 epoch 6 class 4 rung 0 v5 6000 bps (weakest 4333) up weakest 4000 blocks/sink per node 418/10827166 418/10827166 418/10827166 418/10827166
18:58:13.197 epoch 6 -> 7 at daa 420, 496.4 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 6000 bps (weakest of 7: 4333), ladder up weakest 3953 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:58:15.228 stream for epoch 7 (seed 11d92497b13d2123): root 0x400a2286f9564b97 17 records, from n0
18:58:22.416 t=505.6 s daa 420 epoch 7 class 4 rung 0 v5 5789 bps (weakest 4426) up weakest 3953 blocks/sink per node 420/7d89f8b5 420/7d89f8b5 420/7d89f8b5 420/7d89f8b5
18:58:37.482 t=520.6 s daa 426 epoch 7 class 4 rung 0 v5 5833 bps (weakest 4406) up weakest 3809 blocks/sink per node 426/840c840c 422/f75d16a3 426/840c840c 426/840c840c
18:58:52.564 t=535.7 s daa 453 epoch 7 class 4 rung 0 v5 5500 bps (weakest 4500) up weakest 4000 blocks/sink per node 453/619b0c51 453/619b0c51 453/619b0c51 453/619b0c51
18:58:55.581 RESTART STEP: n2 and its miner stopped at daa 455 (538.7 s), 25 DAA before the expected v5 boundary 480
18:59:07.613 t=550.8 s daa 464 epoch 7 class 4 rung 0 v5 6333 bps (weakest 4406) up weakest 4000 blocks/sink per node 464/59002a68 464/59002a68 down 464/59002a68
18:59:21.654 epoch 7 -> 8 at daa 480, 564.8 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 6333 bps (weakest of 7: 4237), ladder up weakest 4444 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
18:59:22.656 t=565.8 s daa 480 epoch 8 class 4 rung 0 v5 6333 bps (weakest 4237) up weakest 4444 blocks/sink per node 480/e28ec830 480/e28ec830 down 480/e28ec830
18:59:24.662 stream for epoch 8 (seed 80d0a620a6ecebf4): root 0x23f3a5625aecf82b 17 records, from n0
18:59:37.695 t=580.9 s daa 486 epoch 8 class 4 rung 0 v5 6949 bps (weakest 4406) up weakest 4347 blocks/sink per node 486/b694cd0f 482/e28ec830 down 486/b694cd0f
18:59:46.244 n2 RESTARTED pid 1096346 json 29882 p2p 29881 exec 29883, byte 7, ladder none (own datadir kept)
18:59:46.249 RESTART STEP: n2 started on its own datadir at daa 501 (589.4 s), 21 DAA after the boundary; its miner restarted with its key
18:59:53.286 t=596.4 s daa 505 epoch 8 class 4 rung 0 v5 7333 bps (weakest 4500) up weakest 4444 blocks/sink per node 505/0f80d069 505/0f80d069 456/f6d64bdb 505/0f80d069
19:00:01.331 RESTART STEP: n2 at n0's sink 957b89377fe547b6 (519 blocks) 15.1 s after its start
19:00:08.347 t=611.5 s daa 527 epoch 8 class 4 rung 0 v5 7586 bps (weakest 4333) up weakest 4583 blocks/sink per node 527/29b1d727 527/29b1d727 527/29b1d727 527/29b1d727
19:00:24.128 t=627.3 s daa 537 epoch 8 class 4 rung 0 v5 7241 bps (weakest 4166) up weakest 4000 blocks/sink per node 537/11ebd781 537/11ebd781 537/11ebd781 537/11ebd781
19:00:30.219 epoch 8 -> 9 at daa 540, 633.4 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 7241 bps (weakest of 7: 4333), ladder up weakest 3777 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:00:30.227 stream for epoch 9 (seed 2c34fdd80a5de8d8): root 0xf9458bd37c65f386 17 records, from n0
19:00:39.260 t=642.4 s daa 542 epoch 9 class 4 rung 0 v5 6964 bps (weakest 4426) up weakest 3777 blocks/sink per node 542/ef2a92b9 542/ef2a92b9 542/ef2a92b9 542/ef2a92b9
19:00:54.301 t=657.5 s daa 567 epoch 9 class 4 rung 0 v5 6896 bps (weakest 4500) up weakest 4222 blocks/sink per node 567/47ef8e5f 567/47ef8e5f 567/47ef8e5f 567/47ef8e5f
19:01:09.338 t=672.5 s daa 590 epoch 9 class 4 rung 0 v5 6333 bps (weakest 4237) up weakest 4222 blocks/sink per node 590/a1afb013 590/a1afb013 590/a1afb013 590/a1afb013
19:01:21.371 epoch 9 -> 10 at daa 600, 684.5 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 6333 bps (weakest of 7: 4166), ladder up weakest 5111 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:01:22.402 stream for epoch 10 (seed a1afb01312e90a4f): root 0x87714d81b7e08c17 17 records, from n0
19:01:24.407 t=687.6 s daa 600 epoch 10 class 4 rung 0 v5 6333 bps (weakest 4166) up weakest 5111 blocks/sink per node 600/30ce4a24 600/30ce4a24 600/30ce4a24 600/30ce4a24
19:01:39.449 t=702.6 s daa 620 epoch 10 class 4 rung 0 v5 5500 bps (weakest 4666) up weakest 4651 blocks/sink per node 620/77292bfc 620/77292bfc 620/77292bfc 620/77292bfc
19:01:54.504 t=717.7 s daa 638 epoch 10 class 4 rung 0 v5 5666 bps (weakest 4166) up weakest 4888 blocks/sink per node 638/0bd164ff 638/0bd164ff 638/0bd164ff 638/0bd164ff
19:02:09.564 t=732.7 s daa 659 epoch 10 class 4 rung 0 v5 5000 bps (weakest 4333) up weakest 5333 blocks/sink per node 659/26536916 659/26536916 659/26536916 659/26536916
19:02:10.568 epoch 10 -> 11 at daa 660, 733.7 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 5000 bps (weakest of 7: 4333), ladder up weakest 5111 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
19:02:10.586 stream for epoch 11 (seed d821a50c62df258f): root 0x5903044325aaf3af 17 records, from n0
19:04:01.533 SUMMARY FAIL (failed-case) 2026-10-07T19:04:01.528Z: no rung step; no v5 epoch; epochs e0:v3:r0:0bps e1:v3:r0:0bps e2:v4:r0:0bps e3:v4:r0:0bps e4:v4:r0:0bps e5:v4:r0:0bps e6:v4:r0:0bps e7:v4:r0:4333bps e8:v4:r0:4237bps e9:v4:r0:4333bps e10:v4:r0:4166bps e11:v4:r0:4333bps; chain bytes {"0":1,"6":369,"7":291} (5582 bps at byte 6), ladder bits {"none":292,"up":369}; blocks 485 / 176 across DAA 480; rejected miners 0/0/0/0 nodes 0/0/0/0; accepted after the flip ///; sinks 4b4d2285243d00a4 4b4d2285243d00a4 4b4d2285243d00a4 4b4d2285243d00a4 (agree) at 660/660/660/660; step lines 0/4 (epochs -/-/-/-), v5 signal lines 0/4 (epochs -/-/-/-, weakest -/-/-/-); stale n3 accepted after the first refresh n/a rejected 0; restart n2 stopped at DAA 455, started at DAA 501, resynced 15.1 s later, lock faults 0, accepted after 58; dn3 id OK
19:04:01.533 PROGRAM ID epoch 0 v3 rung 0 (27 passes) seed 234e082d653dc69d: miners 736e1544a578165e (4 of 4) cli 83dab9f31f2e2dba other-class 343287b6df5996ed MISMATCH
19:04:01.533 PROGRAM ID epoch 1 v3 rung 0 (27 passes) seed f5b449908005b8d9: miners 3795b871498fca4f (4 of 4) cli 66b9498fd159346d other-class 324e660e35997420 state 0x69b9c20f49a7f21c (17 records) MISMATCH
19:04:01.533 PROGRAM ID epoch 2 v4 rung 0 (27 passes) seed c1ca08dd6e1ac0a1: miners 5d46d36e92bc4963 (4 of 4) cli 5d46d36e92bc4963 other-class 6bbd8a57c5d758f2 state 0x5b283e914f5a8ab4 (17 records) OK
19:04:01.533 PROGRAM ID epoch 3 v4 rung 0 (27 passes) seed a0c9f37d64ca5a4f: miners 6ba924bdb96aff91 (4 of 4) cli 6ba924bdb96aff91 other-class e1f249833988bee8 state 0x2921bfa4d9bf96d0 (17 records) OK
19:04:01.533 PROGRAM ID epoch 4 v4 rung 0 (27 passes) seed 8431fb1f5a5d7c00: miners 5d82434d62649f19 (4 of 4) cli 5d82434d62649f19 other-class 2e75be8c92311d6c state 0x95c89acb30737c65 (17 records) OK
19:04:01.533 PROGRAM ID epoch 5 v4 rung 0 (27 passes) seed 3dbc67dbab8b0eb7: miners 94a7615748b3a7b9 (4 of 4) cli 94a7615748b3a7b9 other-class c28758ddda1b7258 state 0xf0db5830942a19dd (17 records) OK
19:04:01.533 PROGRAM ID epoch 6 v4 rung 0 (27 passes) seed cc1c2be8363ee669: miners be7db3263c95f529 (4 of 4) cli be7db3263c95f529 other-class 1efd27c008b2eb1c state 0xf00e490c8f9e6b7b (17 records) OK
19:04:01.533 PROGRAM ID epoch 7 v4 rung 0 (27 passes) seed 11d92497b13d2123: miners 01912cd0c89b2200 (4 of 4) cli 01912cd0c89b2200 other-class 51134da8d0547c27 state 0x400a2286f9564b97 (17 records) OK
19:04:01.533 PROGRAM ID epoch 8 v4 rung 0 (27 passes) seed 80d0a620a6ecebf4: miners 6bd9e774d0c78b4e (4 of 4) cli 6bd9e774d0c78b4e other-class 753558f23d83b45d state 0x23f3a5625aecf82b (17 records) OK
19:04:01.533 PROGRAM ID epoch 9 v4 rung 0 (27 passes) seed 2c34fdd80a5de8d8: miners e6a63807fa80cae2 (4 of 4) cli e6a63807fa80cae2 other-class 40292dc9af89ccfd state 0xf9458bd37c65f386 (17 records) OK
19:04:01.533 PROGRAM ID epoch 10 v4 rung 0 (27 passes) seed a1afb01312e90a4f: miners 07b2b938722d0700 (4 of 4) cli 07b2b938722d0700 other-class 6183a23bce607b93 state 0x87714d81b7e08c17 (17 records) OK
19:04:01.533 FAILED CHECK started_on_class_v4_at_rung_0
19:04:01.533 FAILED CHECK template_stepped_to_rung_1
19:04:01.533 FAILED CHECK stepped_at_the_first_full_window_epoch
19:04:01.533 FAILED CHECK step_line_on_every_node_same_epoch
19:04:01.533 FAILED CHECK step_weakest_up_at_or_above_threshold
19:04:01.533 FAILED CHECK no_second_step
19:04:01.533 FAILED CHECK rung1_ids_equal_the_cli_rung1_id
19:04:01.533 FAILED CHECK template_switched_to_v5
19:04:01.533 FAILED CHECK crossed_at_the_first_full_window_epoch
19:04:01.533 FAILED CHECK crossed_after_the_rung_step
19:04:01.533 FAILED CHECK crossed_before_the_floor
19:04:01.534 FAILED CHECK signal_line_on_every_honest_node_same_epoch
19:04:01.534 FAILED CHECK signal_weakest_at_or_above_threshold
19:04:01.534 FAILED CHECK byte_6_counted_exactly_on_chain
19:04:01.534 FAILED CHECK v5_rides_at_rung_1
19:04:01.534 FAILED CHECK a_stream_for_every_v5_epoch
19:04:01.534 FAILED CHECK v5_ids_equal_the_cli_v5_id
19:04:01.534 FAILED CHECK v5_ids_differ_from_the_same_seed_v4_id
19:04:01.534 FAILED CHECK v5_ids_differ_from_the_rung0_v5_id
19:04:01.534 FAILED CHECK a_v5_epoch_per_window_refresh
19:04:01.534 FAILED CHECK honest_miners_accepted_after_the_flip
19:04:01.534 FAILED CHECK stale_node_refused_from_the_first_refresh
19:04:01.534 summary: /tmp/igneum-fast-time-v5x-failed-case/summary.json and /srv/builds/igneum-wt-v5-fasttime/docs/design/class-v5-harness/fasttime/failed-case.json

15
infra/fast-time/class-v5-crossing.mjs
View file

@ -213,7 +213,8 @@ let flip = null, step = null, lastEpoch = -1, lastReport = 0, lastDaa = 0, endAt
const restart = { stopped: null, started: null, resynced: null };
const samples = [];
async function fetchStream(node, seedHash) {
const r = await fetch(node.execRpc, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'igneum_getPowStateLeaves', params: [seedHash] }) });
// five seconds: the restarted node's exec RPC hung a fetch at 19:18Z on 7 October and the end-of-run reads with it
const r = await fetch(node.execRpc, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'igneum_getPowStateLeaves', params: [seedHash] }), signal: AbortSignal.timeout(5000) });
const j = await r.json();
if (!j.result?.streamHex) throw new Error(JSON.stringify(j.error || j).slice(0, 160));
return j.result;
@ -278,6 +279,8 @@ while (Date.now() - t0 < SECS * 1000) {
}
await sleep(3000);
await pollStreams();
// the end-of-run reads under a cap: a hung RPC must not hold the box (the first cross run, 19:18Z)
setTimeout(async () => { log('FAILED: the end-of-run reads did not finish inside 180 s'); await finish(5); }, 180_000).unref();
// ---- the end-of-run reads ------------------------------------------------------------------------------------------
const dag = await Promise.all(nodes.map(n => n.rpc ? n.dag() : Promise.resolve({ error: 'down' })));
@ -354,6 +357,11 @@ const epochRows = [...epochs.entries()].sort((a, b) => a[0] - b[0]);
const afterStep = step ? epochRows.filter(([e]) => e > step.epoch).map(([, v]) => v.rung) : [];
const v5Epochs = epochRows.filter(([, v]) => v.class === 5);
const restartFaults = restartNode && restart.started && !restart.started.failed ? restartNode.grepLog(LOCK_FAULT, restartNode.restartLine) : (restartNode?.grepLog(LOCK_FAULT) ?? []);
// the first cross run (19:14Z): the restarted node's IBD aborted on every try with "proof-of-work check refused, class v5 needs
// the execution state after the epoch's seed block ... no published state stream" (headers-first IBD validates the v5 headers
// before its executor has passed the seed block it stopped short of), and with unsynced mining on it mined its own fork
const IBD_REFUSED = /IBD with peer .* completed with error|proof-of-work check refused|class v5 needs the execution state/;
const restartIbdRefusals = restartNode && restart.started && !restart.started.failed ? restartNode.grepLog(IBD_REFUSED, restartNode.restartLine) : [];
const restartAcceptedAfter = restartNode && restart.started ? acceptedAfter(RESTART, restart.started.wall) : null;
// the stream roots agree across the executing honest nodes wherever two or more answered
const rootRows = [...streams.entries()].map(([e, st]) => ({ epoch: +e, roots: Object.values(st.roots) })).filter(r => r.roots.length >= 2);
@ -405,6 +413,7 @@ const checks = {
restarted_across_the_boundary: restartNode == null ? null : (!!restart.stopped && !!restart.started && !restart.started.failed && restart.stopped.daa < BOUNDARY && restart.started.daa >= BOUNDARY),
no_lock_fault_after_the_restart: restartNode == null ? null : (!!restart.started && !restart.started.failed && restartFaults.length === 0),
restarted_node_resynced: restartNode == null ? null : (!!restart.resynced && sinks[RESTART] === sinks[0]),
restarted_node_ibd_not_refused: restartNode == null ? null : (!!restart.started && !restart.started.failed && restartIbdRefusals.length === 0),
restarted_node_mines_v5_after_the_restart: restartNode == null ? null : (restartAcceptedAfter > 0 && rejectedNode[RESTART] === 0),
};
const pass = Object.values(checks).every(v => v === true || v === null);
@ -416,12 +425,12 @@ const summary = {
blocks: { total: blocks.length, before_boundary: before.length, after_boundary: after.length, object_bytes: objectBytes, byte_6_share_bps: byte6Share, ladder_bits: ladderBits, up_share_bps: upShare },
program_id_rows: idRows, accepted_per_miner: accepted, accepted_after_flip_per_miner: acceptedAfterFlip, rejected_by_miners: rejectedMiner, rejected_by_nodes: rejectedNode,
stale: stale ? { node: STALE, accepted_total: accepted[STALE], accepted_after_first_refresh: staleAcceptedAfterRefresh, rejected_by_its_node: rejectedNode[STALE], rejected_lines: rejectedMiner[STALE], first_refresh_epoch: flip ? flip.epoch + 1 : null } : null,
restart: restartNode ? { node: RESTART, ...restart, lock_fault_lines: restartFaults.slice(0, 5), accepted_after_restart: restartAcceptedAfter, rejected_by_its_node: rejectedNode[RESTART] } : null,
restart: restartNode ? { node: RESTART, ...restart, lock_fault_lines: restartFaults.slice(0, 5), ibd_refusal_lines: restartIbdRefusals.length, ibd_refusal_example: restartIbdRefusals[0]?.replace(/^.*?\] /, '').slice(0, 300) ?? null, accepted_after_restart: restartAcceptedAfter, rejected_by_its_node: rejectedNode[RESTART] } : null,
sinks, block_counts: counts, step_lines: stepLines.map(ls => ls.map(l => l.slice(0, 300))), signal_lines: signalLines.map(ls => ls.map(l => l.slice(0, 300))), floor_lines: floorLines, samples,
};
writeFileSync(`${TMP}/summary.json`, JSON.stringify(summary, null, 2));
try { mkdirSync(OUT, { recursive: true }); writeFileSync(`${OUT}/${CASE}.json`, JSON.stringify(summary, null, 2)); } catch (e) { log(`could not write ${OUT}/${CASE}.json: ${e.message}`); }
log(`SUMMARY ${pass ? 'PASS' : 'FAIL'} (${CASE}) ${summary.utc}: ${step ? `rung ${step.rung} (${step.reps} passes) from epoch ${step.epoch} at DAA ${step.daa}` : 'no rung step'}; ${flip ? `v5 from epoch ${flip.epoch} at DAA ${flip.daa} at rung ${flip.rung}` : 'no v5 epoch'}; epochs ${epochRows.map(([e, v]) => `e${e}:v${v.class}:r${v.rung}:${v.v5_weakest_bps}bps`).join(' ')}; chain bytes ${JSON.stringify(objectBytes)} (${byte6Share} bps at byte 6), ladder bits ${JSON.stringify(ladderBits)}; blocks ${before.length} / ${after.length} across DAA ${BOUNDARY}; rejected miners ${rejectedMiner.join('/')} nodes ${rejectedNode.join('/')}; accepted after the flip ${acceptedAfterFlip.join('/')}; sinks ${sinks.join(' ')} (${checks.sinks_agree ? 'agree' : 'DIFFER'}) at ${counts.join('/')}; step lines ${stepParsed.filter(Boolean).length}/${N} (epochs ${stepParsed.map(s => s?.epoch ?? '-').join('/')}), v5 signal lines ${signalParsed.filter(Boolean).length}/${N} (epochs ${signalParsed.map(s => s?.epoch ?? '-').join('/')}, weakest ${signalParsed.map(s => s?.weakest ?? '-').join('/')}); stale n${STALE ?? '-'} accepted after the first refresh ${staleAcceptedAfterRefresh ?? 'n/a'} rejected ${STALE != null ? rejectedNode[STALE] : 'n/a'}; restart n${RESTART ?? '-'} ${restart.stopped ? `stopped at DAA ${restart.stopped.daa}` : 'not stopped'}${restart.started ? `, started at DAA ${restart.started.daa}` : ''}${restart.resynced ? `, resynced ${restart.resynced.after_s} s later` : ''}, lock faults ${restartFaults.length}, accepted after ${restartAcceptedAfter ?? 'n/a'}; dn3 id ${dn3 ? (dn3.ok ? 'OK' : 'MISMATCH') : 'skipped'}`);
log(`SUMMARY ${pass ? 'PASS' : 'FAIL'} (${CASE}) ${summary.utc}: ${step ? `rung ${step.rung} (${step.reps} passes) from epoch ${step.epoch} at DAA ${step.daa}` : 'no rung step'}; ${flip ? `v5 from epoch ${flip.epoch} at DAA ${flip.daa} at rung ${flip.rung}` : 'no v5 epoch'}; epochs ${epochRows.map(([e, v]) => `e${e}:v${v.class}:r${v.rung}:${v.v5_weakest_bps}bps`).join(' ')}; chain bytes ${JSON.stringify(objectBytes)} (${byte6Share} bps at byte 6), ladder bits ${JSON.stringify(ladderBits)}; blocks ${before.length} / ${after.length} across DAA ${BOUNDARY}; rejected miners ${rejectedMiner.join('/')} nodes ${rejectedNode.join('/')}; accepted after the flip ${acceptedAfterFlip.join('/')}; sinks ${sinks.join(' ')} (${checks.sinks_agree ? 'agree' : 'DIFFER'}) at ${counts.join('/')}; step lines ${stepParsed.filter(Boolean).length}/${N} (epochs ${stepParsed.map(s => s?.epoch ?? '-').join('/')}), v5 signal lines ${signalParsed.filter(Boolean).length}/${N} (epochs ${signalParsed.map(s => s?.epoch ?? '-').join('/')}, weakest ${signalParsed.map(s => s?.weakest ?? '-').join('/')}); stale n${STALE ?? '-'} accepted after the first refresh ${staleAcceptedAfterRefresh ?? 'n/a'} rejected ${STALE != null ? rejectedNode[STALE] : 'n/a'}; restart n${RESTART ?? '-'} ${restart.stopped ? `stopped at DAA ${restart.stopped.daa}` : 'not stopped'}${restart.started ? `, started at DAA ${restart.started.daa}` : ''}${restart.resynced ? `, resynced ${restart.resynced.after_s} s later` : ''}, lock faults ${restartFaults.length}, IBD refusals ${restartIbdRefusals.length}, accepted after ${restartAcceptedAfter ?? 'n/a'}; dn3 id ${dn3 ? (dn3.ok ? 'OK' : 'MISMATCH') : 'skipped'}`);
for (const r of idRows) log(`PROGRAM ID epoch ${r.epoch} ${r.class} rung ${r.rung ?? '?'} (${r.reps ?? '?'} passes) seed ${r.seed}: miners ${r.miners_id} (${r.miners} of ${N}${r.disagree ? ', DISAGREE' : ''}) cli ${r.cli} other-class ${r.cli_other}${r.cli_rung0 ? ` rung0 ${r.cli_rung0}` : ''}${r.state_root ? ` state ${String(r.state_root).slice(0, 18)} (${r.state_records} records)` : ''} ${r.cli === r.miners_id ? 'OK' : 'MISMATCH'}`);
for (const [k, v] of Object.entries(checks)) if (v === false) log(`FAILED CHECK ${k}`);
log(`summary: ${TMP}/summary.json and ${OUT}/${CASE}.json`);