igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity

fast-time crossing on pair 63524e28 (20:19:59Z to 20:34:12Z): rung 1 at epoch 6, v5 at epoch 8 at rung 1, the stale node refused 132 of 132, the prelude v4; the restart step: the relay refusal gone, the IBD catch-up engaged and never completed because the node's own miner, restarted at once with unsynced mining on, extended its branch from the split and the executor followed it; the harness holds the restarted node's miner until its sink equals n0's (cap 300 s) and reads the catch-up done line

Browse source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-07 20:35:41 +00:00
parent de213c19b5
commit a73898a90c
3 changed files with 1673 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1548
docs/design/class-v5-harness/fasttime/cross-63524e28.json Normal file
View file

File diff suppressed because it is too large Load diff

108
docs/design/class-v5-harness/fasttime/cross-63524e28.log Normal file
View file

@ -0,0 +1,108 @@
20:19:59.037 override: every key known to the binary
20:19:59.039 case cross-63524e28: 4 nodes, bytes 6/6/6/6, ladder up/up/up/up; class v4 sub-version 3 from genesis at rung 0; class window 60 DAA x 7 (the first epoch that can cross to v5 is 8, DAA 480), ladder window 45 DAA x 7 (the first epoch that can step is 6, DAA 360); v5 floor 900 (epoch 15); stale node 3; restart node 2 (stop at DAA 455, start at 500); 60 DAA per epoch, lead 10; run 960 s or 11 epochs; node /srv/artefacts/v5-pair-63524e28/igneumd
20:20:03.371 CPU verifier on the Devnet 3 genesis pack: cli e5a4ac5978462156 pack e5a4ac5978462156 expected e5a4ac5978462156: OK
20:20:04.927 n0 up pid 1819608 json 29862 p2p 29861 exec 29863, byte 6, ladder up
20:20:06.438 n1 up pid 1820581 json 29872 p2p 29871 exec 29873, byte 6, ladder up
20:20:07.944 n2 up pid 1820976 json 29882 p2p 29881 exec 29883, byte 6, ladder up
20:20:09.449 n3 up pid 1821375 json 29892 p2p 29891 exec 29893, byte 6, ladder up
20:20:09.450 n0: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
20:20:09.450 n1: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
20:20:09.450 n2: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
20:20:09.450 n3: Program class v5 from the override file: enabled, the floor at epoch 15 (DAA score 900 rounded up to the epoch boundary at 900); the class v | this node signals object version 6 | this node signals up
20:20:09.450 n0 digest: 6fa8e71d7fb9aa0d
20:20:10.462 epoch -1 -> 0 at daa 0, 7.1 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
20:20:10.488 t=7.1 s daa 0 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 0/234e082d 0/234e082d 0/234e082d 0/234e082d
20:20:25.537 t=22.2 s daa 0 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 0/234e082d 0/234e082d 0/234e082d 0/234e082d
20:20:40.581 t=37.2 s daa 6 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 6/bf9d3893 6/bf9d3893 6/bf9d3893 6/bf9d3893
20:20:55.619 t=52.2 s daa 8 epoch 0 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 8/b3ebb4f3 8/b3ebb4f3 8/b3ebb4f3 8/b3ebb4f3
20:21:10.655 t=67.3 s daa 11 epoch 0 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 11/5f433a9d 11/5f433a9d 11/5f433a9d 11/5f433a9d
20:21:25.688 t=82.3 s daa 13 epoch 0 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 13/9fee3e7e 13/9fee3e7e 13/9fee3e7e 13/9fee3e7e
20:21:40.724 t=97.4 s daa 19 epoch 0 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 19/40be078d 19/40be078d 19/40be078d 19/40be078d
20:21:55.761 t=112.4 s daa 37 epoch 0 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 37/eea0659c 37/eea0659c 37/eea0659c 37/eea0659c
20:22:09.784 epoch 0 -> 1 at daa 60, 126.4 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
20:22:10.792 stream for epoch 1 (seed 625a96ec783479e4): root 0xa955444f8e5d0dcc 16 records, from n0
20:22:10.797 t=127.4 s daa 60 epoch 1 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 60/dbccaf52 60/dbccaf52 60/dbccaf52 60/dbccaf52
20:22:25.835 t=142.5 s daa 74 epoch 1 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 74/3ec27fce 73/3ec27fce 73/3ec27fce 65/a31d5de7
20:22:40.869 t=157.5 s daa 87 epoch 1 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 87/774f9e3b 87/774f9e3b 87/774f9e3b 87/774f9e3b
20:22:55.907 t=172.5 s daa 106 epoch 1 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 106/18947ed6 106/18947ed6 106/18947ed6 106/18947ed6
20:23:06.925 epoch 1 -> 2 at daa 120, 183.6 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
20:23:06.930 stream for epoch 2 (seed 10d331863788ace9): root 0xd00cb0e4afc17720 17 records, from n0
20:23:10.947 t=187.6 s daa 122 epoch 2 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 122/f1087793 122/f1087793 122/f1087793 122/f1087793
20:23:25.989 t=202.6 s daa 141 epoch 2 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 141/28b3e442 141/28b3e442 141/28b3e442 141/28b3e442
20:23:41.031 t=217.7 s daa 162 epoch 2 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 162/ad45a3c3 162/ad45a3c3 162/ad45a3c3 162/ad45a3c3
20:23:56.074 t=232.7 s daa 175 epoch 2 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 175/9d02d409 175/9d02d409 175/9d02d409 175/9d02d409
20:24:07.095 epoch 2 -> 3 at daa 180, 243.7 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
20:24:07.102 stream for epoch 3 (seed a75213ceb811abad): root 0x6888bdeb6d14cad6 17 records, from n0
20:24:11.120 t=247.7 s daa 180 epoch 3 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 180/a8887618 180/a8887618 180/a8887618 180/a8887618
20:24:26.153 t=262.8 s daa 186 epoch 3 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 186/ba589261 186/ba589261 186/ba589261 182/f071a047
20:24:41.193 t=277.8 s daa 206 epoch 3 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 206/c78bd601 206/c78bd601 206/c78bd601 206/c78bd601
20:24:56.238 t=292.9 s daa 222 epoch 3 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 222/c81c5f01 222/c81c5f01 222/c81c5f01 222/c81c5f01
20:25:09.278 epoch 3 -> 4 at daa 240, 305.9 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
20:25:11.285 stream for epoch 4 (seed 6bc39c0fed38c1c7): root 0xae27f1438d314ff3 17 records, from n0
20:25:11.287 t=307.9 s daa 240 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 240/b1c24a5c 240/b1c24a5c 240/b1c24a5c 240/b1c24a5c
20:25:26.324 t=323.0 s daa 254 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 0 blocks/sink per node 254/f15bd440 254/f15bd440 254/f15bd440 254/f15bd440
20:25:41.369 t=338.0 s daa 274 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 274/ac0b69be 274/ac0b69be 274/ac0b69be 274/ac0b69be
20:25:56.408 t=353.0 s daa 292 epoch 4 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 292/27b33be6 292/27b33be6 292/27b33be6 292/27b33be6
20:26:06.426 epoch 4 -> 5 at daa 300, 363.1 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 0
20:26:07.435 stream for epoch 5 (seed 0a07dca05d53beb1): root 0xe2164c5c9665c4b7 17 records, from n0
20:26:11.448 t=368.1 s daa 302 epoch 5 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 302/cc2d11a2 302/cc2d11a2 302/cc2d11a2 302/cc2d11a2
20:26:26.499 t=383.1 s daa 315 epoch 5 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 315/781e3dbd 315/781e3dbd 315/781e3dbd 315/781e3dbd
20:26:41.536 t=398.2 s daa 332 epoch 5 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 332/5c6a9a53 332/5c6a9a53 332/5c6a9a53 332/5c6a9a53
20:26:56.578 t=413.2 s daa 349 epoch 5 class 4 rung 0 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 349/e8dd041d 350/2f47a974 349/e8dd041d 349/e8dd041d
20:27:04.598 epoch 5 -> 6 at daa 360, 421.2 s: template class 4 (next 4) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
20:27:04.599 LADDER STEP: the template is rung 1 (35 shadow passes) from epoch 6 (daa 360) at 421.2 s wall
20:27:07.609 stream for epoch 6 (seed 2f47a9742c88e289): root 0xd1aa568c26ac7b3b 17 records, from n0
20:27:11.623 t=428.3 s daa 363 epoch 6 class 4 rung 1 v5 10000 bps (weakest 0) up weakest 10000 blocks/sink per node 363/d72a73bb 363/d72a73bb 363/d72a73bb 363/d72a73bb
20:27:26.658 t=443.3 s daa 381 epoch 6 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 381/b27f7832 381/b27f7832 381/b27f7832 381/b27f7832
20:27:41.689 t=458.3 s daa 404 epoch 6 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 404/3e9c16ab 404/3e9c16ab 404/3e9c16ab 404/3e9c16ab
20:27:56.751 epoch 6 -> 7 at daa 421, 473.4 s: template class 4 (next 4) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
20:27:56.753 t=473.4 s daa 421 epoch 7 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 421/cbbb1809 421/cbbb1809 421/cbbb1809 421/cbbb1809
20:27:59.767 stream for epoch 7 (seed 6b977f4d0016495d): root 0x83da097a19d54213 17 records, from n0
20:28:11.805 t=488.4 s daa 427 epoch 7 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 427/f9cd53e0 427/f9cd53e0 427/f9cd53e0 427/f9cd53e0
20:28:26.833 t=503.5 s daa 452 epoch 7 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 452/633bb74b 452/633bb74b 452/633bb74b 452/633bb74b
20:28:29.847 RESTART STEP: n2 and its miner stopped at daa 456 (506.5 s), 24 DAA before the expected v5 boundary 480
20:28:41.878 t=518.5 s daa 468 epoch 7 class 4 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 468/4ce9e243 468/4ce9e243 down 468/4ce9e243
20:28:53.898 epoch 7 -> 8 at daa 480, 530.5 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch 8, step at epoch 6
20:28:53.898 CLASS SWITCH: the template is class v5 from epoch 8 (daa 480) at rung 1 (35 passes), 530.5 s wall
20:28:55.905 stream for epoch 8 (seed 2f7944518dd2ad3b): root 0x234a1f9127c9c03b 17 records, from n0
20:28:56.906 t=533.5 s daa 480 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 480/eef725c5 480/eef725c5 down 480/eef725c5
20:29:11.938 t=548.6 s daa 487 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 487/7cc69b2b 487/7cc69b2b down 487/7cc69b2b
20:29:20.473 n2 RESTARTED pid 1935290 json 29882 p2p 29881 exec 29883, byte 6, ladder up (own datadir kept)
20:29:20.478 RESTART STEP: n2 started on its own datadir at daa 500 (557.1 s), 20 DAA after the boundary; its miner restarted with its key
20:29:27.512 t=564.1 s daa 510 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 510/235198aa 510/235198aa 456/31cb84da 510/235198aa
20:29:42.568 t=579.2 s daa 528 epoch 8 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 528/ea673569 528/ea673569 459/d67fa17e 528/ea673569
20:29:57.620 epoch 8 -> 9 at daa 540, 594.2 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
20:29:57.628 stream for epoch 9 (seed 1161f6e6a0b99b13): root 0x97edd42aa6892428 17 records, from n0
20:29:57.631 t=594.3 s daa 540 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 540/506de4de 540/506de4de 467/2c0a4d36 540/506de4de
20:30:12.705 t=609.3 s daa 544 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 544/591f9b61 544/591f9b61 480/5c6c258f 544/591f9b61
20:30:27.761 t=624.4 s daa 561 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 561/84216ce0 561/84216ce0 484/63497de0 561/84216ce0
20:30:42.831 t=639.5 s daa 576 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 576/03baf326 576/03baf326 506/aa98decc 576/03baf326
20:30:57.876 t=654.5 s daa 591 epoch 9 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 592/f8bc65b0 592/f8bc65b0 518/55ab116d 591/f1518b75
20:31:00.879 epoch 9 -> 10 at daa 600, 657.5 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
20:31:01.889 stream for epoch 10 (seed 1588ab6d08450df2): root 0x04fcc1dc625f7d0a 17 records, from n0
20:31:12.931 t=669.6 s daa 603 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 603/94b5b4f2 601/f761ab9c 536/d1da20d5 603/94b5b4f2
20:31:28.003 t=684.6 s daa 619 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 619/200fcc10 619/200fcc10 543/0b801fc8 619/200fcc10
20:31:43.069 t=699.7 s daa 639 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 639/ba88533f 639/ba88533f 563/49843199 639/ba88533f
20:31:58.130 t=714.8 s daa 655 epoch 10 class 5 rung 1 v5 10000 bps (weakest 10000) up weakest 10000 blocks/sink per node 655/c3cff9ec 655/c3cff9ec 586/dc829f77 655/c3cff9ec
20:32:03.144 epoch 10 -> 11 at daa 660, 719.8 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 10000 bps (weakest of 7: 10000), ladder up weakest 10000 bps, this node signals byte 6, v5 by signal at epoch none, step at epoch 6
20:32:06.153 stream for epoch 11 (seed 7fda1c7f1e3c69ad): root 0x254ee9192c821ec7 17 records, from n0
20:34:12.804 SUMMARY FAIL (cross-63524e28) 2026-10-07T20:34:12.803Z: rung 1 (35 passes) from epoch 6 at DAA 360; v5 from epoch 8 at DAA 480 at rung 1; epochs e0:v4:r0:0bps e1:v4:r0:0bps e2:v4:r0:0bps e3:v4:r0:0bps e4:v4:r0:0bps e5:v4:r0:0bps e6:v4:r1:0bps e7:v4:r1:10000bps e8:v5:r1:10000bps e9:v5:r1:10000bps e10:v5:r1:10000bps e11:v5:r1:10000bps; chain bytes {"0":1,"6":660} (9985 bps at byte 6), ladder bits {"none":1,"up":660}; blocks 483 / 178 across DAA 480; rejected miners 0/0/0/132 nodes 0/0/0/132 (submit timeouts 1/1/1/0); accepted after the flip 142/144/259/20; sinks f7551068d959220d f7551068d959220d 57b763ae4b44eca7 f7551068d959220d (DIFFER) at 660/660/594/660; step lines 4/4 (epochs 6/6/6/6), v5 signal lines 4/4 (epochs 8/8/8/8, weakest 10000/10000/10000/10000); stale n3 accepted after the first refresh 0 rejected 132; restart n2 stopped at DAA 456, started at DAA 500, lock faults 0, IBD refusals 1, accepted after 259; dn3 id OK
20:34:12.804 PROGRAM ID epoch 0 v4 rung 0 (27 passes) seed 234e082d653dc69d: miners 83dab9f31f2e2dba (4 of 4) cli 83dab9f31f2e2dba other-class 343287b6df5996ed OK
20:34:12.804 PROGRAM ID epoch 1 v4 rung 0 (27 passes) seed 625a96ec783479e4: miners 749276ac1f80b56c (4 of 4) cli 749276ac1f80b56c other-class 2678a4201322a433 state 0xa955444f8e5d0dcc (16 records) OK
20:34:12.804 PROGRAM ID epoch 2 v4 rung 0 (27 passes) seed 10d331863788ace9: miners 9d8446f3db777c4e (4 of 4) cli 9d8446f3db777c4e other-class 9cb656d981c4463d state 0xd00cb0e4afc17720 (17 records) OK
20:34:12.804 PROGRAM ID epoch 3 v4 rung 0 (27 passes) seed a75213ceb811abad: miners eaf4807764312238 (4 of 4) cli eaf4807764312238 other-class 7631d160beabddc3 state 0x6888bdeb6d14cad6 (17 records) OK
20:34:12.804 PROGRAM ID epoch 4 v4 rung 0 (27 passes) seed 6bc39c0fed38c1c7: miners 2512871226631fdc (4 of 4) cli 2512871226631fdc other-class 5f0adc0d4bf6142b state 0xae27f1438d314ff3 (17 records) OK
20:34:12.804 PROGRAM ID epoch 5 v4 rung 0 (27 passes) seed 0a07dca05d53beb1: miners 2a9b4ac909e477c0 (4 of 4) cli 2a9b4ac909e477c0 other-class 0316aefe8106dccb state 0xe2164c5c9665c4b7 (17 records) OK
20:34:12.804 PROGRAM ID epoch 6 v4 rung 1 (35 passes) seed 2f47a9742c88e289: miners a1585cedf2b5ed51 (4 of 4) cli a1585cedf2b5ed51 other-class 593e4af6b6c9b106 state 0xd1aa568c26ac7b3b (17 records) OK
20:34:12.804 PROGRAM ID epoch 7 v4 rung 1 (35 passes) seed 6b977f4d0016495d: miners b5b948c26001debc (4 of 4) cli b5b948c26001debc other-class cf3d6e0e9b4e02c1 state 0x83da097a19d54213 (17 records) OK
20:34:12.804 PROGRAM ID epoch 8 v5 rung 1 (35 passes) seed 2f7944518dd2ad3b: miners 46e976c9d74abaae (3 of 4) cli 46e976c9d74abaae other-class 0c113d6179ad63bb rung0 799ecc6cad703841 state 0x234a1f9127c9c03b (17 records) OK
20:34:12.805 PROGRAM ID epoch 8 FORK v5 rung 1 (35 passes) seed 31f43d19d7a7b9ba: miners 7b53d4c975269d6c (1 of 4) cli null other-class 233af331f8f8b63d rung0 7a6769fd8ce91e3b MISMATCH
20:34:12.805 PROGRAM ID epoch 9 v5 rung 1 (35 passes) seed 1161f6e6a0b99b13: miners 4ffc98b46cc939b6 (3 of 4) cli 4ffc98b46cc939b6 other-class 86a2636289c7a56b rung0 3077727680243f35 state 0x97edd42aa6892428 (17 records) OK
20:34:12.805 PROGRAM ID epoch 9 FORK v5 rung 1 (35 passes) seed fd530b1bb0145eae: miners a840102915b7bfa3 (1 of 4) cli null other-class a3024b5cba1c5076 rung0 a3997b2070aee97a MISMATCH
20:34:12.805 PROGRAM ID epoch 10 v5 rung 1 (35 passes) seed 1588ab6d08450df2: miners 814782883265643a (3 of 4) cli 814782883265643a other-class 72a4a39971470ea3 rung0 fd2e3da4f9dff9d9 state 0x04fcc1dc625f7d0a (17 records) OK
20:34:12.805 FAILED CHECK no_second_step
20:34:12.805 FAILED CHECK sinks_agree
20:34:12.805 FAILED CHECK block_counts_agree
20:34:12.805 FAILED CHECK restarted_node_resynced
20:34:12.805 FAILED CHECK restarted_node_ibd_not_refused
20:34:12.805 summary: /tmp/igneum-fast-time-v5x-cross-63524e28/summary.json and /srv/builds/igneum-wt-v5-fasttime/docs/design/class-v5-harness/fasttime/cross-63524e28.json

23
infra/fast-time/class-v5-crossing.mjs
View file

@ -27,7 +27,7 @@
// the checks.
//
// node infra/fast-time/class-v5-crossing.mjs [--case cross] [--signal 6,6,6,6] [--ladder up,up,up,up] [--class-window 60]
// [--ladder-window 45] [--floor 900|never] [--stale 3|none] [--restart 2|none] [--restart-before 25] [--restart-after 20]
// [--ladder-window 45] [--floor 900|never] [--stale 3|none] [--restart 2|none] [--restart-before 25] [--restart-after 20] [--restart-miner-hold-s 300]
// [--epochs 11] [--secs 960] [--out <dir>] [--genesis-bits 0x1f010000]
// IGNEUMD, IGNEUM_MINER, IGNEUM_POW name the binaries (defaults: vendor/igneum-node-v5-fasttime/target/release and
// igneum-pow/target/release/igneum-pow, the layout on igneum-build-1 under /srv/builds/igneum-wt-v5-fasttime).
@ -66,7 +66,7 @@ const LADDER_WINDOW = flag('ladder-window', 45);
const FLOOR = actflag('floor', 900);
const STALE = optidx('stale', 3);
const RESTART = optidx('restart', 2);
const RESTART_BEFORE = flag('restart-before', 25), RESTART_AFTER = flag('restart-after', 20);
const RESTART_BEFORE = flag('restart-before', 25), RESTART_AFTER = flag('restart-after', 20), MINER_HOLD = flag('restart-miner-hold-s', 300);
const SIGNAL = (sflag('signal') || '6,6,6,6').split(',').map(Number);
const LADDER = (sflag('ladder') || 'up,up,up,up').split(',').map(s => s.trim().toLowerCase());
const N = SIGNAL.length;
@ -261,7 +261,15 @@ while (Date.now() - t0 < SECS * 1000) {
}
if (restartNode && restart.stopped && !restart.started && daa != null && daa >= BOUNDARY + RESTART_AFTER) {
try { await restartNode.start(); } catch (e) { log(`RESTART STEP FAILED: ${e.message}`); restart.started = { daa, at: +since(), wall: Date.now(), failed: e.message }; }
if (!restart.started) { miner(RESTART, restartNode, Math.max(60, SECS - Math.floor((Date.now() - t0) / 1000))); restart.started = { daa, at: +since(), wall: Date.now() }; log(`RESTART STEP: n${RESTART} started on its own datadir at daa ${daa} (${since()} s), ${daa - BOUNDARY} DAA after the boundary; its miner restarted with its key`); }
// the miner waits for the resync (--restart-miner-hold-s cap): on the 63524e28 run a miner restarted at once, with
// unsynced mining on, extended the node's own branch from the split, which outweighed the 24 honest blocks the node could
// validate (the v5 headers past the boundary deferred), so the executor followed the fork and the IBD catch-up waited forever
if (!restart.started) { restart.started = { daa, at: +since(), wall: Date.now() }; log(`RESTART STEP: n${RESTART} started on its own datadir at daa ${daa} (${since()} s), ${daa - BOUNDARY} DAA after the boundary; its miner waits for the resync (at most ${MINER_HOLD} s)`); }
}
if (restartNode && restart.started && !restart.started.failed && !restart.minerStarted && (restart.resynced || Date.now() - restart.started.wall > MINER_HOLD * 1000)) {
miner(RESTART, restartNode, Math.max(60, SECS - Math.floor((Date.now() - t0) / 1000)));
restart.minerStarted = { daa, at: +since(), wall: Date.now(), after_resync: !!restart.resynced };
log(`RESTART STEP: n${RESTART}'s miner restarted with its key at daa ${daa} (${since()} s), ${restart.resynced ? 'after the resync' : `without a resync inside ${MINER_HOLD} s`}`);
}
if (restartNode && restart.started && !restart.started.failed && !restart.resynced && restartNode.rpc) {
const [a, b] = await Promise.all([n0.dag(), restartNode.dag()]);
@ -387,7 +395,9 @@ const restartFaults = restartNode && restart.started && !restart.started.failed
// before its executor has passed the seed block it stopped short of), and with unsynced mining on it mined its own fork
const IBD_REFUSED = /IBD with peer .* completed with error|proof-of-work check refused|class v5 needs the execution state/;
const restartIbdRefusals = restartNode && restart.started && !restart.started.failed ? restartNode.grepLog(IBD_REFUSED, restartNode.restartLine) : [];
const restartAcceptedAfter = restartNode && restart.started ? acceptedAfter(RESTART, restart.started.wall) : null;
const restartAcceptedAfter = restartNode && restart.minerStarted ? acceptedAfter(RESTART, restart.minerStarted.wall) : null;
const CATCHUP_DONE = /IBD: class v5 catch-up done/;
const restartCatchupDone = restartNode && restart.started && !restart.started.failed ? restartNode.grepLog(CATCHUP_DONE, restartNode.restartLine) : [];
// the stream roots agree across the executing honest nodes wherever two or more answered
const rootRows = [...streams.entries()].map(([e, st]) => ({ epoch: +e, roots: Object.values(st.roots) })).filter(r => r.roots.length >= 2);
@ -439,7 +449,8 @@ const checks = {
no_lock_fault_after_the_restart: restartNode == null ? null : (!!restart.started && !restart.started.failed && restartFaults.length === 0),
restarted_node_resynced: restartNode == null ? null : (!!restart.resynced && sinks[RESTART] === sinks[0]),
restarted_node_ibd_not_refused: restartNode == null ? null : (!!restart.started && !restart.started.failed && restartIbdRefusals.length === 0),
restarted_node_mines_v5_after_the_restart: restartNode == null ? null : (restartAcceptedAfter > 0 && rejectedNode[RESTART] === 0),
restarted_node_mines_v5_after_the_restart: restartNode == null ? null : (restartAcceptedAfter > 0 && rejectedNode[RESTART] === 0 && !!restart.minerStarted?.after_resync),
restarted_node_catch_up_done_line: restartNode == null ? null : (restartCatchupDone.length > 0),
};
const pass = Object.values(checks).every(v => v === true || v === null);
const summary = {
@ -450,7 +461,7 @@ const summary = {
blocks: { total: blocks.length, before_boundary: before.length, after_boundary: after.length, object_bytes: objectBytes, byte_6_share_bps: byte6Share, ladder_bits: ladderBits, up_share_bps: upShare },
program_id_rows: idRows, fork_rows: forkRows.length, miners_expected_on_a_row: MINERS_EXPECTED, accepted_per_miner: accepted, accepted_after_flip_per_miner: acceptedAfterFlip, rejected_by_miners: rejectedMiner, submit_timeouts_per_miner: submitTimeouts, rejected_by_nodes: rejectedNode,
stale: stale ? { node: STALE, accepted_total: accepted[STALE], accepted_after_first_refresh: staleAcceptedAfterRefresh, old_template_blocks_after_refresh: staleOldTemplateAfterRefresh, rejected_by_its_node: rejectedNode[STALE], rejected_lines: rejectedMiner[STALE], first_refresh_epoch: flip ? flip.epoch + 1 : null } : null,
restart: restartNode ? { node: RESTART, ...restart, lock_fault_lines: restartFaults.slice(0, 5), ibd_refusal_lines: restartIbdRefusals.length, ibd_refusal_example: restartIbdRefusals[0]?.replace(/^.*?\] /, '').slice(0, 300) ?? null, accepted_after_restart: restartAcceptedAfter, rejected_by_its_node: rejectedNode[RESTART] } : null,
restart: restartNode ? { node: RESTART, ...restart, lock_fault_lines: restartFaults.slice(0, 5), ibd_refusal_lines: restartIbdRefusals.length, catch_up_done_lines: restartCatchupDone.map(l => l.replace(/^.*?\] /, '').slice(0, 200)), ibd_refusal_example: restartIbdRefusals[0]?.replace(/^.*?\] /, '').slice(0, 300) ?? null, accepted_after_restart: restartAcceptedAfter, rejected_by_its_node: rejectedNode[RESTART] } : null,
sinks, block_counts: counts, step_lines: stepLines.map(ls => ls.map(l => l.slice(0, 300))), signal_lines: signalLines.map(ls => ls.map(l => l.slice(0, 300))), floor_lines: floorLines, samples,
};
writeFileSync(`${TMP}/summary.json`, JSON.stringify(summary, null, 2));