igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity

fast-time crossing, the class v6 object's cold-restart known-failed shape on 617cb441 (15:24:48Z to 15:38:43Z: a 60 s cold wait, FAIL as it must on cold_restart_line_on_every_node; the crossing v4 to v5 by floor at epoch 8 to v6 at epoch 9, 'Class v6 object active: floor DAA 540' on every node, byte 8 from genesis, 11 of 11 ids) and three harness faults it showed: the cold restart moved the restart step's read window (a cold line index of its own now), the stale and restart counts ran past the cold stop (bounded now), a pair prepared one lead early was attributed to the previous epoch by its DAA (the template's seed map decides now)

Browse source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-08 15:40:15 +00:00
parent 82139d417d
commit cda58d341f
3 changed files with 1715 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1582
docs/design/class-v5-harness/fasttime/cross-v6-617cb441-failed-cold.json Normal file
View file

File diff suppressed because it is too large Load diff

117
docs/design/class-v5-harness/fasttime/cross-v6-617cb441-failed-cold.log Normal file
View file

@ -0,0 +1,117 @@
15:24:48.657 override: every key known to the binary
15:24:48.661 case cross-v6-617cb441-failed-cold: 4 nodes, bytes the node's own default, ladder up/up/up/up; v5 by floor; class v6 floor 540 (epoch 9), dataset step [[540, 5632 MiB]], flags 15; cold restart 60 s; class v4 sub-version 3 from genesis at rung 0; class window 60 DAA x 7 (the first epoch that can cross to v5 is 8, DAA 480), ladder window 45 DAA x 7 (the first epoch that can step is 6, DAA 360); v5 floor 480 (epoch 8); stale node 3; restart node 2 (stop at DAA 455, start at 500); 60 DAA per epoch, lead 10; run 960 s or 11 epochs; node /srv/builds/igneum-wt-v5-fasttime/vendor/igneum-node-v6/target/release/igneumd
15:24:51.297 CPU verifier on the Devnet 3 genesis pack: cli e5a4ac5978462156 pack e5a4ac5978462156 expected e5a4ac5978462156: OK
15:24:52.866 n0 up pid 1059406 json 29862 p2p 29861 exec 29863, byte 6, ladder up
15:24:54.378 n1 up pid 1059794 json 29872 p2p 29871 exec 29873, byte 6, ladder up
15:24:55.883 n2 up pid 1060549 json 29882 p2p 29881 exec 29883, byte 6, ladder up
15:24:57.389 n3 up pid 1060909 json 29892 p2p 29891 exec 29893, byte 6, ladder up
15:24:57.390 n0: Program class v5 from the override file: enabled, the floor at epoch 8 (DAA score 480 rounded up to the epoch boundary at 480); the class v5 | (no class signal line) | this node signals up
15:24:57.390 n1: Program class v5 from the override file: enabled, the floor at epoch 8 (DAA score 480 rounded up to the epoch boundary at 480); the class v5 | (no class signal line) | this node signals up
15:24:57.391 n2: Program class v5 from the override file: enabled, the floor at epoch 8 (DAA score 480 rounded up to the epoch boundary at 480); the class v5 | (no class signal line) | this node signals up
15:24:57.391 n3: Program class v5 from the override file: enabled, the floor at epoch 8 (DAA score 480 rounded up to the epoch boundary at 480); the class v5 | (no class signal line) | this node signals up
15:24:57.391 n0 digest: 329435836576000b
15:24:58.403 epoch -1 -> 0 at daa 0, 7.1 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 0
15:24:58.421 stream for epoch 0 (seed 234e082d653dc69d): root 0x7e37a9fb19b154d3 11 records, from n0
15:24:58.431 t=7.1 s daa 0 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 0/234e082d 0/234e082d 0/234e082d 0/234e082d
15:25:13.454 t=22.2 s daa 3 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 3/8bb8972f 3/8bb8972f 3/8bb8972f 3/8bb8972f
15:25:28.471 t=37.2 s daa 11 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 11/61a16514 11/61a16514 11/61a16514 11/61a16514
15:25:43.489 t=52.2 s daa 20 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 20/6425bc4e 20/6425bc4e 20/6425bc4e 20/6425bc4e
15:25:58.510 t=67.2 s daa 40 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 40/a574cdf5 40/a574cdf5 40/a574cdf5 40/a574cdf5
15:26:13.526 t=82.2 s daa 58 epoch 0 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 58/25dc52aa 58/25dc52aa 58/25dc52aa 58/25dc52aa
15:26:15.528 epoch 0 -> 1 at daa 60, 84.2 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 0
15:26:18.535 stream for epoch 1 (seed e7fd08aff7c76704): root 0x4b8d6d1e7f022f85 16 records, from n0
15:26:28.557 t=97.3 s daa 74 epoch 1 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 74/1f5c7b12 74/1f5c7b12 74/1f5c7b12 74/1f5c7b12
15:26:43.573 t=112.3 s daa 92 epoch 1 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 92/30ce34c9 92/30ce34c9 92/30ce34c9 92/30ce34c9
15:26:58.588 t=127.3 s daa 103 epoch 1 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 103/ac396397 103/ac396397 103/ac396397 103/ac396397
15:27:13.606 t=142.3 s daa 117 epoch 1 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 117/c0584cd3 117/c0584cd3 117/c0584cd3 117/c0584cd3
15:27:16.609 epoch 1 -> 2 at daa 120, 145.3 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 0
15:27:18.617 stream for epoch 2 (seed c5d0f983c4f9ec4d): root 0xe76bb279e491b424 17 records, from n0
15:27:28.634 t=157.3 s daa 129 epoch 2 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 129/fe7891ef 129/fe7891ef 129/fe7891ef 130/1aaeea69
15:27:43.648 t=172.4 s daa 146 epoch 2 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 146/1ef12256 146/1ef12256 146/1ef12256 146/1ef12256
15:27:58.663 t=187.4 s daa 166 epoch 2 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 166/f52c89cc 166/f52c89cc 166/f52c89cc 166/f52c89cc
15:28:11.678 epoch 2 -> 3 at daa 180, 200.4 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 0
15:28:13.680 t=202.4 s daa 180 epoch 3 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 180/4524f8cd 180/4524f8cd 180/4524f8cd 180/4524f8cd
15:28:14.682 stream for epoch 3 (seed 80af352c328c1d29): root 0x976bfb2f43b95d61 17 records, from n0
15:28:28.701 t=217.4 s daa 197 epoch 3 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 197/d9b737a0 197/d9b737a0 197/d9b737a0 197/d9b737a0
15:28:43.716 t=232.4 s daa 213 epoch 3 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 213/5ea13b29 213/5ea13b29 213/5ea13b29 213/5ea13b29
15:28:58.733 t=247.4 s daa 234 epoch 3 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 234/678e245c 234/678e245c 234/678e245c 234/678e245c
15:29:03.738 epoch 3 -> 4 at daa 240, 252.4 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 0 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 0
15:29:06.743 stream for epoch 4 (seed be2a2bbcc512daa3): root 0x187b0a75a0e2871a 17 records, from n0
15:29:13.755 t=262.5 s daa 247 epoch 4 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 247/2e26f850 247/2e26f850 247/2e26f850 247/2e26f850
15:29:28.771 t=277.5 s daa 267 epoch 4 class 4 rung 0 v5 0 bps (weakest 0) up weakest 0 blocks/sink per node 267/50775877 267/50775877 267/50775877 267/50775877
15:29:43.787 t=292.5 s daa 288 epoch 4 class 4 rung 0 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 288/cba5bbe1 288/cba5bbe1 288/cba5bbe1 288/cba5bbe1
15:29:58.802 epoch 4 -> 5 at daa 300, 307.5 s: template class 4 (next 4) rung 0 (27 passes, next rung 0), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 0
15:29:58.804 stream for epoch 5 (seed a639a03ad1034ccd): root 0xca1dae1a2fa5928b 17 records, from n0
15:29:58.808 t=307.5 s daa 300 epoch 5 class 4 rung 0 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 300/e95baa25 300/e95baa25 300/e95baa25 300/e95baa25
15:30:13.823 t=322.5 s daa 313 epoch 5 class 4 rung 0 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 313/43282a33 313/43282a33 313/43282a33 313/43282a33
15:30:28.838 t=337.5 s daa 325 epoch 5 class 4 rung 0 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 325/0849b93f 325/0849b93f 325/0849b93f 325/0849b93f
15:30:43.856 t=352.6 s daa 340 epoch 5 class 4 rung 0 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 340/cdbd9737 340/cdbd9737 340/cdbd9737 340/cdbd9737
15:30:58.870 t=367.6 s daa 357 epoch 5 class 4 rung 0 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 357/e88a2c04 357/e88a2c04 357/e88a2c04 357/e88a2c04
15:31:03.876 epoch 5 -> 6 at daa 360, 372.6 s: template class 4 (next 4) rung 1 (35 passes, next rung 1), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 6
15:31:03.877 LADDER STEP: the template is rung 1 (35 shadow passes) from epoch 6 (daa 360) at 372.6 s wall
15:31:06.882 stream for epoch 6 (seed fe574fe6ebbee7ef): root 0x4952f5674960f4cb 17 records, from n0
15:31:13.894 t=382.6 s daa 370 epoch 6 class 4 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 370/3b343b9e 370/3b343b9e 369/e058f084 369/e058f084
15:31:28.911 t=397.6 s daa 383 epoch 6 class 4 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 383/5dbe453a 383/5dbe453a 383/5dbe453a 383/5dbe453a
15:31:43.926 t=412.6 s daa 398 epoch 6 class 4 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 398/25ad380f 398/25ad380f 398/25ad380f 398/25ad380f
15:31:58.942 t=427.6 s daa 415 epoch 6 class 4 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 415/d142170d 415/d142170d 415/d142170d 414/1fbdaaf0
15:32:06.950 epoch 6 -> 7 at daa 420, 435.7 s: template class 4 (next 4) rung 1 (35 passes, next rung 1), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 6
15:32:06.952 stream for epoch 7 (seed 574cb32a4370e1c7): root 0x683b242435f6f33b 17 records, from n0
15:32:13.961 t=442.7 s daa 423 epoch 7 class 4 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 423/6543aa25 422/e1317f32 422/e1317f32 422/e1317f32
15:32:28.976 t=457.7 s daa 441 epoch 7 class 4 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 441/2d6fc543 440/6a4a2260 440/6a4a2260 441/2d6fc543
15:32:39.992 RESTART STEP: n2 and its miner stopped at daa 457 (468.7 s), 23 DAA before the expected v5 boundary 480
15:32:43.998 t=472.7 s daa 459 epoch 7 class 4 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 459/7e31f9fb 459/7e31f9fb down 459/7e31f9fb
15:32:59.018 t=487.7 s daa 469 epoch 7 class 4 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 469/a28bbb56 469/a28bbb56 down 469/a28bbb56
15:33:10.029 epoch 7 -> 8 at daa 480, 498.7 s: template class 5 (next 5) rung 1 (35 passes, next rung 1), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 6
15:33:10.030 CLASS SWITCH: the template is class v5 from epoch 8 (daa 480) at rung 1 (35 passes), 498.7 s wall
15:33:11.032 stream for epoch 8 (seed 08ee8b5ead0c6bce): root 0x6ae7d010e24ed060 17 records, from n0
15:33:14.037 t=502.7 s daa 481 epoch 8 class 5 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 481/bfedbfdd 481/bfedbfdd down 481/bfedbfdd
15:33:29.053 t=517.8 s daa 497 epoch 8 class 5 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 497/13271eed 497/13271eed down 497/13271eed
15:33:33.566 n2 RESTARTED pid 1161931 json 29882 p2p 29881 exec 29883, byte 6, ladder up (own datadir kept)
15:33:33.566 RESTART STEP: n2 started on its own datadir at daa 500 (522.3 s), 20 DAA after the boundary; its miner waits for the resync (at most 0 s)
15:33:34.573 RESTART STEP: n2's miner restarted with its key at daa 502 (523.3 s), without a resync inside 0 s
15:33:41.594 RESTART STEP: n2 at n0's sink 07c4f75a9b5f6707 (511 blocks) 8 s after its start
15:33:44.600 t=533.3 s daa 513 epoch 8 class 5 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 513/490b5cc7 513/490b5cc7 513/490b5cc7 513/490b5cc7
15:33:59.616 t=548.3 s daa 533 epoch 8 class 5 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 533/e01578c4 533/e01578c4 533/e01578c4 533/e01578c4
15:34:06.624 epoch 8 -> 9 at daa 540, 555.3 s: template class 6 (next 6) rung 1 (35 passes, next rung 1), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 6
15:34:06.624 CLASS SWITCH: the template is class v6 from epoch 9 (daa 540) at rung 1 (35 passes), 555.3 s wall
15:34:07.627 stream for epoch 9 (seed 82589b42295de101): root 0xeec249bbbdb575b0 17 records, from n0
15:34:14.641 t=563.3 s daa 543 epoch 9 class 6 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 543/47d2954c 540/5a8d5eec 540/5a8d5eec 540/5a8d5eec
15:34:29.657 t=578.4 s daa 561 epoch 9 class 6 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 561/a0187ced 561/a0187ced 561/a0187ced 561/a0187ced
15:34:44.673 t=593.4 s daa 580 epoch 9 class 6 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 580/4d6c6765 580/4d6c6765 580/4d6c6765 580/4d6c6765
15:34:58.687 epoch 9 -> 10 at daa 600, 607.4 s: template class 6 (next 6) rung 1 (35 passes, next rung 1), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 6
15:34:59.690 stream for epoch 10 (seed adb8cad8b1a93103): root 0x699988a80fa24e7c 17 records, from n0
15:34:59.694 t=608.4 s daa 600 epoch 10 class 6 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 600/ed11cdc5 600/ed11cdc5 600/ed11cdc5 600/ed11cdc5
15:35:14.712 t=623.4 s daa 609 epoch 10 class 6 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 609/937ffe54 609/937ffe54 609/937ffe54 609/937ffe54
15:35:29.732 t=638.4 s daa 623 epoch 10 class 6 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 623/5a6cb45f 623/5a6cb45f 623/5a6cb45f 623/5a6cb45f
15:35:44.751 t=653.5 s daa 650 epoch 10 class 6 rung 1 v5 0 bps (weakest 0) up weakest 10000 blocks/sink per node 650/5204aafe 650/5204aafe 650/5204aafe 650/5204aafe
15:35:56.764 epoch 10 -> 11 at daa 660, 665.5 s: template class 6 (next 6) rung 1 (35 passes, next rung 1), v5 share at the sink 0 bps (weakest of 7: 0), ladder up weakest 10000 bps, this node signals byte 8, v5 by signal at epoch none, step at epoch 6
15:35:59.766 stream for epoch 11 (seed 5204aafe5dbc5705): root 0x08cdb9e13a0c7762 17 records, from n0
15:36:03.792 COLD RESTART: 4 nodes and their miners stopped at daa 660 (672.5 s), 8 exec snapshot file(s) removed; waiting 60 s
15:37:05.321 n0 RESTARTED pid 1193185 json 29862 p2p 29861 exec 29863, byte 6, ladder up (own datadir kept)
15:37:06.834 n1 RESTARTED pid 1193591 json 29872 p2p 29871 exec 29873, byte 6, ladder up (own datadir kept)
15:37:08.343 n2 RESTARTED pid 1193879 json 29882 p2p 29881 exec 29883, byte 6, ladder up (own datadir kept)
15:37:09.854 n3 RESTARTED pid 1194152 json 29892 p2p 29891 exec 29893, byte 6, ladder up (own datadir kept)
15:37:24.899 COLD RESTART: every node at one sink ad1b1b226460fccf (671 blocks) 15 s after the restart, mining on
15:37:24.915 COLD RESTART: n0 NO cold-restart line; wait lines 0; faults 0; accepted after 2
15:37:24.915 COLD RESTART: n1 NO cold-restart line; wait lines 0; faults 0; accepted after 3
15:37:24.915 COLD RESTART: n2 NO cold-restart line; wait lines 0; faults 0; accepted after 4
15:37:24.915 COLD RESTART: n3 NO cold-restart line; wait lines 0; faults 0; accepted after 2
15:38:43.121 SUMMARY FAIL (cross-v6-617cb441-failed-cold) 2026-10-08T15:38:43.119Z: rung 1 (35 passes) from epoch 6 at DAA 360; v5 from epoch 8 at DAA 480 at rung 1 by floor; v6 from epoch 9 at DAA 540 at rung 1 (floor 540); cold restart synced 15 s after the restart, cold-restart lines 0/4, wait lines 0, faults 0; epochs e0:v4:r0:0bps e1:v4:r0:0bps e2:v4:r0:0bps e3:v4:r0:0bps e4:v4:r0:0bps e5:v4:r0:0bps e6:v4:r1:0bps e7:v4:r1:0bps e8:v5:r1:0bps e9:v6:r1:0bps e10:v6:r1:0bps e11:v6:r1:0bps; chain bytes {"0":1,"8":672} (0 bps at byte 6), ladder bits {"none":1,"up":672}; blocks 483 / 190 across DAA 480; rejected miners 0/0/0/63 nodes 0/0/0/63 (submit timeouts 0/0/0/0); accepted after the flip 95/98/78/32; sinks ad1b1b226460fccf ad1b1b226460fccf ad1b1b226460fccf ad1b1b226460fccf (agree) at 671/671/671/671; step lines 4/4 (epochs 6/6/6/6; later steps logged 12, earliest allowed 12), v5 signal lines 0/4 (epochs -/-/-/-, weakest -/-/-/-); stale n3 accepted after the first refresh 15 rejected 63; restart n2 stopped at DAA 457, started at DAA 500, resynced 8 s later, lock faults 0, IBD refusals 0, catch-up done no, hold refusals to its miner 0, accepted before/after done n/a/n/a, accepted after the miner start 78; dn3 id OK; late joiner off
15:38:43.121 PROGRAM ID epoch 0 v4 rung 0 (27 passes) seed 234e082d653dc69d: miners 83dab9f31f2e2dba (4 of 4) cli 83dab9f31f2e2dba other-class 343287b6df5996ed state 0x7e37a9fb19b154d3 (11 records) OK
15:38:43.121 PROGRAM ID epoch 1 v4 rung 0 (27 passes) seed e7fd08aff7c76704: miners 9f208aefde3f9f56 (4 of 4) cli 9f208aefde3f9f56 other-class e906997d7bb083f5 state 0x4b8d6d1e7f022f85 (16 records) OK
15:38:43.121 PROGRAM ID epoch 2 v4 rung 0 (27 passes) seed c5d0f983c4f9ec4d: miners 6891ed8ff3ec6026 (4 of 4) cli 6891ed8ff3ec6026 other-class c21d01e11bee510d state 0xe76bb279e491b424 (17 records) OK
15:38:43.121 PROGRAM ID epoch 3 v4 rung 0 (27 passes) seed 80af352c328c1d29: miners 5a065150debdb06c (4 of 4) cli 5a065150debdb06c other-class f0db07cd3ce1ae83 state 0x976bfb2f43b95d61 (17 records) OK
15:38:43.121 PROGRAM ID epoch 4 v4 rung 0 (27 passes) seed be2a2bbcc512daa3: miners b0c6b39989083653 (4 of 4) cli b0c6b39989083653 other-class 7147df1b83e3625a state 0x187b0a75a0e2871a (17 records) OK
15:38:43.121 PROGRAM ID epoch 5 v4 rung 0 (27 passes) seed a639a03ad1034ccd: miners 50c390064ed72541 (4 of 4) cli 50c390064ed72541 other-class ab760d3c60a86468 state 0xca1dae1a2fa5928b (17 records) OK
15:38:43.121 PROGRAM ID epoch 6 v4 rung 1 (35 passes) seed fe574fe6ebbee7ef: miners a83a780a01a205e9 (4 of 4) cli a83a780a01a205e9 other-class ce20c4f7964cfd32 state 0x4952f5674960f4cb (17 records) OK
15:38:43.121 PROGRAM ID epoch 7 v4 rung 1 (35 passes) seed 574cb32a4370e1c7: miners 301cbc44e8a66ec1 (4 of 4) cli 301cbc44e8a66ec1 other-class aff4131a758dba7a state 0x683b242435f6f33b (17 records) OK
15:38:43.121 PROGRAM ID epoch 7 FORK v5 rung 1 (35 passes) seed 08ee8b5ead0c6bce: miners cbff2a3b7cfea321 (4 of 4) cli null other-class cccf092b13de35cc rung0 8bc7593246d5fea8 MISMATCH
15:38:43.122 PROGRAM ID epoch 9 v6 rung 1 (35 passes) seed 82589b42295de101: miners f8c26ad7e6c82019 (4 of 4) cli f8c26ad7e6c82019 other-class 80f09f9b42c02380 rung0 7deb8144c0381ec0 state 0xeec249bbbdb575b0 (17 records) OK
15:38:43.122 PROGRAM ID epoch 10 v6 rung 1 (35 passes) seed adb8cad8b1a93103: miners 451abf1ec61b4832 (4 of 4) cli 451abf1ec61b4832 other-class 25925d3c89ffa1f7 rung0 d4a6549e183024d9 state 0x699988a80fa24e7c (17 records) OK
15:38:43.122 PROGRAM ID epoch 11 v6 rung 1 (35 passes) seed 5204aafe5dbc5705: miners 837cd4bc81267720 (4 of 4) cli 837cd4bc81267720 other-class 563ca6f8f4e14c2d rung0 bd168654f35d6c63 state 0x08cdb9e13a0c7762 (17 records) OK
15:38:43.122 FAILED CHECK cold_restart_line_on_every_node
15:38:43.122 FAILED CHECK stale_node_refused_from_the_first_refresh
15:38:43.122 FAILED CHECK restarted_node_mines_v5_after_the_restart
15:38:43.122 FAILED CHECK restarted_node_held_its_own_mining_during_the_catch_up
15:38:43.122 FAILED CHECK restarted_node_catch_up_done_line
15:38:43.122 summary: /tmp/igneum-fast-time-v5x-cross-v6-617cb441-failed-cold/summary.json and /srv/builds/igneum-wt-v5-fasttime/docs/design/class-v5-harness/fasttime/cross-v6-617cb441-failed-cold.json

25
infra/fast-time/class-v5-crossing.mjs
View file

@ -151,7 +151,8 @@ class Node {
get execRpc() { return `http://127.0.0.1:${this.execPort}`; }
async start() {
mkdirSync(this.dir, { recursive: true });
if (this.starts > 0) this.restartLine = this.logLines().length;
// the first restart's line index is the restart step's read window; a later (cold) restart keeps its own
if (this.starts > 0) { const at = this.logLines().length; if (this.starts === 1) this.restartLine = at; this.coldLine = at; }
this.starts++;
const a = ['--devnet', `--devnet-suffix=${SUFFIX}`, '--nodnsseed', '--disable-upnp', '--nologfiles', '--enable-unsynced-mining', '--utxoindex',
`--appdir=${this.dir}`, `--rpclisten=127.0.0.1:${this.grpcPort}`, `--rpclisten-json=127.0.0.1:${this.jsonPort}`, `--evm-rpclisten=127.0.0.1:${this.execPort}`,
@ -203,7 +204,10 @@ process.on('unhandledRejection', async (e) => { log(`FAILED: ${e?.stack || e}`);
const minerLog = (i) => { try { return readFileSync(`${TMP}/cpu${i}.log`, 'utf8').split('\n'); } catch { return []; } };
// the miner stamps every line with epoch seconds (`1791374997.353 ACCEPTED block ...`)
const lineWall = (l) => { const m = /^(\d{10}\.\d+) /.exec(l); return m ? +m[1] * 1000 : null; };
const acceptedAfter = (i, wallMs, freshOnly = false) => wallMs == null ? null : minerLog(i).filter(l => /ACCEPTED block/.test(l) && (lineWall(l) ?? 0) >= wallMs && !(freshOnly && /gens old/.test(l))).length;
// the main run's counts stop at the cold stop (a miner restarted after the cold wait is a new process with a fresh stream)
let COLD_STOP_WALL = Infinity;
const acceptedAfter = (i, wallMs, freshOnly = false) => wallMs == null ? null : minerLog(i).filter(l => /ACCEPTED block/.test(l) && (lineWall(l) ?? 0) >= wallMs && (lineWall(l) ?? 0) < COLD_STOP_WALL && !(freshOnly && /gens old/.test(l))).length;
const acceptedSince = (i, wallMs) => minerLog(i).filter(l => /ACCEPTED block/.test(l) && (lineWall(l) ?? 0) >= wallMs).length;
const STEP_LINE = /Latency ladder step by miner signal: epoch (\d+) moves to rung (\d+) \((\d+) shadow passes, from rung (\d+)\).*?weakest up (\d+) bps/;
const SIGNAL_LINE = /Program class v5 by miner signal: epoch (\d+) \(share (\d+) bps.*?weakest (\d+) bps/;
const FLOOR_LINE = /Program class v5 from the override file: enabled, the floor at epoch (\d+)/;
@ -348,7 +352,7 @@ if (COLD) {
const coldNodes = nodes.slice(0, N);
for (const n of coldNodes) await stopMiner(n.i);
for (const n of coldNodes) await n.stop();
const stoppedAt = Date.now();
const stoppedAt = Date.now(); COLD_STOP_WALL = stoppedAt;
let removed = 0;
for (const n of coldNodes) { for (const f of (await import('node:fs')).readdirSync(n.dir, { recursive: true })) { if (/exec-snapshot[^/]*\.bin$/.test(String(f))) { rmSync(`${n.dir}/${f}`, { force: true }); removed++; } } }
log(`COLD RESTART: ${coldNodes.length} nodes and their miners stopped at daa ${lastDaa} (${since()} s), ${removed} exec snapshot file(s) removed; waiting ${COLD_WAIT_S} s`);
@ -365,10 +369,10 @@ if (COLD) {
if (new Set(sinks2).size === 1 && d[0].sink && d[0].blockCount > lastDaa + 5) { cold.synced = { at: +since(), after_s: +(((Date.now() - t2) / 1000).toFixed(1)), blocks: d[0].blockCount }; log(`COLD RESTART: every node at one sink ${String(d[0].sink).slice(0, 16)} (${d[0].blockCount} blocks) ${cold.synced.after_s} s after the restart, mining on`); break; }
}
for (const n of coldNodes) {
cold.lines[n.i] = n.grepLog(/replaying from genesis whatever the sink's age/, n.restartLine)[0]?.replace(/^.*?\] /, '').slice(0, 220) ?? null;
cold.wait_lines[n.i] = n.grepLog(WAIT_LINE, n.restartLine).map(l => l.replace(/^.*?\] /, '').slice(0, 200)).slice(0, 2);
cold.faults[n.i] = n.grepLog(LOCK_FAULT, n.restartLine).length;
cold.accepted_after[n.i] = acceptedAfter(n.i, minerStart);
cold.lines[n.i] = n.grepLog(/replaying from genesis whatever the sink's age/, n.coldLine)[0]?.replace(/^.*?\] /, '').slice(0, 220) ?? null;
cold.wait_lines[n.i] = n.grepLog(WAIT_LINE, n.coldLine).map(l => l.replace(/^.*?\] /, '').slice(0, 200)).slice(0, 2);
cold.faults[n.i] = n.grepLog(LOCK_FAULT, n.coldLine).length;
cold.accepted_after[n.i] = acceptedSince(n.i, minerStart);
}
if (!cold.synced) log(`COLD RESTART: the nodes did NOT reach one sink with new blocks inside ${COLD_SYNC_S} s`);
for (const n of coldNodes) log(`COLD RESTART: n${n.i} ${cold.lines[n.i] ? 'cold-restart line: ' + cold.lines[n.i].slice(0, 140) : 'NO cold-restart line'}; wait lines ${cold.wait_lines[n.i].length}; faults ${cold.faults[n.i]}; accepted after ${cold.accepted_after[n.i]}`);
@ -407,7 +411,9 @@ const programs = new Map();
for (const n of nodes) for (const l of minerLog(n.i)) {
const m = /epoch seed ([0-9a-f]{64}) day (\d+) \(daa (\d+)\): program and 256 MiB cache ready in ([\d.]+) ms; class (v\d) program id ([0-9a-f]{16})/.exec(l);
if (!m) continue;
const k = m[1]; const e = programs.get(k) || { seedHex: k, seed: k.slice(0, 16), epoch: Math.floor(+m[3] / EPOCH), class: m[5], id: m[6], miners: new Set() };
// the epoch by the template's seed map (a pair prepared one lead early carries the previous epoch's DAA), else by the DAA
const k = m[1]; const byMap = [...epochSeeds.entries()].find(([, sd]) => sd === k)?.[0];
const e = programs.get(k) || { seedHex: k, seed: k.slice(0, 16), epoch: byMap != null ? +byMap : Math.floor(+m[3] / EPOCH), class: m[5], id: m[6], miners: new Set() };
if (e.id !== m[6] || e.class !== m[5]) e.disagree = true;
e.miners.add(n.i); programs.set(k, e);
}
@ -446,7 +452,8 @@ const rung0Rows = v4Rows.filter(r => r.reps === RUNG0 || r.reps === 0), rung1Row
// the restarted node's own refusals of its miner during a class v5 catch-up (IsInIBD, not synced) are the node-side hold at
// work (pair 432ea3d6), counted apart from a PoW refusal
const IBD_HOLD = /IsInIBD|not synced|is_synced|in IBD/i;
const rejectedMiner = nodes.map(n => minerLog(n.i).filter(l => /rejected nonce=/.test(l) && !(n.i === RESTART && IBD_HOLD.test(l))).length);
const beforeCold = (l) => (lineWall(l) ?? 0) < COLD_STOP_WALL;
const rejectedMiner = nodes.map(n => minerLog(n.i).filter(l => /rejected nonce=/.test(l) && beforeCold(l) && !(n.i === RESTART && IBD_HOLD.test(l))).length);
const holdRefusals = nodes.map(n => minerLog(n.i).filter(l => /rejected nonce=|submit error/.test(l) && IBD_HOLD.test(l)).length);
const submitTimeouts = nodes.map(n => minerLog(n.i).filter(l => /submit error/.test(l)).length);
const rejectedNode = nodes.map(n => n.grepLog(REJECT_LINE).length);