diff --git a/docs/plans/miner-faults.md b/docs/plans/miner-faults.md index 438430ef..42e29cef 100644 --- a/docs/plans/miner-faults.md +++ b/docs/plans/miner-faults.md @@ -38,12 +38,30 @@ Standing rules behind every row (branch `miner-reliability`, off `release-0.3.19 | Repo | Branch | Commit | What | |---|---|---|---| -| igneum | `miner-reliability` (off `release-0.3.19` db6f0964, with release-0.3.20's `igneum-pow` and master's build tooling) | `38a30397`, `c5cb5cd5` | the app side of every row, the register, the injector, the cards job kind, the LG-4 job, the CI check | +| igneum | `miner-reliability` (off `release-0.3.19` db6f0964, with release-0.3.20's `igneum-pow` and master's build tooling) | `38a30397` to `9d2c267a` (the engine at `89d117ce`) | the app side of every row, the register, the injector, the cards job kind, the LG-4 job, the CI check | | igneum-node | `miner-reliability-20` (off `release-0.3.20-node` dc141409) | `f067f7c1` | MF-5's miner side: STATUS while waiting, identities from the template time, the fetch back-off | Box lines: app tests 198 + 27 + 8 green on igneum-build-2 (the tree gate green, 33 checks on the Mac); igneum-miner 19 -green on igneum-build-2; both Linux binaries built on igneum-build-1 (app sha256 6d4ee013, miner b3bf3590). Injector: -the pod run's lines are appended below when it ends. +green on igneum-build-2; both Linux binaries built on igneum-build-1 (app sha256 28650614 at 89d117ce, miner b3bf3590 at +f067f7c1). + +Injector, run 4 on a one-shot RunPod pod (RTX 3070, Ubuntu 24.04, 7 October 2026 12:19Z to 12:48Z, engine 89d117ce, +the 0.3.17-line miner, the stand-in worker; the pod's real GPU switched off through the app's own card path; the private +node on devnet suffix 9960 with a 2-thread CPU block producer): seven of eight steps PASS, one on the stand-in's listing +(fixed at ecd79874, rerun below). + +| Step | Class | Result | Seconds | +|---|---|---|---| +| catch-up | MF-1, MF-2 | no worker started while the execution layer held no record (120 s), the card said it waits for the executed tip, the node was not restarted, the worker started on its own once the record existed | synced after 4.1; blocks to mining 18.0 | +| own-restart | (the miner's own worker restart) | the card showed the worker fault; the app did not restart the miner (same pid) | fault on the card 1.0; mining again 4.0 | +| zero-ladder | MF-2 | three watchdog restarts at 10, 30, 120 s on the row, the reason in plain words, no faulted state, no "once already" words, mining back on its own | zero to rung 1 77.3; rung gaps 81.3, 101.9; healthy to mining 129.8 | +| no-status | MF-2 | the miner stopped with SIGSTOP: restarted for "no status line for 90 s", the stopped process killed, mining on a new process | quiet to restart 92.2; restart to mining 311.7 (the ladder's 300 s rung, carried from the step before: no five healthy minutes between them) | +| node-silent | node watchdog | the node stopped with SIGSTOP: restarted by the app (120 s rule plus the 30 s terminate grace), synced, mining again | quiet to restart 150.3; restart to synced 7.0; quiet to mining 170.4 | +| one-card-fails | MF-4 | two new cards listed, two healthy ones mining, the third held 30 minutes with "not usable on this driver: ", the healthy two still mining 90 s on, the pack exported once and reused 7 times | listed 48.1; two mining 48.1 | +| orphan-miner | MF-7 | a stray igneum-miner on the engine's node killed by the minute sweep, one log line per kill, the engine's own miner left alone | inject to kill 21.1 | +| card-appears | MF-3 | run 4: the card that appeared was listed (40.1 s) and mined with no tap (50.1 s); the card that left was NOT marked removed: the stand-in's `--list` lacked the real worker's header line, which the engine reads as "the tool did not answer" (right for a crashed driver). Rerun with the header (ecd79874): see run 5 | | + +FAULT lines the engine posted to the intake in run 4: 13 (watchdog, worker-fault, node-exit, orphan-miner classes). Which side: MF-1, MF-2, MF-3, MF-4, MF-6, MF-7 and the cards kind are app-side (0.3.20's app). MF-5 is both: the miner (the fork branch) prints the fields and caps the identities; the app reads them and keeps the worker. An app