Merge remote-tracking branch 'origin/miner-reliability-21' into release-0.3.21

docs/plans/miner-faults.md

@@ -32,10 +32,13 @@ Standing rules behind every row (branch `miner-reliability`, off `release-0.3.19
 | MF-8 | 7 Oct 2026, every node build from 10db4b61 on a kept 0.3.17 datadir | "Node refuses its own kept datadir after an update": the app updates, the node dies at once (`DeserializationError(Io(Kind(UnexpectedEof)))` at `consensus/src/model/stores/virtual_state.rs:250`), the app restarts it in a loop, the miner never starts | `silent: bool` was added to `BlockRewardData` under `serde(default)` (10db4b61, the 0.3.16 vote-or-burn commit) and bincode ignores serde defaults, so the old row reads short; no canary saw it because every canary wiped. Fixed as ledger N13 on `release-0.3.20-node` b7cc37e7 (the store reads the v1 row and rewrites it) | Every schema change to a stored row ships with a versioned read path (the old shape read, rewritten in the new one); every release's gate starts the pinned binary on a COPY of a standing box's datadir, Linux-shaped and Windows-shaped (a copy of PC 2's), with the rewrite line or the synced line as the pass; the app never asks the user to wipe: a node that dies inside 10 s of its start is reported home (`node-exit` FAULT line) and the row says "the node cannot read its data after the update; the team has the report" | The node line's N13 test (the v1 row read and rewritten); the gate's kept-datadir start on both shapes; the injector step `kept-datadir` (`app-run.mjs --node-old <previous igneumd>`: the previous release's node writes the datadir to 150 blocks and stops, the engine's new node must open it, read synced, with one start and no exit line; its pod seconds are appended with the 0.3.21 run) | `canary kept-datadir start PASS (linux, windows)` in every cut's plan; `LG-4` row kept=true beside the nine wiped |
 | MF-9 | 7 Oct 2026, release-0.3.20-node b7cc37e7, the mixed-version gate | A node that stops slower than its restart: the restart died at once on the datadir LOCK of the stopping node | The listener watchdog slept its whole 10 s poll before it checked the shutdown flag, so a stop took up to 10 s while the app's restart followed inside it | Every poll loop in the node returns the moment shutdown is set (a `select` on the shutdown signal, never a sleep then a check); "a shutdown returns within one second" is a named gate of every release beside the kept-datadir start; the app's `stop_node` waits for the exit before the restart (`restart_node` schedules the start after the stop returns) | The node line's shutdown-latency test (stop at a random moment of the poll, return under 1 s); the mixed-version gate's restart case; `app-run.mjs` step `node-silent` reads the restart-to-synced seconds | `node shutdown under 1 s PASS` in every cut's plan |
 | MF-10 | 7 Oct 2026, the prover roll (3080, 3090, 4070, 5090) | Every proof fails in 12 s with `CudaRustError: named symbol not found`; the miner never notices and the box proves nothing for hours | The prover's `sp1-gpu-server` was built for another card's compute capability (sm_86 on the 3080 and 3090, sm_89 on the 4070, sm_120 on the 5090), so every kernel load fails the same way | The prover reads the card's compute capability at start (`nvidia-smi --query-gpu=compute_cap`) and refuses to start a server that does not match, with the reason on the prover tile and a FAULT line home ("the proving server is built for sm_89, this card is sm_120"); a proof that fails inside 15 s with the symbol error marks the server mismatched the same way; the 0.3.21 kit ships one server per architecture chosen at install, or a fat binary | `provedefault` test `a_server_for_another_card_is_refused` (sm_86 on 8.9 and 12.0 refused, 8.6 passes, a fat binary passes, unknowns pass); the fleet roll's paired line per box | `prover server matches the card PASS` per box in the roll, beside the kept-datadir and the shutdown gates |
-| MF-11 | 7 Oct 2026, PC 2 (1ccfe586), silent 10:46Z to 13:38Z (Kernel-Power 41 and 6008 "unexpected shutdown", no bugcheck, no dump; two such events on that box today; the 0.3.19 update-now at 10:34:38Z had returned cleanly, so the update is not the cause) | The machine goes silent and nothing tells anyone: no job, no relay task, no line reaches the team for three hours; the restart path today is a hand at the PC | A power loss took the whole PC; the relay agent runs under the app's lifetime instead of beside it, and no watcher raises a line when a machine stops reporting | An app or agent that does not report within 10 minutes raises a line somewhere that still runs (the intake's own silence watch per machine id: a FAULT line "no report from <id8> for N min" to the team's channel); the relay agent runs as a service that survives the app and restarts the app on boot (the relay lane, 0.3.21); the app's first act after any start is the read-back line "app <version> up, node <commit>" to the intake | the intake's silence-watch test (a machine whose last upload is older than 10 min is listed once); the relay lane's service test (the app killed, the agent still answers; the box rebooted, the app back) | `silence watch` line in the team channel per quiet box; PC 2's own line when it is back. Note for MF-8: PC 2's datadir, cut off mid-write twice by power loss, started clean on c4459193 (the PC 2 job's PASS) |
+| MF-11 | 7 Oct 2026, PC 2 (1ccfe586), silent from 10:46Z after the 0.3.19 update-now (the update-return lane's row; code on update-return 0b423697, the app half on update-return-21b a64c193f) | The app did not come back and nothing reached the PC for hours; the only recovery was a hand on the power button; the stale relay logon task popped "Windows cannot find 'igneum-agent'" at every boot | A power loss or hard reset of the whole PC (Kernel-Power 41, EventLog 6008, no BugCheck 1001, no minidump; three such events that day, the third with the 5060 Ti enclosure attached and no TDR, WHEA or Thunderbolt trace), while the update itself had returned in 10 s (10:34:28Z quit, 10:34:38Z "[ok] updated to 0.3.19"); the relay agent dead since 6 October behind a UAC prompt; the Windows helper's return path checked nothing after the installer's exit; the tuner at 575 W with proving on the same card nine minutes before the first drop | (1) the helper owns the return: exe set kept beside the app, the app launched by the helper (/IGNOTA=2), api/state polled 120 s, the kept set restored, one intake line either way; the host restarts a dead engine and answers the Restart Manager; the first act after an update is the read-back line; a boot after a power loss posts FAULT pc-restart; (2) the relay agent as the per-user logon task IgneumRelayService (LeastPrivilege, no UAC, restart on failure, full path, stale IgneumRelayAgent* removed), the start-app kind, the per-install hostname; (3) every wake request carries the ping; the console says "job channel silent since <time>, last job <name>" after 15 min; (4) the tuner never asks above the card's measured efficient point (5090 308 W, floored at the vendor's 400 W) unless Power control is on and the user raised the cap; a refused cap is asked again at 2 and 10 min then FAULT power-cap | ota::return_tests (the legacy known-failed first, ok, rolled-back, rolled-back-silent, relaunched, installer-failed, every sequence, the helper carries every step); bootcheck (pc2_boot_reads_as_a_power_loss, a_6008_alone, a_clean_boot); ember (without_a_ceiling_the_full_plan_asks_the_5090_for_575_w, the_tuner_never_asks_above_the_measured_efficient_point, power_control_on_with_a_cap_the_user_raised, a_card_not_in_the_table); engine a_refused_cap_climbs_the_retry_ladder_then_faults; jobrun wake_query/ping_query; relay service.test.mjs, handler start-app, wake ping | "update-return: ok" per machine in the rollout table, "job channel polled" on every console card, PC 2's own line when it is back. Note for MF-8: PC 2's datadir, cut off mid-write by power loss, started clean on c4459193 (the PC 2 job's PASS) |
 | MF-12 | 7 Oct 2026, the fleet lane's ten-member pool window (ten rented 3070s, `igneum-miner mine none ... --pool pool-1:4463`, daemon 03457d96, miner 9829bdf7); the pool lane's row (its branch pool-mf-row 343dd83b called it MF-11; renumbered here so the register has one number per class) | A pool member without a node of its own stopped hashing at the epoch boundary 78 to 79 and never resumed: every member printed `POOL SEEDS epoch c1fc2c7c... class 3` and `worker: info prepare started for epoch c1fc2c7c160f8a19 ... (NVRTC sm_86 in the background)`, that prepare never answered `prepared` or `prepare-failed`, the worker sat at 0 percent GPU serving the epoch-78 pack, the daemon's STATUS read `workers=9 accepted=0`, vardiff eased every member from shift 10 to 27 with no share, and no error line was printed anywhere (12:50Z to 13:08Z; cleared only by a restart on a pack exported from pool-1's node) | Open (the pool lane): the member's `prepare` line is the solo miner's shape (`igneum/miner/src/pool.rs` `prepare_line`), so the fault is either the member's pack write under `--prepare-packs` or the worker's NVRTC prepare on that pack; the fleet lane holds the worker lines and a solo control on the same box is the split | A member that sent a `prepare` and heard nothing for 120 s treats it as `prepare-failed`: it re-exports the pack, re-sends the prepare once, and on a second silence restarts its worker with the reason on its STATUS line and in the pool's `stats`; the pool daemon flags a member whose accepted count stays 0 across an epoch roll (`EPOCH STALL` line, the member's `online` false on the page); no pool member is ever silent across a boundary | owed (the pool lane): the pool's measure harness crosses two epoch boundaries with a member on `none` and a worker whose prepare is held (the stand-in worker of `tools/reliability/fake-worker.mjs`), and reads shares on both sides | the ten-member window's capture: every member's accepted count rises across every boundary the window crosses |
+| MF-13 | 7 Oct 2026, PC 2, every tune since the 14:35Z boot (and the 11:37 local tune of the 0.3.19 run before it); the update-return lane's row | "tune: request N refused: the helper did not run sequence N within 15 s (no line in helper.log)" on every request; the ladder never runs; the 5090 sits at "cap NOT applied" | The Power Helper's stale-command rule counted lines: the lines present at its start are skipped, the count reset only when the file shrank; the engine writes its four command lines with fs::write over a stale four-line cmd.txt, so the count never dropped and every new command read as "present at start"; helper.log shows "helper started" three times after the boot with no command executed; the task itself was Running with its exe on disk and a fresh heartbeat (not the stale-task class) | A rewrite is a new command: the skipped prefix must still read as it did at the start (powertask::effective_skip), else the skip is 0; a helper that gives no heartbeat or no line inside its window is a "FAULT power-helper:" line with its registration re-read; the registered probe wants the task enabled and its action exe on disk; a refused restore after a tune re-applies the cap on the FAULT power-cap ladder | powertask a_stale_command_file_does_not_run_at_start (extended: the same-length rewrite is the known-failed case first, then effective_skip resets it; a grown file keeps the stale prefix skipped); engine a_helper_that_does_not_answer_is_a_fault_line; powertask the_registration_is_per_user_highest_no_trigger_fixed_action (the probe's Test-Path) | the tune's first request acknowledged ("helper: <seq> nvidia-smi ...") on PC 1 and PC 2 after the 0.3.21 install; no "did not run sequence" line in a cut's PC runs |
 | MF-3 | 7 Oct 2026, PC 1, Intel Arc | The Intel driver's first install did not bind: the device sat in Code 12 at install time; the card never mined until a reboot | A driver installed while the device reports a problem code (12, 43, 31) does not bind; nothing re-scanned the device afterwards, and the app only re-enumerates | The app re-enumerates every 60 s and starts the worker the minute the OS drives the card (`hotplug::diff` recovered / revived, `settle_new`); the row says what to do while it does not ("reboot with the card attached; if it persists, reinstall the driver with the card attached"); a Windows host asks for a re-scan (`pnputil /scan-devices`) after a problem code is seen, every 5 minutes, at most 6 times (follow-up, host side) | `hotplug` test `a_driven_card_that_turns_faulty_is_errored_and_recovers_later`; `app-run.mjs` step `card-appears` (a card listed after 2 minutes starts without a tap) | `app-run card-appears PASS` |
 
+A kind, not a fault (the update-return lane): the "external" card kind, a USB4 or Thunderbolt router in the device's parent chain, so an eGPU reads eGPU on the Cards page and in the cards line (`detect.rs` ADAPTERS_SCRIPT, `classify_kind`; the view test "a card behind a USB4 or Thunderbolt router is an eGPU").
+
 ## Commits (7 October 2026)
 
 | Repo | Branch | Commit | What |