089b078a3a
docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (97255a4e) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
28 lines
1.5 KiB
Bash
Executable file
28 lines
1.5 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# No fault is permanent (the project lead, 7 October 2026, docs/plans/miner-faults.md MF-2): the app's engine may never mark a card
|
|
# "faulted" or say "restarted once already"; every fault retries on the ladder. This check fails when either word comes
|
|
# back into the engine's sources or its UI.
|
|
# tools/ci/permanent-fault-check.sh the tree
|
|
# tools/ci/permanent-fault-check.sh --self-test fires on a fixture with the words, passes on one without
|
|
set -uo pipefail
|
|
cd "$(git rev-parse --show-toplevel)" || exit 1
|
|
check() {
|
|
# $1: directory; a hit is a line in a .rs or .js file that sets or tests the faulted state or carries the words
|
|
grep -rnE --include='*.rs' --include='*.js' -e 'restarted once already' -e '"faulted"\.into\(\)' -e "state === 'faulted'" -e 'c\.state = "faulted"' "$1" 2>/dev/null
|
|
}
|
|
if [ "${1:-}" = "--self-test" ]; then
|
|
d="$(mktemp -d)"; trap 'rm -rf "$d"' EXIT
|
|
mkdir -p "$d/bad" "$d/good"
|
|
printf 'fn x() { c.state = "faulted".into(); }\n' > "$d/bad/a.rs"
|
|
printf 'fn x() { c.state = "restarting".into(); }\n' > "$d/good/a.rs"
|
|
if ! check "$d/bad" >/dev/null; then echo "self-test failed: the fixture with the faulted state passed"; exit 1; fi
|
|
if check "$d/good" >/dev/null; then echo "self-test failed: the clean fixture was flagged"; exit 1; fi
|
|
echo "self-test ok"; exit 0
|
|
fi
|
|
hits="$(check app/igneum-app/src; check app/igneum-app/ui)"
|
|
if [ -n "$hits" ]; then
|
|
echo "permanent fault words in the engine (docs/plans/miner-faults.md MF-2: no fault is permanent):"
|
|
echo "$hits"
|
|
exit 1
|
|
fi
|
|
exit 0
|