docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (97255a4e) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
103 lines
5.6 KiB
JavaScript
Executable file
103 lines
5.6 KiB
JavaScript
Executable file
#!/usr/bin/env node
|
|
// A GPU worker stand-in that speaks the miner's serve protocol (ready / job / done / found / prepare / prepared /
|
|
// quit) and misbehaves on command, so the miner's fault guards and the app's watchdog can be measured on a Mac with
|
|
// no faulty GPU at hand. It never hashes: every `done` is invented, every `found` is wrong on purpose.
|
|
//
|
|
// FAKE_WORKER_CTL=<file> the file's first word is the mode, re-read before every job:
|
|
// ok every job: `done` after 100 ms with the job's nonce count as hashes (about 167 MH/s at 2^24 nonces)
|
|
// slow `done` after 5,000 ms (a self-heal build inside a job, a game on the GPU)
|
|
// fast `done` after 0.3 ms with the full count: a runtime answering success while running nothing
|
|
// (the 4 October 2026 gfx1036 fault)
|
|
// zero `done` after 100 ms with 0 hashes: the worker completes jobs and produces nothing
|
|
// silent never answers a job (the queue fills and nothing comes back)
|
|
// badfound a `found` with a wrong hash before every `done`: a worker on a wrong program
|
|
// preparefail every `prepare` is answered `prepare-failed`; jobs run as in ok
|
|
// exit the process exits with code 7 at the next job
|
|
// absent `--list` shows no device (the card is unplugged or in a problem code); jobs run as in ok
|
|
// selftest the worker prints a self-test failure and exits 3 before ready (MF-4)
|
|
// Extra lines in the control file: `devices N` (how many --list shows), `dev N <mode>` (a mode for device N only)
|
|
// The mode is read from argv `--mode <m>` when no control file is set. `--serve` is accepted and ignored.
|
|
// Lines on stderr are prefixed `fake-worker:` and never match a miner pattern.
|
|
|
|
import { readFileSync, existsSync, writeSync } from 'node:fs';
|
|
import { createRequire } from 'node:module';
|
|
const require = createRequire(import.meta.url);
|
|
import { createInterface } from 'node:readline';
|
|
|
|
// `--list` (the engine's OpenCL enumeration on Windows and Linux, src/detect.rs): one fake GPU, or none while the
|
|
// control file says `absent` (the hot-plug step: a card that appears later must start without a tap)
|
|
if (process.argv.includes('--list')) {
|
|
const m = (() => { try { return require('node:fs').readFileSync(process.env.FAKE_WORKER_CTL, 'utf8').trim().split(/\s+/)[0]; } catch { return 'ok'; } })();
|
|
const n = m === 'absent' ? 0 : (() => { try { const l = require('node:fs').readFileSync(process.env.FAKE_WORKER_CTL, 'utf8').split('\n').map(x => x.trim().split(/\s+/)).find(f => f[0] === 'devices'); return l ? Number(l[1]) : 1; } catch { return 1; } })();
|
|
for (let i = 0; i < n; i++) {
|
|
process.stdout.write(`[${i}] Fake GPU ${i + 1} | Fake Platform (OpenCL 1.2)\n GPU, vendor Fake Silicon, driver 1.0.0, OpenCL C 1.2, 8 compute units, 8192 MB\n`);
|
|
}
|
|
process.exit(0);
|
|
}
|
|
const ctl = process.env.FAKE_WORKER_CTL;
|
|
const argMode = (() => { const i = process.argv.indexOf('--mode'); return i >= 0 ? process.argv[i + 1] : 'ok'; })();
|
|
// the device this process serves (`--device N` from the engine's OpenCL path); the control file's first word is the
|
|
// mode for every device, a line `dev N <mode>` overrides it for device N, a line `devices N` sets how many --list shows
|
|
const device = (() => { const i = process.argv.indexOf('--device'); return i >= 0 ? process.argv[i + 1] : '0'; })();
|
|
function mode() {
|
|
if (ctl && existsSync(ctl)) {
|
|
const lines = readFileSync(ctl, 'utf8').trim().split('\n');
|
|
const own = lines.map(l => l.trim().split(/\s+/)).find(f => f[0] === 'dev' && f[1] === device);
|
|
if (own && own[2]) return own[2];
|
|
const m = (lines[0] || '').trim().split(/\s+/)[0];
|
|
if (m) return m;
|
|
}
|
|
return argMode;
|
|
}
|
|
function deviceCount() {
|
|
try { const l = readFileSync(ctl, 'utf8').split('\n').map(x => x.trim().split(/\s+/)).find(f => f[0] === 'devices'); return l ? Number(l[1]) : 1; } catch { return 1; }
|
|
}
|
|
// synchronous writes: a piped stdout is asynchronous in Node and process.exit would drop pending lines
|
|
const out = (s) => { writeSync(1, s + '\n'); };
|
|
const sleep = (ms) => new Promise(r => setTimeout(r, ms));
|
|
|
|
if (mode() === 'selftest') {
|
|
// MF-4: the worker fails its self-test and exits before it is ready (the Arc B580 on PC 1, 7 October 2026)
|
|
out('error 0 self-test FAIL: 3 of 96 vectors mismatched (fake worker, device ' + device + ')');
|
|
process.exit(3);
|
|
}
|
|
out('ready fake Fake_GPU dataset-log2 28 batch 16777216 prepare 1');
|
|
|
|
const queue = [];
|
|
let running = false;
|
|
async function pump() {
|
|
if (running) return;
|
|
running = true;
|
|
while (queue.length) {
|
|
const f = queue.shift();
|
|
const [id, , , nonceStart, nonces] = f;
|
|
const m = mode();
|
|
if (m === 'exit') process.exit(7);
|
|
if (m === 'silent') { await sleep(1_000_000); continue; }
|
|
const t0 = performance.now();
|
|
let hashes = Number(nonces);
|
|
let ms = 100;
|
|
if (m === 'slow') ms = 5000;
|
|
else if (m === 'fast') ms = 0.3;
|
|
else if (m === 'zero') hashes = 0;
|
|
await sleep(ms);
|
|
if (m === 'badfound') out(`found ${id} ${BigInt(nonceStart) + 17n} 0000000000000001`);
|
|
out(`done ${id} ${hashes} ${(performance.now() - t0).toFixed(2)}`);
|
|
}
|
|
running = false;
|
|
}
|
|
|
|
const rl = createInterface({ input: process.stdin });
|
|
rl.on('line', (line) => {
|
|
const f = line.trim().split(/\s+/);
|
|
switch (f[0]) {
|
|
case 'job': queue.push(f.slice(1)); pump(); break;
|
|
case 'prepare':
|
|
if (mode() === 'preparefail') out(`prepare-failed ${f[1]} ${f[2]} fake worker refuses`);
|
|
else setTimeout(() => out(`prepared ${f[1]} ${f[2]} 50 fake resident 2 programs 2 datasets`), 50);
|
|
break;
|
|
case 'quit': process.exit(0);
|
|
default: break;
|
|
}
|
|
});
|
|
rl.on('close', () => process.exit(0));
|