igneum/tools/reliability/fake-worker.mjs
igneum-labs 089b078a3a Miner app: plug, tune, play (the project lead, 7 October 2026): node readiness gate, the retry ladder, no permanent fault, fault lines to the intake, the signed cards job, the fault-class register
docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
  executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
  once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
  30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
  budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
  card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
  exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
  the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
  after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
  through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
  no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (97255a4e) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:11:48 +00:00

103 lines
5.6 KiB
JavaScript
Executable file

#!/usr/bin/env node
// A GPU worker stand-in that speaks the miner's serve protocol (ready / job / done / found / prepare / prepared /
// quit) and misbehaves on command, so the miner's fault guards and the app's watchdog can be measured on a Mac with
// no faulty GPU at hand. It never hashes: every `done` is invented, every `found` is wrong on purpose.
//
// FAKE_WORKER_CTL=<file> the file's first word is the mode, re-read before every job:
// ok every job: `done` after 100 ms with the job's nonce count as hashes (about 167 MH/s at 2^24 nonces)
// slow `done` after 5,000 ms (a self-heal build inside a job, a game on the GPU)
// fast `done` after 0.3 ms with the full count: a runtime answering success while running nothing
// (the 4 October 2026 gfx1036 fault)
// zero `done` after 100 ms with 0 hashes: the worker completes jobs and produces nothing
// silent never answers a job (the queue fills and nothing comes back)
// badfound a `found` with a wrong hash before every `done`: a worker on a wrong program
// preparefail every `prepare` is answered `prepare-failed`; jobs run as in ok
// exit the process exits with code 7 at the next job
// absent `--list` shows no device (the card is unplugged or in a problem code); jobs run as in ok
// selftest the worker prints a self-test failure and exits 3 before ready (MF-4)
// Extra lines in the control file: `devices N` (how many --list shows), `dev N <mode>` (a mode for device N only)
// The mode is read from argv `--mode <m>` when no control file is set. `--serve` is accepted and ignored.
// Lines on stderr are prefixed `fake-worker:` and never match a miner pattern.
import { readFileSync, existsSync, writeSync } from 'node:fs';
import { createRequire } from 'node:module';
const require = createRequire(import.meta.url);
import { createInterface } from 'node:readline';
// `--list` (the engine's OpenCL enumeration on Windows and Linux, src/detect.rs): one fake GPU, or none while the
// control file says `absent` (the hot-plug step: a card that appears later must start without a tap)
if (process.argv.includes('--list')) {
const m = (() => { try { return require('node:fs').readFileSync(process.env.FAKE_WORKER_CTL, 'utf8').trim().split(/\s+/)[0]; } catch { return 'ok'; } })();
const n = m === 'absent' ? 0 : (() => { try { const l = require('node:fs').readFileSync(process.env.FAKE_WORKER_CTL, 'utf8').split('\n').map(x => x.trim().split(/\s+/)).find(f => f[0] === 'devices'); return l ? Number(l[1]) : 1; } catch { return 1; } })();
for (let i = 0; i < n; i++) {
process.stdout.write(`[${i}] Fake GPU ${i + 1} | Fake Platform (OpenCL 1.2)\n GPU, vendor Fake Silicon, driver 1.0.0, OpenCL C 1.2, 8 compute units, 8192 MB\n`);
}
process.exit(0);
}
const ctl = process.env.FAKE_WORKER_CTL;
const argMode = (() => { const i = process.argv.indexOf('--mode'); return i >= 0 ? process.argv[i + 1] : 'ok'; })();
// the device this process serves (`--device N` from the engine's OpenCL path); the control file's first word is the
// mode for every device, a line `dev N <mode>` overrides it for device N, a line `devices N` sets how many --list shows
const device = (() => { const i = process.argv.indexOf('--device'); return i >= 0 ? process.argv[i + 1] : '0'; })();
function mode() {
if (ctl && existsSync(ctl)) {
const lines = readFileSync(ctl, 'utf8').trim().split('\n');
const own = lines.map(l => l.trim().split(/\s+/)).find(f => f[0] === 'dev' && f[1] === device);
if (own && own[2]) return own[2];
const m = (lines[0] || '').trim().split(/\s+/)[0];
if (m) return m;
}
return argMode;
}
function deviceCount() {
try { const l = readFileSync(ctl, 'utf8').split('\n').map(x => x.trim().split(/\s+/)).find(f => f[0] === 'devices'); return l ? Number(l[1]) : 1; } catch { return 1; }
}
// synchronous writes: a piped stdout is asynchronous in Node and process.exit would drop pending lines
const out = (s) => { writeSync(1, s + '\n'); };
const sleep = (ms) => new Promise(r => setTimeout(r, ms));
if (mode() === 'selftest') {
// MF-4: the worker fails its self-test and exits before it is ready (the Arc B580 on PC 1, 7 October 2026)
out('error 0 self-test FAIL: 3 of 96 vectors mismatched (fake worker, device ' + device + ')');
process.exit(3);
}
out('ready fake Fake_GPU dataset-log2 28 batch 16777216 prepare 1');
const queue = [];
let running = false;
async function pump() {
if (running) return;
running = true;
while (queue.length) {
const f = queue.shift();
const [id, , , nonceStart, nonces] = f;
const m = mode();
if (m === 'exit') process.exit(7);
if (m === 'silent') { await sleep(1_000_000); continue; }
const t0 = performance.now();
let hashes = Number(nonces);
let ms = 100;
if (m === 'slow') ms = 5000;
else if (m === 'fast') ms = 0.3;
else if (m === 'zero') hashes = 0;
await sleep(ms);
if (m === 'badfound') out(`found ${id} ${BigInt(nonceStart) + 17n} 0000000000000001`);
out(`done ${id} ${hashes} ${(performance.now() - t0).toFixed(2)}`);
}
running = false;
}
const rl = createInterface({ input: process.stdin });
rl.on('line', (line) => {
const f = line.trim().split(/\s+/);
switch (f[0]) {
case 'job': queue.push(f.slice(1)); pump(); break;
case 'prepare':
if (mode() === 'preparefail') out(`prepare-failed ${f[1]} ${f[2]} fake worker refuses`);
else setTimeout(() => out(`prepared ${f[1]} ${f[2]} 50 fake resident 2 programs 2 datasets`), 50);
break;
case 'quit': process.exit(0);
default: break;
}
});
rl.on('close', () => process.exit(0));