089b078a3a
docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (97255a4e) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
92 lines
6.3 KiB
JavaScript
Executable file
92 lines
6.3 KiB
JavaScript
Executable file
#!/usr/bin/env node
|
|
// Launch gate LG-4 as a job: "first share within 10 minutes of download in 9 of 10 fresh Windows installs"
|
|
// (docs/plans/miner-ui-5-first-share-runbook.md, docs/plans/miner-faults.md). Runs relay/playbooks/first-share.ps1 on
|
|
// Windows boxes over SSH, ten installs in all, and writes the evidence table the shipper's publish reads.
|
|
//
|
|
// node tools/fleet/first-share-gate.mjs run --version 0.3.20 --url https://dl.igneum.network/<token>/Igneum-Miner-Setup-0.3.20.exe
|
|
// [--boxes ~/igneum-fleet/windows-boxes.json] [--installs 10] [--allow-installed]
|
|
// node tools/fleet/first-share-gate.mjs check --version 0.3.20 exit 0 on a PASS line for that version, 1 otherwise
|
|
//
|
|
// Boxes file: a JSON array of the fleet registry's Windows rows ({kind:"windows", label, host, port, user, key, transport:"ssh"});
|
|
// with no row the gate writes "LG-4 not run: no Windows box" and `check` fails. Each box runs installs in turn; boxes run in
|
|
// parallel. Output: site/evidence/first-share-<version>.json (the runbook's keys per row, the medians, the pass count, the
|
|
// installer's sha256) and the gate line on stdout and in the file: "LG-4 PASS k/10" or "LG-4 FAIL k/10".
|
|
// Never touches PC 1 (the project lead's desk) and refuses a box whose installed app would be replaced unless --allow-installed is given
|
|
// and the box's row says "job_box": true (PC 2's case needs the project lead's word, recorded in the row by the fleet lane).
|
|
|
|
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
|
|
import { spawnSync } from 'node:child_process';
|
|
import { homedir } from 'node:os';
|
|
import { join, dirname, resolve } from 'node:path';
|
|
import { fileURLToPath } from 'node:url';
|
|
|
|
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
|
const args = process.argv.slice(2);
|
|
const cmd = args[0];
|
|
const opt = (n, d) => { const i = args.indexOf(n); return i >= 0 ? args[i + 1] : d; };
|
|
const flag = (n) => args.includes(n);
|
|
const version = opt('--version');
|
|
if (!version) { console.error('need --version'); process.exit(2); }
|
|
const evidence = join(ROOT, 'site', 'evidence', `first-share-${version}.json`);
|
|
|
|
function gateLine(rows, installs) {
|
|
const passed = rows.filter(r => r.pass).length;
|
|
return `LG-4 ${passed >= Math.ceil(installs * 0.9) ? 'PASS' : 'FAIL'} ${passed}/${installs}`;
|
|
}
|
|
|
|
if (cmd === 'check') {
|
|
if (!existsSync(evidence)) { console.log(`LG-4 not run for ${version}: no ${evidence}`); process.exit(1); }
|
|
const e = JSON.parse(readFileSync(evidence, 'utf8'));
|
|
console.log(e.gate);
|
|
process.exit(/^LG-4 PASS/.test(e.gate) ? 0 : 1);
|
|
}
|
|
if (cmd !== 'run') { console.log(readFileSync(fileURLToPath(import.meta.url), 'utf8').split('\n').slice(1, 16).map(l => l.replace(/^\/\/ ?/, '')).join('\n')); process.exit(2); }
|
|
|
|
const url = opt('--url');
|
|
if (!url) { console.error('need --url (the public installer)'); process.exit(2); }
|
|
const installs = Number(opt('--installs', '10'));
|
|
const boxesFile = opt('--boxes', join(homedir(), 'igneum-fleet', 'windows-boxes.json')).replace(/^~/, homedir());
|
|
const boxes = existsSync(boxesFile) ? JSON.parse(readFileSync(boxesFile, 'utf8')).filter(b => b.kind === 'windows' && b.transport === 'ssh' && b.host) : [];
|
|
mkdirSync(dirname(evidence), { recursive: true });
|
|
if (!boxes.length) {
|
|
const out = { version, url, date: new Date().toISOString(), installs, rows: [], gate: 'LG-4 not run: no Windows box', note: 'the fleet lane has no rented Windows GPU box; PC 2 over the relay needs the owner\'s word (docs/plans/miner-faults.md)' };
|
|
writeFileSync(evidence, JSON.stringify(out, null, 2) + '\n');
|
|
console.log(out.gate);
|
|
process.exit(1);
|
|
}
|
|
for (const b of boxes) {
|
|
if (/ae432dc7|pc1|pc-1/i.test(`${b.label} ${b.machine || ''}`)) { console.error(`refusing ${b.label}: PC 1 is never a job box`); process.exit(2); }
|
|
}
|
|
const allowInstalled = flag('--allow-installed');
|
|
const parse = (text) => {
|
|
const r = { pass: false };
|
|
for (const l of text.split('\n')) {
|
|
const m = /^RESULT (.*)$/.exec(l.trim()); if (!m) continue;
|
|
for (const kv of m[1].split(/\s+/)) { const [k, v] = kv.split('='); if (k === 'step') r._step = v; else if (r._step && k !== 'step') r[`${r._step}_${k}`] = v; else r[k] = v; }
|
|
}
|
|
r.pass = r.pass === 'true';
|
|
r.total_s = Number(r.total_s || 0);
|
|
return r;
|
|
};
|
|
const rows = [];
|
|
const perBox = Math.ceil(installs / boxes.length);
|
|
const started = Date.now();
|
|
// boxes in parallel (one ssh each, installs in turn inside it), the Mac waits
|
|
const procs = boxes.map((b) => {
|
|
const n = Math.min(perBox, installs - rows.length);
|
|
const remote = `$ErrorActionPreference='Continue'; $f=\"$env:TEMP\\first-share.ps1\"; [IO.File]::WriteAllText($f, [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String('${Buffer.from(readFileSync(join(ROOT, 'relay', 'playbooks', 'first-share.ps1'))).toString('base64')}'))); 1..${n} | ForEach-Object { Write-Host \"INSTALL $_\"; powershell -ExecutionPolicy Bypass -File $f -Url '${url}' ${allowInstalled && b.job_box ? '-AllowInstalled' : ''} }`;
|
|
const key = (b.key || '~/.ssh/igneum-fleet').replace(/^~/, homedir());
|
|
const p = spawnSync('ssh', ['-o', 'ConnectTimeout=20', '-o', 'StrictHostKeyChecking=accept-new', '-i', key, '-p', String(b.port || 22), `${b.user}@${b.host}`, 'powershell', '-NoProfile', '-Command', remote], { encoding: 'utf8', timeout: (n * 1900 + 120) * 1000 });
|
|
return { box: b, out: (p.stdout || '') + (p.stderr || ''), status: p.status };
|
|
});
|
|
for (const { box, out } of procs) {
|
|
for (const chunk of out.split(/^INSTALL \d+\r?$/m).slice(1)) {
|
|
const r = parse(chunk); r.box = box.label; r.gpu = box.gpu || ''; rows.push(r);
|
|
}
|
|
if (!out.includes('RESULT')) rows.push({ box: box.label, gpu: box.gpu || '', pass: false, total_s: 0, error: out.trim().split('\n').slice(-3).join(' | ') });
|
|
}
|
|
const med = (k) => { const v = rows.map(r => Number(r[k])).filter(x => x > 0).sort((a, b) => a - b); return v.length ? v[Math.floor(v.length / 2)] : null; };
|
|
const out = { version, url, date: new Date().toISOString(), installs, boxes: boxes.map(b => b.label), rows, median_total_s: med('total_s'), pass_count: rows.filter(r => r.pass).length, sha256: rows.find(r => r.download_sha256)?.download_sha256 || '', gate: gateLine(rows, installs), took_s: Math.round((Date.now() - started) / 1000) };
|
|
writeFileSync(evidence, JSON.stringify(out, null, 2) + '\n');
|
|
console.log(out.gate);
|
|
process.exit(/^LG-4 PASS/.test(out.gate) ? 0 : 1);
|