igneum/relay/lib/parse.mjs
igneum-labs 41631431c0 Workers: a pack's seed words are its program attempt's words, not the bare seed's (epoch 34 incident, 5 October 2026)
From 18:23Z both Windows workers (CUDA on PC 1 and PC 2, OpenCL on PC 1 after the 18:34Z node restart)
refused every pack for epoch 34 with "the epoch seed bytes do not give the pack's IGNEUM_SEEDW_INIT", and
the miner and the app restarted them every 5 to 60 s until 18:44Z and beyond. The packs were correct.
The generator retries a rejected candidate with seed || k_le32 (attempt_words); epoch 34's attempt 0 was
rejected (246 of 16384 final register values saturated, limit 163) and attempt 1 accepted, so the pack
carried attempt 1's words while pf_load (proto-cuda/nvrtc/packfile.h) derived the expected words from
the bare seed. Both workers also matched jobs to pairs by those bare-seed words, so even a loaded pack
of a retried program would have answered "epoch seed mismatch" on every job.

The rule, in one place per language:
- packfile.h: pf_program_words(bytes, attempt); pf_load reads IGNEUM_PROGRAM_ATTEMPT and checks the
  attempt's words; the refusal says "program pack and its seeds disagree: IGNEUM_SEEDW_INIT is not
  attempt N of the epoch seed ..." in plain words.
- worker.cpp and proto-opencl/host.c: a job belongs to a pair when the seed hex the node sent is the
  pair's (pairIs); the compiled-in placeholder pack keeps the word comparison.
- igneum-pow/src/packcheck.rs: verify_pack_texts / verify_pack_dir, the same rule in Rust; the miner
  checks every pack it writes with it before a worker sees it (vendor/igneum-node pack-loop branch).
  Tests pin the attempt vectors of epoch 34 on both sides (one vector, two implementations), that
  epoch 34 is attempt 1 and epoch 33 attempt 0, a known-good pack of a later attempt, a known-mismatched
  (out of date) pack, and self-contradicting packs.
- proto-cuda/nvrtc/emu/packfile-test.c (+ .sh, in CI): pf_load on a known-good attempt-1 pack, the
  checked-in attempt-0 pack, and the known-mismatched bare-words pack.

The app (app/igneum-app):
- watchdog.rs: PACK_OUT_OF_DATE_CODE 44, PackRebuilds (at most 3 pack exports per epoch, then the card
  shows the reason), pack_refusal (the worker's "error 0 pack" line and the miner's "PACK OUT OF DATE"
  line), pack_epoch_of; tests on the incident lines, known-good and known-mismatched.
- engine.rs: exit 44 exports the pack again before the restart instead of a blind restart, the strip
  says "program pack out of date, rebuilding", the card and the log name the condition; at the cap the
  card is marked failed with the reason and tries again in 10 minutes.

The relay (relay/lib/parse.mjs): PACK_MISMATCH; the card reads "pack mismatch, rebuilding (N refusals
in the tail, M restarts)" in `node tools/console.mjs machines` instead of a bare restart count; tests
on the PC 2 tail of 18:27Z and a healthy tail.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 19:10:07 +00:00

139 lines
11 KiB
JavaScript

// Pure parsers of the console (no dependencies, so `node --test relay/test` covers them): the log-intake labels,
// a miner log's tail, and the stale mark on a card whose last STATUS line is old.
export const kv = (text, k) => { const m = new RegExp(`(?:^|[\\s(])${k}=([^\\s,)]+)`).exec(text); return m ? m[1] : null; };
export const kvNum = (text, k) => { const v = kv(text, k); return v === null ? null : Number(v); };
export const lastMatch = (lines, re) => { for (let i = lines.length - 1; i >= 0; i--) { const m = re.exec(lines[i]); if (m) return m; } return null; };
export const FAULT = /WORKER MISMATCH|worker error|worker exited|panicked|CUDA error|submit error/;
// A program pack the worker refused at start or the miner found stale (5 October 2026, epoch 34 on both PCs: the
// bare restart count hid an hour-long loop). The card says "pack mismatch, rebuilding" instead of a bare count.
export const PACK_MISMATCH = /error 0 pack |PACK OUT OF DATE|program pack out of date|program pack and its seeds disagree/;
// A card's hash is "now" only while its worker's STATUS line (every 30 s, uploaded every 60 s) is this fresh; older
// than this the card is marked stale and its hash leaves the machine total (4 October 2026: PC 2 showed 117 MH/s "now"
// from a STATUS line four minutes old while the shard prover held the GPU; its upload gap was 1,860 s). One missed
// upload (120 s gap, seen on the Mac the same evening) plus a 30 s STATUS age plus the 10 s API cache is 160 s, so the
// mark starts at 180 s, the same as SILENT_S for a machine.
export const STALE_S = 180;
export function markStale(cards, now = Date.now()) {
for (const c of cards) {
const age = c.status_at ? Math.round((now - Date.parse(c.status_at)) / 1000) : null;
c.status_age_s = age;
c.stale = age === null || age > STALE_S;
}
return cards;
}
// Labels the app uploads: nodelog-<os>-<id8>, miner-<vendor>-<id8>-<n> (vendor as the app names the worker: nvidia, amd,
// mac/metal, opencl, and other/intel for the OpenCL fallback on an iGPU; 4 October 2026: PC 37ba0461's other-* worker
// was dropped and the card showed 0 MH/s), <os>-<id8> for the app log; the launchers before the app are "legacy".
export function parseLabel(label, machine) {
let m;
if ((m = /^nodelog-(win|mac)-([0-9a-f]{8})$/.exec(label))) return { id: m[2], platform: m[1], stream: 'node' };
if ((m = /^miner-(nvidia|amd|mac|metal|opencl|other|intel)-([0-9a-f]{8})-(\d+)$/.exec(label))) return { id: m[2], platform: m[1] === 'mac' || m[1] === 'metal' ? 'mac' : 'win', stream: 'miner', vendor: m[1], card: Number(m[3]) };
if ((m = /^(win|mac)-([0-9a-f]{8})$/.exec(label))) return { id: m[2], platform: m[1], stream: 'app' };
// the launchers before the app: nodelog-<host>, nvidia-<host>[-n], amd-<host>[-n], igneum-<host>, launcher-<host>
if ((m = /^nodelog-(.+)$/.exec(label))) return { id: 'legacy:' + m[1], platform: 'legacy', stream: 'node' };
if ((m = /^(nvidia|amd)-(.+?)(?:-(\d+))?$/.exec(label))) return { id: 'legacy:' + m[2], platform: 'legacy', stream: 'miner', vendor: m[1], card: Number(m[3] || 1) };
if ((m = /^(igneum|launcher)-(.+)$/.exec(label))) return { id: 'legacy:' + m[2], platform: 'legacy', stream: 'app' };
return null;
}
export function parseMinerTail(tail) {
const lines = tail.split('\n');
const c = { hash_now: 0, hash_avg: 0, accepted: 0, rejected: 0, mismatched: 0, restarts: 0, identities: null, synced: null, uptime_s: null, status_at: null, fault: null, faults: 0, template_age_s: null };
const st = lastMatch(lines, /^(\d+(?:\.\d+)?) STATUS '([^']+)'.*$/);
if (st) {
const t = st[0];
c.status_at = new Date(Number(st[1]) * 1000).toISOString();
c.hash_avg = kvNum(t, 'hash') ?? 0;
c.hash_now = kvNum(t, 'now') ?? c.hash_avg;
c.accepted = kvNum(t, 'accepted') ?? 0;
c.rejected = kvNum(t, 'rejected') ?? 0;
c.mismatched = kvNum(t, 'mismatched') ?? 0;
c.restarts = kvNum(t, 'restarts') ?? 0;
c.identities = kvNum(t, 'identities');
c.synced = kv(t, 'synced') === 'true';
c.template_age_s = (() => { const v = kv(t, 'template_age'); return v ? Number(v.replace('s', '')) : null; })();
const up = /STATUS '[^']+' \[[^\]]*\]: (\d+)s /.exec(t); if (up) c.uptime_s = Number(up[1]);
} else {
// no STATUS line in the tail: count the ACCEPTED lines at least
c.accepted = lines.filter(l => l.includes(' ACCEPTED block')).length;
}
const faults = lines.filter(l => FAULT.test(l));
c.faults = faults.length;
if (faults.length) c.fault = faults[faults.length - 1].replace(/^\d+(\.\d+)? /, '').slice(0, 200);
const packs = lines.filter(l => PACK_MISMATCH.test(l));
c.pack_mismatch = packs.length > 0;
if (packs.length) c.fault = `pack mismatch, rebuilding (${packs.length} refusal${packs.length === 1 ? '' : 's'} in the tail${c.restarts ? ', ' + c.restarts + ' restarts' : ''})`;
const acc = lastMatch(lines, /^(\d+(?:\.\d+)?) ACCEPTED block/);
if (acc) c.last_accepted_at = new Date(Number(acc[1]) * 1000).toISOString();
return c;
}
// The app log: the IGNEUM-APP header, the status line, OTA and power-cap lines, telemetry, events, jobs, and a clean stop.
export const HEADER = /IGNEUM-APP version=(\S+) machine=(\S+) platform=(\S+) node=(\S+)/;
export function parseHeader(text) {
const m = HEADER.exec(text || '');
return m ? { version: m[1], id8: m[2], platform: m[3], node: m[4].replace(/^igneumd[_\/]/, '') } : null;
}
export function parseAppTail(tail) {
const lines = tail.split('\n');
const a = { telemetry: [], events: [], jobs: [] };
let m;
if ((m = /Igneum Miner (\S+) on /.exec(tail))) a.version = m[1];
const h = lastMatch(lines, HEADER); if (h) { a.version = h[1]; a.node_version = h[4].replace(/^igneumd[_\/]/, ''); }
const stampOf = l => { const t = /^(\d+)(?:\.\d+)? /.exec(l); return t ? new Date(Number(t[1]) * 1000).toISOString() : null; };
const strip = l => l.replace(/^\d+(\.\d+)? /, '');
// OTA: the newest line decides the state shown on the card
const ota = lastMatch(lines, /^\d+(?:\.\d+)? (?:\[\w+\] )?(OTA: .+)$/);
if (ota) { const t = ota[1]; a.ota = { text: t.slice(0, 220), at: stampOf(ota[0]), state: /waiting for administrator approval|approval not given/.test(t) ? 'waiting for approval' : /deferred/.test(t) ? 'deferred' : /updated to|installed|applied/.test(t) ? 'updated' : /download/.test(t) ? 'downloading' : 'note' }; }
// The app's other update lines (app/igneum-app/src/ota.rs): the newest of them, the OTA line above and the plain
// "update check" decides the state shown on the card. 4 October 2026, 20:25: the Mac had downloaded, verified and
// staged 0.3.4 ("update: ... downloaded and verified", "[ok] ... is ready; it installs at the next safe moment") and
// its card said "0.3.3 is current" from a check 70 minutes older; a finished update's new run says only "is current".
const SHAPES = [
[/^(\d+)(?:\.\d+)? \[info\] Igneum Miner (\d+\.\d+\.\d+)\S* is available: downloading/, 'downloading'],
[/^(\d+)(?:\.\d+)? update: .*Igneum-Miner-(\d+\.\d+\.\d+)\S* downloaded and verified/, 'downloaded'],
[/^(\d+)(?:\.\d+)? \[ok\] Igneum Miner (\d+\.\d+\.\d+)\S* is ready; it installs at the next safe moment/, 'staged'],
[/^(\d+)(?:\.\d+)? \[info\] installing Igneum Miner (\d+\.\d+\.\d+)/, 'installing'],
[/^(\d+)(?:\.\d+)? update to (\d+\.\d+\.\d+)\S* complete/, 'updated'],
[/^(\d+)(?:\.\d+)? update: (\d+\.\d+\.\d+)\S* is marked failed/, 'failed'],
[/^(\d+)(?:\.\d+)? update check: (\d+\.\d+\.\d+)\S* is published but has no \S+ build yet/, 'no build'],
[/^(\d+)(?:\.\d+)? update check: (\d+\.\d+\.\d+)\S* is current \(manifest \S+\)/, 'current'],
];
let best = a.ota ? { at: Number((/^(\d+)/.exec(ota[0]) || [])[1]) } : null;
for (const [re, state] of SHAPES) {
const m = lastMatch(lines, re);
if (m && (!best || Number(m[1]) >= best.at)) best = { at: Number(m[1]), ota: { text: strip(m[0]).replace(/^\[\w+\] /, '').slice(0, 220), at: stampOf(m[0]), state, version: m[2] } };
}
if (best && best.ota) a.ota = best.ota;
const pc = lastMatch(lines, /^\d+(?:\.\d+)? (?:\[\w+\] )?(?:GPU )?(power cap.*)$/i);
if (pc) a.power_cap = { text: pc[1].slice(0, 220), at: stampOf(pc[0]), ok: !/NOT applied|nothing to read back/.test(pc[1]) };
const jf = lastMatch(lines, /^\d+(?:\.\d+)? jobs: (.+)$/);
if (jf) a.jobs_file = { text: jf[1].slice(0, 220), at: stampOf(jf[0]) };
a.jobs = lines.filter(l => /^\d+(?:\.\d+)? (?:\[\w+\] )?job [^:]+[:(]/.test(l) || /^\d+(?:\.\d+)? (?:\[\w+\] )?\d+ remote jobs? received/.test(l)).slice(-6).map(l => ({ at: stampOf(l), text: strip(l).replace(/^\[\w+\] /, '').slice(0, 220) }));
for (const l of lines) {
const s = /stability: (.+?): draw p95 (\d+) W, max (\d+) W \(cap (\d+) W\), max GPU (\d+) C, max memory (\d+) C, (\d+) samples/.exec(l);
if (s) { const i = a.telemetry.findIndex(t => t.name === s[1]); const t = { name: s[1], p95_w: Number(s[2]), max_w: Number(s[3]), cap_w: Number(s[4]), max_gpu_c: Number(s[5]), max_mem_c: Number(s[6]), samples: Number(s[7]) }; if (i >= 0) a.telemetry[i] = t; else a.telemetry.push(t); }
const e = /^(\d+(?:\.\d+)?) \[(error|ok|block|warn)\] (.+)$/.exec(l);
if (e && e[2] !== 'block') a.events.push({ at: new Date(Number(e[1]) * 1000).toISOString(), kind: e[2], text: e[3].slice(0, 200) });
}
a.events = a.events.slice(-6);
// A clean stop: the app logs "quit: ..." (Quit, Stop, or an update: "[info] installing Igneum Miner ...") and uploads
// once more before it exits; when no status line follows, the machine was stopped on purpose, not lost (4 October
// 2026: a Mac stopped at 14:47 UTC read "silent 4h" for the whole afternoon, the same as a crash or a lost network).
const stop = lastMatch(lines, /^(\d+)(?:\.\d+)? (quit: .+|stopped|\[info\] installing Igneum Miner .+)$/);
const lastStatus = lastMatch(lines, /^(\d+)(?:\.\d+)? status: /);
if (stop && (!lastStatus || Number(stop[1]) >= Number(lastStatus[1]))) {
const update = lines.some(l => /\[info\] installing Igneum Miner/.test(l) && Number((/^(\d+)/.exec(l) || [])[1]) >= Number(stop[1]) - 30);
a.stopped = { at: stampOf(stop[0]), reason: update ? 'update' : 'quit', text: strip(stop[0]).slice(0, 120) };
}
if ((m = lastMatch(lines, /^(\d+)(?:\.\d+)? status: (.+?) \| node (.+?) \| up (.+)$/))) {
a.status = { miner: m[2], node: m[3], up: m[4], at: new Date(Number(m[1]) * 1000).toISOString() };
const acc = /accepted (\d+) blocks \((\d+) this run\), ([\d.]+) MH\/s, (\w+)/.exec(m[2]);
if (acc) Object.assign(a.status, { accepted_total: Number(acc[1]), accepted_run: Number(acc[2]), mhs: Number(acc[3]), mining: acc[4] });
const nd = /(\d+) blocks, (\d+) peers, (\w+)/.exec(m[3]);
if (nd) Object.assign(a.status, { node_blocks: Number(nd[1]), node_peers: Number(nd[2]), node_state: nd[3] });
}
return a;
}