From c4770b20b81a8b5b9efceba3ef2e772b65e6c27a Mon Sep 17 00:00:00 2001 From: igneum-labs <337424239+igneum-labs@users.noreply.github.com> Date: Sun, 4 Oct 2026 20:02:48 +0000 Subject: [PATCH 1/3] bugs.md: the observer double-lock fix proven on the live stream (21 locks, 0 duplicates) Co-Authored-By: Claude Fable 5.1 --- docs/bugs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/bugs.md b/docs/bugs.md index d436523ce..37d2a586f 100644 --- a/docs/bugs.md +++ b/docs/bugs.md @@ -15,7 +15,7 @@ shard run reported as exit 0, 7a7e873). | 4 Oct 2026 | `vercel env add` from `site/` fails with "Could not retrieve Project Settings" | `site/.vercel/project.json` links the [other-business] team's `igneum` project; the live site (igneum.network, igneum.com, the GitHub integration) is the `igneum` team's project of the same name, which the igneum login reads and the [other-business] link does not | documented in `packaging/README-ship.md` (link, env ls, env add, deploy); no env set | `env ls` from a scratch link to the igneum-team project lists the two names; a branch push produced `igneum-git-` | | 4 Oct 2026 | `tools/jobs.mjs` and `publish-jobs.sh` print the downloads-folder token inside URLs on every run (X24 pattern) | the tokened base URL is echoed as is | the token masked as `` in every printed URL | by eye, this log's own transcript | | 4 Oct 2026 | round 4 X28 and X24, the parts under an hour: `===` on secrets, no HSTS on the relay, the relay token printed by `tools/relay.mjs list` and `watch` | as the review said | c1f59fb: `sameSecret` (timingSafeEqual, `relay/lib/auth.mjs`, test in CI), `Strict-Transport-Security` in `relay/vercel.json`, `/r/` printed (only `url` prints the real one) | relay deployed: key auth 200, wrong key and token 401, token path 200, HSTS header present | -| 4 Oct 2026 | the live feed showed two "checkpoint N locked" events 30 ms apart (1122, 1172, 1230, 1258, 1259 in 400 observer lines), and 708 of 764 locked checkpoints in `live_checkpoints` had `votes_seen` 0 | `finalityTick` read the state, awaited two SQL writes, then set the map; the `finalityLockNotification` handler checked the same map synchronously in between and recorded the lock too; a lock claimed by the notification was never upserted again, so the poll's `votes_seen` never landed | 7de1bdb: the poll claims the state before its first await; a `checkpointDetailed` set makes the poll fill `votes_seen` once | duplicates before the restart: 5 in 400 lines; after: see the next row's restart and pass 5 | +| 4 Oct 2026 | the live feed showed two "checkpoint N locked" events 30 ms apart (1122, 1172, 1230, 1258, 1259 in 400 observer lines), and 708 of 764 locked checkpoints in `live_checkpoints` had `votes_seen` 0 | `finalityTick` read the state, awaited two SQL writes, then set the map; the `finalityLockNotification` handler checked the same map synchronously in between and recorded the lock too; a lock claimed by the notification was never upserted again, so the poll's `votes_seen` never landed | 7de1bdb: the poll claims the state before its first await; a `checkpointDetailed` set makes the poll fill `votes_seen` once | before: 5 duplicates in 400 lines; after the 19:50:33 UTC restart: 21 locks (1297 to 1317), 0 duplicates, 0 write failures; 1300 was notification-first and the poll filled it to 17 votes. The zeros that remain are the node's own count (`finality.rs:770`, its vote map for that hash, empty when the lock came by certificate), not the observer's. Index 1296 appears twice on the feed: it locked inside the restart window, one write per process, a restart-boundary one-off | | 4 Oct 2026 | the observer kept running old code after its fix was on master and HEAD had moved past it | `autosync.sh` restarted the observer only when its own fast-forward moved HEAD; a pull by hand (19:35 UTC, HEAD to 8a77b85) bypassed it | autosync compares the checked-out `tools/observer` tree id with a marker written at each restart and restarts on any difference; `autosync.sh check` says what it would do | `check` with no marker: "restart due", exit 3; with the marker equal to the tree: "not due", exit 0; the shared checkout at that moment: due | | 4 Oct 2026 | the Mac card read 0.0 MH/s for a minute while its blocks were still accepted (after the stale mark shipped) | STALE_S 120 s left no margin: one missed 60 s upload (a 120 s gap at 19:45:59 UTC) plus a 30 s STATUS age plus the 10 s cache | STALE_S 180 s (one missed upload is not stale; PC 2's real case was a 1,860 s gap) | test: 150 s is fresh, 240 s stale | From 21ccb65f40257ad585bb1340c1358712b2e642be Mon Sep 17 00:00:00 2001 From: igneum-labs <337424239+igneum-labs@users.noreply.github.com> Date: Sun, 4 Oct 2026 20:04:18 +0000 Subject: [PATCH 2/3] Console: a machine whose app logged a clean quit or an update, with no status line after it, shows 'stopped (quit|update) N ago' instead of 'silent' (parseAppTail moved to relay/lib/parse.mjs, test) Co-Authored-By: Claude Fable 5.1 --- relay/api/console.mjs | 40 ++------------------------------ relay/lib/parse.mjs | 48 +++++++++++++++++++++++++++++++++++++++ relay/test/parse.test.mjs | 22 ++++++++++++++++++ relay/ui.html | 5 ++-- tools/console.mjs | 2 +- 5 files changed, 76 insertions(+), 41 deletions(-) diff --git a/relay/api/console.mjs b/relay/api/console.mjs index 111024789..06275da40 100644 --- a/relay/api/console.mjs +++ b/relay/api/console.mjs @@ -12,7 +12,7 @@ // POST post {kind,title,body,who,key?,meta?} one item; with key it upserts // POST sync {items:[...]} bulk upsert by key import { neon, authed, readJson, str, iso } from '../lib/relay.mjs'; -import { kv, kvNum, lastMatch, FAULT, parseLabel, parseMinerTail, STALE_S, markStale } from '../lib/parse.mjs'; +import { kv, kvNum, lastMatch, FAULT, parseLabel, parseMinerTail, parseHeader, parseAppTail, STALE_S, markStale } from '../lib/parse.mjs'; const json = (res, status, obj) => { res.status(status).setHeader('Content-Type', 'application/json; charset=utf-8'); res.end(JSON.stringify(obj)); }; const CACHE_MS = 10_000; @@ -55,43 +55,6 @@ function parseNodeTail(tail) { n.last_warning = errs.length ? errs[errs.length - 1].replace(/^\S+ \S+ /, '').slice(0, 200) : null; return n; } -const HEADER = /IGNEUM-APP version=(\S+) machine=(\S+) platform=(\S+) node=(\S+)/; -function parseHeader(text) { - const m = HEADER.exec(text || ''); - return m ? { version: m[1], id8: m[2], platform: m[3], node: m[4].replace(/^igneumd[_\/]/, '') } : null; -} -function parseAppTail(tail) { - const lines = tail.split('\n'); - const a = { telemetry: [], events: [], jobs: [] }; - let m; - if ((m = /Igneum Miner (\S+) on /.exec(tail))) a.version = m[1]; - const h = lastMatch(lines, HEADER); if (h) { a.version = h[1]; a.node_version = h[4].replace(/^igneumd[_\/]/, ''); } - const stampOf = l => { const t = /^(\d+)(?:\.\d+)? /.exec(l); return t ? new Date(Number(t[1]) * 1000).toISOString() : null; }; - const strip = l => l.replace(/^\d+(\.\d+)? /, ''); - // OTA: the newest line decides the state shown on the card - const ota = lastMatch(lines, /^\d+(?:\.\d+)? (?:\[\w+\] )?(OTA: .+)$/); - if (ota) { const t = ota[1]; a.ota = { text: t.slice(0, 220), at: stampOf(ota[0]), state: /waiting for administrator approval|approval not given/.test(t) ? 'waiting for approval' : /deferred/.test(t) ? 'deferred' : /updated to|installed|applied/.test(t) ? 'updated' : /download/.test(t) ? 'downloading' : 'note' }; } - const pc = lastMatch(lines, /^\d+(?:\.\d+)? (?:\[\w+\] )?(?:GPU )?(power cap.*)$/i); - if (pc) a.power_cap = { text: pc[1].slice(0, 220), at: stampOf(pc[0]), ok: !/NOT applied|nothing to read back/.test(pc[1]) }; - const jf = lastMatch(lines, /^\d+(?:\.\d+)? jobs: (.+)$/); - if (jf) a.jobs_file = { text: jf[1].slice(0, 220), at: stampOf(jf[0]) }; - a.jobs = lines.filter(l => /^\d+(?:\.\d+)? (?:\[\w+\] )?job [^:]+[:(]/.test(l) || /^\d+(?:\.\d+)? (?:\[\w+\] )?\d+ remote jobs? received/.test(l)).slice(-6).map(l => ({ at: stampOf(l), text: strip(l).replace(/^\[\w+\] /, '').slice(0, 220) })); - for (const l of lines) { - const s = /stability: (.+?): draw p95 (\d+) W, max (\d+) W \(cap (\d+) W\), max GPU (\d+) C, max memory (\d+) C, (\d+) samples/.exec(l); - if (s) { const i = a.telemetry.findIndex(t => t.name === s[1]); const t = { name: s[1], p95_w: Number(s[2]), max_w: Number(s[3]), cap_w: Number(s[4]), max_gpu_c: Number(s[5]), max_mem_c: Number(s[6]), samples: Number(s[7]) }; if (i >= 0) a.telemetry[i] = t; else a.telemetry.push(t); } - const e = /^(\d+(?:\.\d+)?) \[(error|ok|block|warn)\] (.+)$/.exec(l); - if (e && e[2] !== 'block') a.events.push({ at: new Date(Number(e[1]) * 1000).toISOString(), kind: e[2], text: e[3].slice(0, 200) }); - } - a.events = a.events.slice(-6); - if ((m = lastMatch(lines, /^(\d+)(?:\.\d+)? status: (.+?) \| node (.+?) \| up (.+)$/))) { - a.status = { miner: m[2], node: m[3], up: m[4], at: new Date(Number(m[1]) * 1000).toISOString() }; - const acc = /accepted (\d+) blocks \((\d+) this run\), ([\d.]+) MH\/s, (\w+)/.exec(m[2]); - if (acc) Object.assign(a.status, { accepted_total: Number(acc[1]), accepted_run: Number(acc[2]), mhs: Number(acc[3]), mining: acc[4] }); - const nd = /(\d+) blocks, (\d+) peers, (\w+)/.exec(m[3]); - if (nd) Object.assign(a.status, { node_blocks: Number(nd[1]), node_peers: Number(nd[2]), node_state: nd[3] }); - } - return a; -} async function machines(sql) { // node logs get a longer tail: the connection-manager (peers) line is rare on a busy node const rows = await sql(`SELECT DISTINCT ON (label) label, machine, run_id, received_at, left(lines, 300) AS head, @@ -133,6 +96,7 @@ async function machines(sql) { name: id8 ? (NAMES[id8] || (g.platform === 'mac' ? 'Mac' : 'PC ' + id8)) : g.id.slice(7), host: [...g.hosts][0] || null, legacy: !id8, last_seen: g.last_seen, silent_s, silent: silent_s === null || silent_s > SILENT_S, + stopped: (silent_s === null || silent_s > SILENT_S) && g.app && g.app.stopped ? g.app.stopped : null, app_version: (g.app && g.app.version) || (g.header && g.header.version) || null, app: g.app ? { at: g.app.at, status: g.app.status || null, ota: g.app.ota || null, power_cap: g.app.power_cap || null, jobs_file: g.app.jobs_file || null, jobs: g.app.jobs || [] } : null, node: { version: g.node.version || (g.header && g.header.node) || (g.app && g.app.node_version) || null, daa: g.node.daa ?? null, blue_score: g.node.blue_score ?? null, checkpoint: g.node.checkpoint ?? null, diff --git a/relay/lib/parse.mjs b/relay/lib/parse.mjs index e185bd489..9565ca731 100644 --- a/relay/lib/parse.mjs +++ b/relay/lib/parse.mjs @@ -63,3 +63,51 @@ export function parseMinerTail(tail) { if (acc) c.last_accepted_at = new Date(Number(acc[1]) * 1000).toISOString(); return c; } + +// The app log: the IGNEUM-APP header, the status line, OTA and power-cap lines, telemetry, events, jobs, and a clean stop. +export const HEADER = /IGNEUM-APP version=(\S+) machine=(\S+) platform=(\S+) node=(\S+)/; +export function parseHeader(text) { + const m = HEADER.exec(text || ''); + return m ? { version: m[1], id8: m[2], platform: m[3], node: m[4].replace(/^igneumd[_\/]/, '') } : null; +} +export function parseAppTail(tail) { + const lines = tail.split('\n'); + const a = { telemetry: [], events: [], jobs: [] }; + let m; + if ((m = /Igneum Miner (\S+) on /.exec(tail))) a.version = m[1]; + const h = lastMatch(lines, HEADER); if (h) { a.version = h[1]; a.node_version = h[4].replace(/^igneumd[_\/]/, ''); } + const stampOf = l => { const t = /^(\d+)(?:\.\d+)? /.exec(l); return t ? new Date(Number(t[1]) * 1000).toISOString() : null; }; + const strip = l => l.replace(/^\d+(\.\d+)? /, ''); + // OTA: the newest line decides the state shown on the card + const ota = lastMatch(lines, /^\d+(?:\.\d+)? (?:\[\w+\] )?(OTA: .+)$/); + if (ota) { const t = ota[1]; a.ota = { text: t.slice(0, 220), at: stampOf(ota[0]), state: /waiting for administrator approval|approval not given/.test(t) ? 'waiting for approval' : /deferred/.test(t) ? 'deferred' : /updated to|installed|applied/.test(t) ? 'updated' : /download/.test(t) ? 'downloading' : 'note' }; } + const pc = lastMatch(lines, /^\d+(?:\.\d+)? (?:\[\w+\] )?(?:GPU )?(power cap.*)$/i); + if (pc) a.power_cap = { text: pc[1].slice(0, 220), at: stampOf(pc[0]), ok: !/NOT applied|nothing to read back/.test(pc[1]) }; + const jf = lastMatch(lines, /^\d+(?:\.\d+)? jobs: (.+)$/); + if (jf) a.jobs_file = { text: jf[1].slice(0, 220), at: stampOf(jf[0]) }; + a.jobs = lines.filter(l => /^\d+(?:\.\d+)? (?:\[\w+\] )?job [^:]+[:(]/.test(l) || /^\d+(?:\.\d+)? (?:\[\w+\] )?\d+ remote jobs? received/.test(l)).slice(-6).map(l => ({ at: stampOf(l), text: strip(l).replace(/^\[\w+\] /, '').slice(0, 220) })); + for (const l of lines) { + const s = /stability: (.+?): draw p95 (\d+) W, max (\d+) W \(cap (\d+) W\), max GPU (\d+) C, max memory (\d+) C, (\d+) samples/.exec(l); + if (s) { const i = a.telemetry.findIndex(t => t.name === s[1]); const t = { name: s[1], p95_w: Number(s[2]), max_w: Number(s[3]), cap_w: Number(s[4]), max_gpu_c: Number(s[5]), max_mem_c: Number(s[6]), samples: Number(s[7]) }; if (i >= 0) a.telemetry[i] = t; else a.telemetry.push(t); } + const e = /^(\d+(?:\.\d+)?) \[(error|ok|block|warn)\] (.+)$/.exec(l); + if (e && e[2] !== 'block') a.events.push({ at: new Date(Number(e[1]) * 1000).toISOString(), kind: e[2], text: e[3].slice(0, 200) }); + } + a.events = a.events.slice(-6); + // A clean stop: the app logs "quit: ..." (Quit, Stop, or an update: "[info] installing Igneum Miner ...") and uploads + // once more before it exits; when no status line follows, the machine was stopped on purpose, not lost (4 October + // 2026: a Mac stopped at 14:47 UTC read "silent 4h" for the whole afternoon, the same as a crash or a lost network). + const stop = lastMatch(lines, /^(\d+)(?:\.\d+)? (quit: .+|stopped|\[info\] installing Igneum Miner .+)$/); + const lastStatus = lastMatch(lines, /^(\d+)(?:\.\d+)? status: /); + if (stop && (!lastStatus || Number(stop[1]) >= Number(lastStatus[1]))) { + const update = lines.some(l => /\[info\] installing Igneum Miner/.test(l) && Number((/^(\d+)/.exec(l) || [])[1]) >= Number(stop[1]) - 30); + a.stopped = { at: stampOf(stop[0]), reason: update ? 'update' : 'quit', text: strip(stop[0]).slice(0, 120) }; + } + if ((m = lastMatch(lines, /^(\d+)(?:\.\d+)? status: (.+?) \| node (.+?) \| up (.+)$/))) { + a.status = { miner: m[2], node: m[3], up: m[4], at: new Date(Number(m[1]) * 1000).toISOString() }; + const acc = /accepted (\d+) blocks \((\d+) this run\), ([\d.]+) MH\/s, (\w+)/.exec(m[2]); + if (acc) Object.assign(a.status, { accepted_total: Number(acc[1]), accepted_run: Number(acc[2]), mhs: Number(acc[3]), mining: acc[4] }); + const nd = /(\d+) blocks, (\d+) peers, (\w+)/.exec(m[3]); + if (nd) Object.assign(a.status, { node_blocks: Number(nd[1]), node_peers: Number(nd[2]), node_state: nd[3] }); + } + return a; +} diff --git a/relay/test/parse.test.mjs b/relay/test/parse.test.mjs index ade4eadf5..991e0a774 100644 --- a/relay/test/parse.test.mjs +++ b/relay/test/parse.test.mjs @@ -41,3 +41,25 @@ test('stale: a card is "now" within STALE_S of its STATUS line and stale after i const oneMissed = { status_at: new Date(now - 150_000).toISOString(), hash_now: 18 }; markStale([oneMissed], now); assert.equal(oneMissed.stale, false); }); + +test('app tail: a clean quit after the last status line is "stopped (quit)", an update is "stopped (update)", a running app is not stopped', async () => { + const { parseAppTail } = await import('../lib/parse.mjs'); + const running = ['1791126700 status: accepted 10 blocks (5 this run), 18.00 MH/s, mining | node 100 blocks, 4 peers, synced | up 00:10:00'].join('\n'); + assert.equal(parseAppTail(running).stopped, undefined); + const quit = [ + '1791126700 status: accepted 10 blocks (5 this run), 18.00 MH/s, mining | node 100 blocks, 4 peers, synced | up 00:10:00', + '1791126735 quit: stopping the miners, then the node', '1791126735 stopping the miners (quit)', '1791126736 node stopped (exit Some(0))', + '1791126736 SUMMARY after 00:26:53: node started 1 time(s), restarts 0, 1165 accepted blocks this run (1270 lifetime); data stays in x', '1791126737 stopped', + ].join('\n'); + const q = parseAppTail(quit); + assert.deepEqual([q.stopped.reason, q.stopped.at], ['quit', new Date(1791126737000).toISOString()]); + const update = [ + '1791125100 status: accepted 1 blocks (1 this run), 1.00 MH/s, mining | node 1 blocks, 1 peers, synced | up 00:01:00', + '1791125117 [info] installing Igneum Miner 0.3.2: the miners stop, then the node, then the app opens again', + '1791125118 quit: stopping the miners, then the node', '1791125121 stopped', + ].join('\n'); + assert.equal(parseAppTail(update).stopped.reason, 'update'); + // a quit line from an earlier run in the same tail, with a status line after it: still running + const resumed = quit + '\n1791126800 status: accepted 11 blocks (1 this run), 18.00 MH/s, mining | node 120 blocks, 4 peers, synced | up 00:01:00'; + assert.equal(parseAppTail(resumed).stopped, undefined); +}); diff --git a/relay/ui.html b/relay/ui.html index 716c0f770..46e971f4e 100644 --- a/relay/ui.html +++ b/relay/ui.html @@ -42,6 +42,7 @@ main{padding-block:14px 48px} .card{background:var(--graphite);border:1px solid var(--line);border-radius:var(--r);padding:14px;min-width:0} .card.red{border-color:var(--red);box-shadow:inset 3px 0 0 var(--red)} .card.live{box-shadow:inset 3px 0 0 var(--green)} +.card.stopped{box-shadow:inset 3px 0 0 var(--ash)} .card h3{margin:0;font-family:'Unbounded',sans-serif;font-weight:700;font-size:14px;letter-spacing:.02em;display:flex;align-items:center;gap:10px;flex-wrap:wrap} .card h3 .id{font-family:'IBM Plex Mono',monospace;font-weight:400;font-size:11px;color:var(--ash);letter-spacing:.04em} .card h3 .seen{margin-left:auto;font-family:'IBM Plex Mono',monospace;font-weight:400;font-size:11px;color:var(--ash)} @@ -266,10 +267,10 @@ input[type=text]{width:100%} } function machineCard(mc) { const n = mc.node || {}; - const silentTxt = mc.never ? 'never seen' : (mc.silent ? `silent ${ago(mc.last_seen)}` : `seen ${ago(mc.last_seen)} ago`); + const silentTxt = mc.never ? 'never seen' : mc.stopped ? `stopped (${esc(mc.stopped.reason)}) ${ago(mc.stopped.at)} ago` : (mc.silent ? `silent ${ago(mc.last_seen)}` : `seen ${ago(mc.last_seen)} ago`); const tele = mc.telemetry && mc.telemetry.length ? `
${mc.telemetry.map(t => ``).join('')}
telemetryp95maxcapGPUmem
${esc(t.name)}${t.p95_w} W${t.max_w} W${t.cap_w} W${t.max_gpu_c} C${t.max_mem_c} C
` : (mc.cards.length ? `
${mc.app ? 'no stability lines yet (power and temperatures come from nvidia-smi every 5 min; none on a Mac)' : 'the app log is not in the intake (an app before 0.3.3)'}
` : ''); const events = mc.events && mc.events.length ? `
${mc.events.map(e => `
${when(e.at)}${e.kind}${esc(e.text)}
`).join('')}
` : ''; - return `
+ return `

${esc(mc.name)}${esc(mc.id8 || '')}${mc.host ? ' ' + esc(mc.host) : ''}${mc.legacy ? ' (old launcher)' : ''}${silentTxt}

${mc.never ? '
nothing from this machine in the log intake in the last 48 h
' : kvs([ ['app', mc.app_version ? esc(mc.app_version) : 'not in the intake'], diff --git a/tools/console.mjs b/tools/console.mjs index c8929007e..40feee60d 100644 --- a/tools/console.mjs +++ b/tools/console.mjs @@ -110,7 +110,7 @@ try { else if (cmd === 'machines') { const j = await api('machines'); for (const m of j.machines) { - console.log(`${m.name.padEnd(8)} ${m.id8 || '-'} ${m.silent ? 'SILENT' : 'live'} seen ${ago(m.last_seen)} ago | app ${m.app_version || '?'} node ${m.node.version || '?'} daa ${m.node.daa ?? '?'} peers ${m.node.peers ?? '?'} ${m.node.synced ? 'synced' : 'not synced'} | ${m.hash_total.toFixed(1)} MH/s, ${m.accepted_total} accepted, ${m.faults} faults`); + console.log(`${m.name.padEnd(8)} ${m.id8 || '-'} ${m.stopped ? `STOPPED (${m.stopped.reason}) ${ago(m.stopped.at)} ago,` : m.silent ? 'SILENT' : 'live'} seen ${ago(m.last_seen)} ago | app ${m.app_version || '?'} node ${m.node.version || '?'} daa ${m.node.daa ?? '?'} peers ${m.node.peers ?? '?'} ${m.node.synced ? 'synced' : 'not synced'} | ${m.hash_total.toFixed(1)} MH/s, ${m.accepted_total} accepted, ${m.faults} faults`); for (const c of m.cards) console.log(` ${c.label.padEnd(22)} ${c.stale ? 'last ' + c.hash_now.toFixed(1) + ' MH/s (stale, not in the total)' : c.hash_now.toFixed(1) + ' MH/s now'}, ${c.accepted} accepted, ${c.rejected} rejected, ${c.mismatched} mismatched, ${c.restarts} restarts, status ${ago(c.status_at)} ago${c.fault ? ' | ' + c.fault : ''}`); for (const t of m.telemetry) console.log(` ${t.name}: p95 ${t.p95_w} W, max ${t.max_w} W (cap ${t.cap_w} W), GPU ${t.max_gpu_c} C, memory ${t.max_mem_c} C`); } From afa57f20b89ea6d3c90093c08b25199f4546d8ea Mon Sep 17 00:00:00 2001 From: igneum-labs <337424239+igneum-labs@users.noreply.github.com> Date: Sun, 4 Oct 2026 20:05:03 +0000 Subject: [PATCH 3/3] bugs.md: the stopped-versus-silent console row Co-Authored-By: Claude Fable 5.1 --- docs/bugs.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/bugs.md b/docs/bugs.md index 37d2a586f..83e61ffd9 100644 --- a/docs/bugs.md +++ b/docs/bugs.md @@ -18,8 +18,9 @@ shard run reported as exit 0, 7a7e873). | 4 Oct 2026 | the live feed showed two "checkpoint N locked" events 30 ms apart (1122, 1172, 1230, 1258, 1259 in 400 observer lines), and 708 of 764 locked checkpoints in `live_checkpoints` had `votes_seen` 0 | `finalityTick` read the state, awaited two SQL writes, then set the map; the `finalityLockNotification` handler checked the same map synchronously in between and recorded the lock too; a lock claimed by the notification was never upserted again, so the poll's `votes_seen` never landed | 7de1bdb: the poll claims the state before its first await; a `checkpointDetailed` set makes the poll fill `votes_seen` once | before: 5 duplicates in 400 lines; after the 19:50:33 UTC restart: 21 locks (1297 to 1317), 0 duplicates, 0 write failures; 1300 was notification-first and the poll filled it to 17 votes. The zeros that remain are the node's own count (`finality.rs:770`, its vote map for that hash, empty when the lock came by certificate), not the observer's. Index 1296 appears twice on the feed: it locked inside the restart window, one write per process, a restart-boundary one-off | | 4 Oct 2026 | the observer kept running old code after its fix was on master and HEAD had moved past it | `autosync.sh` restarted the observer only when its own fast-forward moved HEAD; a pull by hand (19:35 UTC, HEAD to 8a77b85) bypassed it | autosync compares the checked-out `tools/observer` tree id with a marker written at each restart and restarts on any difference; `autosync.sh check` says what it would do | `check` with no marker: "restart due", exit 3; with the marker equal to the tree: "not due", exit 0; the shared checkout at that moment: due | | 4 Oct 2026 | the Mac card read 0.0 MH/s for a minute while its blocks were still accepted (after the stale mark shipped) | STALE_S 120 s left no margin: one missed 60 s upload (a 120 s gap at 19:45:59 UTC) plus a 30 s STATUS age plus the 10 s cache | STALE_S 180 s (one missed upload is not stale; PC 2's real case was a 1,860 s gap) | test: 150 s is fresh, 240 s stale | +| 4 Oct 2026 | a machine stopped on purpose (Sam's Mac, 14:47 UTC) read "silent 4h" on the console, the same as a crash or a lost network | the app logs `quit: stopping the miners, then the node` and `stopped` and uploads once more before it exits (so it does report), but the console never read those lines | 4d208c9: `parseAppTail` (now in `relay/lib/parse.mjs`) sets `stopped {at, reason quit\|update}` when a quit or OTA-install line has no status line after it; the card says "stopped (quit) N ago" with a grey stripe, `tools/console.mjs` says STOPPED | unit test: quit, update, running, and a quit followed by a later status line; live: five running machines show no false stop; a live stopped case is pending the next real quit | ## Open -- Sam's Mac (3a9bf309): CLOSED 19:15 UTC, the app came back on 0.3.3 and mines (11.4 MH/s): it was stopped by its user for 4.5 h. What stays: the last node upload (14:47:02 UTC) ends with `SIGTERM - shutting down` and `igneumd has stopped`, the miner upload stops at 14:46:56, nothing after: a clean app-driven stop (quit or Stop), not a crash and not a network loss (the stop lines reached the intake). Its app predates the app-log upload (`app ?` on the console), so there is no app stream to say which. The app posts nothing on a clean quit, so the console shows "silent 4h" for a machine that was stopped on purpose. Proposed: one `[ok] app quit by the user` line uploaded before the engine stops, and the console card saying "stopped (quit) at 14:47" instead of "silent". +- Sam's Mac (3a9bf309): CLOSED 19:15 UTC; the console side (stopped versus silent) shipped in 4d208c9. Earlier note: CLOSED 19:15 UTC, the app came back on 0.3.3 and mines (11.4 MH/s): it was stopped by its user for 4.5 h. What stays: the last node upload (14:47:02 UTC) ends with `SIGTERM - shutting down` and `igneumd has stopped`, the miner upload stops at 14:46:56, nothing after: a clean app-driven stop (quit or Stop), not a crash and not a network loss (the stop lines reached the intake). Its app predates the app-log upload (`app ?` on the console), so there is no app stream to say which. The app posts nothing on a clean quit, so the console shows "silent 4h" for a machine that was stopped on purpose. Proposed: one `[ok] app quit by the user` line uploaded before the engine stops, and the console card saying "stopped (quit) at 14:47" instead of "silent". - `collect` jobs: the Format-List / Out-String loss did not reproduce (board2 printed `Sum`, board4 printed `Capacity` and `Speed`); only the `$_` case failed. Closed unless it shows again.