bugs.md: the stopped detector's second live firing (a plain quit)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-05 00:18:05 +00:00
parent 23124efc40
commit 52096a02fe

View file

@ -18,7 +18,7 @@ shard run reported as exit 0, 7a7e873).
| 4 Oct 2026 | the live feed showed two "checkpoint N locked" events 30 ms apart (1122, 1172, 1230, 1258, 1259 in 400 observer lines), and 708 of 764 locked checkpoints in `live_checkpoints` had `votes_seen` 0 | `finalityTick` read the state, awaited two SQL writes, then set the map; the `finalityLockNotification` handler checked the same map synchronously in between and recorded the lock too; a lock claimed by the notification was never upserted again, so the poll's `votes_seen` never landed | 7de1bdb: the poll claims the state before its first await; a `checkpointDetailed` set makes the poll fill `votes_seen` once | before: 5 duplicates in 400 lines; after the 19:50:33 UTC restart: 21 locks (1297 to 1317), 0 duplicates, 0 write failures; 1300 was notification-first and the poll filled it to 17 votes. The zeros that remain are the node's own count (`finality.rs:770`, its vote map for that hash, empty when the lock came by certificate), not the observer's. Index 1296 appears twice on the feed: it locked inside the restart window, one write per process, a restart-boundary one-off At the 20:15:58 restart index 1319 (locked 14 min before) was recorded again: open, the seed should have held it locked; f22870a logs the seeded states and the earlier state on such a record. The 20:24:46 restart seeded 'proposed 500, locked 864' and re-recorded nothing (11 locks, 0 duplicates) | | 4 Oct 2026 | the live feed showed two "checkpoint N locked" events 30 ms apart (1122, 1172, 1230, 1258, 1259 in 400 observer lines), and 708 of 764 locked checkpoints in `live_checkpoints` had `votes_seen` 0 | `finalityTick` read the state, awaited two SQL writes, then set the map; the `finalityLockNotification` handler checked the same map synchronously in between and recorded the lock too; a lock claimed by the notification was never upserted again, so the poll's `votes_seen` never landed | 7de1bdb: the poll claims the state before its first await; a `checkpointDetailed` set makes the poll fill `votes_seen` once | before: 5 duplicates in 400 lines; after the 19:50:33 UTC restart: 21 locks (1297 to 1317), 0 duplicates, 0 write failures; 1300 was notification-first and the poll filled it to 17 votes. The zeros that remain are the node's own count (`finality.rs:770`, its vote map for that hash, empty when the lock came by certificate), not the observer's. Index 1296 appears twice on the feed: it locked inside the restart window, one write per process, a restart-boundary one-off At the 20:15:58 restart index 1319 (locked 14 min before) was recorded again: open, the seed should have held it locked; f22870a logs the seeded states and the earlier state on such a record. The 20:24:46 restart seeded 'proposed 500, locked 864' and re-recorded nothing (11 locks, 0 duplicates) |
| 4 Oct 2026 | the observer kept running old code after its fix was on master and HEAD had moved past it | `autosync.sh` restarted the observer only when its own fast-forward moved HEAD; a pull by hand (19:35 UTC, HEAD to 8a77b85) bypassed it | autosync compares the checked-out `tools/observer` tree id with a marker written at each restart and restarts on any difference; `autosync.sh check` says what it would do | `check` with no marker: "restart due", exit 3; with the marker equal to the tree: "not due", exit 0; the shared checkout at that moment: due | | 4 Oct 2026 | the observer kept running old code after its fix was on master and HEAD had moved past it | `autosync.sh` restarted the observer only when its own fast-forward moved HEAD; a pull by hand (19:35 UTC, HEAD to 8a77b85) bypassed it | autosync compares the checked-out `tools/observer` tree id with a marker written at each restart and restarts on any difference; `autosync.sh check` says what it would do | `check` with no marker: "restart due", exit 3; with the marker equal to the tree: "not due", exit 0; the shared checkout at that moment: due |
| 4 Oct 2026 | the Mac card read 0.0 MH/s for a minute while its blocks were still accepted (after the stale mark shipped) | STALE_S 120 s left no margin: one missed 60 s upload (a 120 s gap at 19:45:59 UTC) plus a 30 s STATUS age plus the 10 s cache | STALE_S 180 s (one missed upload is not stale; PC 2's real case was a 1,860 s gap) | test: 150 s is fresh, 240 s stale | | 4 Oct 2026 | the Mac card read 0.0 MH/s for a minute while its blocks were still accepted (after the stale mark shipped) | STALE_S 120 s left no margin: one missed 60 s upload (a 120 s gap at 19:45:59 UTC) plus a 30 s STATUS age plus the 10 s cache | STALE_S 180 s (one missed upload is not stale; PC 2's real case was a 1,860 s gap) | test: 150 s is fresh, 240 s stale |
| 4 Oct 2026 | a machine stopped on purpose (Sam's Mac, 14:47 UTC) read "silent 4h" on the console, the same as a crash or a lost network | the app logs `quit: stopping the miners, then the node` and `stopped` and uploads once more before it exits (so it does report), but the console never read those lines | 4d208c9: `parseAppTail` (now in `relay/lib/parse.mjs`) sets `stopped {at, reason quit\|update}` when a quit or OTA-install line has no status line after it; the card says "stopped (quit) N ago" with a grey stripe, `tools/console.mjs` says STOPPED | unit test: quit, update, running, and a quit followed by a later status line; live: five running machines show no false stop; 20:45:01 UTC Sam's Mac quit for the 0.3.4 install and the card read "STOPPED (update) 3m ago" (the quit line and the `[info] installing` line in its last upload), while PC 2's earlier canary quit had not been caught because the detector shipped after it | | 4 Oct 2026 | a machine stopped on purpose (Sam's Mac, 14:47 UTC) read "silent 4h" on the console, the same as a crash or a lost network | the app logs `quit: stopping the miners, then the node` and `stopped` and uploads once more before it exits (so it does report), but the console never read those lines | 4d208c9: `parseAppTail` (now in `relay/lib/parse.mjs`) sets `stopped {at, reason quit\|update}` when a quit or OTA-install line has no status line after it; the card says "stopped (quit) N ago" with a grey stripe, `tools/console.mjs` says STOPPED | unit test: quit, update, running, and a quit followed by a later status line; live: five running machines show no false stop; 20:45:01 UTC Sam's Mac quit for the 0.3.4 install and the card read "STOPPED (update) 3m ago" (the quit line and the `[info] installing` line in its last upload), while PC 2's earlier canary quit had not been caught because the detector shipped after it; 00:04 UTC on 5 Oct Sam's Mac quit for the night and the card read "STOPPED (quit)" |
| 4 Oct 2026 | `publish-manifest.sh --deploy` called the live manifest "verified" after one signature check: any validly signed manifest, including the previous version still at the edge, passed; a failed deploy was hidden by `\|\| true` | one-shot check, signature only, no byte compare | `verify_live_manifest`: up to `--tries` checks 5 s apart, byte-identical to the folder's file, then the signature, each failure named; `--verify-only` runs just the check; a failed deploy stops first (the ship tool's own verify step already compared bytes and version, so a cut through `tools/ship-app.mjs` was covered; a hand publish was not) | finished: `--verify-only --tries 2` against the live 0.3.3 (try 1); failed: a local server with an altered copy ("differs", both stamps named) and a closed port ("is not reachable") | | 4 Oct 2026 | `publish-manifest.sh --deploy` called the live manifest "verified" after one signature check: any validly signed manifest, including the previous version still at the edge, passed; a failed deploy was hidden by `\|\| true` | one-shot check, signature only, no byte compare | `verify_live_manifest`: up to `--tries` checks 5 s apart, byte-identical to the folder's file, then the signature, each failure named; `--verify-only` runs just the check; a failed deploy stops first (the ship tool's own verify step already compared bytes and version, so a cut through `tools/ship-app.mjs` was covered; a hand publish was not) | finished: `--verify-only --tries 2` against the live 0.3.3 (try 1); failed: a local server with an altered copy ("differs", both stamps named) and a closed port ("is not reachable") |
| 4 Oct 2026 | PC 2 came back on 0.3.4 (the canary) and its card's OTA state read "none" | the OTA state parsed only `OTA: …` lines, and the install lines sit in the previous run's log file; the new run only says `update check: 0.3.4 is current (manifest 0.3.4)` | 797a844 then 2nd commit: every update line the app logs (`is available: downloading`, `downloaded and verified`, `is ready; it installs at the next safe moment`, `installing`, `update to X complete`, `is marked failed`, `has no build yet`, `is current`) is parsed and the newest decides the state; the Mac's card had read "0.3.3 is current" from a 70-minute-old check while 0.3.4 was downloaded, verified and staged | test: staged beats the older check, a finished run reads current, a newer OTA line wins, failed and updated; live cards after the deploy | | 4 Oct 2026 | PC 2 came back on 0.3.4 (the canary) and its card's OTA state read "none" | the OTA state parsed only `OTA: …` lines, and the install lines sit in the previous run's log file; the new run only says `update check: 0.3.4 is current (manifest 0.3.4)` | 797a844 then 2nd commit: every update line the app logs (`is available: downloading`, `downloaded and verified`, `is ready; it installs at the next safe moment`, `installing`, `update to X complete`, `is marked failed`, `has no build yet`, `is current`) is parsed and the newest decides the state; the Mac's card had read "0.3.3 is current" from a 70-minute-old check while 0.3.4 was downloaded, verified and staged | test: staged beats the older check, a finished run reads current, a newer OTA line wins, failed and updated; live cards after the deploy |
| 4 Oct 2026 | the live 0.3.3 Mac bundle's `Info.plist` said 0.3.2 while the engine reported 0.3.3 (Finder and Get Info showed the wrong version) | the 0.3.3 cut was by hand and the plist was not bumped | nothing to do: 0.3.4 was cut by `tools/ship-app.mjs`, which bumps the six version files, and its plist says 0.3.4 | `PlistBuddy` on the staged 0.3.4 bundle | | 4 Oct 2026 | the live 0.3.3 Mac bundle's `Info.plist` said 0.3.2 while the engine reported 0.3.3 (Finder and Get Info showed the wrong version) | the 0.3.3 cut was by hand and the plist was not bumped | nothing to do: 0.3.4 was cut by `tools/ship-app.mjs`, which bumps the six version files, and its plist says 0.3.4 | `PlistBuddy` on the staged 0.3.4 bundle |