docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (97255a4e) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A node before 0.3.18's exec RPC fix dies on eth_getBlockByNumber when its exec follower holds no record (rpc.rs indexes records[0] on an empty vector; the panic hook exits the process), and this sample ran every 9 s once blocks arrived, so a fresh install's IBD and PC 1's restart crash-looped. The sample now waits for executedTipHash. No node change; the pin stays at 5899f603. Version 0.3.18 on the 0.3.17 app tree.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's rule 7 October 2026 03:3xZ: the version scheme is three-part everywhere, so the hotfix (f1ea7a38 plus the IBD-guard fix) ships as 0.3.17; the feature tree becomes 0.3.18, decimals 0.3.19.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GNU tar on HiveOS materialised the Mac's extended headers as ._ files next to every binary.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit c5187a70693415a6bb68d94972008272bbdf50b1)
The Mac and PC 1 took earlier 0.3.15 builds (nodes 713ef876 and 7961c5f1) through deploys of the shared downloads folder before the publish; the updater compares version strings only, so the number moves them to the final build. Node pin unchanged at f1ea7a38.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's rule of 6 October 2026 from the 0.3.14 repro (docs/evidence/reproduced/0.3.14.md): prost's protowire.rs embeds OUT_DIR,
libmimalloc-sys embeds __DATE__/__TIME__, sccache hid both. lib.sh bs_repro_env exports SOURCE_DATE_EPOCH=<author time> TZ=UTC in
front of every remote command (build-remote.sh, cross-remote.sh, workers-remote.sh); remote-run.sh exports BR_SDE too and logs it
as source_date_epoch; proto-cuda/windows-node/cross-build.sh exports the same; the PC job carries node.commit_time in the manifest
(push-build-inputs.sh) and exports it before every cargo build of a stage (jobbuild.rs, unit test asserts it; 4 of 4 pass on the
box). Target dirs stay one fixed path per target. tools/build-remote.sh --self-test-repro [--full] from a fork worktree, run on
the box: igneum-miner twice a minute apart without sccache (RUSTC_WRAPPER=/usr/bin/env, an empty value is unset to cargo) MATCH
91e130f5..., a per-run target path differs (OUT_DIR shown); --full: kaspad with libmimalloc-sys recompiled a minute later MATCH
70219bc2..., without the epoch differs (__DATE__ shown).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The CI payload had never carried igneum-prove-host or igneum-prove-export for WSL2 (make-payload.sh's warning branch), so every PC ran a stale pair built by setup-wsl.sh from an old package; after 0.3.14 no PC verified peer proofs or exported segments. push-inputs.sh refuses to publish without the pair (IGNEUM_PROVE_LINUX names the folder) and make-payload.sh refuses a payload without it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The node pin moves to 4c6b129d (release-0.3.14-node), which is what the live payload inputs carry, so windows-ci's payload-inputs step is green again (red since e9eca23 on master's 0.3.13 pin against the 0.3.14 inputs).
Conflicts: infra/fast-time/override-60x.json keeps master's side (the fresh-rule field was already there at u64::MAX; the release line would be a duplicate key); tools/ci/playbook-quit-check.sh keeps master's rule 2 and pre-rule list with the release side's Ember allow entry and the 0.3.15 expiry check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main's decision of 6 October 2026. CLAUDE.md 'Running agents on this Mac': every Linux and Windows cargo build and every
Linux test suite goes to igneum-build-1 through tools/build-remote.sh and tools/cross-remote.sh; the PCs keep GPU and
Windows-runtime jobs; the Mac keeps macOS binaries, the DMG and Metal tests under the lock. -Wl,--no-insert-timestamp in
cross-remote.sh, proto-cuda/windows-node/cross-build.sh and jobbuild.rs (two box builds byte-identical, verified).
The empty-commit class: tools/ci/commit-string-check.sh (self-test in ci.yml, shown firing on a Mac worktree build and
passing on a box build) runs on every igneumd from the three build scripts; push-build-inputs.sh adds node.commit_full,
the PC job writes a minimal node/.git from it at extract and cleans kaspa-build-info on a new commit (4 jobbuild tests
pass, run on the box); cross-build.sh refuses a worktree and cleans on a new commit. Plan: second worktree's clean build
1 min 18 s with sccache 604 hits of 993.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The finding (release-0.3.11.md section 5): h-run.sh started the rig's node with --devnet --appdir
--rpclisten --listen and the peers and no --override-params-file, so a HiveOS rig in local mode ran
on genesis parameters, printed the no-override digest and was refused by every devnet peer; no
package ever carried the override.
h-config.sh: OVERRIDE=<json object> in the Flight Sheet's extra config, read as a whole line (JSON
may carry spaces; single or double quotes around it are stripped), refused unless it is {...},
written to igneum.conf single-quoted (sq helper; EXTRA gets the same quoting, the same class: a
value with shell characters sourced unquoted). Still sourceable (return, never exit).
h-run.sh, local branch: writes data/override-params.json from OVERRIDE when set and passes
--override-params-file=<that path> to igneumd; when empty, a WARNING in the main log that the node
runs on genesis parameters and devnet peers will refuse it. After the node answers, the switch
lines and the "Consensus params digest" line from node.log are copied into the main log as
"node: ..." so the operator can compare the digest with the downloads page.
README: the Flight Sheet table gains the OVERRIDE row with the four-field devnet object as the
example (nine fields after the 0.3.11 switch; the downloads page carries the live one), the rule
"set OVERRIDE from the downloads page when it changes", the sentence that a rig without it is
refused, the digest check in the requirements, and the gap entry. No "every override publish
needs a package republish" sentence exists in packaging/hive/README.md on this branch or master,
so nothing was replaced; the new rule stands alone.
selftest.sh: a fake igneumd that records its argv and prints the real digest line; OVERRIDE
single-quoted on its own line beside other keys round-trips through the conf; OVERRIDE=notjson
refused; empty OVERRIDE named in the summary; the main h-run run checks the file, the flag on the
node and the digest and switch lines in the main log; a second short run without OVERRIDE checks
the warning and the absence of the flag. bash packaging/hive/selftest.sh on this Mac:
== h-config.sh OVERRIDE
OVERRIDE (single-quoted, own line) sourced back intact beside the other keys ok
OVERRIDE that is not {...} refused ok
no OVERRIDE: empty in the conf and named in the summary ok
== h-run.sh (fake GPUs: 2 NVIDIA, fake node, fake miner)
data/override-params.json written from OVERRIDE ok
the node got --override-params-file ok
the node's digest and switch lines reached the main log ok
NVIDIA cards got the cuda worker ok
exit 42 restarted the miner and re-exported the pack ok
== h-stats.sh (sourced)
stats JSON ok: hs [118500.0, 118500.0] temp [61, 58] ar [24, 0] bus [1, 2]
== h-run.sh without OVERRIDE (the warning)
no OVERRIDE: warning in the main log, no flag on the node ok
== self-test passed (scripts and stats shape; Hive itself is untested)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The app's install clears the jobs folder on a PC, so a run job whose kit was fetched by an earlier fetch job finds
nothing after an update and fails in seconds (5 October 2026, 21:49Z, the AMD kit; bench-log 4df339f). Rule: a run
playbook that reaches a path under the jobs folder other than its own tests the kit is there before its first use,
and the fetch is republished under a new id after any app update.
tools/ci/kit-path-check.sh reads every *.ps1 under relay/playbooks/ and tools/. A kit root is a path derived from
the jobs folder (`$jobs = Split-Path $env:IGNEUM_JOB_DIR` then `Join-Path $jobs '<fetch id>'`, the race-5090.ps1
shape) or one carrying a literal `jobs\` (the amd-card-test.ps1 shape); every path built from it belongs to that kit.
A presence check (Test-Path, [IO.File]::Exists, [IO.Directory]::Exists, Get-Item or Get-ChildItem with -ErrorAction)
on the root or anything under it covers the whole kit. A use before that line fails with "kit path used before a
presence check: republish the fetch after any app update", as does a literal jobs\ path in a command with no check.
The job's own folder ($env:IGNEUM_JOB_DIR) is not a kit path.
Fixtures: kit-path-ok.ps1 (both shapes, checked; a sibling pack file covered by the worker's check) and
kit-path-unchecked.ps1 (the worker run before its check, a literal never checked); --self-test asserts the lines.
Wired into ci.yml after the bash-body step, and into publish-jobs.sh add --kind run beside the other two checks;
test-publish-jobs.sh gains the refusal (34 passed, 0 failed). The current tree: race-5090.ps1 is the one playbook
with a kit, checked before use. README-ship.md: the rule.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A PC job is published from a worktree by packaging/ota/publish-jobs.sh and never passes CI before it runs; tonight
the root-socket fault came back from a job on a branch without the check. `add --kind run` now runs, on the script
being published and before anything is signed: tools/ci/bash-body-check.sh for a PowerShell script (every inline
bash body parses; a body it cannot read fails, never skips), `bash -n` for a .sh script, and
tools/ci/prover-socket-check.sh for both (a root prover run kills sp1-gpu-server and unlinks its socket). A failure
refuses the publish with the check's output; a missing check file refuses too. Kinds without a script (fetch,
collect, restart, update-now, shard-benchmark, build) are untouched.
tools/ci/prover-socket-check.sh is copied from proving-v1 (344cba8; master lacks it) with two additions: file
arguments check those files only (the publisher's call), and an allow list for packaging/ota/test-publish-jobs.sh,
which carries a known-bad root prover script on purpose. Its ci.yml step is left to proving-v1 to avoid a duplicate.
packaging/ota/test-publish-jobs.sh: four refusals (a lost quote in a PowerShell bash body, an unreadable body, a
.sh with a lost quote, a root prover script without the cleanup) and the envelope unchanged after a refusal.
32 passed, 0 failed on this Mac with the main checkout's signer. packaging/README-ship.md: the publish-time gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Twice on 5 October 2026 a PowerShell job script carried a bash body inside a string, a quote was lost on the way
through PowerShell, and bash refused the body: pc1-cpu-prove.ps1 (first version) reported exit 0 having done
nothing, the 0.3.10 installer job failed in 4 s. tools/amd-prove/check-job-bash.sh covered only its own here-string.
tools/ci/bash-body-check.sh reads every *.ps1 under relay/playbooks/ and tools/, finds each bash body however it is
handed over (bash -c "...", bash -lc '...', bash -c $var, a + concatenation in parentheses, the Start-Process argument
list, a here-string written to a file that is later run with bash), unescapes it the way PowerShell would (backtick
escapes and "" in double-quoted strings, '' in single-quoted strings, here-strings verbatim; $var left as-is, a $(...)
subexpression replaced by ${PS_SUBEXPR}), and runs bash -n on it. One line per body with the file line of the error.
A body it sees but cannot read is "unextractable body" and fails too: a skip would be a hole in the class check.
bash 3.2 compatible; python3 for the extractor.
--self-test runs three fixtures under tools/ci/fixtures/: the correct shapes (8 bodies, must pass), the lost quotes
(the awk apostrophe, a dropped closing quote in a literal and in a variable; must fail with the line), and three
unreadable bodies (must fail). Wired into ci.yml next to the copied-sources check, self-test first. The current tree:
7 inline bodies in 3 playbooks, all parse. packaging/README-ship.md: the job-script rule (body to a file, bash <file>).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
