Commit graph

1127 commits

Author SHA1 Message Date
igneum-labs
06eadb406d miner-faults MF-10: a prover server built for another card's architecture (named symbol not found); the capability check at start and the per-architecture kit
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 9d2c267af7a7b097199e6232b221b91081f52bca)
2026-10-07 13:12:08 +00:00
igneum-labs
87c17c84fc miner-faults MF-9: a node that stops slower than its restart (the listener watchdog's sleep-then-check); every poll loop returns on shutdown, a sub-second shutdown is a named release gate
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 7a0115617b9fa7b7e8e9930c8cfb530af71d4d55)
2026-10-07 13:12:08 +00:00
igneum-labs
635e8576a8 miner-faults MF-8: the class in the shipper's words (node refuses its own kept datadir after an update), N13 b7cc37e7, the gate on both datadir shapes, the injector step owed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 291ea5418f)
2026-10-07 13:12:08 +00:00
igneum-labs
ae35330f42 miner-faults MF-8: a node dying at start on a kept datadir (the serde(default) row, N13); the LG-4 job's tenth install keeps the datadir and a node restart loop fails the row
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit d82de28e40)
2026-10-07 13:12:08 +00:00
igneum-labs
c95e854265 miner-faults register: the commits table and which side each class is on
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 43cbec0d63)
2026-10-07 13:12:08 +00:00
igneum-labs
089b078a3a Miner app: plug, tune, play (the project lead, 7 October 2026): node readiness gate, the retry ladder, no permanent fault, fault lines to the intake, the signed cards job, the fault-class register
docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
  executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
  once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
  30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
  budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
  card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
  exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
  the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
  after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
  through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
  no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (97255a4e) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:11:48 +00:00
igneum-labs
015e6d5a6a Driver table: the NVIDIA 617.42 and AMD 26.9.2 rows measured on igneum-build-2 (sha256, size, Authenticode subject from the vendors' own files); the stray #[test] above the Intel test's doc comment removed
NVIDIA GeForce Game Ready 617.42 WHQL (6 October 2026): us.download.nvidia.com/Windows/617.42/..., 990,853,168
bytes, sha256 f115c927..., subject CN=NVIDIA Corporation (DigiCert G4). AMD Software Adrenalin 26.9.2 WHQL
(29 September 2026, the win11-b build): drivers.amd.com/drivers/whql-amd-software-adrenalin-edition-26.9.2-win11-b.exe
(an amd.com Referer required), 1,000,800,840 bytes, sha256 593c1d73..., subject CN=Advanced Micro Devices (Sectigo).
Read on the box: curl, sha256sum, the PKCS7 out of the PE security directory through openssl pkcs7 -print_certs.
The table's placeholders are gone: every row installs. The drivertable test sample, the mock and the view test name
the real NVIDIA release. detect.rs:990 carried a #[test] above the doc comment of the Intel test from the cherry-pick,
the test build's one warning ("duplicated attribute"): removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 80787ea024)
2026-10-07 13:09:47 +00:00
igneum-labs
d00772bf20 Driver check: the app detects a missing or old driver per card and installs it on one click from the vendor's own server (branch driver-check, 7 October 2026)
the project lead: can the drivers be packaged with the miner, for all cards, with the system knowing which to install if not
present. Not bundled: detected and installed on one click. A per-vendor table rides the signed manifest (drivers.json:
min_version, the version on offer, the vendor's URL, size, sha256 and its source page, the silent arguments, the
restart exit codes, the Authenticode signer, the Linux and HiveOS package), validated by the signer and the app alike
(src/drivertable.rs, shared), written to <app data>/drivers.json by ota.rs. Each card's driver version is read at
every detection (nvidia-smi's driver_version; Windows' DriverVersion for AMD and Intel) and compared; a missing or
old driver puts the offer on the card's row, on the dashboard and on the first-run list. The click downloads with
curl (resume), checks size, sha256 and the Authenticode subject, runs the installer through one elevated prompt
(platform::elevated_command, the PC 1 driver job's shape), reports restart required with a Restart now button, and
never restarts by itself; the miners keep mining. macOS: no step; Linux and HiveOS: the package line. Dry run through
IGNEUM_DRIVER_DRY_RUN or the table. Tests: the table, the versions, the offers per tier, the exit codes, the
Authenticode verdicts, the download against a mocked vendor server on 127.0.0.1, the UI's strip per state; the mock's
drivers scenarios; captures light and dark in docs/plans/driver-check-shots. publish-manifest.sh --drivers carries the
table. Also the doubled #[test] in detect.rs from the cherry-pick.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 3f0ef65786)
2026-10-07 13:09:46 +00:00
igneum-labs
53e62a4bb8 release-0.3.20 plan: section 35, the project lead's orders (the floor file publishes, the sweep in waves, the Arc-tested worker exe, 0.3.21 tonight)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:09:09 +00:00
igneum-labs
957606f8ec release rules: 4a every gate starts on every candidate as it builds, 4b warm pods per gate class, 5 the sweep in waves (the project lead, 7 October 2026)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:07:49 +00:00
igneum-labs
650cc59669 release-0.3.20 plan: section 34, every artefact on c4459193 with shas, the workers with the Arc fix, the hive package and its smoke, the Windows inputs and run, the staging plan
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:00:53 +00:00
igneum-labs
ec06347123 release-0.3.20 plan: the cases rerun's slip to about 16:10 BST, the choice put to main
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:53:10 +00:00
igneum-labs
37cb92e928 release-0.3.20 plan: the c4459193 seed pair; the pair set complete
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:48:51 +00:00
igneum-labs
b528038e45 release-0.3.20 plan: c20-1's kept start on the pin, the Mac build started
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:47:20 +00:00
igneum-labs
e499277bc3 release-0.3.20 plan: c4459193's two node gates pass; the Mac binaries start
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:46:47 +00:00
igneum-labs
990acbd2ba release-0.3.20 plan: the c4459193 hands pair
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:41:51 +00:00
igneum-labs
c61213f30f release-0.3.20 plan: the cases rerun started on c20-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:41:44 +00:00
igneum-labs
49e7ed8b2c release-0.3.20 plan: main accepts the three lines; the cut set stands
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:41:24 +00:00
igneum-labs
9a814457f1 release-0.3.20 plan: the cases rerun on c4459193 (the class byte touch), the 610 read (the driver proves; the server per architecture is the rule)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:40:45 +00:00
igneum-labs
ed30f72984 release-0.3.20 plan: the wipe canary's start stamp on c19-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:40:00 +00:00
igneum-labs
11e05c2dfe release-0.3.20 plan: CASES END pass on dc141409, the carry question to c4459193
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:39:47 +00:00
igneum-labs
09d7e1b6e2 release-0.3.20 plan: the wipe canary pod c19-1 on c4459193, the 10 GB tier closed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:38:30 +00:00
igneum-labs
edb8f3feeb release-0.3.20 plan: main accepts the wipe canary's clock
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:36:56 +00:00
igneum-labs
c2d9c9e579 release-0.3.20 plan: the wipe canary moves to a second pod so its line lands inside the checkpoint
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:36:32 +00:00
igneum-labs
2ab92ce6bf release-0.3.20 plan: the 3080 hour's number (a 10 GB card proves alone, not beside its miner), the build-server sequence on c4459193
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:34:34 +00:00
igneum-labs
4bc1ac28b7 release-0.3.20 plan: section 33, b7cc37e7 struck (the 10 s stop), the pin c4459193, the carry ruling, main's F8 and slot rulings, the 12 GB rule in and green, the roll complete, the server-architecture fact, the 0.3.21 driver-check tip
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:33:26 +00:00
igneum-labs
e4cc7fd503 release-0.3.20 plan: the driver check to 0.3.21, the clock rule, the 10 GB prover tier and the rule put to main
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:11:42 +00:00
igneum-labs
b00dc19380 release-0.3.20 plan: 6b94c823's two gate lines (clean on the checks that bear on the binary), the harness sequencing fixes
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:10:20 +00:00
igneum-labs
ea8d0cc77b release rules file (the standing rules as main set them, the kept-datadir gate as rule 4); the 0.3.20 plan's row for main's three additions
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:07:42 +00:00
igneum-labs
2ce9b05bbd release-0.3.20 plan: section 32, N12 and N13 on the line, the pin b7cc37e7 with no fallback commit, the kept-datadir gate, the stale packs
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:06:22 +00:00
igneum-labs
690b9d8415 release-0.3.20 plan: the build-server lane's four pairs in flight, the node line's missing rust-toolchain.toml owed for 0.3.21
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:54:38 +00:00
igneum-labs
e7eb49e992 release-0.3.20 plan: clock correction (the UK stamps since 12:40 BST ran ahead; true times from the commit times)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:54:22 +00:00
igneum-labs
d8cd4d8e42 release-0.3.20 plan: the sixteen-field interop fact (0.3.17 relays byte-5 headers), the void run's failed checks explained
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:54:01 +00:00
igneum-labs
330d6ad062 release-0.3.20 plan: the amended v4 packs' ids and fingerprints (the hash lane's table), G1, the pairing line in flight
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:53:51 +00:00
igneum-labs
a02f3ad2a4 release-0.3.20 plan: section 31, the candidate pin, igneum-pow pairing, the speculative builds
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:52:21 +00:00
igneum-labs
e0dca71eb5 release-0.3.20 plan: the fallback pin 6b94c823 on the mirror
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:47:57 +00:00
igneum-labs
2095b32368 release-0.3.20 plan: the stale finality test on 8097d600, the fallback pin is a test-only commit on top
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:47:10 +00:00
igneum-labs
87703a8750 release-0.3.20 plan: epoch 231 on the old file confirmed with the reference, the staging recipe for the floor-moved file
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:43:30 +00:00
igneum-labs
3253f7f6a7 release-0.3.20 plan: main's rulings (watchdog test meets the condition, the floor-moved file staged not published, the 13 October sweep date in the lock lines, the epoch-231 question for the report)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:41:45 +00:00
igneum-labs
9a822854f2 release-0.3.20 plan: the node lane's clock estimate, how the two conditions are met, the kit's wait rule, the 13 October floor deadline
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:41:15 +00:00
igneum-labs
707b38e1b7 release-0.3.20 plan: main's two conditions on the second node commit, the publish order
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:40:07 +00:00
igneum-labs
acac7b0a37 release-0.3.20 plan: the node line at 8097d600 (what it carries), the second commit, the 15:30 UK cut rule
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:39:21 +00:00
igneum-labs
8c59ebbe9a release-0.3.20 plan: the fleet's prover-roll read (12 GB provers paired and unpaid, the claim race, the fix on the node line)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:36:26 +00:00
igneum-labs
8e9da582c8 release-0.3.20 plan: cards_leave_off in (00e8c2d1), the third app gate, the reliability lane's hashes named
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:32:39 +00:00
igneum-labs
4ee4bc69d3 release-0.3.20 plan: main's hold for the isSynced fix (16:00 UK checkpoint)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:29:39 +00:00
igneum-labs
1d26c4e54c release-0.3.20 plan: the app gate green twice on the box, the push
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:28:48 +00:00
igneum-labs
2711ee3036 release-0.3.20 plan: the Arc rotr fix and the Intel app files are in (bench d numbers, the two hashes, the box tests)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:25:31 +00:00
igneum-labs
d2028b9454 release-0.3.20 plan: the app side assembled (the picks, the five unions, the rebuilt signer, the box UI tests), the isSynced hold, main's Arc rotr_var order, the fleet's case timing
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:19:45 +00:00
igneum-labs
1b912385ad ember-heat: the gate reader, the PC 1 four-hour runbook, the plan and the light and dark captures
tools/heat-gate.mjs reads the HEAT lines of an app log and passes a hold within 1 degree for 4 hours with the hash
following the slice; its self-test fires on a known hold, a drift, a hash through the rest, a short log, an empty log
and a log without readings, and sits on the one gate beside heat-region.test.mjs. docs/plans/ember-heat.md carries
the words, the loop, the sources, the per-tier table and the runbook for the project lead's desk (this lane never touches PC 1).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 0d5fc6d258dee4774ebe7ea760736c9f05026d06)
2026-10-07 10:50:23 +00:00
igneum-labs
bb3649a266 ui-ota: the publish order rule in the plan (1.0.1 with min_engine 0.3.20 only after 0.3.20 is on the manifest)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit ee09ae8b633206c8a7ab438eb4b0ee57362de000)
2026-10-07 10:44:51 +00:00