Commit graph

1127 commits

Author SHA1 Message Date
igneum-labs
a57c36e55e miner-faults MF-10: a prover server built for another card's architecture (named symbol not found); the capability check at start and the per-architecture kit
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 9d2c267af7a7b097199e6232b221b91081f52bca)
2026-10-07 13:12:08 +00:00
igneum-labs
43a226fe54 miner-faults MF-9: a node that stops slower than its restart (the listener watchdog's sleep-then-check); every poll loop returns on shutdown, a sub-second shutdown is a named release gate
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 7a0115617b9fa7b7e8e9930c8cfb530af71d4d55)
2026-10-07 13:12:08 +00:00
igneum-labs
579e834873 miner-faults MF-8: the class in the shipper's words (node refuses its own kept datadir after an update), N13 b7cc37e7, the gate on both datadir shapes, the injector step owed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit a9a39b7b55)
2026-10-07 13:12:08 +00:00
igneum-labs
75742c512b miner-faults MF-8: a node dying at start on a kept datadir (the serde(default) row, N13); the LG-4 job's tenth install keeps the datadir and a node restart loop fails the row
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 5d69de12dc)
2026-10-07 13:12:08 +00:00
igneum-labs
3d8c3ac308 miner-faults register: the commits table and which side each class is on
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit f44a9c1e6e)
2026-10-07 13:12:08 +00:00
igneum-labs
9786f3c4c9 Miner app: plug, tune, play (the project lead, 7 October 2026): node readiness gate, the retry ladder, no permanent fault, fault lines to the intake, the signed cards job, the fault-class register
docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
  executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
  once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
  30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
  budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
  card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
  exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
  the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
  after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
  through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
  no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (18a086b9) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:11:48 +00:00
igneum-labs
f549564c61 Driver table: the NVIDIA 617.42 and AMD 26.9.2 rows measured on igneum-build-2 (sha256, size, Authenticode subject from the vendors' own files); the stray #[test] above the Intel test's doc comment removed
NVIDIA GeForce Game Ready 617.42 WHQL (6 October 2026): us.download.nvidia.com/Windows/617.42/..., 990,853,168
bytes, sha256 f115c927..., subject CN=NVIDIA Corporation (DigiCert G4). AMD Software Adrenalin 26.9.2 WHQL
(29 September 2026, the win11-b build): drivers.amd.com/drivers/whql-amd-software-adrenalin-edition-26.9.2-win11-b.exe
(an amd.com Referer required), 1,000,800,840 bytes, sha256 593c1d73..., subject CN=Advanced Micro Devices (Sectigo).
Read on the box: curl, sha256sum, the PKCS7 out of the PE security directory through openssl pkcs7 -print_certs.
The table's placeholders are gone: every row installs. The drivertable test sample, the mock and the view test name
the real NVIDIA release. detect.rs:990 carried a #[test] above the doc comment of the Intel test from the cherry-pick,
the test build's one warning ("duplicated attribute"): removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit afafbcd52e)
2026-10-07 13:09:47 +00:00
igneum-labs
fac6db03de Driver check: the app detects a missing or old driver per card and installs it on one click from the vendor's own server (branch driver-check, 7 October 2026)
the project lead: can the drivers be packaged with the miner, for all cards, with the system knowing which to install if not
present. Not bundled: detected and installed on one click. A per-vendor table rides the signed manifest (drivers.json:
min_version, the version on offer, the vendor's URL, size, sha256 and its source page, the silent arguments, the
restart exit codes, the Authenticode signer, the Linux and HiveOS package), validated by the signer and the app alike
(src/drivertable.rs, shared), written to <app data>/drivers.json by ota.rs. Each card's driver version is read at
every detection (nvidia-smi's driver_version; Windows' DriverVersion for AMD and Intel) and compared; a missing or
old driver puts the offer on the card's row, on the dashboard and on the first-run list. The click downloads with
curl (resume), checks size, sha256 and the Authenticode subject, runs the installer through one elevated prompt
(platform::elevated_command, the PC 1 driver job's shape), reports restart required with a Restart now button, and
never restarts by itself; the miners keep mining. macOS: no step; Linux and HiveOS: the package line. Dry run through
IGNEUM_DRIVER_DRY_RUN or the table. Tests: the table, the versions, the offers per tier, the exit codes, the
Authenticode verdicts, the download against a mocked vendor server on 127.0.0.1, the UI's strip per state; the mock's
drivers scenarios; captures light and dark in docs/plans/driver-check-shots. publish-manifest.sh --drivers carries the
table. Also the doubled #[test] in detect.rs from the cherry-pick.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit e49f988c5c)
2026-10-07 13:09:46 +00:00
igneum-labs
b891b34f2a release-0.3.20 plan: section 35, the project lead's orders (the floor file publishes, the sweep in waves, the Arc-tested worker exe, 0.3.21 tonight)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:09:09 +00:00
igneum-labs
d5641562b5 release rules: 4a every gate starts on every candidate as it builds, 4b warm pods per gate class, 5 the sweep in waves (the project lead, 7 October 2026)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:07:49 +00:00
igneum-labs
01426b9f03 release-0.3.20 plan: section 34, every artefact on c4459193 with shas, the workers with the Arc fix, the hive package and its smoke, the Windows inputs and run, the staging plan
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:00:53 +00:00
igneum-labs
7cdd11067e release-0.3.20 plan: the cases rerun's slip to about 16:10 BST, the choice put to main
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:53:10 +00:00
igneum-labs
f298e22a8e release-0.3.20 plan: the c4459193 seed pair; the pair set complete
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:48:51 +00:00
igneum-labs
e04a4460b8 release-0.3.20 plan: c20-1's kept start on the pin, the Mac build started
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:47:20 +00:00
igneum-labs
a256168307 release-0.3.20 plan: c4459193's two node gates pass; the Mac binaries start
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:46:47 +00:00
igneum-labs
95cd505413 release-0.3.20 plan: the c4459193 hands pair
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:41:51 +00:00
igneum-labs
bd7dbf5f53 release-0.3.20 plan: the cases rerun started on c20-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:41:44 +00:00
igneum-labs
136122042b release-0.3.20 plan: main accepts the three lines; the cut set stands
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:41:24 +00:00
igneum-labs
c477971e9c release-0.3.20 plan: the cases rerun on c4459193 (the class byte touch), the 610 read (the driver proves; the server per architecture is the rule)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:40:45 +00:00
igneum-labs
75945ca427 release-0.3.20 plan: the wipe canary's start stamp on c19-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:40:00 +00:00
igneum-labs
2b545d1360 release-0.3.20 plan: CASES END pass on dc141409, the carry question to c4459193
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:39:47 +00:00
igneum-labs
91545d8378 release-0.3.20 plan: the wipe canary pod c19-1 on c4459193, the 10 GB tier closed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:38:30 +00:00
igneum-labs
125e58711f release-0.3.20 plan: main accepts the wipe canary's clock
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:36:56 +00:00
igneum-labs
0ba41a50ab release-0.3.20 plan: the wipe canary moves to a second pod so its line lands inside the checkpoint
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:36:32 +00:00
igneum-labs
5a3beaf921 release-0.3.20 plan: the 3080 hour's number (a 10 GB card proves alone, not beside its miner), the build-server sequence on c4459193
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:34:34 +00:00
igneum-labs
2e0c978979 release-0.3.20 plan: section 33, b7cc37e7 struck (the 10 s stop), the pin c4459193, the carry ruling, main's F8 and slot rulings, the 12 GB rule in and green, the roll complete, the server-architecture fact, the 0.3.21 driver-check tip
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:33:26 +00:00
igneum-labs
b8f87a0911 release-0.3.20 plan: the driver check to 0.3.21, the clock rule, the 10 GB prover tier and the rule put to main
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:11:42 +00:00
igneum-labs
80edb84068 release-0.3.20 plan: 6b94c823's two gate lines (clean on the checks that bear on the binary), the harness sequencing fixes
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:10:20 +00:00
igneum-labs
3f106ae626 release rules file (the standing rules as main set them, the kept-datadir gate as rule 4); the 0.3.20 plan's row for main's three additions
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:07:42 +00:00
igneum-labs
7d1dedd818 release-0.3.20 plan: section 32, N12 and N13 on the line, the pin b7cc37e7 with no fallback commit, the kept-datadir gate, the stale packs
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 12:06:22 +00:00
igneum-labs
e59a748741 release-0.3.20 plan: the build-server lane's four pairs in flight, the node line's missing rust-toolchain.toml owed for 0.3.21
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:54:38 +00:00
igneum-labs
1a6ce0ac6b release-0.3.20 plan: clock correction (the UK stamps since 12:40 BST ran ahead; true times from the commit times)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:54:22 +00:00
igneum-labs
1a1e3dc5f2 release-0.3.20 plan: the sixteen-field interop fact (0.3.17 relays byte-5 headers), the void run's failed checks explained
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:54:01 +00:00
igneum-labs
4d27156a82 release-0.3.20 plan: the amended v4 packs' ids and fingerprints (the hash lane's table), G1, the pairing line in flight
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:53:51 +00:00
igneum-labs
6b261c8b4f release-0.3.20 plan: section 31, the candidate pin, igneum-pow pairing, the speculative builds
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:52:21 +00:00
igneum-labs
8d6061ba76 release-0.3.20 plan: the fallback pin 6b94c823 on the mirror
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:47:57 +00:00
igneum-labs
8ddacb5f76 release-0.3.20 plan: the stale finality test on 8097d600, the fallback pin is a test-only commit on top
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:47:10 +00:00
igneum-labs
41ae291ba3 release-0.3.20 plan: epoch 231 on the old file confirmed with the reference, the staging recipe for the floor-moved file
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:43:30 +00:00
igneum-labs
7121b568b2 release-0.3.20 plan: main's rulings (watchdog test meets the condition, the floor-moved file staged not published, the 13 October sweep date in the lock lines, the epoch-231 question for the report)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:41:45 +00:00
igneum-labs
cc7f6a5367 release-0.3.20 plan: the node lane's clock estimate, how the two conditions are met, the kit's wait rule, the 13 October floor deadline
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:41:15 +00:00
igneum-labs
4162ccff42 release-0.3.20 plan: main's two conditions on the second node commit, the publish order
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:40:07 +00:00
igneum-labs
d9d434904a release-0.3.20 plan: the node line at 8097d600 (what it carries), the second commit, the 15:30 UK cut rule
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:39:21 +00:00
igneum-labs
16c91a0aeb release-0.3.20 plan: the fleet's prover-roll read (12 GB provers paired and unpaid, the claim race, the fix on the node line)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:36:26 +00:00
igneum-labs
c41346df09 release-0.3.20 plan: cards_leave_off in (de314e37), the third app gate, the reliability lane's hashes named
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:32:39 +00:00
igneum-labs
2f7cafb73b release-0.3.20 plan: main's hold for the isSynced fix (16:00 UK checkpoint)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:29:39 +00:00
igneum-labs
ebe0ccac7d release-0.3.20 plan: the app gate green twice on the box, the push
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:28:48 +00:00
igneum-labs
4081775254 release-0.3.20 plan: the Arc rotr fix and the Intel app files are in (bench d numbers, the two hashes, the box tests)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:25:31 +00:00
igneum-labs
467b555054 release-0.3.20 plan: the app side assembled (the picks, the five unions, the rebuilt signer, the box UI tests), the isSynced hold, main's Arc rotr_var order, the fleet's case timing
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:19:45 +00:00
igneum-labs
4182bcaf14 ember-heat: the gate reader, the PC 1 four-hour runbook, the plan and the light and dark captures
tools/heat-gate.mjs reads the HEAT lines of an app log and passes a hold within 1 degree for 4 hours with the hash
following the slice; its self-test fires on a known hold, a drift, a hash through the rest, a short log, an empty log
and a log without readings, and sits on the one gate beside heat-region.test.mjs. docs/plans/ember-heat.md carries
the words, the loop, the sources, the per-tier table and the runbook for the project lead's desk (this lane never touches PC 1).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 0d5fc6d258dee4774ebe7ea760736c9f05026d06)
2026-10-07 10:50:23 +00:00
igneum-labs
12b0463ded ui-ota: the publish order rule in the plan (1.0.1 with min_engine 0.3.20 only after 0.3.20 is on the manifest)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit ee09ae8b633206c8a7ab438eb4b0ee57362de000)
2026-10-07 10:44:51 +00:00