igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity
2017 commits 305 branches 1 tag 1.5 GiB
Commit graph

79 commits

Author SHA1 Message Date
igneum-labs
59fe7d08b4 Merge remote-tracking branch 'origin/miner-reliability-21' into release-0.3.21 2026-10-07 13:16:12 +00:00
igneum-labs
ae35330f42 miner-faults MF-8: a node dying at start on a kept datadir (the serde(default) row, N13); the LG-4 job's tenth install keeps the datadir and a node restart loop fails the row 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit d82de28e40)
 2026-10-07 13:12:08 +00:00
igneum-labs
089b078a3a Miner app: plug, tune, play (the project lead, 7 October 2026): node readiness gate, the retry ladder, no permanent fault, fault lines to the intake, the signed cards job, the fault-class register 
docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
  executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
  once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
  30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
  budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
  card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
  exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
  the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
  after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
  through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
  no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (97255a4e) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 13:11:48 +00:00
igneum-labs
89300aafe6 Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:33:54 +00:00
igneum-labs
db1cebd1f8 Fleet: the last two process patterns (a top-level alternation; the canary's node path by its basename) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:33:54 +00:00
igneum-labs
9e6a325128 Merge branch 'gpu-fleet' into HEAD 
# Conflicts:
#	.github/workflows/ci.yml
 2026-10-06 23:33:28 +00:00
igneum-labs
22ef7c3f97 Fleet: the last process patterns in the bracket-first form; the fleet's own pgrep check withdrawn (master's kill-by-name check covers the class) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:31:55 +00:00
igneum-labs
d41cb3876c Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:29:39 +00:00
igneum-labs
238fccfce6 Fleet scripts: no path into another worktree (sys.path from the file, the ship tree from the environment) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:29:39 +00:00
igneum-labs
6aad8d8a53 Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:28:55 +00:00
igneum-labs
ccfa6e6cb4 Fleet scripts find their helpers inside their own tree (the CI checks, the Discord hook, the publish script), env-overridable; the library's node patterns inline 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:28:55 +00:00
igneum-labs
0cb0e413b8 Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:25:53 +00:00
igneum-labs
c704c8263c Fleet: the last three process patterns in the bracket-first form 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:25:53 +00:00
igneum-labs
b8b807ca15 Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:25:10 +00:00
igneum-labs
0e10fcfe30 Fleet scripts: process patterns in master's bracket-first form (the kill-by-name check), anchors dropped 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:25:10 +00:00
igneum-labs
859b79b2d4 Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:23:46 +00:00
igneum-labs
4d5b80f791 Merge gpu-fleet: the rented fleet's tooling (box library, standing supervisor and library, the 10 percent and 75 percent gates, the deploy gate, publish and canary scripts, Devnet 2 scripts and gate reader, the wind-down) and the night's analyses (prover tiers, block rate, the fleet night rows 1 to 18, the bench-log entries) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:23:46 +00:00
igneum-labs
ddaeaf076a lib/box.py: the node path pattern in the bracket form; the pgrep check allows the variable it is read through 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:17:48 +00:00
igneum-labs
25d2fe8275 Fleet scripts: every pgrep/pkill -f literal in master's bracket form (the kill-by-name check); the rehearsal chain's end script 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:14:41 +00:00
igneum-labs
4573b6f3ed Fleet: the deploy gate (syntax + master's defaults-line check + the comment-with-code class) in lib.box.put; the 75 percent table gate (standing.table_gate, the wind-down holds under it) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:04:29 +00:00
igneum-labs
7b3594ab46 box-dn2.sh: the UNSYNCED comment moved above its line (it had swallowed the port variables; three Devnet 2 nodes failed to start under set -u) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:02:49 +00:00
igneum-labs
b180ac0403 dn2-check: the reorg read ignores the exec follower's replay of old history after a restart (lines far below the exec tip) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:52:30 +00:00
igneum-labs
5e4dc171e9 box-dn2.sh: unsynced mining is opt-in (a fresh genesis only); the standing Devnet 2 nodes restart synced-first 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:51:54 +00:00
igneum-labs
b53b217e56 Fleet: the publish-2 object move for the standing boxes (file swap, node restart by the supervisor, digest table) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:50:03 +00:00
igneum-labs
0b52c1b3da Standing supervisor: the miner runs only on a synced node past block 1,000 and is stopped while the node syncs (a box never mines from a genesis-only view) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:39:29 +00:00
igneum-labs
f20bda7804 dn2-check: the since stamp follows each log's own UTC offset (the build-server seed logs +02:00) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:29:23 +00:00
igneum-labs
7a025825f2 Standing check: every box's handshake digest against the live object (~/Desktop/fleet/expected-digest) every five minutes, one #incidents line per box per hour on a mismatch 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:19:23 +00:00
igneum-labs
165b0ff588 box-dn2.sh: the program-ids comment moved off the node line (it had swallowed the flags and the backgrounding, the same class as 17:18Z) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:15:29 +00:00
igneum-labs
a86b532736 Standing supervisor: a crash loop on a corrupt datadir moves it aside and rejoins fresh (p2-4090-1b, 24 restarts on a missing sst) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:13:09 +00:00
igneum-labs
552ef811f9 Devnet 2 nodes carry IGNEUM_PROOF_PROGRAM_IDS (a zero shard program id refused every segment record) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:10:32 +00:00
igneum-labs
a54d58dce2 Devnet 2 prover restart as a file (one instance; an inline pkill with the host's name killed its own ssh shell) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:01:29 +00:00
igneum-labs
76723b3264 Fleet: the wave's wind-down by the 10 percent rule (hourly slices sized by weight share, weight_check gated, held under two thirds signing, the lock read after each); the poisoned-peer confirmation script 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:58:04 +00:00
igneum-labs
29076448f2 Standing supervisor: one miner per GPU (the miner loop kills any other miner before starting its own) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:55:36 +00:00
igneum-labs
34fc638266 box-prover.py: one instance per box (pid file) and a per-process state tmp (two instances raced and killed each other's GPU server on Devnet 2); dn2-check.py: the Devnet 2 gate reads for the build-server seed 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:53:51 +00:00
igneum-labs
c1b6ab1971 fleet-bg.sh: the macOS note moved off the case line (a trailing comment swallowed the ;;) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:43:37 +00:00
igneum-labs
608bfa4a11 fleet-bg.sh: setsid optional (macOS has none) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:43:14 +00:00
igneum-labs
5b1531c15a Fleet: pid-file job helper (kill by pid, never by a name), the supervisor's P2P_LISTEN knob for inbound-port boxes, publish 1 on f1ea7a38; CI README row for the kill rule 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:42:59 +00:00
igneum-labs
03287bcc78 box-prover.py: a statement mismatch names the hex offset and both tails (the 34-char prefix hid the difference on Devnet 2) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:40:28 +00:00
igneum-labs
b7144f5dc6 Poisoned-peer canary: the already-running check uses a marker (the Vast proxy's banner on stdout defeated a bare yes) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:07:53 +00:00
igneum-labs
ee2391c9a4 Poisoned-peer canary: idempotent on a box already running the binary; a 45-minute sync wait for a fresh datadir (IBD of 128,000 blocks takes about 12 minutes) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:03:29 +00:00
igneum-labs
86c9854863 box-prover.py: the export directory owns its disk (20 GB and 7-day caps, export deleted on record accepted or paid, no export under 10 percent free) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:56:19 +00:00
igneum-labs
a99617bd9f Poisoned-peer canary: anchor the poisoned node's pgrep (the bare pattern matched its own shell and the peer never started) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:54:08 +00:00
igneum-labs
853c0217d9 Fleet: disk sweep every five minutes (disk per box on the page, one #incidents line per box per hour at 85 percent); fleet night row 15 (the hub's three deaths) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:49:41 +00:00
igneum-labs
fe5e481528 Standing supervisor: prunes the prover's segment exports and trims the node log (the hub died on a full disk at 20:42Z); disk in the status line 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:45:32 +00:00
igneum-labs
498370d07f Fleet: the 0.3.15 canary in the poisoned-peer form (new node beside the hub with a 1026-block peer, mid-window restart, the hub's count by peer) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:42:21 +00:00
igneum-labs
09cf92db48 Fleet page publish: the Fleet | Workers nav and its CSS are re-inserted on every publish when a regeneration dropped them 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:38:29 +00:00
igneum-labs
07d0ed185c Standing supervisor restarts the miner with the node (dead template subscription); block-rate collector keeps difficulty and reads red share as 1 minus blue over DAA 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:32:40 +00:00
igneum-labs
66602299df Devnet 2 box script: SEED accepts a comma list (run A2, two relays per box) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:18:56 +00:00
igneum-labs
23b840165f Standing fleet: weight_check reads vote-key shares from the hub's inspect and maps boxes by key-hash (tested: two boxes = 1.2 percent of 2,000 blocks) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:03:41 +00:00
igneum-labs
2021de8262 Standing supervisor: the live-node pattern excludes the rehearsal and Devnet 2 nodes (the hub's live node died unnoticed at 19:5xZ behind the old pattern) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:01:54 +00:00