igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity
2050 commits 395 branches 1 tag 2.7 GiB
Commit graph

79 commits

Author SHA1 Message Date
igneum-labs
5e8df956f3 Merge remote-tracking branch 'origin/miner-reliability-21' into release-0.3.21 2026-10-07 13:16:12 +00:00
igneum-labs
13911191d1 miner-faults MF-8: a node dying at start on a kept datadir (the serde(default) row, N13); the LG-4 job's tenth install keeps the datadir and a node restart loop fails the row 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 625fc4e06d)
 2026-10-07 13:12:08 +00:00
igneum-labs
f3238c9b3e Miner app: plug, tune, play (the project lead, 7 October 2026): node readiness gate, the retry ladder, no permanent fault, fault lines to the intake, the signed cards job, the fault-class register 
docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
  executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
  once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
  30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
  budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
  card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
  exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
  the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
  after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
  through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
  no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (d4bc5a94) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 13:11:48 +00:00
igneum-labs
aaaa5058dc Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:33:54 +00:00
igneum-labs
39db713f99 Fleet: the last two process patterns (a top-level alternation; the canary's node path by its basename) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:33:54 +00:00
igneum-labs
5e6cc45cfc Merge branch 'gpu-fleet' into HEAD 
# Conflicts:
#	.github/workflows/ci.yml
 2026-10-06 23:33:28 +00:00
igneum-labs
c9fa98bb6c Fleet: the last process patterns in the bracket-first form; the fleet's own pgrep check withdrawn (master's kill-by-name check covers the class) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:31:55 +00:00
igneum-labs
488149e5c5 Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:29:39 +00:00
igneum-labs
0c51a27c15 Fleet scripts: no path into another worktree (sys.path from the file, the ship tree from the environment) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:29:39 +00:00
igneum-labs
0d2379a68c Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:28:55 +00:00
igneum-labs
4dda6f2a4f Fleet scripts find their helpers inside their own tree (the CI checks, the Discord hook, the publish script), env-overridable; the library's node patterns inline 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:28:55 +00:00
igneum-labs
9242e94d8c Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:25:53 +00:00
igneum-labs
a837acfe0d Fleet: the last three process patterns in the bracket-first form 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:25:53 +00:00
igneum-labs
e8199048d0 Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:25:10 +00:00
igneum-labs
ff33e4a592 Fleet scripts: process patterns in master's bracket-first form (the kill-by-name check), anchors dropped 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:25:10 +00:00
igneum-labs
0aad6a0975 Merge branch 'gpu-fleet' into HEAD 2026-10-06 23:23:46 +00:00
igneum-labs
05eedeac29 Merge gpu-fleet: the rented fleet's tooling (box library, standing supervisor and library, the 10 percent and 75 percent gates, the deploy gate, publish and canary scripts, Devnet 2 scripts and gate reader, the wind-down) and the night's analyses (prover tiers, block rate, the fleet night rows 1 to 18, the bench-log entries) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:23:46 +00:00
igneum-labs
28042dc996 lib/box.py: the node path pattern in the bracket form; the pgrep check allows the variable it is read through 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:17:48 +00:00
igneum-labs
37b9c2a6d4 Fleet scripts: every pgrep/pkill -f literal in master's bracket form (the kill-by-name check); the rehearsal chain's end script 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:14:41 +00:00
igneum-labs
3ced860f96 Fleet: the deploy gate (syntax + master's defaults-line check + the comment-with-code class) in lib.box.put; the 75 percent table gate (standing.table_gate, the wind-down holds under it) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:04:29 +00:00
igneum-labs
25dac23a12 box-dn2.sh: the UNSYNCED comment moved above its line (it had swallowed the port variables; three Devnet 2 nodes failed to start under set -u) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 23:02:49 +00:00
igneum-labs
aa9a1a6e9c dn2-check: the reorg read ignores the exec follower's replay of old history after a restart (lines far below the exec tip) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:52:30 +00:00
igneum-labs
ffef7fbfc2 box-dn2.sh: unsynced mining is opt-in (a fresh genesis only); the standing Devnet 2 nodes restart synced-first 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:51:54 +00:00
igneum-labs
1af1a7d80c Fleet: the publish-2 object move for the standing boxes (file swap, node restart by the supervisor, digest table) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:50:03 +00:00
igneum-labs
897be899e1 Standing supervisor: the miner runs only on a synced node past block 1,000 and is stopped while the node syncs (a box never mines from a genesis-only view) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:39:29 +00:00
igneum-labs
ea614bb7f4 dn2-check: the since stamp follows each log's own UTC offset (the build-server seed logs +02:00) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:29:23 +00:00
igneum-labs
4f8e85f665 Standing check: every box's handshake digest against the live object (~/Desktop/fleet/expected-digest) every five minutes, one #incidents line per box per hour on a mismatch 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:19:23 +00:00
igneum-labs
6a8ac209ca box-dn2.sh: the program-ids comment moved off the node line (it had swallowed the flags and the backgrounding, the same class as 17:18Z) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:15:29 +00:00
igneum-labs
a93b09ec8e Standing supervisor: a crash loop on a corrupt datadir moves it aside and rejoins fresh (p2-4090-1b, 24 restarts on a missing sst) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:13:09 +00:00
igneum-labs
0e19e63143 Devnet 2 nodes carry IGNEUM_PROOF_PROGRAM_IDS (a zero shard program id refused every segment record) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:10:32 +00:00
igneum-labs
9be0dfd4ca Devnet 2 prover restart as a file (one instance; an inline pkill with the host's name killed its own ssh shell) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 22:01:29 +00:00
igneum-labs
00879d7363 Fleet: the wave's wind-down by the 10 percent rule (hourly slices sized by weight share, weight_check gated, held under two thirds signing, the lock read after each); the poisoned-peer confirmation script 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:58:04 +00:00
igneum-labs
22837c22a6 Standing supervisor: one miner per GPU (the miner loop kills any other miner before starting its own) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:55:36 +00:00
igneum-labs
211eeb05d9 box-prover.py: one instance per box (pid file) and a per-process state tmp (two instances raced and killed each other's GPU server on Devnet 2); dn2-check.py: the Devnet 2 gate reads for the build-server seed 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:53:51 +00:00
igneum-labs
4b709db7ee fleet-bg.sh: the macOS note moved off the case line (a trailing comment swallowed the ;;) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:43:37 +00:00
igneum-labs
55660ecde8 fleet-bg.sh: setsid optional (macOS has none) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:43:14 +00:00
igneum-labs
9808520f4d Fleet: pid-file job helper (kill by pid, never by a name), the supervisor's P2P_LISTEN knob for inbound-port boxes, publish 1 on f1ea7a38; CI README row for the kill rule 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:42:59 +00:00
igneum-labs
4a741da85d box-prover.py: a statement mismatch names the hex offset and both tails (the 34-char prefix hid the difference on Devnet 2) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:40:28 +00:00
igneum-labs
5b0066a2a8 Poisoned-peer canary: the already-running check uses a marker (the Vast proxy's banner on stdout defeated a bare yes) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:07:53 +00:00
igneum-labs
438a282bff Poisoned-peer canary: idempotent on a box already running the binary; a 45-minute sync wait for a fresh datadir (IBD of 128,000 blocks takes about 12 minutes) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 21:03:29 +00:00
igneum-labs
c649036bca box-prover.py: the export directory owns its disk (20 GB and 7-day caps, export deleted on record accepted or paid, no export under 10 percent free) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:56:19 +00:00
igneum-labs
611cbe5edd Poisoned-peer canary: anchor the poisoned node's pgrep (the bare pattern matched its own shell and the peer never started) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:54:08 +00:00
igneum-labs
512d6e25f0 Fleet: disk sweep every five minutes (disk per box on the page, one #incidents line per box per hour at 85 percent); fleet night row 15 (the hub's three deaths) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:49:41 +00:00
igneum-labs
8e010d84d6 Standing supervisor: prunes the prover's segment exports and trims the node log (the hub died on a full disk at 20:42Z); disk in the status line 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:45:32 +00:00
igneum-labs
61928774a0 Fleet: the 0.3.15 canary in the poisoned-peer form (new node beside the hub with a 1026-block peer, mid-window restart, the hub's count by peer) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:42:21 +00:00
igneum-labs
436cc3c7a8 Fleet page publish: the Fleet | Workers nav and its CSS are re-inserted on every publish when a regeneration dropped them 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:38:29 +00:00
igneum-labs
de5b91f682 Standing supervisor restarts the miner with the node (dead template subscription); block-rate collector keeps difficulty and reads red share as 1 minus blue over DAA 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:32:40 +00:00
igneum-labs
b6d132bc1d Devnet 2 box script: SEED accepts a comma list (run A2, two relays per box) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:18:56 +00:00
igneum-labs
1dda56b491 Standing fleet: weight_check reads vote-key shares from the hub's inspect and maps boxes by key-hash (tested: two boxes = 1.2 percent of 2,000 blocks) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:03:41 +00:00
igneum-labs
f0a7a9249a Standing supervisor: the live-node pattern excludes the rehearsal and Devnet 2 nodes (the hub's live node died unnoticed at 19:5xZ behind the old pattern) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 20:01:54 +00:00