Commit graph

181 commits

Author SHA1 Message Date
igneum-josh
27f4aa090f Fleet: disk sweep every five minutes (disk per box on the page, one #incidents line per box per hour at 85 percent); fleet night row 15 (the hub's three deaths)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:49:41 +01:00
igneum-josh
353cc5fbbb Standing supervisor: prunes the prover's segment exports and trims the node log (the hub died on a full disk at 20:42Z); disk in the status line
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:45:32 +01:00
igneum-josh
36e8d7ae56 Fleet: the 0.3.15 canary in the poisoned-peer form (new node beside the hub with a 1026-block peer, mid-window restart, the hub's count by peer)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:42:21 +01:00
igneum-josh
9a294b42e7 Standing supervisor restarts the miner with the node (dead template subscription); block-rate collector keeps difficulty and reads red share as 1 minus blue over DAA
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:32:40 +01:00
igneum-josh
9e348366d7 Devnet 2 box script: SEED accepts a comma list (run A2, two relays per box)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:18:56 +01:00
igneum-josh
95fa60dd48 Standing fleet: weight_check reads vote-key shares from the hub's inspect and maps boxes by key-hash (tested: two boxes = 1.2 percent of 2,000 blocks)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:03:41 +01:00
igneum-josh
db58804adb Standing supervisor: the live-node pattern excludes the rehearsal and Devnet 2 nodes (the hub's live node died unnoticed at 19:5xZ behind the old pattern)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:01:54 +01:00
igneum-josh
b02ac61d08 Standing fleet: the 10 percent weight rule (gpu-fleet.md, lib/standing.py weight_check), the supervisor's node pattern narrowed to the live node
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 20:59:40 +01:00
igneum-josh
f4edd1bbcd Fleet: the class v4 rehearsal tooling (CUDA worker wrapper, generator-4 worker, pack re-export), the standing-fleet rule (supervisor, library, page block, gpu-fleet.md), the 0.3.15 publish script, the Devnet 2 script's ports and bps knobs, the collector's seed knobs
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 20:46:32 +01:00
igneum-josh
697297dfce Block rate on Devnet 2: the plan file (runs A 10 bps and B 1 bps on the 38 wave cards, the readings, the per-tier payout arithmetic) and bps-collect.py; the remote-build tooling from master 2026-10-06 19:01:29 +01:00
igneum-josh
90dd8e27e6 tools/fleet/lib: the one box library (ssh with ServerAlive, scp with retries, install_payload checked on the box, start_node with anchored kills, chain-side readers: height, peers, exec tip, state root, paid segments, rejects and reorgs since a stamp, miner and prover start, destroy with the registry row) and its Devnet 2 test (known-failed and known-finished cases) 2026-10-06 18:58:18 +01:00
igneum-josh
a461f0d6b6 devnet2-gate: rejects and reorgs counted from the step-1 restart (a joining box's lone-chain reorg is not the release's), bash 3.2 (no mapfile, no associative arrays), the crossing line rewritten 2026-10-06 18:55:13 +01:00
igneum-josh
3f9e9e3a0f pgrep-self-match-check: the fleet's scripts anchored (every pattern on the binary path or -x); the live devnet's operational scripts and finished PC playbooks on the owed list by path, named in the README; the check is green over the tree 2026-10-06 18:53:19 +01:00
igneum-josh
41107eba3b fleet: box-matrix.sh kill_server uses pkill -x; the self-match check is green (the previous commit's claim was premature) 2026-10-06 18:47:34 +01:00
igneum-josh
d568a3f9ff fleet: box-matrix.sh patterns anchored; pgrep-self-match-check passes over the whole tree 2026-10-06 18:45:08 +01:00
igneum-josh
5c5ae403a1 fleet: box-node-swap.sh patterns anchored; the self-match check is clean over the tree 2026-10-06 18:43:01 +01:00
igneum-josh
f92c629abe fleet: the last three bare process patterns anchored; the self-match check passes over the tree 2026-10-06 18:40:30 +01:00
igneum-josh
350d9783d0 fleet scripts: every process pattern anchored (the pgrep-self-match check passes on tools/fleet); the observer's autosync on the owed list 2026-10-06 18:38:23 +01:00
igneum-josh
3a83c5daea CI: pgrep-self-match-check (a process pattern never matches its own shell; the wave-script class of 6 October), wired into ci.yml, README row; fleet: the 0.3.14 canary (bash 3.2, retries, overridable boxes), Devnet 2 fixes, the wave and pool scripts 2026-10-06 18:35:58 +01:00
igneum-josh
813f0a6fd5 Devnet 2 (the staging chain): box-dn2.sh (igneum-devnet-2, own genesis bits, every activation low, no exec-restart fields), devnet2-override.json, devnet2-gate.sh (install, activation at tip + margin, N minutes, PASS only on zero rejects, reorg depth at most 3, equal exec roots, a paid segment, equal versions), the page's Devnet 2 panel; the wave, pool and snapshot scripts 2026-10-06 17:50:34 +01:00
igneum-josh
d3647aad76 fleet prover: export from the node's exec restart block, not genesis (the exporter refuses the unexecuted blocks' zero roots on a restarted 0.3.13 node) 2026-10-06 17:10:15 +01:00
igneum-josh
deafabf78a fleet: night.py (the prover night loop: launches on the executed tip, re-runs the exec restart on a reset, 15-minute night rows), the tarball sha check in the swap, the prover script's FLOOR line restored 2026-10-06 16:52:59 +01:00
igneum-josh
f5ccfb9f59 fleet: box-node-swap.sh rewritten clean (the edited copy called stamp before its definition and misquoted the sha line; no node was touched) 2026-10-06 16:32:35 +01:00
igneum-josh
d70f152021 fleet: swap.py (one call per shipper word over the fleet, the hub first), the rig as N provers (per-card loops, distinct sockets), the rig script's device and chain fixes, restage.py 2026-10-06 16:20:00 +01:00
igneum-josh
2ad424d3e9 fleet: the node swap's two steps (publish 1 binary with the ten-field file, publish 2 the thirteen-field file), the expected digest checked, igneum_getExecStatus read 2026-10-06 15:59:19 +01:00
igneum-josh
2fe2560498 fleet: the 0.3.13 node swap for phase-2 boxes (fixed binary over the full-history datadir, the replay timed); the prover launcher prefers the swapped binary 2026-10-06 14:58:53 +01:00
igneum-josh
22ed1f70a5 Prover tiers on real cards: all eleven cards (the 3090 added); bench-log entry complete 2026-10-06 14:28:55 +01:00
igneum-josh
a0a0f0dc8b fleet: the collector keyed by card and VRAM with a raw-rows fallback; the matrix script's final write guarded against a card that reports no power 2026-10-06 14:22:15 +01:00
igneum-josh
38d2db63e7 Prover tiers on real cards: ten cards, the 8 GB beside-the-miner rows (core-only 2^25 fits); fleet: the rig script and retry 2026-10-06 14:20:06 +01:00
igneum-josh
d1d5f5a6f8 Prover tiers on real cards: eight cards (3060, 3080, 4060 Ti 16 GB, 5090 added), the 8 and 10 GB rows; fleet: the datadir joiner swap, the v5 known-failed runner 2026-10-06 14:07:05 +01:00
igneum-josh
587e531278 Prover tiers on real cards: the first four cards (4090, A5000, 4070, 5070), the method, the per-tier consequences; fleet: RunPod pods, companions pushed with every stage 2026-10-06 13:56:51 +01:00
igneum-josh
3fe3a1145f GPU fleet: box-kill.sh (anchored kills as a file), log rotation before every stage start, parallel orchestrator probes, the listed-after-rent check 2026-10-06 13:46:14 +01:00
igneum-josh
83b71e8c65 GPU fleet: the orchestrator loop, a killable per-second GPU sampler, the STATUS-line rate parser, an atomic locked registry, the p2p hub peer 2026-10-06 13:34:25 +01:00
igneum-josh
78e93d4c50 GPU fleet: RunPod client, the phase-1 collector, the phase-2 launcher (node restarted with the proof verifier), parallel status, the toolchain fix 2026-10-06 13:14:57 +01:00
igneum-josh
627a2256c7 GPU fleet: Vast client, box setup (0.3.12 node + patched SP1 server + cuda host), the phase-1 matrix, the Ember ladder, the Linux segment prover, the fleet page builder 2026-10-06 13:01:23 +01:00
igneum-josh
c2b922c1e6 prover-floor: the three other PC 2 scripts test their kit file before the first use (C32)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 00:06:38 +01:00
igneum-josh
40cff807ad prover-floor: pc2-floor-sweep3-miner.ps1 tests its kit file before the first use (the kit-path check, C32)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 00:04:52 +01:00
igneum-josh
88e28f4d2e Merge prover-floor (cfe3d80) into release-0.3.11: docs and standalone probe sources the evidence rows cite (C38); nothing a built artefact reads 2026-10-06 00:03:46 +01:00
igneum-josh
cc72f4ab29 CI on the merged 0.3.11 tree: MacBook reworded in the analysis prose (the identity check's hostname pattern), a presence check before the kit's first use in the three proving-v1 PC 2 scripts (C32), pc1-cpu-prove.ps1 on the socket check's allow list (the CPU path starts no GPU server)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 23:44:09 +01:00
igneum-josh
cfe3d805d3 Prover floor: the beside-the-miner sweep playbook (sweep 3)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:42:31 +00:00
igneum-josh
eda49ab357 Prover floor patch v2: every trace buffer sized to its padded need (setup keys and shards), the sweep-2 points
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:41:37 +00:00
igneum-josh
fe1ecdbd11 Merge branch 'bash-body-check' into release-0.3.11
# Conflicts:
#	.github/workflows/ci.yml
#	tools/ci/prover-socket-check.sh
2026-10-05 23:40:22 +01:00
igneum-josh
5cbb796051 Merge commit '22c2363' into release-0.3.11
# Conflicts:
#	docs/bench-log.md
#	docs/evidence.md
#	site/litepaper.html
2026-10-05 23:39:34 +01:00
igneum-josh
18605f425c Merge branch 'ca2-coord' into release-0.3.11
# Conflicts:
#	docs/bench-log.md
2026-10-05 23:39:12 +01:00
igneum-josh
e3bd761790 CI: run jobs test their fetched kit before use (the wiped-jobs-folder class)
The app's install clears the jobs folder on a PC, so a run job whose kit was fetched by an earlier fetch job finds
nothing after an update and fails in seconds (5 October 2026, 21:49Z, the AMD kit; bench-log 39f02ff). Rule: a run
playbook that reaches a path under the jobs folder other than its own tests the kit is there before its first use,
and the fetch is republished under a new id after any app update.

tools/ci/kit-path-check.sh reads every *.ps1 under relay/playbooks/ and tools/. A kit root is a path derived from
the jobs folder (`$jobs = Split-Path $env:IGNEUM_JOB_DIR` then `Join-Path $jobs '<fetch id>'`, the race-5090.ps1
shape) or one carrying a literal `jobs\` (the amd-card-test.ps1 shape); every path built from it belongs to that kit.
A presence check (Test-Path, [IO.File]::Exists, [IO.Directory]::Exists, Get-Item or Get-ChildItem with -ErrorAction)
on the root or anything under it covers the whole kit. A use before that line fails with "kit path used before a
presence check: republish the fetch after any app update", as does a literal jobs\ path in a command with no check.
The job's own folder ($env:IGNEUM_JOB_DIR) is not a kit path.

Fixtures: kit-path-ok.ps1 (both shapes, checked; a sibling pack file covered by the worker's check) and
kit-path-unchecked.ps1 (the worker run before its check, a literal never checked); --self-test asserts the lines.
Wired into ci.yml after the bash-body step, and into publish-jobs.sh add --kind run beside the other two checks;
test-publish-jobs.sh gains the refusal (34 passed, 0 failed). The current tree: race-5090.ps1 is the one playbook
with a kit, checked before use. README-ship.md: the rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:37:29 +00:00
igneum-josh
49c7e78307 Merge remote-tracking branch 'origin/master' into ca2-v3
# Conflicts:
#	docs/bench-log.md
#	proto-opencl/host.c
2026-10-05 22:34:07 +00:00
igneum-josh
965584f54a Prover floor: the build recipe as it ran (Go pinned, the binary's sha256 and targets), the RISC Zero contingency figures with their source, playbook fixes
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:33:31 +00:00
igneum-josh
6805125d29 Jobs publisher: the class checks run on the script before it is signed (row C27)
A PC job is published from a worktree by packaging/ota/publish-jobs.sh and never passes CI before it runs; tonight
the root-socket fault came back from a job on a branch without the check. `add --kind run` now runs, on the script
being published and before anything is signed: tools/ci/bash-body-check.sh for a PowerShell script (every inline
bash body parses; a body it cannot read fails, never skips), `bash -n` for a .sh script, and
tools/ci/prover-socket-check.sh for both (a root prover run kills sp1-gpu-server and unlinks its socket). A failure
refuses the publish with the check's output; a missing check file refuses too. Kinds without a script (fetch,
collect, restart, update-now, shard-benchmark, build) are untouched.

tools/ci/prover-socket-check.sh is copied from proving-v1 (c2544be; master lacks it) with two additions: file
arguments check those files only (the publisher's call), and an allow list for packaging/ota/test-publish-jobs.sh,
which carries a known-bad root prover script on purpose. Its ci.yml step is left to proving-v1 to avoid a duplicate.

packaging/ota/test-publish-jobs.sh: four refusals (a lost quote in a PowerShell bash body, an unreadable body, a
.sh with a lost quote, a root prover script without the cleanup) and the envelope unchanged after a refusal.
32 passed, 0 failed on this Mac with the main checkout's signer. packaging/README-ship.md: the publish-time gate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:14:19 +00:00
igneum-josh
c062166071 Prover floor: SP1 6.8.1 GPU server memory model from source (the 24 GB panic, the threshold-sized trace buffers), the floor patch for sp1-gpu, the PC 2 toolchain, build and sweep playbooks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:11:24 +00:00
igneum-josh
a223ca9aef App: the resume path re-arms every stopped card, re-exports its pack and checks 90 s later that every enabled card mines (PC 2 at 21:25:11Z and the Mac that afternoon stayed at 0 MH/s after resume); the PC 2 socket-fix job
The known-failed case, from PC 2's 0.3.9 log (run win-1ccfe586-20261005-200114): 1791234223 pause -> 'stopping the
miners (paused)' (every slot's restart_at cleared, the 5090 'off'); 1791235511 '[ok] mining resumed'; then
'0.00 MH/s, waiting' at every 30-s status line until the 0.3.10 restart at 21:49:41Z. Cause: Cmd::Resume re-armed
only slots whose watchdog said faulted; the 5090's slot was healthy and stopped, so nothing restarted it. The test
the_pc2_resume_of_21_25_11z_restarts_under_the_new_rule_and_not_the_old encodes that slot (faulted false, live
false): the old rule returns [] (the defect), the new rule [0]. cargo test -p igneum-app resume: 3 passed;
provedefault: 6 passed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:56:20 +01:00