Commit graph

173 commits

Author SHA1 Message Date
igneum-josh
f4edd1bbcd Fleet: the class v4 rehearsal tooling (CUDA worker wrapper, generator-4 worker, pack re-export), the standing-fleet rule (supervisor, library, page block, gpu-fleet.md), the 0.3.15 publish script, the Devnet 2 script's ports and bps knobs, the collector's seed knobs
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 20:46:32 +01:00
igneum-josh
697297dfce Block rate on Devnet 2: the plan file (runs A 10 bps and B 1 bps on the 38 wave cards, the readings, the per-tier payout arithmetic) and bps-collect.py; the remote-build tooling from master 2026-10-06 19:01:29 +01:00
igneum-josh
90dd8e27e6 tools/fleet/lib: the one box library (ssh with ServerAlive, scp with retries, install_payload checked on the box, start_node with anchored kills, chain-side readers: height, peers, exec tip, state root, paid segments, rejects and reorgs since a stamp, miner and prover start, destroy with the registry row) and its Devnet 2 test (known-failed and known-finished cases) 2026-10-06 18:58:18 +01:00
igneum-josh
a461f0d6b6 devnet2-gate: rejects and reorgs counted from the step-1 restart (a joining box's lone-chain reorg is not the release's), bash 3.2 (no mapfile, no associative arrays), the crossing line rewritten 2026-10-06 18:55:13 +01:00
igneum-josh
3f9e9e3a0f pgrep-self-match-check: the fleet's scripts anchored (every pattern on the binary path or -x); the live devnet's operational scripts and finished PC playbooks on the owed list by path, named in the README; the check is green over the tree 2026-10-06 18:53:19 +01:00
igneum-josh
41107eba3b fleet: box-matrix.sh kill_server uses pkill -x; the self-match check is green (the previous commit's claim was premature) 2026-10-06 18:47:34 +01:00
igneum-josh
d568a3f9ff fleet: box-matrix.sh patterns anchored; pgrep-self-match-check passes over the whole tree 2026-10-06 18:45:08 +01:00
igneum-josh
5c5ae403a1 fleet: box-node-swap.sh patterns anchored; the self-match check is clean over the tree 2026-10-06 18:43:01 +01:00
igneum-josh
f92c629abe fleet: the last three bare process patterns anchored; the self-match check passes over the tree 2026-10-06 18:40:30 +01:00
igneum-josh
350d9783d0 fleet scripts: every process pattern anchored (the pgrep-self-match check passes on tools/fleet); the observer's autosync on the owed list 2026-10-06 18:38:23 +01:00
igneum-josh
3a83c5daea CI: pgrep-self-match-check (a process pattern never matches its own shell; the wave-script class of 6 October), wired into ci.yml, README row; fleet: the 0.3.14 canary (bash 3.2, retries, overridable boxes), Devnet 2 fixes, the wave and pool scripts 2026-10-06 18:35:58 +01:00
igneum-josh
813f0a6fd5 Devnet 2 (the staging chain): box-dn2.sh (igneum-devnet-2, own genesis bits, every activation low, no exec-restart fields), devnet2-override.json, devnet2-gate.sh (install, activation at tip + margin, N minutes, PASS only on zero rejects, reorg depth at most 3, equal exec roots, a paid segment, equal versions), the page's Devnet 2 panel; the wave, pool and snapshot scripts 2026-10-06 17:50:34 +01:00
igneum-josh
d3647aad76 fleet prover: export from the node's exec restart block, not genesis (the exporter refuses the unexecuted blocks' zero roots on a restarted 0.3.13 node) 2026-10-06 17:10:15 +01:00
igneum-josh
deafabf78a fleet: night.py (the prover night loop: launches on the executed tip, re-runs the exec restart on a reset, 15-minute night rows), the tarball sha check in the swap, the prover script's FLOOR line restored 2026-10-06 16:52:59 +01:00
igneum-josh
f5ccfb9f59 fleet: box-node-swap.sh rewritten clean (the edited copy called stamp before its definition and misquoted the sha line; no node was touched) 2026-10-06 16:32:35 +01:00
igneum-josh
d70f152021 fleet: swap.py (one call per shipper word over the fleet, the hub first), the rig as N provers (per-card loops, distinct sockets), the rig script's device and chain fixes, restage.py 2026-10-06 16:20:00 +01:00
igneum-josh
2ad424d3e9 fleet: the node swap's two steps (publish 1 binary with the ten-field file, publish 2 the thirteen-field file), the expected digest checked, igneum_getExecStatus read 2026-10-06 15:59:19 +01:00
igneum-josh
2fe2560498 fleet: the 0.3.13 node swap for phase-2 boxes (fixed binary over the full-history datadir, the replay timed); the prover launcher prefers the swapped binary 2026-10-06 14:58:53 +01:00
igneum-josh
22ed1f70a5 Prover tiers on real cards: all eleven cards (the 3090 added); bench-log entry complete 2026-10-06 14:28:55 +01:00
igneum-josh
a0a0f0dc8b fleet: the collector keyed by card and VRAM with a raw-rows fallback; the matrix script's final write guarded against a card that reports no power 2026-10-06 14:22:15 +01:00
igneum-josh
38d2db63e7 Prover tiers on real cards: ten cards, the 8 GB beside-the-miner rows (core-only 2^25 fits); fleet: the rig script and retry 2026-10-06 14:20:06 +01:00
igneum-josh
d1d5f5a6f8 Prover tiers on real cards: eight cards (3060, 3080, 4060 Ti 16 GB, 5090 added), the 8 and 10 GB rows; fleet: the datadir joiner swap, the v5 known-failed runner 2026-10-06 14:07:05 +01:00
igneum-josh
587e531278 Prover tiers on real cards: the first four cards (4090, A5000, 4070, 5070), the method, the per-tier consequences; fleet: RunPod pods, companions pushed with every stage 2026-10-06 13:56:51 +01:00
igneum-josh
3fe3a1145f GPU fleet: box-kill.sh (anchored kills as a file), log rotation before every stage start, parallel orchestrator probes, the listed-after-rent check 2026-10-06 13:46:14 +01:00
igneum-josh
83b71e8c65 GPU fleet: the orchestrator loop, a killable per-second GPU sampler, the STATUS-line rate parser, an atomic locked registry, the p2p hub peer 2026-10-06 13:34:25 +01:00
igneum-josh
78e93d4c50 GPU fleet: RunPod client, the phase-1 collector, the phase-2 launcher (node restarted with the proof verifier), parallel status, the toolchain fix 2026-10-06 13:14:57 +01:00
igneum-josh
627a2256c7 GPU fleet: Vast client, box setup (0.3.12 node + patched SP1 server + cuda host), the phase-1 matrix, the Ember ladder, the Linux segment prover, the fleet page builder 2026-10-06 13:01:23 +01:00
igneum-josh
c2b922c1e6 prover-floor: the three other PC 2 scripts test their kit file before the first use (C32)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 00:06:38 +01:00
igneum-josh
40cff807ad prover-floor: pc2-floor-sweep3-miner.ps1 tests its kit file before the first use (the kit-path check, C32)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 00:04:52 +01:00
igneum-josh
88e28f4d2e Merge prover-floor (cfe3d80) into release-0.3.11: docs and standalone probe sources the evidence rows cite (C38); nothing a built artefact reads 2026-10-06 00:03:46 +01:00
igneum-josh
cc72f4ab29 CI on the merged 0.3.11 tree: MacBook reworded in the analysis prose (the identity check's hostname pattern), a presence check before the kit's first use in the three proving-v1 PC 2 scripts (C32), pc1-cpu-prove.ps1 on the socket check's allow list (the CPU path starts no GPU server)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 23:44:09 +01:00
igneum-josh
cfe3d805d3 Prover floor: the beside-the-miner sweep playbook (sweep 3)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:42:31 +00:00
igneum-josh
eda49ab357 Prover floor patch v2: every trace buffer sized to its padded need (setup keys and shards), the sweep-2 points
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:41:37 +00:00
igneum-josh
fe1ecdbd11 Merge branch 'bash-body-check' into release-0.3.11
# Conflicts:
#	.github/workflows/ci.yml
#	tools/ci/prover-socket-check.sh
2026-10-05 23:40:22 +01:00
igneum-josh
5cbb796051 Merge commit '22c2363' into release-0.3.11
# Conflicts:
#	docs/bench-log.md
#	docs/evidence.md
#	site/litepaper.html
2026-10-05 23:39:34 +01:00
igneum-josh
18605f425c Merge branch 'ca2-coord' into release-0.3.11
# Conflicts:
#	docs/bench-log.md
2026-10-05 23:39:12 +01:00
igneum-josh
e3bd761790 CI: run jobs test their fetched kit before use (the wiped-jobs-folder class)
The app's install clears the jobs folder on a PC, so a run job whose kit was fetched by an earlier fetch job finds
nothing after an update and fails in seconds (5 October 2026, 21:49Z, the AMD kit; bench-log 39f02ff). Rule: a run
playbook that reaches a path under the jobs folder other than its own tests the kit is there before its first use,
and the fetch is republished under a new id after any app update.

tools/ci/kit-path-check.sh reads every *.ps1 under relay/playbooks/ and tools/. A kit root is a path derived from
the jobs folder (`$jobs = Split-Path $env:IGNEUM_JOB_DIR` then `Join-Path $jobs '<fetch id>'`, the race-5090.ps1
shape) or one carrying a literal `jobs\` (the amd-card-test.ps1 shape); every path built from it belongs to that kit.
A presence check (Test-Path, [IO.File]::Exists, [IO.Directory]::Exists, Get-Item or Get-ChildItem with -ErrorAction)
on the root or anything under it covers the whole kit. A use before that line fails with "kit path used before a
presence check: republish the fetch after any app update", as does a literal jobs\ path in a command with no check.
The job's own folder ($env:IGNEUM_JOB_DIR) is not a kit path.

Fixtures: kit-path-ok.ps1 (both shapes, checked; a sibling pack file covered by the worker's check) and
kit-path-unchecked.ps1 (the worker run before its check, a literal never checked); --self-test asserts the lines.
Wired into ci.yml after the bash-body step, and into publish-jobs.sh add --kind run beside the other two checks;
test-publish-jobs.sh gains the refusal (34 passed, 0 failed). The current tree: race-5090.ps1 is the one playbook
with a kit, checked before use. README-ship.md: the rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:37:29 +00:00
igneum-josh
49c7e78307 Merge remote-tracking branch 'origin/master' into ca2-v3
# Conflicts:
#	docs/bench-log.md
#	proto-opencl/host.c
2026-10-05 22:34:07 +00:00
igneum-josh
965584f54a Prover floor: the build recipe as it ran (Go pinned, the binary's sha256 and targets), the RISC Zero contingency figures with their source, playbook fixes
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:33:31 +00:00
igneum-josh
6805125d29 Jobs publisher: the class checks run on the script before it is signed (row C27)
A PC job is published from a worktree by packaging/ota/publish-jobs.sh and never passes CI before it runs; tonight
the root-socket fault came back from a job on a branch without the check. `add --kind run` now runs, on the script
being published and before anything is signed: tools/ci/bash-body-check.sh for a PowerShell script (every inline
bash body parses; a body it cannot read fails, never skips), `bash -n` for a .sh script, and
tools/ci/prover-socket-check.sh for both (a root prover run kills sp1-gpu-server and unlinks its socket). A failure
refuses the publish with the check's output; a missing check file refuses too. Kinds without a script (fetch,
collect, restart, update-now, shard-benchmark, build) are untouched.

tools/ci/prover-socket-check.sh is copied from proving-v1 (c2544be; master lacks it) with two additions: file
arguments check those files only (the publisher's call), and an allow list for packaging/ota/test-publish-jobs.sh,
which carries a known-bad root prover script on purpose. Its ci.yml step is left to proving-v1 to avoid a duplicate.

packaging/ota/test-publish-jobs.sh: four refusals (a lost quote in a PowerShell bash body, an unreadable body, a
.sh with a lost quote, a root prover script without the cleanup) and the envelope unchanged after a refusal.
32 passed, 0 failed on this Mac with the main checkout's signer. packaging/README-ship.md: the publish-time gate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:14:19 +00:00
igneum-josh
c062166071 Prover floor: SP1 6.8.1 GPU server memory model from source (the 24 GB panic, the threshold-sized trace buffers), the floor patch for sp1-gpu, the PC 2 toolchain, build and sweep playbooks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:11:24 +00:00
igneum-josh
a223ca9aef App: the resume path re-arms every stopped card, re-exports its pack and checks 90 s later that every enabled card mines (PC 2 at 21:25:11Z and the Mac that afternoon stayed at 0 MH/s after resume); the PC 2 socket-fix job
The known-failed case, from PC 2's 0.3.9 log (run win-1ccfe586-20261005-200114): 1791234223 pause -> 'stopping the
miners (paused)' (every slot's restart_at cleared, the 5090 'off'); 1791235511 '[ok] mining resumed'; then
'0.00 MH/s, waiting' at every 30-s status line until the 0.3.10 restart at 21:49:41Z. Cause: Cmd::Resume re-armed
only slots whose watchdog said faulted; the 5090's slot was healthy and stopped, so nothing restarted it. The test
the_pc2_resume_of_21_25_11z_restarts_under_the_new_rule_and_not_the_old encodes that slot (faulted false, live
false): the old rule returns [] (the defect), the new rule [0]. cargo test -p igneum-app resume: 3 passed;
provedefault: 6 passed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 22:56:20 +01:00
igneum-josh
7adb1cacb9 CI: bash bodies in PowerShell jobs pass bash -n (the lost-quote class)
Twice on 5 October 2026 a PowerShell job script carried a bash body inside a string, a quote was lost on the way
through PowerShell, and bash refused the body: pc1-cpu-prove.ps1 (first version) reported exit 0 having done
nothing, the 0.3.10 installer job failed in 4 s. tools/amd-prove/check-job-bash.sh covered only its own here-string.

tools/ci/bash-body-check.sh reads every *.ps1 under relay/playbooks/ and tools/, finds each bash body however it is
handed over (bash -c "...", bash -lc '...', bash -c $var, a + concatenation in parentheses, the Start-Process argument
list, a here-string written to a file that is later run with bash), unescapes it the way PowerShell would (backtick
escapes and "" in double-quoted strings, '' in single-quoted strings, here-strings verbatim; $var left as-is, a $(...)
subexpression replaced by ${PS_SUBEXPR}), and runs bash -n on it. One line per body with the file line of the error.
A body it sees but cannot read is "unextractable body" and fails too: a skip would be a hole in the class check.
bash 3.2 compatible; python3 for the extractor.

--self-test runs three fixtures under tools/ci/fixtures/: the correct shapes (8 bodies, must pass), the lost quotes
(the awk apostrophe, a dropped closing quote in a literal and in a variable; must fail with the line), and three
unreadable bodies (must fail). Wired into ci.yml next to the copied-sources check, self-test first. The current tree:
7 inline bodies in 3 playbooks, all parse. packaging/README-ship.md: the job-script rule (body to a file, bash <file>).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 21:38:41 +00:00
igneum-josh
ff873f6d94 Merge origin/master (the explorer pages, the public API check, the CLAUDE.md note) into release-0.3.10; docs, site, observer and ci.yml only 2026-10-05 22:32:17 +01:00
igneum-josh
43ca289ebc fast-time: the proving v1 fields and pow_genesis_dataset_log2 in both override files (the fork's every-field test), README rows; class-v3.mjs reports the build time per epoch; docs/plans/counter-asic-2-node.md (the node side of Counter ASIC 2.0, gate result pending)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 21:04:39 +00:00
igneum-josh
f1d7a7d32a docs/analysis/amd-proving.md: no zkVM proves on AMD (SP1, RISC Zero, Jolt, OpenVM, ICICLE cited), the SP1 CPU prover measured on PC 1 beside the miners (282 s a shard at any size, 30 GB RSS: no CPU tier), the tier consequences and the public line; PC 1 job scripts with the bash -n gate; bench-log entry
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 21:02:59 +00:00
igneum-josh
6c75dade0e Metal worker compiles a class v3 program from the pack a prepare line names; igneum-pow chain_dataset_day seam; pow_genesis_dataset_log2 in the override files
main.swift: servePackProgram reads program.h with the packfile.h checks (generator 2 or 3, the class line against
the generator, the seed bytes, IGNEUM_SEEDW_INIT against attempt_words, class and era against the line) and
compiles program_bound.metal; the program store keys on (seed, class, era); a v3 job with no resident v3 pack
program answers need + error; v2 lines unchanged (Swift generation, the variant race); a pack program never races.
verify.rs: Epoch::chain_dataset_day(day, class, days_since_genesis, genesis_dataset_log2) and days_since_genesis,
the entry the node builds every day cache through (the ca2-mixer growth rule fills the body).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 20:51:24 +00:00
igneum-josh
464d6e140f fast-time: program_class_v3_activation_daa in both override files (never), the README row, and class-v3.mjs, the rollout gate G4 script
class-v3.mjs: a 3-node private network (ports 29600 and up, igneum-devnet-960) on override-60x.json merged with
a CPU genesis difficulty (0x1f010000) and the class switch a few epochs ahead (default 150: inside epoch 2 at
60 DAA per epoch, so the switch rounds up to epoch 3 at DAA 180); one real CPU miner per node; reports blocks on
each side of the boundary, the class and program id of every epoch, rejected blocks (miners and nodes), the
sinks and block counts of every node, and every node's switch line; exit 0 when every check passes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 20:40:30 +00:00
igneum-josh
c2544be3bf Proving v1: the memory sweep and the miner-on peaks, the root-socket class fix (cleanup lines, tools/ci/prover-socket-check.sh in CI), the host's --budget re-plan and the S_p curve job, the RAM and aggregation-card gates, N = 8 in the fast-time file and spec 7.4
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 21:35:13 +01:00
igneum-josh
b3997089ff Proving v1: the chain of 8 on the 5090 measured (N = 2, 4, 8: 32.6, 66.8, 135.6 s; chained aggregation 9.7 s a block on a mining card), the fleet table re-cut on the measured rows
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 21:21:14 +01:00