Commit graph

78 commits

Author SHA1 Message Date
igneum-labs
c461dcc2cd Merge build-server b787ff72 into master on the box mirror (GitHub suspended; the box gate stands in for CI)
# Conflicts:
#	infra/build-server/remote-run.sh
2026-10-07 19:49:37 +00:00
igneum-labs
b787ff7249 lease pool: four priority classes (main, 7 Oct 2026 20:37 BST: the class v5 census sat behind eleven adversarial waiters): release > v5 > measure > adv, from --class or the owner and label; a higher class takes the next freed cores before any lower class whatever the arrival order (lower classes yield while a higher one waits); a lower-class holder above 32 threads is pre-empted (TERM to its command, the lane re-queues) when a higher class has waited 120 s; the class in every line; self-test: the known-failed class-order case (a sweep must not take cores a waiting v5 gate asked for) and the pre-emption case. Also: provision.sh ROLE=sweep (a rented cloud sweep worker, minus runner, caddy, cuda, night, chromium, zig) and infra/build-server/cloud-sweep.sh (Hetzner Cloud CCX workers up/list/cost/down with the lease tool)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 19:40:32 +00:00
igneum-labs
279d76932a lease pool <threads> -- cmd (main, 7 Oct 2026 20:17 BST: adversarial binaries started by hand at 64 to 89 threads, several beside each other, read load 601): a sweep takes up to its thread count of FREE cores from the same 88-core bounded pool as the builds, never fewer than --min, waits while fewer are free, runs pinned at nice 10 with {cores} and {cpuset} substituted and LEASE_CORES exported; the rule is no sweep starts except through it; self-test case added; installed on both boxes
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 19:23:56 +00:00
igneum-labs
9fdc181e53 Merge remote-tracking branch 'box/master' into scrub
# Conflicts:
#	CLAUDE.md
#	docs/plans/counter-asic-3-status.md
#	docs/plans/release-0.3.21.md
#	docs/plans/release-0.3.22.md
#	tools/ci/merge-to-master.sh
#	tools/ci/pre-push.sh
#	tools/ci/red-watch.mjs
2026-10-07 19:16:14 +00:00
igneum-labs
a3ba41b360 Build boxes: the bounded POOL (main, 7 Oct 2026 20:0x BST: every bounded run took 88 threads and the boxes read load 280 to 527): cores 8 to 95 form one pool of 88 per box; a bounded run leases free cores from it (one flock per core, shared with the measurement leases), up to its class cap or --jobs and never fewer than 16, waiting in the queue when fewer are free; CARGO_BUILD_JOBS and taskset follow the cores taken, so the sum of bounded threads on a box never passes 88; self-test updated
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 19:03:35 +00:00
igneum-labs
ed7c3de8e8 Pre-public scrub, second pass (7 October 2026, 20:0x UK, main's rulings 2 and 4): the public tree names igneum-labs only; the public ledger generated from the full ledger
Ruling 2: every tracked text mention of the login's pre-rename spelling becomes igneum-labs (29 files); the commit identity in the scripts becomes igneum-labs <337424239+igneum-labs@users.noreply.github.com> (the same noreply id); the three scripts that hand the stored gh keyring name to gh (register.sh, fetch-ci-artifacts.sh, ship-app.mjs) read it from ~/.config/igneum/gh-user (never in the repository), default igneum-labs; fresh-repo.sh reads the login to rewrite from the encoded list's row 10; the old spelling joins the founder-strings list, so no tracked file may spell it; CLAUDE.md's GitHub paragraph rewritten. The red watcher's self-test fixtures use a neutral author.

Ruling 4: docs/ledger-public.md, one row per ledger item (id, the claim in one line, status, what was done in one line, the evidence link or the evidence in words), generated by tools/ledger/export-public.mjs from docs/fud-ledger.md: no round or status-update sections, no commit ids, no time of day, no lane, agent or branch names. 190 items. The gate runs its self-test (a fixture with a commit id, a time, a lane and a branch name comes out clean; --check fails on drift) and its --check on every merge; the founder check reads the output like every tracked file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 18:56:08 +00:00
igneum-labs
7355d53fde Pre-public scrub, the text pass (7 October 2026, 19:5x UK): no founder name, personal login, earlier business or personal address in any tracked text file, and a gate check that keeps it so
The sweep (main's item 1): 199 tracked text files, 783 lines. The founder's full name, first name and possessive become "the founder" (sentence starts capitalised); the lowercase operating-system user name in WSL paths and commands becomes <user>; the second owner login becomes "the second owner login"; the three earlier businesses and the two other brands become "the other business", "the earlier entity", "the earlier business" and "another brand"; the Chrome profile rule names the igneum.network profile, not the profile's label. The standing commit login igneum-labs is not a founder term here: the fresh-repository step renames it in the history (docs/plans/history-rewrite.md, tools/repo/fresh-repo.sh).

The patterns never appear in plain text in the tree (a plaintext list would be the hit): tools/ci/founder-strings.b64 (perl regex, tab, a sample per row) is read by tools/ci/founder-strings-check.sh (every tracked text file, perl, known-failed first: the self-test plants each row's sample in a fixture and the hit must name the file), by tools/community/discord-hooks.mjs (the guard's founder and business rows; the test takes its fixtures from the samples) and by tools/repo/fresh-repo.sh (the business names of the rewrite rules). site/forbidden-strings.txt carries the same patterns as b64: lines, decoded case-insensitive by site/scrub.mjs and tools/ci/launch-gates-check.mjs (whose fixture now plants an encoded made-up name). The check runs in the gate's tree checks on every merge.

Not in this commit, by main's word: the 105 commit messages and 40 personal-identity commits that need the history rewrite (listed, not run), and the secrets found by gitleaks over the history (reported with owners).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 18:39:50 +00:00
igneum-labs
a85874524f Build boxes to near max (the project lead, 7 Oct 2026 19:4x BST): the bounded class is 88 of 96 cores with -j 88 (8 reserved for release builds, the seed and the observers), the jobs rule 88 alone / 44 shared, the spill line 80; checks updated. testnet-go.md: LG-2 waived by the owner, the new gate, the seeds held armed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 18:23:14 +00:00
igneum-labs
95b82fba69 Devnet 3: devnet3.sh passes the argument line base64-encoded over ssh (ssh flattens its argument list: the first --go started the seed on the OLD devnet, digest c562d70e, port 26611, 7 Oct 2026 18:22 BST) and refuses a line without --devnet-suffix=3
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 17:26:14 +00:00
igneum-labs
7801dc1ceb Devnet 3: devnet3.conf names the 0.3.22 candidate 69d1b56e (genesis timestamp fix)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 17:06:20 +00:00
igneum-labs
d0155e1f96 Devnet 3: devnet3.env becomes devnet3.conf (the no-secrets gate refuses any tracked *.env by name; the file holds ports and a binary path, no secret)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 16:59:56 +00:00
igneum-labs
e9592587b3 Devnet 3: devnet3.env names the 0.3.22 candidate 21d8f454 artefact (the file is tracked on purpose: ports and the binary path, no secret; a *.env ignore rule caught it)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 16:54:12 +00:00
igneum-labs
9d0c5e1b00 Devnet 3 on build-1: ports reconciled with the fleet lane (its observer-node-dn3 on 26650/28650/26651/26850 is the observer instance; node1-dn3 moves to 26671/26670/28670/26870; ufw 26671 open 16:39Z); the units installed as root, disabled
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 16:40:39 +00:00
igneum-labs
f860a54844 Devnet 3 (0.3.22) staged on build-1: the seed as a bare process and the hands' second instances (infra/build-server/devnet3, dry run by default; NET_FLAGS and IGNEUMD placeholders until the node lane names the object and the candidate); ufw and P2P_PORTS carry 26631 and 26651
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 16:28:34 +00:00
igneum-labs
65e5a20976 Build boxes: provision defaults for build-4 (an AX162-1 beside build-3: three slots, the runner pool at 32 cores); the Hetzner cart of 7 Oct 2026 carries both
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 16:21:16 +00:00
igneum-labs
6f8e366569 Build boxes: ufw opens 26621 (the Devnet 2 seed on build-1 listened behind the firewall since it started; opened by hand 7 Oct 2026 16:40 BST, found by the fleet lane)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 15:42:28 +00:00
igneum-labs
ae028d5076 Gate: the box lock self-tests read the same on an idle box and on one at load 120 (private lock directories, no pinning in the fixture, file-driven waits, a 3 s settle, a quiet that outlives the slot settle); the check runs them one at a time with one retry each (the horizon lane, 7 Oct 2026: one red on a full gate, green a minute later)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 15:05:56 +00:00
igneum-labs
678d9835f2 Merge remote-tracking branch 'origin/master' into ui-overlap 2026-10-07 13:57:31 +00:00
igneum-labs
d9417023c4 Merge remote-tracking branch 'origin/master' into ui-overlap 2026-10-07 13:55:13 +00:00
igneum-labs
8e158fa633 Text-overlap sweep (tools/ci/overlap-check.mjs) and the first fixes it found: the home hero's step pill no longer sits on the caption (pill at the corners' baseline, caption above it, at every width), the address page title wraps, the ledger's status badges wrap, the litepaper's mobile contents bar bleeds by the real gutter, the verify harness table scrolls (7 October 2026, 15:5x UK)
The sweep renders every served page in a headless Chromium at 390, 768, 1024, 1280 and 1600 in light and dark (the hero at rest and at each step), reads every visible run of text and flags text covered by another element, clipped by overflow hidden, or past the viewport; a fixture with one deliberate overlap of each kind is flagged before any sweep is trusted. It runs on a build box when this machine has no browser (infra/build-server/overlap-browser.sh installs Chromium there without root).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:55:13 +00:00
igneum-labs
b79d15c724 Gate: the green stamp and the fast path for merges (a full gate that ends GREEN over a clean tree records its commit; the hook lets a merge of a stamped branch onto the exact remote tip through on the light gate, CI runs the full one on landing); tools/ci/merge-to-master.sh; the wave script's seeds mode takes --wipe-genesis with the genesis and base-unit read-backs for the testnet go
Main, 7 October 2026: a 100 s hook gate on the merge commit against a master that moves every minute lost six pushes in a row.
Self-test: a fixture repository (unstamped branch, stamped branch, stale stamp, wrong tip, plain commit, dirty tree).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:54:10 +00:00
igneum-labs
000c82810a CI and boxes: the simulators job runs on master and release-* pushes and pull requests into them only (a feature-branch code push runs the igneum-pow tests alone; tools/ci/sims-branch-check.sh); the box lock self-tests run in parallel in one ssh; build-2 and build-3 join the runner pool bounded to 32 cores; the 0.3.20 first-wave script (seeds, hands, RPC-filter lift; dry run by default)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:46:42 +00:00
igneum-labs
eca43d7b4d Merge remote-tracking branch 'origin/master' into build-server 2026-10-07 13:33:24 +00:00
igneum-labs
0d496d8438 Build boxes: the measure file is retired; a pinned measurement leases its cores only (lease.sh cores), a whole-box quiet measurement is its own class (refused beside a slot or a lease, 20-minute cap, named owner), bounded suites never take it, every run keeps off leased cores, stopped holders are reaped after 5 minutes by every keeper, every waiter has a label file; the box-side self-tests in the gate; provision installs the lease tool and the headless Chromium libraries
Main, 7 October 2026, 15:07 UK: one global exclusive measure flock across unrelated measurements stalled build-1 at load 120 with
free slots (a stopped probe held it 5.5 h; an exclusive waiter queued every new shared taker) and build-2 behind a one-core VDF
bench. lease.sh (installed at /srv/builds/_bin/lease by provision.sh and by hand on both boxes) takes one flock per core for a
pinned measurement and the quiet file for a whole-box one; remote-run.sh takes quiet shared only for unbounded runs, excludes
leased cores from its set, and its keeper refreshes the holder file and calls lease reap (a STOPPED holder of a lease, quiet or a
slot for 5 minutes is killed with a line in _log/reaped.log). Keepers close the lock descriptors they inherit (an orphaned sleep
held a slot and the worktree lock 20 s past the release; the slot self-test had rotted on that since the worktree lock landed).
tools/ci/box-locks-check.sh runs lease.sh --self-test and remote-run.sh --self-test-slots on build-1. provision.sh also carries
the 16 libraries headless Chromium needs (installed by hand on both boxes at 14:5x UK) and a headless self-test step.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:30:40 +00:00
igneum-labs
a638cc516a CI queue: a second self-hosted runner (igneum-build-2 joins the pool label), docs-only pushes skip the compile jobs, the red watcher pinned to the box that posts
31 runs were queued on igneum-build-1's one runner at 13:15 UK on 7 October 2026 and nothing had concluded since 13:03Z, so no lane could read a conclusion.
- ci.yml: a `changes` job (ubuntu-latest) classifies the push with tools/ci/docs-only-check.sh (docs/, site/, *.md only = code=false; a new branch, a pull request, a force push or an API error = code=true); pow and sims need it and run only on code=true. The site job is unchanged on ubuntu-latest for every push. The classifier's self-test is in the gate.
- provision.sh and runner/register.sh: `--host <ip>` registers another box, forwarding BOX_HOSTNAME, RUNNER_NAME, RUNNER_LABELS, RUNNER_CPUS and RUNNER_JOBS (plain words only); RUNNER_CPUS writes AllowedCPUs into the service drop-in beside Nice=10, so igneum-build-2's runner is bounded like a suite (32 cores). The pool label is igneum-build-1 (both boxes carry it); ci-red marks the box with the record file and the poster, the default labels carry it, and igneum-build-1 got it through the runners API today.
- ci-red.yml runs on the ci-red label, so the red line always lands where the poster reads it.
- CLAUDE.md: the rule reads "read the conclusion when it lands, own a red before the next push"; pushes are never held.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:20:18 +00:00
igneum-labs
3794a04c6a Build boxes: the box probe carries its own ssh options (bash 3.2 under set -u refused the unset BS_SSH_OPTS array before bs_host built it; every unpinned route on the Mac died, the pool lane 7 Oct 2026); the router check runs the ssh path under /bin/bash
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:16:39 +00:00
igneum-labs
19ed3eacef Build boxes: the class router is a preference with spill-over (a held or overloaded box hands the job to the other one); build-2 gets a third slot and every run there is bounded on its own 32-core band; the spill decision in the first route line, the slot label and the JSONL row
the project lead, 7 October 2026 15:02 UK: build-1 at load 139 / 114 / 90 with both slots held and a 1 h 40 min queue while build-2 read 4.5 with
free slots, because the class router pinned each class to its box. Now lib.sh bs_route_spill reads the preferred box (free slots,
1-minute load) with one ssh and hands the job to the other box when the preferred one has no free slot or sits above load 64 and the
other qualifies; neither qualifying queues on the class's own box. The decision travels as BR_ROUTE_* into the JSONL "route" object
for the dashboard. build-2's slots file reads 3; everything on box 2 runs at nice 10 / 32 cores / -j 32, and a bounded run takes
the band its slot owns so three never share a core. Self-test tools/ci/route-spill-check.sh (thirteen cases) in the gate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:08:19 +00:00
igneum-labs
d4bc5a9430 Build boxes: the slot holder keeps its own line (a keeper, with its self-test in the gate); an explicit --jobs is clamped for bounded classes; one git remote per box; the host-file double suffix
The dashboard lane, 7 October 2026 10:39Z: both slots of build-1 flock-held and EMPTY while two suites ran. Cause: a run from a
worktree without last night's append-mode fix opens a busy sibling's slot file with > on every probe. The holder now keeps its own
line: a keeper re-writes it within BR_KEEP_S (20 s) whenever the file is empty, until release; remote-run.sh --self-test-keeper
(in the gate) truncates a held line and sees it return, and sees nothing written after release; live on build-1 at 11:19Z (the
line came back in 25 s). The first version deadlocked the runner's bare wait with the keeper (build-2's first run hung 15 min
after its test passed): the keeper stops before the wait. The two running suites' -j 90 came from explicit --jobs 90: a bounded
class now clamps it to its cap with a log line (pass --priority gate for the full set). run-from-mac.sh --box N: the host file
was suffixed twice (build-server-2-2) and every box's mirror would have shared one remote name; one remote per box (build-N).
build-2's first green run: a suite at nice 10 on 32 cores, jobs 32, 986 s cold.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:22:12 +00:00
igneum-labs
73fdbc0fcc Merge remote-tracking branch 'origin/workers-dash-2'
# Conflicts:
#	tools/workers/page/workers.html
2026-10-07 10:43:34 +00:00
igneum-labs
838a14596c Worker dashboard: one server section per box, per-box live feeds, installer and pusher take build-2 and build-3
The build-server agent has igneum-build-2 and -3 on order (suites and benches; proving and the fast-time nodes). The
installer takes a host argument (N reads ~/.config/igneum/build-server-N, or build@<ip>); the pusher drops the Mac and
PC facts on every box it has a host file for, pulls each box's file and publishes boxes[] (box stays the first for the
old shape); the page draws one server section and one crew card per box, reads every box's Caddy feed in parallel
(build, build-2, build-3.igneum.network) with the edge copy filling any box that does not answer, and merges every
box's builds into the lanes, the timeline and the analytics. Nothing changes for build-1 until the host files exist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 10:37:33 +00:00
igneum-labs
dc95dd5059 run-from-mac.sh --box N: the box's dashboard feed name build-N.igneum.network travels to provision as WORKERS_HOST
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 10:35:45 +00:00
igneum-labs
d90cdfdc6f Build boxes: one host file per box and a route by class (suites and benches to build-2, proving to build-3, the rest to build-1); the no-mining rule in a box README and as a guard in the capacity layer
Main's order of 7 October 2026 (igneum-build-2, AX162-1, and igneum-build-3, AX102-1, on order). lib.sh: bs_box_file N and
bs_route <class>; a class whose box has no host file yet falls back to box 1 and says so. run-from-mac.sh --box N <ip> provisions
igneum-build-N and writes build-server-N. tools/build-remote.sh --box N overrides the route; --priority gate always runs on box 1;
the proving crate routes to box 3. README.md: the kind map and the project lead's rule, no mining on any Hetzner box, ever (nodes, builds, tests,
benchmarks and CPU proving only; the pool's fast-time network mines on rented GPU pods, never on build-3). capacity/run.sh refuses a
job that would start igneum-miner mine or a GPU worker, whatever SEQUENCE says.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 10:34:16 +00:00
igneum-labs
c03d000ab8 Build box scheduling: suites and benches at nice 10 on 32 cores with -j 32; a gate kind at nice 0 on the full set ahead of queued suites; kind, nice and cores on the slot label and the JSONL line; the default-class CI check
Main's order of 7 October 2026 after a load of 190 on 96 threads (a release join bench and the 0.3.19 app gate starving each
other, 'builds' of 16 minutes). tools/build-remote.sh resolves a class from the cargo subcommand and --priority: test and bench are
the bounded class (nice 10, the last 32 cores, -j 32) unless --priority gate (nice 0, the full set, the box's own jobs rule);
builds and checks are unchanged. remote-run.sh applies renice and taskset to the command's subshell, caps the jobs, lets a queued
gate (gate-pending-<pid>) take the next slot ahead of suites and benches, and prints nice and cores in the RESULT line and the
JSONL line (nice, cores, priority). The slot label carries '; kind=<k> nice=<n> cores=<c>' before '; agent=', so the dashboard's
job card shows why a job is slow. --plan prints the resolved class without the box; tools/ci/build-kind-default-check.sh (in the
gate) holds the five shapes. Smoke on the box: a suite at jobs=32 nice=10 cores=32, a gate at nice=0 cores=96.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 10:29:26 +00:00
igneum-labs
5e5a309a04 build server: the remote checkout's clean spares a lane's scratch (AP-H1, the lost-scratch class)
remote-run.sh checkout_tree ran `git clean -fd` on the box mirror before every build from any agent, so the attack rows
lost attack-f3/, attack-f1-venv/ and tools/attack/*/target to each other's builds (7 October 2026, 09:2x UK). The clean now
also spares the fixed prefixes attack-*, scratch-*, target-attack-*, .build-remote.log and every glob in the mirror-local
.igneum-scratch-spare (one per line, # comments, the file itself spared), keeps the target and stamp excludes and still runs
without -x. The clean-tree test asks `git clean -nd` with the same excludes instead of filtering the status list, so a spared
dir is not "not clean". --self-test: a fixed-prefix dir at the root and nested, a declared dir and the spare file survive, an
undeclared dir is removed. tools/ci/scratch-spare-check.sh in the pre-push gate fails when the clean line loses the spare
arguments, spare_args stops reading the file, a fixed prefix goes, or -x appears. docs/plans/build-server.md R4a says how a
lane declares its prefix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 08:24:07 +00:00
igneum-labs
b99a867e17 Observer sync follows GitHub: the deploy-key probe captured before grep (ssh -T exits 1 under pipefail); plan 5c done
the project lead added the read-only deploy key on 7 October 2026 (SHA256:51ice3W8...). The sync still said 'mirror' because ssh -T to GitHub
exits 1 after its greeting and observer-sync.sh runs under pipefail, so su ... | grep -q reported failure although grep had
matched (the same line by hand, without pipefail, said authenticated; shown under the unit's environment with systemd-run -v).
The probe's output is captured first. First github pass 07:30:54 UTC: the clone moved from the mirror's 99a98ee to origin/master
13bb606c, the observer restarted on it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 07:31:42 +00:00
igneum-labs
6e91b00882 Hands mover: on 0.3.18 trees the restart read-back takes powEngine and the blockrate object from igneum_getNodeInfo; a stub answer stops the sequence
The shipper's read-back for 0.3.18 (7 October 2026): igneum_getNodeInfo exists again and must answer powEngine igneum-pow (a stub
is a FAIL) with a blockrate object, which the mover prints; a tree before 0.3.18 answers -32601 and the engine is still read from
the binary's igneum-pow source paths. The parser is checked against the three answer shapes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 06:25:12 +00:00
igneum-labs
d008726510 lib.sh: bs_pin gives an empty pin for a tree without rust-toolchain.toml (a failed sed under pipefail ended the caller silently)
The b3c228fa builds under the 0.3.16 app tree 5d118f58 (no rust-toolchain.toml yet) died right after the pairing line with no
message: sed on the missing file failed, pipefail carried its status into the assignment, set -e ended the script. A guard and a
tolerant pipeline; checked alive against a tree without the file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 03:44:26 +00:00
igneum-labs
e22b41ad4a Build tools: whole-body blocks (the edited-while-running class, with its check), the igneum-pow pairing line, move-hand.sh restart with its readbacks
Three classes from the 0.3.17 night. (1) bash reads a script incrementally: tools/build-remote.sh was edited while a four-minute remote
build ran, the running copy continued at shifted bytes and died with a syntax error after the build had succeeded on the box; the
four long-running tools (build-remote, cross-remote, workers-remote, move-hand) now keep their body in one brace block ending in exit,
parsed whole before a line runs; tools/ci/whole-body-check.sh (in the gate, self-test with a block-less copy) holds the shape.
(2) A fork build pairs with the igneum-pow of the igneum worktree it sits in: a fork at 12153428 under a master worktree failed in
kaspa-pow four minutes in (no chain_program_shadow; master's igneum-pow predates release-0.3.17's); build-remote.sh says the
pairing on its first line ('pairs with igneum 6f8d7a7e (detached): igneum-pow 0.2.0') and the JSONL line carries pairs_with.
The first version of that line used '[ -n ... ] && echo' inside an assignment's $( ) and set -e ended the script on the false
status; fixed. (3) move-hand.sh restart <hand> [--digest <hex>] [--go]: after binary installed a release, restart ONE unit and read
it back (first exec line, commit string in the running binary, digest against the wanted one, igneum_getNodeInfo powEngine over the
node's loopback EVM RPC); the digest readers tolerate a missing line.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 03:12:51 +00:00
igneum-labs
0c87b223bf glibc ceilings per artefact class: hive and rig 2.31, seed and linux 2.35, native unchecked; proven in ubuntu:20.04 and 22.04 on the box
Main's order of 7 October 2026 after RunPod's Ubuntu 22.04 canaries (glibc 2.35) refused the box's GLIBC_2.38 binaries and HiveOS
turned out Ubuntu 20.04 based (2.31). The table lives once, in tools/ci/glibc-ceiling-check.sh (--class, --ceiling-of; self-test
covers the table, an unknown class and a 2.34 need against hive); tools/build-remote.sh --ship hive|rig|seed|linux (default seed)
and tools/workers-remote.sh --class (default rig) build with zig at the class's glibc and check against it. provision.sh installs
docker.io (user build in the docker group) for the proof. Proof: fork 3bfe346f at class hive, igneumd and igneum-miner need
GLIBC_2.30; the workers at class rig need GLIBC_2.17; all four run in ubuntu:20.04 (ldd 2.31) and ubuntu:22.04 (ldd 2.35) and
print their version or usage lines (the OpenCL worker its own no-libOpenCL message, the binary running in a GPU-less container).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 00:35:18 +00:00
igneum-labs
2161c76968 Deploy key: the plan's section 5c (the project lead's four steps and the public half) that c47e9a3c named but did not carry; observer-sync.sh runs git as build
The previous commit's plan edit aborted on a changed anchor, so section 5c was missing; written now with the public half
(ssh-ed25519 ... igneum-build-1 observer read-only) and the CI-runner row closed. observer-sync.sh ran git rev-parse as root
against the build-owned clone (safe.directory refused it, 'up to date at' with no commit); every git call runs as build now.
Verified on the box: one sync pass prints 'source: mirror (the deploy key is not accepted by GitHub yet ...)' and the commit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 23:49:20 +00:00
igneum-labs
c47e9a3cbc Observer clone: a read-only deploy key made on the box, GitHub preferred over the mirror once accepted; the project lead's steps in the plan
Main's order of 7 October 2026. install-hands.sh creates /srv/observer/.ssh/deploy_igneum (ed25519, user build, mode 600; the private
half never leaves the box and nothing prints it) and the ssh alias github-igneum-observer; observer-sync.sh tests the key with
ssh -T on every pass, pulls from GitHub when it is accepted and from the mirror otherwise, saying which. docs/plans/build-server.md
5c: the four steps for the project lead (print the public half, Settings > Deploy keys > Add, read-only, start one sync pass) and the public
half itself. Verified on the box: key created, alias written, one sync pass reads 'source: mirror (the deploy key is not accepted by
GitHub yet)'. The CI-runner row is closed (the box-work agent's runner service).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 23:47:10 +00:00
igneum-labs
08264fcb4d rust-toolchain.toml: one pin for every side (1.99.0, the two cross targets); the mismatch refusal reads it
Main's order of 7 October 2026. The repo root carries rust-toolchain.toml (channel 1.99.0, targets x86_64-pc-windows-gnu and
x86_64-unknown-linux-gnu); rustup resolves the nearest file walking up from the crate, so the fork worktrees under vendor/ are
covered, and the fork's master carries its own copy (vendor/igneum-node 37f1206b). lib.sh bs_toolchain_check reads the pin
and refuses a build when the pin, the Mac's rustc as resolved in the crate dir, or the box's rustc differ (the message says
the three commands that align them); run-from-mac.sh passes the pin to provision.sh as RUST_TOOLCHAIN. The 1.99.0 toolchain
with both targets is installed on the Mac so no agent's build stalls on rustup's auto-install. Verified: the Mac resolves
1.99.0 in a fork worktree, the box runs 1.99.0, the check prints 'pinned 1.99.0 by rust-toolchain.toml'.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 23:44:20 +00:00
igneum-labs
7e34ca10b6 Build server: zig and cargo-zigbuild on the box; build-remote.sh --ship builds glibc 2.36 Linux artefacts for seeds and HiveOS; the glibc ceiling check
Main's order of 7 October 2026 after a seed took 14 restarts and three minutes down on a glibc 2.39 binary. provision.sh:
step_zig (zig 0.17.0 from ziglang.org, sha256 from the official download index) and cargo-zigbuild 0.23.4 in the cargo tools.
tools/build-remote.sh --ship [--glibc 2.36]: cargo zigbuild --target x86_64-unknown-linux-gnu.2.36 with zig as the C/C++
toolchain (the Mac's infra/cross/build-linux.sh recipe), artefacts from the target-triple dir, each checked by
tools/ci/glibc-ceiling-check.sh (need at most the ceiling; self-test fires on 2.38 against 2.36, passes 2.34 and 2.36;
--symbols reads a saved objdump -T text so CI needs no ELF tools). tools/workers-remote.sh builds the two GPU workers with
zig at 2.36 by default (GLIBC=native for clang). Proof on the box: fork 3bfe346f igneumd needs GLIBC_2.34 (47,023,120 B,
sha256 345dfb95...), igneum-miner GLIBC_2.34 (9,248,808 B, d09dc27b...), 3 min 17 s cold through zig; the workers at 2.36.
Rule: anything that ships to a seed or a HiveOS rig is built with --ship; a plain build (glibc 2.39) is for the box and
Ubuntu 24.04 hosts only.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 23:37:33 +00:00
igneum-labs
f525835a51 Hands moved to igneum-build-1 (run log); the overlay carries directories a crate reaches by include_bytes!; two mover fixes
docs/plans/hands-on-build-1.md section 6: the move of 6 October 2026 23:06 to 23:22 UTC, observer node, observer, node 1, unload,
each hand's first executing line, digest eada4bda MATCH, IBD and acceptance, the open readback (the Mac's Miner app has not
started its own node since 26610/26611 were freed). lib.sh: the shipper's class from the 0.3.17 tree, the fork's igneum-exec
embeds proving/igneum-prove/elf/*.vk by include_bytes! five levels up, which is no path dependency; every .rs in the trees that
travel is scanned for include_bytes!/include_str! paths leaving the crate's repository and their directories join the overlay;
proving/igneum-prove/elf is the fixed fallback for a fork build. move-hand.sh: igneumd --version exits 1 (tolerated; it ended
the binary step before the override was written), and the launchd pid lookup used \s in macOS awk (printed 'pid none').

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 23:25:36 +00:00
igneum-labs
210b2f892b Hands mover: the node tree defaults to the Mac agent's own build, a digest readback per hand against the Mac hand, a 16-field guard on the override
Main's hold of 6 October 2026 (23:xx UK): the hands move only on the shipper's 'publish 2 live: digest <x>' line, each hand read back by
commit string and consensus digest. move-hand.sh binary now derives --node from the fork tree behind the Mac's node1 launchd agent
(igneum-wt-ship0315/vendor/igneum-node-0315 at f1ea7a38 tonight; the box's igneumd rebuilt from it is sha256 7f0bde70...), warns when the
Mac's override still has 13 fields (publish 2 writes the sixteen-field object when it restarts the hands), and after each hand's first
executing line prints the box binary's commit string and the hand's digest against the Mac hand's last digest (MATCH or DIFFER, DIFFER
stops the sequence). Dry run clean against the live Mac.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 22:51:06 +00:00
igneum-labs
f1c0e6b9dd Kill by exact command line or pid file, never by a name: tools/ci/kill-by-name-check.sh in the gate; the 36 pgrep/pkill literals in the tree fixed
The fleet's 22:09 UK incident (a Mac-side pkill -f <log file name> matched nothing, the roll-everything script lived on and wiped a held box) and the day's two pgrep self-matches are one class. The check flags pgrep -f / pkill -f with a plain literal (every one on a line), any pgrep/pkill on a file-name shape, and ps | grep with a literal; it allows the bracket form, -x, -F pidfile, kill $(cat pidfile), a variable and a full path; 11 banned and 16 allowed shapes in its self-test; 0.15 s over the tree. The 25 pkill -f sp1-gpu-server inside bash -c bodies (which matched the calling bash) are pkill -x; the other 11 literals take the bracket form; prover-socket-check accepts both. Row R in the record; the CLAUDE.md rule names the check and covers pkill and file names.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 22:10:31 +00:00
igneum-labs
8f2b56abfc Merge ci-hardening 9bfaa57: one gate script for the hook and CI, the research exclusion list, the Windows paths check, the red watcher and the box's red-row classes with pre-flight
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:43:37 +00:00
igneum-labs
9bfaa57cbb Build box: every red run classified and kept, pre-flight before the slot, one run per worktree, box reds in the red-run file, a 09:00 UK digest
The box's 34 red rows of 6 October classified (docs/analysis/ci-failures-2026-10-06.md section 6): 22 iterations, 12 in three real classes (instant deaths with nothing kept, a shared worktree directory, an unread dry run). remote-run.sh now: pre-flight (subcommand, manifest, -p package, --features) refuses in a second with exit 3 and a class; the last 400 lines of every run kept in /srv/builds/_log/runs; a class on every row (compile-error, link-error, test-failure, instant, no-test-matched, slot-timeout, no-dir, preflight-*); a cargo test whose filter matched no test exits 3; a per-worktree lock in checkout and run mode; every red row appended to /srv/ci-red/red.jsonl as source box. red-watch.mjs never posts a box row alone and sends one digest a day (counts per class with each class's guard); the timer runs tick. Shared group cired on the box so the runner and build append to one file. Shown in a sandbox on the box: pass, failing test, empty filter, bad package, bad feature, missing subcommand, compile error, broken manifest, two concurrent runs of one worktree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:40:07 +00:00
igneum-labs
576136b97b Merge box-capacity 585e87b: the builder's background capacity layer (fuzz, sims, sweeps, clippy; yields to builds)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:35:54 +00:00
igneum-labs
585e87b027 Capacity layer: fixes from the smoke runs
- pow-fuzz: list saved mismatches from the directory, not an ls|node pipe (pipefail ran
  the || echo too, giving invalid JSON [][]); drop the dead inner accumulation line
- sync-fuzz: merge the override with python, not node (node rounds the u64::MAX activation
  fields to a float the Rust parser rejects); retention-period-days 2 (the 2-day minimum);
  grep -c without || echo (pipefail doubled the count)
- clippy-audit: cap_cargo_t (timeout cannot run the cap_cargo shell function); grep -c fix
- all jobs sync the checkout unconditionally and lib.sh defaults the branch vars so a
  standalone job or smoke never trips set -u on CAP_NODE_BRANCH

Smokes on igneum-build-1, all 0 panics: pow-fuzz 98 rounds / 19,600 programs / 78,400
units; sync-fuzz 142,883 requests, node alive, every kind disconnected or answered;
sim-sweeps 5 rows; model-sweeps 7 sections; clippy-audit 69 branches, 1,192 warnings,
1 error (ca2-coord), 0 advisories.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 21:32:55 +00:00