Commit graph

21 commits

Author SHA1 Message Date
igneum-labs
d60d27525b Merge remote-tracking branch 'box/master' into scrub
# Conflicts:
#	CLAUDE.md
#	docs/plans/counter-asic-3-status.md
#	docs/plans/release-0.3.21.md
#	docs/plans/release-0.3.22.md
#	tools/ci/merge-to-master.sh
#	tools/ci/pre-push.sh
#	tools/ci/red-watch.mjs
2026-10-07 19:16:14 +00:00
igneum-labs
7eed16a29a Pre-public scrub, the text pass (7 October 2026, 19:5x UK): no founder name, personal login, earlier business or personal address in any tracked text file, and a gate check that keeps it so
The sweep (main's item 1): 199 tracked text files, 783 lines. The founder's full name, first name and possessive become "the founder" (sentence starts capitalised); the lowercase operating-system user name in WSL paths and commands becomes <user>; the second owner login becomes "the second owner login"; the three earlier businesses and the two other brands become "the other business", "the earlier entity", "the earlier business" and "another brand"; the Chrome profile rule names the igneum.network profile, not the profile's label. The standing commit login igneum-labs is not a founder term here: the fresh-repository step renames it in the history (docs/plans/history-rewrite.md, tools/repo/fresh-repo.sh).

The patterns never appear in plain text in the tree (a plaintext list would be the hit): tools/ci/founder-strings.b64 (perl regex, tab, a sample per row) is read by tools/ci/founder-strings-check.sh (every tracked text file, perl, known-failed first: the self-test plants each row's sample in a fixture and the hit must name the file), by tools/community/discord-hooks.mjs (the guard's founder and business rows; the test takes its fixtures from the samples) and by tools/repo/fresh-repo.sh (the business names of the rewrite rules). site/forbidden-strings.txt carries the same patterns as b64: lines, decoded case-insensitive by site/scrub.mjs and tools/ci/launch-gates-check.mjs (whose fixture now plants an encoded made-up name). The check runs in the gate's tree checks on every merge.

Not in this commit, by main's word: the 105 commit messages and 40 personal-identity commits that need the history rewrite (listed, not run), and the secrets found by gitleaks over the history (reported with owners).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 18:39:50 +00:00
igneum-labs
65fdd39d5b Build boxes to near max (the project lead, 7 Oct 2026 19:4x BST): the bounded class is 88 of 96 cores with -j 88 (8 reserved for release builds, the seed and the observers), the jobs rule 88 alone / 44 shared, the spill line 80; checks updated. testnet-go.md: LG-2 waived by the owner, the new gate, the seeds held armed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 18:23:14 +00:00
igneum-labs
ab364a10c8 Build boxes: the engine gate (igneumd and igneum-miner fetched by build-remote must carry igneum-pow/src/ paths; a commit string alone does not prove the engine: the stub-engine 21d8f454 that rejected every block on the Devnet 3 hub, 7 Oct 2026); tools/ci/engine-check.sh with its self-test in the gate
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 16:55:43 +00:00
igneum-labs
b04c4b3ac9 Build boxes: the class router is a preference with spill-over (a held or overloaded box hands the job to the other one); build-2 gets a third slot and every run there is bounded on its own 32-core band; the spill decision in the first route line, the slot label and the JSONL row
the project lead, 7 October 2026 15:02 UK: build-1 at load 139 / 114 / 90 with both slots held and a 1 h 40 min queue while build-2 read 4.5 with
free slots, because the class router pinned each class to its box. Now lib.sh bs_route_spill reads the preferred box (free slots,
1-minute load) with one ssh and hands the job to the other box when the preferred one has no free slot or sits above load 64 and the
other qualifies; neither qualifying queues on the class's own box. The decision travels as BR_ROUTE_* into the JSONL "route" object
for the dashboard. build-2's slots file reads 3; everything on box 2 runs at nice 10 / 32 cores / -j 32, and a bounded run takes
the band its slot owns so three never share a core. Self-test tools/ci/route-spill-check.sh (thirteen cases) in the gate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 13:08:19 +00:00
igneum-labs
97255a4e8f Build boxes: the slot holder keeps its own line (a keeper, with its self-test in the gate); an explicit --jobs is clamped for bounded classes; one git remote per box; the host-file double suffix
The dashboard lane, 7 October 2026 10:39Z: both slots of build-1 flock-held and EMPTY while two suites ran. Cause: a run from a
worktree without last night's append-mode fix opens a busy sibling's slot file with > on every probe. The holder now keeps its own
line: a keeper re-writes it within BR_KEEP_S (20 s) whenever the file is empty, until release; remote-run.sh --self-test-keeper
(in the gate) truncates a held line and sees it return, and sees nothing written after release; live on build-1 at 11:19Z (the
line came back in 25 s). The first version deadlocked the runner's bare wait with the keeper (build-2's first run hung 15 min
after its test passed): the keeper stops before the wait. The two running suites' -j 90 came from explicit --jobs 90: a bounded
class now clamps it to its cap with a log line (pass --priority gate for the full set). run-from-mac.sh --box N: the host file
was suffixed twice (build-server-2-2) and every box's mirror would have shared one remote name; one remote per box (build-N).
build-2's first green run: a suite at nice 10 on 32 cores, jobs 32, 986 s cold.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 11:22:12 +00:00
igneum-labs
b6e5dffa32 Build boxes: one host file per box and a route by class (suites and benches to build-2, proving to build-3, the rest to build-1); the no-mining rule in a box README and as a guard in the capacity layer
Main's order of 7 October 2026 (igneum-build-2, AX162-1, and igneum-build-3, AX102-1, on order). lib.sh: bs_box_file N and
bs_route <class>; a class whose box has no host file yet falls back to box 1 and says so. run-from-mac.sh --box N <ip> provisions
igneum-build-N and writes build-server-N. tools/build-remote.sh --box N overrides the route; --priority gate always runs on box 1;
the proving crate routes to box 3. README.md: the kind map and the project lead's rule, no mining on any Hetzner box, ever (nodes, builds, tests,
benchmarks and CPU proving only; the pool's fast-time network mines on rented GPU pods, never on build-3). capacity/run.sh refuses a
job that would start igneum-miner mine or a GPU worker, whatever SEQUENCE says.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 10:34:16 +00:00
igneum-labs
69034dc071 Build box scheduling: suites and benches at nice 10 on 32 cores with -j 32; a gate kind at nice 0 on the full set ahead of queued suites; kind, nice and cores on the slot label and the JSONL line; the default-class CI check
Main's order of 7 October 2026 after a load of 190 on 96 threads (a release join bench and the 0.3.19 app gate starving each
other, 'builds' of 16 minutes). tools/build-remote.sh resolves a class from the cargo subcommand and --priority: test and bench are
the bounded class (nice 10, the last 32 cores, -j 32) unless --priority gate (nice 0, the full set, the box's own jobs rule);
builds and checks are unchanged. remote-run.sh applies renice and taskset to the command's subshell, caps the jobs, lets a queued
gate (gate-pending-<pid>) take the next slot ahead of suites and benches, and prints nice and cores in the RESULT line and the
JSONL line (nice, cores, priority). The slot label carries '; kind=<k> nice=<n> cores=<c>' before '; agent=', so the dashboard's
job card shows why a job is slow. --plan prints the resolved class without the box; tools/ci/build-kind-default-check.sh (in the
gate) holds the five shapes. Smoke on the box: a suite at jobs=32 nice=10 cores=32, a gate at nice=0 cores=96.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 10:29:26 +00:00
igneum-labs
db6475def5 Build tools: whole-body blocks (the edited-while-running class, with its check), the igneum-pow pairing line, move-hand.sh restart with its readbacks
Three classes from the 0.3.17 night. (1) bash reads a script incrementally: tools/build-remote.sh was edited while a four-minute remote
build ran, the running copy continued at shifted bytes and died with a syntax error after the build had succeeded on the box; the
four long-running tools (build-remote, cross-remote, workers-remote, move-hand) now keep their body in one brace block ending in exit,
parsed whole before a line runs; tools/ci/whole-body-check.sh (in the gate, self-test with a block-less copy) holds the shape.
(2) A fork build pairs with the igneum-pow of the igneum worktree it sits in: a fork at 12153428 under a master worktree failed in
kaspa-pow four minutes in (no chain_program_shadow; master's igneum-pow predates release-0.3.17's); build-remote.sh says the
pairing on its first line ('pairs with igneum 6f8d7a7e (detached): igneum-pow 0.2.0') and the JSONL line carries pairs_with.
The first version of that line used '[ -n ... ] && echo' inside an assignment's $( ) and set -e ended the script on the false
status; fixed. (3) move-hand.sh restart <hand> [--digest <hex>] [--go]: after binary installed a release, restart ONE unit and read
it back (first exec line, commit string in the running binary, digest against the wanted one, igneum_getNodeInfo powEngine over the
node's loopback EVM RPC); the digest readers tolerate a missing line.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 03:12:51 +00:00
igneum-labs
3720c40bce glibc ceilings per artefact class: hive and rig 2.31, seed and linux 2.35, native unchecked; proven in ubuntu:20.04 and 22.04 on the box
Main's order of 7 October 2026 after RunPod's Ubuntu 22.04 canaries (glibc 2.35) refused the box's GLIBC_2.38 binaries and HiveOS
turned out Ubuntu 20.04 based (2.31). The table lives once, in tools/ci/glibc-ceiling-check.sh (--class, --ceiling-of; self-test
covers the table, an unknown class and a 2.34 need against hive); tools/build-remote.sh --ship hive|rig|seed|linux (default seed)
and tools/workers-remote.sh --class (default rig) build with zig at the class's glibc and check against it. provision.sh installs
docker.io (user build in the docker group) for the proof. Proof: fork 3bfe346f at class hive, igneumd and igneum-miner need
GLIBC_2.30; the workers at class rig need GLIBC_2.17; all four run in ubuntu:20.04 (ldd 2.31) and ubuntu:22.04 (ldd 2.35) and
print their version or usage lines (the OpenCL worker its own no-libOpenCL message, the binary running in a GPU-less container).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 00:35:18 +00:00
igneum-labs
7089aa161f Build server: zig and cargo-zigbuild on the box; build-remote.sh --ship builds glibc 2.36 Linux artefacts for seeds and HiveOS; the glibc ceiling check
Main's order of 7 October 2026 after a seed took 14 restarts and three minutes down on a glibc 2.39 binary. provision.sh:
step_zig (zig 0.17.0 from ziglang.org, sha256 from the official download index) and cargo-zigbuild 0.23.4 in the cargo tools.
tools/build-remote.sh --ship [--glibc 2.36]: cargo zigbuild --target x86_64-unknown-linux-gnu.2.36 with zig as the C/C++
toolchain (the Mac's infra/cross/build-linux.sh recipe), artefacts from the target-triple dir, each checked by
tools/ci/glibc-ceiling-check.sh (need at most the ceiling; self-test fires on 2.38 against 2.36, passes 2.34 and 2.36;
--symbols reads a saved objdump -T text so CI needs no ELF tools). tools/workers-remote.sh builds the two GPU workers with
zig at 2.36 by default (GLIBC=native for clang). Proof on the box: fork 3bfe346f igneumd needs GLIBC_2.34 (47,023,120 B,
sha256 345dfb95...), igneum-miner GLIBC_2.34 (9,248,808 B, d09dc27b...), 3 min 17 s cold through zig; the workers at 2.36.
Rule: anything that ships to a seed or a HiveOS rig is built with --ship; a plain build (glibc 2.39) is for the box and
Ubuntu 24.04 hosts only.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 23:37:33 +00:00
igneum-labs
6cc1bc9d57 cross-remote.sh: the default cross-build runs again (a trailing comment had swallowed the defaults line); the class gets a CI check
The shipper's report: with no arguments, cross-remote.sh died under the Mac's bash 3.2 with 'CARGO_ARGS: unbound variable' and
its chain kept the previous exes. Cause: a comment appended to the defaults line in the box-work merge turned OUT, COMPARE,
TARGET_DIR and CARGO_ARGS=() into comment text, so the array was never declared (the same shape build-remote.sh had at 19:5x).
Fix: the line split, and the default tests use [ -n "${CARGO_ARGS[*]:-}" ] in both tools, safe whether or not the array
exists. Proof: the default cross-build from a fork worktree under /bin/bash 3.2.57 built both exes (igneumd.exe 8f7e2ae3...,
igneum-miner.exe 6f8b49d8...). tools/ci/defaults-line-check.sh fails CI on a line where a comment start is followed by an
assignment list (quoted strings, ${...}, $# and [^#] dropped first); self-test fires on the swallowed shape and passes
${a#b}, sed s#x#y#, $# loops and prose mentions; the tree is clean.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 20:55:44 +00:00
igneum-labs
3fffa083cd build-remote.sh: the sync log line names the touch re-stamp on a code line (the copied-sources check reads code lines only since 141b559)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 20:11:46 +00:00
igneum-labs
7e833e36a1 build-remote.sh: the header names the overlay's touch again (the copied-sources check reads each file for the word)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 20:10:01 +00:00
igneum-labs
67ae1e4c6d Reproducible builds: SOURCE_DATE_EPOCH from the commit's author time, TZ=UTC and one fixed target path in every build path; self-test
Main's rule of 6 October 2026 from the 0.3.14 repro (docs/evidence/reproduced/0.3.14.md): prost's protowire.rs embeds OUT_DIR,
libmimalloc-sys embeds __DATE__/__TIME__, sccache hid both. lib.sh bs_repro_env exports SOURCE_DATE_EPOCH=<author time> TZ=UTC in
front of every remote command (build-remote.sh, cross-remote.sh, workers-remote.sh); remote-run.sh exports BR_SDE too and logs it
as source_date_epoch; proto-cuda/windows-node/cross-build.sh exports the same; the PC job carries node.commit_time in the manifest
(push-build-inputs.sh) and exports it before every cargo build of a stage (jobbuild.rs, unit test asserts it; 4 of 4 pass on the
box). Target dirs stay one fixed path per target. tools/build-remote.sh --self-test-repro [--full] from a fork worktree, run on
the box: igneum-miner twice a minute apart without sccache (RUSTC_WRAPPER=/usr/bin/env, an empty value is unset to cargo) MATCH
91e130f5..., a per-run target path differs (OUT_DIR shown); --full: kaspad with libmimalloc-sys recompiled a minute later MATCH
70219bc2..., without the epoch differs (__DATE__ shown).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 20:09:23 +00:00
igneum-labs
7184e5bfc7 Box: GitHub Actions runner as user runner, two build slots with 90 or 45 jobs, the measure hold, the CPU prover trial
- provision.sh step_runner: actions/runner 2.338.0 (sha256 checked) at /opt/actions-runner under a dedicated user `runner`
  (no sudo, not in build's group), rustup 1.99.0 pinned with both targets, sccache against /srv/sccache in READ_ONLY mode
  on its own server port, Node 22 and mingw from the system, GitHub's svc.sh unit with a Nice 10 drop-in; registered on
  igneum-network/igneum as igneum-build-1 (labels self-hosted, linux, x64, igneum-build-1) through
  infra/build-server/runner/register.sh (gh as igneum-labs, the token on ssh stdin, never logged). Idempotent after
  the env files moved behind svc.sh install (its env.sh rewrites them). libicu74 and python3-numpy added to APT.
- main's slots ruling: SLOTS default 2; remote-run.sh sets CARGO_BUILD_JOBS 90 when it holds the only taken slot and 45
  when both are held, BR_MEASURE=1 takes the `measure` file exclusively and excludes builds (builds hold it shared),
  lock files open in append mode (the old `exec {fd}>` truncated a busy slot's holder line on every probe), env
  IGNEUM_BUILD_SLOTS_DIR and IGNEUM_BUILD_LOG_DIR win over the profile, `--self-test-slots` with five cases (the old
  script fails it with JOBS=none); build-remote.sh and cross-remote.sh pass -j only when --jobs is given.
- infra/build-server/prover/cpu-trial.sh: the SP1 CPU prover on one fixture shard under the measure hold with a VmHWM
  poller; 6 Oct 2026 run: core 34.2 s, compressed 85.9 s, peak RSS 28.2 GB on 96 threads, so no standing CPU prover.
- docs/plans/ci-self-hosted.md: the proposed runs-on change for ci.yml behind the repository variable IGNEUM_CI_RUNNER
  (GitHub-hosted is the fallback), and why windows.yml cannot move to a Linux box. Workflows untouched.
- docs/plans/build-server.md section 7.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 19:32:43 +00:00
igneum-labs
8561fadb33 Build server: the devnet hands' move to igneum-build-1 (plan, installer, mover), a per-worktree lock, the nested-stamp checkout fix
the project lead's decision of 6 October 2026: the Mac runs nothing the network depends on. docs/plans/hands-on-build-1.md plans node 1
and the observer (its node and tools/observer) as systemd units on the box, one hand at a time, with ports, DNS names,
exec recovery, rollback and the decisions for main; infra/build-server/hands/install-hands.sh writes users, dirs, run
scripts and units (inert, run on the box); move-hand.sh (dry run by default) builds 0.3.15 on the box, copies the override
and snapshot, rsyncs each data dir hot then stopped, starts the unit and prints the hand's first executing line, copies the
observer env by scp (mode 600), stops the Mac's observer first, unloads the launchd agents last. CLAUDE.md's running-agents
rule carries the decision. lib.sh: a per-worktree lock on the box across sync, build and fetch (the shipper's collision at
18:48:56Z). remote-run.sh: the clean-tree test excuses stamps and target dirs at any depth with --untracked-files=all (a
nested stamp failed every checkout of /srv/builds/igneum for twenty minutes) and removes a stale .git/index.lock; the
self-test covers both. Plan rows for the three and the cuda prover pair built for the shipper.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:58:14 +00:00
igneum-labs
08e469ff65 Build server: a path dependency inside a vendor repository is synced as a whole repository (the shipper's proving build)
proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a member of the fork's workspace that inherits
from the fork's root manifest; syncing that one directory left cargo without a workspace root on the box. lib.sh now groups
path dependencies by git top level: a repository under vendor/ is pushed to its mirror (a fork worktree to
/srv/igneum-node.git, a repository of its own to /srv/<name>.git, created on first use), checked out whole at
/srv/builds/<worktree>/vendor/<name> and overlaid whole; run-from-mac.sh wires every vendor repository the Cargo.toml files
reach. libprotobuf-dev added (sp1-prover-types imports google/protobuf/empty.proto). build-remote.sh no longer fails on a
default artefact the caller's own -p selection did not build. Proof on the box: igneum-prove-host 71,943,192 B, sha256
e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, 1 min 05 s warm. Plan: gotcha rows for the case, the
protoc miss and one lost ssh session (collector cleared by test).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:36:39 +00:00
igneum-labs
aed5ca9bd7 Build server adopted: box-first build rule in CLAUDE.md, reproducible Windows exes, the commit-string gate, PC and Mac recipes
Main's decision of 6 October 2026. CLAUDE.md 'Running agents on this Mac': every Linux and Windows cargo build and every
Linux test suite goes to igneum-build-1 through tools/build-remote.sh and tools/cross-remote.sh; the PCs keep GPU and
Windows-runtime jobs; the Mac keeps macOS binaries, the DMG and Metal tests under the lock. -Wl,--no-insert-timestamp in
cross-remote.sh, proto-cuda/windows-node/cross-build.sh and jobbuild.rs (two box builds byte-identical, verified).
The empty-commit class: tools/ci/commit-string-check.sh (self-test in ci.yml, shown firing on a Mac worktree build and
passing on a box build) runs on every igneumd from the three build scripts; push-build-inputs.sh adds node.commit_full,
the PC job writes a minimal node/.git from it at extract and cleans kaspa-build-info on a new commit (4 jobbuild tests
pass, run on the box); cross-build.sh refuses a worktree and cleans on a new commit. Plan: second worktree's clean build
1 min 18 s with sccache 604 hits of 993.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 17:50:39 +00:00
igneum-labs
0b3ca31d87 Build server: remote-run.sh with the JSONL build log, benchmark plan docs/plans/build-server.md, commit hash in box builds
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in
/srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot,
wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h
slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and
build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash
(no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that
imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental
7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 17:41:26 +00:00
igneum-labs
7b61483f4b Build server: tools/build-remote.sh, tools/cross-remote.sh, infra/build-server/{lib,run-from-mac}.sh
build-remote.sh runs a cargo command on igneum-build-1 from any crate directory of any worktree: HEAD through the bare
mirror (a real .git for kaspa-build-info), uncommitted changes by rsync --checksum with the written files re-stamped, a
remote slot (/srv/builds/_locks, never the Mac's), sccache, -j 90, artefacts back into target-remote/ with size and sha256.
cross-remote.sh is the Windows cross-build with the PC job's Ubuntu mingw-posix recipe plus the Mac's static flags, DLL
list and sha256 per exe, --compare against the Mac's exes. run-from-mac.sh ships provision.sh, writes
~/.config/igneum/build-server, adds the build remotes and pushes every branch of both repos. shellcheck and the CI checks clean.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 17:26:51 +00:00