Box tooling from master b4214735 (build-remote routes by class and load across build-1 and build-2, --box N pins, infra/build-server with it): the pool lane's branch sits on release-0.3.20 for the ladder's igneum-pow, and a full merge of master conflicts in eight files that are the shipper's merge (bench-log, fud-ledger, testnet-go, restart-hand-nodes, ledger.html, litepaper.html, miner.html, pre-push.sh)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs
parent
cb2c3ee0c7
commit
aa2070ef22
12 changed files with 563 additions and 70 deletions
37
infra/build-server/README.md
Normal file
37
infra/build-server/README.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# The build boxes (infra/build-server)
|
||||
|
||||
Three Hetzner dedicated servers in Falkenstein run everything the Mac must not: builds, test suites, benchmarks, CPU proving, the
|
||||
devnet hands and the observer. The Mac keeps macOS binaries, the DMG and Metal tests (CLAUDE.md, "Running agents on this Mac").
|
||||
|
||||
## No mining on any Hetzner box, ever
|
||||
|
||||
the project lead's rule through main, 7 October 2026: Hetzner's policies forbid crypto mining. The boxes run nodes, builds, tests, benchmarks and
|
||||
CPU proving only. The pool's fast-time network runs its miners on rented GPU pods (tools/fleet), never on a box; a box may run the
|
||||
network's nodes. The capacity layer refuses a job that would start `igneum-miner mine` or a GPU worker (capacity/run.sh), and no
|
||||
hands unit carries a miner. A node started with `--enable-unsynced-mining` is a node flag, not a miner; nothing feeds it blocks here.
|
||||
|
||||
## The boxes and the kind map
|
||||
|
||||
| Box | Host file on the Mac | Takes | Never |
|
||||
|---|---|---|---|
|
||||
| igneum-build-1 (188.40.146.49, AX162-1-LTD) | `~/.config/igneum/build-server` | release gates (`--priority gate`), builds and cross-builds, checks, the GPU workers' host side, the devnet hands (node 1, the observer node, the observer), the Devnet 2 seed, the CI runner, the dashboard feed | suites and benches once box 2 exists |
|
||||
| igneum-build-2 (AX162-1, on order) | `~/.config/igneum/build-server-2` | suites (`cargo test`), benches (`cargo bench`), the attack rows (`--box 2`) | gates, hands |
|
||||
| igneum-build-3 (AX102-1, on order) | `~/.config/igneum/build-server-3` | proving and aggregation CPU work (proving/igneum-prove builds and suites), the second prover's shadow runner, the pool's fast-time NETWORK (nodes only, `--box 3`) | miners of any kind |
|
||||
|
||||
`tools/build-remote.sh` routes by class (lib.sh `bs_route`): suite and bench to box 2, the proving crate to box 3, everything else
|
||||
to box 1; `--box N` overrides; a class whose box has no host file yet falls back to box 1 and says so. `--priority gate` always runs
|
||||
on box 1. Each box has its own mirrors, slots, locks and JSONL log under /srv; `run-from-mac.sh --box N <ip>` provisions a box and
|
||||
writes its host file; the dashboard collector reads every box it is told about.
|
||||
|
||||
## Files
|
||||
|
||||
| File | What |
|
||||
|---|---|
|
||||
| `provision.sh` | the box itself: install mode (rescue system, Ubuntu 24.04, RAID 1, no swap) and provision mode (user build, toolchains, the pin, sccache, zig, CUDA headers, docker, Caddy, mirrors, slots, sshd, ufw) |
|
||||
| `run-from-mac.sh` | ships provision.sh, writes the host file, wires the `build` remotes and pushes every branch |
|
||||
| `lib.sh`, `remote-run.sh` | the Mac and box halves of a remote run: sync, checkout, slots, scheduling classes, the JSONL line |
|
||||
| `hands/` | the devnet hands' units, the mover and the restart read-backs |
|
||||
| `capacity/` | the capacity layer (the box-work lane's): background jobs under the build slots, never a miner |
|
||||
| `repro/`, `night/`, `prover/`, `runner/`, `workers/` | other lanes' pieces that live on the boxes |
|
||||
|
||||
Plan, numbers and the gotchas: docs/plans/build-server.md; the hands: docs/plans/hands-on-build-1.md.
|
||||
|
|
@ -58,6 +58,11 @@ run_slice() {
|
|||
|
||||
once() {
|
||||
for job in $SEQUENCE; do
|
||||
# No mining on any Hetzner box, ever (the project lead through main, 7 October 2026; Hetzner's policies forbid it): a job that would start a
|
||||
# miner or a GPU worker is refused here, whatever SEQUENCE says. Nodes, builds, tests, benchmarks and CPU proving only.
|
||||
if grep -qE 'igneum-miner[[:space:]]+mine|igneum-worker-(cuda|opencl)|igneum-app.*--mine|cargo run.*-p[[:space:]]+igneum-miner' "$JOBS_DIR/$job.sh" 2>/dev/null; then
|
||||
echo "capacity: REFUSED job $job: it would start a miner or a GPU worker; no mining on a Hetzner box (infra/build-server/README.md)" >&2; continue
|
||||
fi
|
||||
[ -f "$JOBS_DIR/$job.sh" ] || { cap_say "no job $job"; continue; }
|
||||
# wait out a running build before starting a slice (do not even launch during a build)
|
||||
while cap_build_active; do
|
||||
|
|
|
|||
|
|
@ -68,16 +68,40 @@ IFS=',' read -r -a peers <<< "${PEERS:-}"; for p in "${peers[@]}"; do [ -n "$p"
|
|||
[ -n "${EXTRA_ARGS_OBSERVER:-}" ] && args+=($EXTRA_ARGS_OBSERVER)
|
||||
exec "$IGNEUMD" "${args[@]}"
|
||||
RUN
|
||||
# the read-only deploy key (main, 7 October 2026): made HERE as user build, the private half never leaves this box and is never
|
||||
# printed; the project lead adds the public half as a read-only deploy key on github.com/igneum-network/igneum (steps in
|
||||
# docs/plans/build-server.md, "Deploy key"). Until GitHub accepts it, observer-sync.sh follows the mirror.
|
||||
install -d -m 700 -o $U -g $U $O/.ssh
|
||||
if [ ! -f $O/.ssh/deploy_igneum ]; then su - $U -c "ssh-keygen -q -t ed25519 -N '' -C 'igneum-build-1 observer read-only' -f $O/.ssh/deploy_igneum"; log "deploy key created at $O/.ssh/deploy_igneum (public half: $O/.ssh/deploy_igneum.pub)"; else log "deploy key ok"; fi
|
||||
chmod 600 $O/.ssh/deploy_igneum; chmod 644 $O/.ssh/deploy_igneum.pub
|
||||
install -d -m 700 -o $U -g $U /home/$U/.ssh
|
||||
if ! grep -q '^Host github-igneum-observer' /home/$U/.ssh/config 2>/dev/null; then
|
||||
printf 'Host github-igneum-observer\n HostName github.com\n User git\n IdentityFile %s/.ssh/deploy_igneum\n IdentitiesOnly yes\n StrictHostKeyChecking accept-new\n' "$O" >> /home/$U/.ssh/config
|
||||
chown $U:$U /home/$U/.ssh/config; chmod 600 /home/$U/.ssh/config; log "ssh alias github-igneum-observer written"
|
||||
fi
|
||||
|
||||
cat > $H/bin/observer-sync.sh <<'RUN'
|
||||
#!/usr/bin/env bash
|
||||
# the box's tools/observer/autosync.sh: fast-forward /srv/observer/igneum from the mirror /srv/igneum.git (fed by every
|
||||
# build-remote.sh and run-from-mac.sh push from the Mac), restart igneum-observer when tools/observer or site/lib changed.
|
||||
# the box's tools/observer/autosync.sh: fast-forward /srv/observer/igneum, then restart igneum-observer when tools/observer or
|
||||
# site/lib changed. Source: GitHub through the read-only deploy key (ssh alias github-igneum-observer, remote `github`) once
|
||||
# the project lead has added the public half; until then, or when GitHub refuses, the mirror /srv/igneum.git (fed by every build-remote.sh
|
||||
# and run-from-mac.sh push from the Mac). One line says which.
|
||||
set -uo pipefail
|
||||
cd /srv/observer/igneum || exit 1
|
||||
before=$(git rev-parse HEAD:tools/observer HEAD:site/lib 2>/dev/null | tr '\n' ' ')
|
||||
su - build -c "git -C /srv/observer/igneum pull -q --ff-only origin master" 2>&1 | head -2
|
||||
after=$(git rev-parse HEAD:tools/observer HEAD:site/lib 2>/dev/null | tr '\n' ' ')
|
||||
if [ "$before" != "$after" ]; then echo "observer files changed ($(git rev-parse --short HEAD)); restarting igneum-observer"; systemctl restart igneum-observer; else echo "observer up to date at $(git rev-parse --short HEAD)"; fi
|
||||
g() { su - build -c "git -C /srv/observer/igneum $*"; } # the clone is build's; root's git refuses it (safe.directory), so every git call runs as build
|
||||
before=$(g rev-parse HEAD:tools/observer HEAD:site/lib 2>/dev/null | tr '\n' ' ')
|
||||
# `ssh -T` to GitHub exits 1 after its greeting, so under this script's pipefail a `su ... | grep -q` reported failure although grep
|
||||
# had matched (7 Oct 2026: the pass said "mirror" while the same line by hand said authenticated); the output is captured first
|
||||
probe=$(su - build -c "ssh -o BatchMode=yes -o ConnectTimeout=10 -T git@github-igneum-observer" 2>&1 || true)
|
||||
if grep -q "successfully authenticated" <<<"$probe"; then
|
||||
su - build -c "git -C /srv/observer/igneum remote get-url github >/dev/null 2>&1 || git -C /srv/observer/igneum remote add github git@github-igneum-observer:igneum-network/igneum.git"
|
||||
if su - build -c "git -C /srv/observer/igneum pull -q --ff-only github master" 2>&1 | head -2; then echo "source: github (deploy key accepted)"; else echo "source: github refused the pull, mirror next"; su - build -c "git -C /srv/observer/igneum pull -q --ff-only origin master" 2>&1 | head -2; fi
|
||||
else
|
||||
echo "source: mirror (the deploy key is not accepted by GitHub yet: docs/plans/build-server.md, Deploy key)"
|
||||
su - build -c "git -C /srv/observer/igneum pull -q --ff-only origin master" 2>&1 | head -2
|
||||
fi
|
||||
after=$(g rev-parse HEAD:tools/observer HEAD:site/lib 2>/dev/null | tr '\n' ' ')
|
||||
if [ "$before" != "$after" ]; then echo "observer files changed ($(g rev-parse --short HEAD)); restarting igneum-observer"; systemctl restart igneum-observer; else echo "observer up to date at $(g rev-parse --short HEAD)"; fi
|
||||
RUN
|
||||
chmod 755 $H/bin/*.sh; chown $U:$U $H/bin/*.sh
|
||||
|
||||
|
|
|
|||
|
|
@ -12,6 +12,13 @@
|
|||
# Mac's run.sh + autosync + observer.mjs, start igneum-observer (step 3)
|
||||
# infra/build-server/hands/move-hand.sh node1 [--go] the same as observer-node for node 1 (step 4)
|
||||
# infra/build-server/hands/move-hand.sh unload [--go] bootout the Mac's two launchd agents for good (step 5, last)
|
||||
# infra/build-server/hands/move-hand.sh restart observer-node|node1 [--digest <hex>] [--go]
|
||||
# a release on the box (7 Oct 2026, 0.3.17): after `binary`
|
||||
# installed the new igneumd and the override, restart ONE
|
||||
# unit and read it back: first executing line, commit string
|
||||
# of the installed binary, digest (against --digest when
|
||||
# given, else the unit's own last digest), igneum_getNodeInfo
|
||||
# powEngine over the node's loopback EVM RPC
|
||||
# infra/build-server/hands/move-hand.sh status both sides: units, pids, tips, peers
|
||||
#
|
||||
# Needs ~/.config/igneum/build-server (build@<ip>) and the ops key; root ssh to the box for systemctl, scp of the env file and chown.
|
||||
|
|
@ -23,6 +30,9 @@ REPO="$(cd "$HERE/../../.." && pwd)"
|
|||
BS_TOOL=move-hand
|
||||
# shellcheck source=../lib.sh
|
||||
. "$HERE/../lib.sh"
|
||||
|
||||
{ # whole-body: bash parses this block entirely before running a line of it, so an edit to this file while a run is in
|
||||
# flight cannot reach the running copy (7 Oct 2026: build-remote.sh was edited mid-run and died on shifted bytes after a 4-min build)
|
||||
bs_host
|
||||
IP="${BS_HOST#*@}"
|
||||
ROOT_SSH=(ssh -i "$BS_KEY" -o BatchMode=yes -o StrictHostKeyChecking=accept-new "root@$IP")
|
||||
|
|
@ -31,11 +41,20 @@ MAC_SNAP=/tmp/igneum-devnet/node1-copy-snapshot.bin
|
|||
MAC_SNAP_SHA=ac101f13576179fd7d7f5e8ee902c9a7b6cc47730e3a3c069f389f0ca46d9221 # the launchd agents' value (6 Oct 2026); recomputed below
|
||||
H=/srv/hands
|
||||
MODE="${1:-}"; shift || true
|
||||
GO=0; NODE_WT=""; OV_JSON=""
|
||||
while [ $# -gt 0 ]; do case "$1" in --go) GO=1; shift ;; --node) NODE_WT="$2"; shift 2 ;; --override-json) OV_JSON="$2"; shift 2 ;; *) bs_die "unknown argument $1" ;; esac; done
|
||||
GO=0; NODE_WT=""; OV_JSON=""; WANT_DIGEST=""; HAND=""
|
||||
while [ $# -gt 0 ]; do case "$1" in --go) GO=1; shift ;; --node) NODE_WT="$2"; shift 2 ;; --override-json) OV_JSON="$2"; shift 2 ;; --digest) WANT_DIGEST="$2"; shift 2 ;; *) if [ "$MODE" = restart ] && [ -z "$HAND" ]; then HAND="$1"; shift; else bs_die "unknown argument $1"; fi ;; esac; done
|
||||
say() { bs_log "$*"; }
|
||||
run() { if [ "$GO" = 1 ]; then "$@"; else say "DRY RUN: $*"; fi; }
|
||||
rssh() { "${ROOT_SSH[@]}" "$@"; }
|
||||
# the digest readback (main, 6 Oct 2026 23:xx UK): the box hand's "Consensus params digest" against the Mac hand's last one
|
||||
# tolerant pipelines: a missing line gives an empty string, never a failed command substitution that ends the script under set -e
|
||||
# (7 Oct 2026: the restart dry run died silently because the unit's digest line was older than the 10-minute window)
|
||||
mac_digest() { { grep 'Consensus params digest' "$HOME/Library/Logs/Igneum/$1.out" 2>/dev/null | tail -1 | grep -oE '[0-9a-f]{64}' | head -1; } || true; }
|
||||
box_digest() { { rssh "journalctl -u $1 --no-pager -o cat --since '7 days ago' | grep 'Consensus params digest' | tail -1" 2>/dev/null | grep -oE '[0-9a-f]{64}' | head -1; } || true; }
|
||||
digest_readback() { # <unit> <mac hand log name>
|
||||
local b m; b=$(box_digest "$1"); m=$(mac_digest "$2")
|
||||
if [ -n "$b" ] && [ "$b" = "$m" ]; then say "$1 digest ${b:0:16}... MATCHES the Mac's $2 hand"; else say "$1 digest ${b:-none} against the Mac's ${m:-none}: DIFFER (stop and read the override before moving the next hand)"; return 1; fi
|
||||
}
|
||||
first_exec_line() { # <unit>: wait up to 180 s for the node's exec line, print it (the proof main wants) and the chain tip
|
||||
local u="$1" line=""
|
||||
for _ in $(seq 1 36); do
|
||||
|
|
@ -50,7 +69,7 @@ sync_dir() { # <mac dir> <box dir>: rsync a data dir (hot or final), keeping R
|
|||
}
|
||||
mac_stop_agent() { # <label>: bootout the launchd agent (KeepAlive would restart a killed process), wait for the pid to end
|
||||
local label="$1" pid
|
||||
pid=$(launchctl print "gui/$(id -u)/$label" 2>/dev/null | awk '/^\s*pid = /{ print $3 }' | head -1 || true)
|
||||
pid=$(launchctl print "gui/$(id -u)/$label" 2>/dev/null | grep -oE 'pid = [0-9]+' | head -1 | grep -oE '[0-9]+' || true) # macOS awk has no \s: the first run printed "pid none" though the bootout worked
|
||||
launchctl bootout "gui/$(id -u)/$label" 2>/dev/null || true
|
||||
if [ -n "$pid" ]; then while kill -0 "$pid" 2>/dev/null; do sleep 1; done; fi
|
||||
say "launchd $label unloaded (igneumd pid ${pid:-none} ended)"
|
||||
|
|
@ -58,7 +77,13 @@ mac_stop_agent() { # <label>: bootout the launchd agent (KeepAlive would resta
|
|||
|
||||
case "$MODE" in
|
||||
binary)
|
||||
[ -n "$NODE_WT" ] || bs_die "binary needs --node <fork worktree> (the release-0.3.15-node tree)"
|
||||
if [ -z "$NODE_WT" ]; then
|
||||
# default: the fork tree that built the hand running on the Mac now (the launchd agent's binary path), so the box runs
|
||||
# the same commit the Mac did (6 Oct 2026: igneum-wt-ship0315/vendor/igneum-node-0315 at f1ea7a38)
|
||||
macbin=$(plutil -p "$HOME/Library/LaunchAgents/network.igneum.devnet.node1.plist" 2>/dev/null | grep -oE '/[^"]*igneumd' | head -1)
|
||||
NODE_WT=$(dirname "$macbin" | sed -E 's|/target[^/]*/release$||'); [ -f "$NODE_WT/Cargo.toml" ] || bs_die "no --node and no fork tree behind the Mac's node1 agent ($macbin)"
|
||||
say "node tree from the Mac's node1 agent: $NODE_WT ($(git -C "$NODE_WT" rev-parse --short HEAD) on $(git -C "$NODE_WT" branch --show-current))"
|
||||
fi
|
||||
[ -f "$NODE_WT/Cargo.toml" ] || bs_die "no Cargo.toml in $NODE_WT"
|
||||
ver=$(grep -m1 '^version' "$NODE_WT/Cargo.toml" | sed 's/.*"\(.*\)".*/\1/'); sha=$(git -C "$NODE_WT" rev-parse --short HEAD)
|
||||
say "building igneumd $ver ($sha) on the box from $NODE_WT"
|
||||
|
|
@ -67,7 +92,7 @@ case "$MODE" in
|
|||
bin_sha=$(bs_sha256 "$out/release/igneumd"); rm -rf "$out"
|
||||
# the binary is already on the box; copy it there, never back and forth
|
||||
ctx=$(cd "$NODE_WT" && BS_TOOL=move-hand bash -c '. "$0"; bs_host >/dev/null; bs_context; echo "$BS_REMOTE_CRATE"' "$HERE/../lib.sh")
|
||||
run rssh "install -m 755 -o build -g build '$ctx/target/release/igneumd' '$H/bin/igneumd-$ver-$sha' && ln -sfn '$H/bin/igneumd-$ver-$sha' '$H/bin/igneumd' && sha256sum '$H/bin/igneumd-$ver-$sha' | cut -c1-16 && '$H/bin/igneumd' --version"
|
||||
run rssh "install -m 755 -o build -g build '$ctx/target/release/igneumd' '$H/bin/igneumd-$ver-$sha' && ln -sfn '$H/bin/igneumd-$ver-$sha' '$H/bin/igneumd' && sha256sum '$H/bin/igneumd-$ver-$sha' | cut -c1-16 && { '$H/bin/igneumd' --version || true; }" # igneumd --version prints and exits 1 (7 Oct 2026: it ended this step under set -e before the override was written)
|
||||
say "box binary sha256 $bin_sha; checking its commit string on the box"
|
||||
run rssh "[ \$(strings '$H/bin/igneumd' | grep -c '$sha') -gt 0 ] && echo 'commit $sha in the binary' || { echo 'NO commit string in the binary'; exit 1; }"
|
||||
if [ -n "$OV_JSON" ]; then
|
||||
|
|
@ -77,7 +102,9 @@ case "$MODE" in
|
|||
ovfile=$(mktemp); printf '%s\n' "$OV_JSON" > "$ovfile"
|
||||
else
|
||||
[ -f "$MAC_OV" ] || bs_die "no override file at $MAC_OV and no --override-json"
|
||||
say "override from the Mac's file: $(cut -c1-200 "$MAC_OV")"; ovfile="$MAC_OV"
|
||||
nf=$(python3 -c 'import json,sys; print(len(json.load(open(sys.argv[1]))))' "$MAC_OV")
|
||||
say "override from the Mac's file ($nf fields): $(cut -c1-200 "$MAC_OV")"; ovfile="$MAC_OV"
|
||||
[ "$nf" -ge 16 ] || say "WARNING: the Mac's override has $nf fields; publish 2 writes the sixteen-field object when it restarts the hands. Move only after the shipper's 'publish 2 live: digest <x>' line, or pass --override-json"
|
||||
fi
|
||||
run bs_rsync -p "$ovfile" "$BS_HOST:$H/override.json"
|
||||
[ "$ovfile" = "$MAC_OV" ] || rm -f "$ovfile"
|
||||
|
|
@ -99,7 +126,11 @@ case "$MODE" in
|
|||
run mac_stop_agent "$label"
|
||||
run sync_dir "$mac_dir" "$H/$MODE"
|
||||
run rssh "systemctl start $unit && sleep 3 && systemctl is-active $unit"
|
||||
[ "$GO" = 1 ] && first_exec_line "$unit"
|
||||
if [ "$GO" = 1 ]; then
|
||||
first_exec_line "$unit"
|
||||
rssh "[ \$(strings $H/bin/igneumd | grep -c \"\$(readlink $H/bin/igneumd | sed -E 's/.*-([0-9a-f]{7,})\$/\\1/')\") -gt 0 ] && echo 'commit string present in the box binary' || echo 'WARNING: no commit string in the box binary'"
|
||||
digest_readback "$unit" "$( [ "$MODE" = node1 ] && echo node1 || echo observer )" || true
|
||||
fi
|
||||
say "$MODE moved; the Mac's agent stays unloaded (step 5 removes the plist from the login)" ;;
|
||||
|
||||
observer)
|
||||
|
|
@ -118,16 +149,53 @@ case "$MODE" in
|
|||
run rssh "systemctl start igneum-observer && sleep 5 && systemctl is-active igneum-observer && journalctl -u igneum-observer --no-pager -o cat -n 6"
|
||||
say "observer moved: /api/live reads Neon, which the box's observer now writes" ;;
|
||||
|
||||
restart)
|
||||
hand="$HAND"
|
||||
case "$hand" in node1) unit=igneum-node1; evm=26791 ;; observer-node) unit=igneum-observer-node; evm=26840 ;; *) bs_die "restart needs observer-node or node1" ;; esac
|
||||
bin=$(rssh "readlink $H/bin/igneumd"); sha=$(printf '%s' "$bin" | sed -E 's/.*-([0-9a-f]{7,})$/\1/')
|
||||
before=$(box_digest "$unit"); want="${WANT_DIGEST:-$before}"
|
||||
say "$hand: restart $unit on $bin (commit $sha); digest before ${before:-none}, wanted ${want:-any}"
|
||||
run rssh "systemctl restart $unit && sleep 3 && systemctl is-active $unit"
|
||||
if [ "$GO" = 1 ]; then
|
||||
first_exec_line "$unit"
|
||||
rssh "[ \$(strings $H/bin/igneumd | grep -c '$sha') -gt 0 ] && echo 'commit string $sha present in the running binary' || { echo 'NO commit string $sha in the binary'; exit 1; }"
|
||||
after=$(box_digest "$unit")
|
||||
if [ -n "$after" ] && [ "$after" = "$want" ]; then say "$unit digest ${after:0:16}... MATCHES"; else say "$unit digest ${after:-none} against wanted ${want:-any}: DIFFER (stop here)"; exit 1; fi
|
||||
# 0.3.18 and later (the shipper, 7 Oct 2026): igneum_getNodeInfo answers with powEngine (igneum-pow; a stub is a FAIL that stops
|
||||
# the sequence) and a blockrate object; a tree before it answers -32601 and the engine is read from the binary below
|
||||
eng=$(rssh "curl -s --max-time 10 -X POST -H 'content-type: application/json' --data '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"igneum_getNodeInfo\",\"params\":[]}' http://127.0.0.1:$evm" | python3 -c '
|
||||
import json, sys
|
||||
d = json.load(sys.stdin); r = d.get("result") or {}
|
||||
if d.get("error"): print("ERROR: " + str(d["error"])); sys.exit()
|
||||
eng = r.get("powEngine") or r.get("pow_engine") or "none"
|
||||
br = r.get("blockrate") or r.get("blockRate")
|
||||
print(eng + "|" + (json.dumps(br, separators=(",", ":"))[:160] if isinstance(br, dict) else "NO blockrate object"))' 2>/dev/null || echo "igneum_getNodeInfo not answered on $evm")
|
||||
case "$eng" in
|
||||
igneum-pow\|*)
|
||||
say "$unit igneum_getNodeInfo powEngine: igneum-pow; blockrate: ${eng#igneum-pow|}"
|
||||
case "$eng" in *"NO blockrate"*) say "WARNING: $unit igneum_getNodeInfo carries no blockrate object" ;; esac ;;
|
||||
stub\|*|none\|*) say "$unit igneum_getNodeInfo powEngine: ${eng%%|*}: FAIL (stop here)"; exit 1 ;;
|
||||
*-32601*|*"not found"*|*"not answered"*)
|
||||
# a tree before 0.3.18 has no igneum_getNodeInfo (the shipper, 7 Oct 2026): the engine is read from the binary instead; a stub
|
||||
# build carries none of the igneum-pow crate's source paths
|
||||
n=$(rssh "strings $H/bin/igneumd | grep -c 'igneum-pow/src/'" || echo 0)
|
||||
if [ "${n:-0}" -gt 0 ]; then say "$unit engine from the binary: igneum-pow ($n igneum-pow/src/ paths; igneum_getNodeInfo is not on this tree)"; else say "WARNING: $unit binary carries no igneum-pow/src/ path: a stub build?"; fi ;;
|
||||
*) say "WARNING: $unit igneum_getNodeInfo powEngine: $eng" ;;
|
||||
esac
|
||||
fi ;;
|
||||
|
||||
unload)
|
||||
say "removing the Mac's launchd agents for good (bootout and the plists moved aside); the hands are on the box"
|
||||
for label in network.igneum.devnet.observer network.igneum.devnet.node1; do
|
||||
run launchctl bootout "gui/$(id -u)/$label" 2>/dev/null || true
|
||||
run mv -f "$HOME/Library/LaunchAgents/$label.plist" "$HOME/Library/LaunchAgents/$label.plist.moved-to-build-1-$(date -u +%Y%m%d)"
|
||||
done
|
||||
say "Mac igneumd processes now: $(pgrep -fl 'igneumd --' | grep -v Wallet | wc -l | tr -d ' ') (the wallet's own node is not a hand)" ;;
|
||||
say "Mac igneumd processes now: $(pgrep -fl '[i]gneumd --' | grep -v Wallet | wc -l | tr -d ' ') (the wallet's own node is not a hand)" ;;
|
||||
|
||||
status)
|
||||
echo "--- box:"; rssh "for u in igneum-node1 igneum-observer-node igneum-observer igneum-observer-sync.timer; do printf '%-26s %s\n' \$u \$(systemctl is-active \$u); done; journalctl -u igneum-node1 -o cat -n 2 --no-pager 2>/dev/null | cut -c1-160; journalctl -u igneum-observer -o cat -n 2 --no-pager 2>/dev/null | cut -c1-160"
|
||||
echo "--- mac:"; launchctl print "gui/$(id -u)/network.igneum.devnet.node1" 2>/dev/null | grep -E 'state|pid' | head -2; launchctl print "gui/$(id -u)/network.igneum.devnet.observer" 2>/dev/null | grep -E 'state|pid' | head -2; pgrep -fl 'observer.mjs|observer/run.sh|autosync.sh' || echo "no observer processes on the Mac" ;;
|
||||
echo "--- mac:"; launchctl print "gui/$(id -u)/network.igneum.devnet.node1" 2>/dev/null | grep -E 'state|pid' | head -2; launchctl print "gui/$(id -u)/network.igneum.devnet.observer" 2>/dev/null | grep -E 'state|pid' | head -2; pgrep -fl '[o]bserver.mjs|[o]bserver/run.sh|[a]utosync.sh' || echo "no observer processes on the Mac" ;;
|
||||
*) sed -n '2,20p' "$0"; exit 2 ;;
|
||||
esac
|
||||
exit 0
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,61 @@
|
|||
|
||||
# shellcheck disable=SC2034 # shared with the scripts that source lib.sh
|
||||
BS_KEY="${IGNEUM_BUILD_KEY:-$HOME/.ssh/igneum_ed25519}"
|
||||
BS_HOST_FILE="${IGNEUM_BUILD_HOST_FILE:-$HOME/.config/igneum/build-server}" # one line: build@<ip>
|
||||
BS_HOST_FILE="${IGNEUM_BUILD_HOST_FILE:-$HOME/.config/igneum/build-server}" # one line: build@<ip> (box 1; box N is build-server-N)
|
||||
# Several boxes (main, 7 October 2026: igneum-build-2 and igneum-build-3 on order). One host file per box: ~/.config/igneum/build-server
|
||||
# is box 1, build-server-2 is box 2, build-server-3 is box 3 (run-from-mac.sh --box N writes it). The route by class, unless the
|
||||
# caller passes --box: gates, builds, checks, cross-builds, the workers, the hands and the observer stay on box 1; suites, benches and
|
||||
# the attack rows go to box 2; proving and aggregation CPU work, the second prover's shadow runner and the pool's fast-time NETWORK go
|
||||
# to box 3. A class whose box has no host file yet falls back to box 1, and the log line says so. No box mines, ever (README.md).
|
||||
# Since 7 October 2026 15:xx UK the class is a preference with spill-over (bs_route_spill below): a full or overloaded box hands
|
||||
# the job to the other one.
|
||||
bs_box_file() { case "${1:-1}" in 1) echo "$BS_HOST_FILE" ;; *) echo "${BS_HOST_FILE}-$1" ;; esac; }
|
||||
bs_route() { # <class: gate|build|check|suite|bench|prove|attack|other> -> the PREFERRED box number, falling back to 1
|
||||
local want=1
|
||||
case "$1" in suite|bench|attack) want=2 ;; prove|shadow|fasttime) want=3 ;; esac
|
||||
if [ "$want" != 1 ] && [ ! -s "$(bs_box_file "$want")" ]; then bs_log "class $1 prefers box $want, which has no host file yet ($(bs_box_file "$want")): box 1"; want=1; fi
|
||||
echo "$want"
|
||||
}
|
||||
# Spill-over (the project lead, 7 October 2026, 15:02 UK: build-1 at load 139 with a queue of 1 h 40 min while build-2 read 4.5 with both
|
||||
# slots free). The class is a PREFERENCE, not a pin: a job goes to its class's box unless that box has no free slot or its 1-minute
|
||||
# load is above BS_SPILL_LOAD (64), in which case it goes to the other box when THAT one has a free slot under the same load
|
||||
# line; when neither qualifies it queues on its own box. The alternate of box 1 is box 2, of box 2 box 1, of box 3 box 1; a box
|
||||
# without a host file is never chosen. The decision is one line on the Mac (bs_log) and travels to the box in BR_ROUTE_* for the
|
||||
# JSONL row ("route": preferred, box, spilled, reason), so the dashboard shows it per job. A box is read with one ssh
|
||||
# (bs_box_state: free slots of the slot count, load1); BS_ROUTE_STATE_<n> in the environment replaces the ssh for the self-test
|
||||
# (tools/ci/route-spill-check.sh), "down" standing for an unreachable box.
|
||||
BS_SPILL_LOAD="${BS_SPILL_LOAD:-64}"
|
||||
bs_box_state() { # <box> -> "free=<n> slots=<n> load1=<x>" | "absent" | "down"
|
||||
local b="$1" v f h
|
||||
v=$(eval "printf '%s' \"\${BS_ROUTE_STATE_$b:-}\""); if [ -n "$v" ]; then printf '%s' "$v"; return 0; fi
|
||||
f=$(bs_box_file "$b"); [ -s "$f" ] || { printf 'absent'; return 0; }
|
||||
h="$(head -1 "$f" | tr -d '[:space:]')"
|
||||
ssh "${BS_SSH_OPTS[@]}" -o ConnectTimeout=8 "$h" 'd=/srv/builds/_locks; n=$(cat $d/slots 2>/dev/null || echo 1); free=0; k=0; while [ $k -lt $n ]; do exec 9>>$d/build-$k; if flock -n 9; then free=$((free+1)); fi; exec 9>&-; k=$((k+1)); done; printf "free=%s slots=%s load1=%s" $free $n "$(cut -d" " -f1 /proc/loadavg)"' 2>/dev/null || printf 'down'
|
||||
}
|
||||
bs_state_ok() { # <state> -> 0 when the box can take a job now (a free slot, load1 at or under the line)
|
||||
local st="$1" free load
|
||||
case "$st" in free=*) ;; *) return 1 ;; esac
|
||||
free=${st#free=}; free=${free%% *}; load=${st##*load1=}
|
||||
[ "$free" -gt 0 ] 2>/dev/null || return 1
|
||||
awk -v l="$load" -v m="$BS_SPILL_LOAD" 'BEGIN { exit !(l + 0 <= m + 0) }'
|
||||
}
|
||||
bs_route_spill() { # <class> [priority] -> the box number; sets BS_ROUTE_PREF, BS_ROUTE_BOX, BS_ROUTE_SPILLED, BS_ROUTE_REASON
|
||||
local class="$1" pref alt ps as
|
||||
pref=$(bs_route "$class" 2>/dev/null)
|
||||
case "$pref" in 1) alt=2 ;; 2) alt=1 ;; *) alt=1 ;; esac
|
||||
BS_ROUTE_PREF=$pref; BS_ROUTE_BOX=$pref; BS_ROUTE_SPILLED=0
|
||||
ps=$(bs_box_state "$pref")
|
||||
if bs_state_ok "$ps"; then BS_ROUTE_REASON="box $pref ($ps) takes it"
|
||||
else
|
||||
as=$(bs_box_state "$alt")
|
||||
if bs_state_ok "$as"; then BS_ROUTE_BOX=$alt; BS_ROUTE_SPILLED=1; BS_ROUTE_REASON="box $pref ($ps) is full or over load $BS_SPILL_LOAD: spilled to box $alt ($as)"
|
||||
else BS_ROUTE_REASON="box $pref ($ps) and box $alt ($as) are both full or over load $BS_SPILL_LOAD: queued on box $pref"; fi
|
||||
fi
|
||||
bs_log "route: class $class prefers box $pref; $BS_ROUTE_REASON"
|
||||
BR_ROUTE_PREF=$BS_ROUTE_PREF BR_ROUTE_BOX=$BS_ROUTE_BOX BR_ROUTE_SPILLED=$BS_ROUTE_SPILLED BR_ROUTE_REASON=$BS_ROUTE_REASON
|
||||
export BR_ROUTE_PREF BR_ROUTE_BOX BR_ROUTE_SPILLED BR_ROUTE_REASON
|
||||
echo "$BS_ROUTE_BOX"
|
||||
}
|
||||
BS_ROOT_REMOTE=/srv/builds
|
||||
BS_MIRROR_REPO=/srv/igneum.git
|
||||
BS_MIRROR_NODE=/srv/igneum-node.git
|
||||
|
|
@ -24,11 +78,13 @@ BS_MIRROR_NODE=/srv/igneum-node.git
|
|||
bs_log() { printf '%s %s: %s\n' "$(date -u +%H:%M:%S)" "${BS_TOOL:-build-server}" "$*" >&2; }
|
||||
bs_die() { bs_log "ERROR: $*"; exit 1; }
|
||||
|
||||
bs_host() {
|
||||
bs_host() { # [box number, default BS_BOX or 1]
|
||||
BS_BOX="${1:-${BS_BOX:-1}}"
|
||||
BS_HOST="${BUILD_HOST:-}"
|
||||
if [ -z "$BS_HOST" ]; then
|
||||
[ -s "$BS_HOST_FILE" ] || bs_die "no build server: write build@<ip> to $BS_HOST_FILE (infra/build-server/run-from-mac.sh does) or set BUILD_HOST"
|
||||
BS_HOST="$(head -1 "$BS_HOST_FILE" | tr -d '[:space:]')"
|
||||
local f; f=$(bs_box_file "$BS_BOX")
|
||||
[ -s "$f" ] || bs_die "no build server for box $BS_BOX: write build@<ip> to $f (infra/build-server/run-from-mac.sh --box $BS_BOX <ip> does) or set BUILD_HOST"
|
||||
BS_HOST="$(head -1 "$f" | tr -d '[:space:]')"
|
||||
fi
|
||||
case "$BS_HOST" in *@*) ;; *) bs_die "BUILD_HOST must be user@host, got '$BS_HOST'" ;; esac
|
||||
[ -r "$BS_KEY" ] || bs_die "no ssh key at $BS_KEY"
|
||||
|
|
@ -43,15 +99,22 @@ bs_rsync() { rsync -e "$BS_SSH_CMD" "$@"; }
|
|||
|
||||
# the two sides' rustc must agree (the box is pinned by provision.sh RUST_TOOLCHAIN; the Mac runs rustup's stable):
|
||||
# a different compiler gives different bytes and, across a minor version, different lints and errors
|
||||
# the pin (main, 7 October 2026): rust-toolchain.toml at the worktree root (and the fork's own copy) names the channel; the Mac's
|
||||
# rustc AS RESOLVED IN THE CRATE DIR (rustup reads the file), the box's rustc and the pin must agree, else the build is refused
|
||||
# a tree without the file (an older release branch) gives an empty pin, never a failed pipeline: under pipefail the sed status would
|
||||
# become the assignment's status and set -e would end the caller silently (7 Oct 2026, 03:36 UTC: the b3c228fa builds under the
|
||||
# 0.3.16 app tree 5d118f58 died right after the pairing line)
|
||||
bs_pin() { local f="${1:-$BS_WT_ROOT}/rust-toolchain.toml"; [ -f "$f" ] || { echo ""; return 0; }; { sed -n 's/^channel *= *"\([^"]*\)".*/\1/p' "$f" 2>/dev/null || true; } | head -1; }
|
||||
bs_toolchain_check() {
|
||||
local mac box
|
||||
mac=$("${CARGO_HOME:-$HOME/.cargo}/bin/rustc" --version 2>/dev/null | awk '{ print $2 }')
|
||||
local mac box pin
|
||||
pin=$(bs_pin "$BS_WT_ROOT"); [ -n "$pin" ] || pin=$(bs_pin "$BS_TOP")
|
||||
mac=$(cd "${BS_CRATE:-.}" && "${CARGO_HOME:-$HOME/.cargo}/bin/rustc" --version 2>/dev/null | awk '{ print $2 }')
|
||||
box=$(bs_ssh '. /etc/profile.d/igneum-build.sh; rustc --version' 2>/dev/null | awk '{ print $2 }')
|
||||
[ -n "$box" ] || bs_die "cannot read rustc on $BS_HOST (is it provisioned? infra/build-server/run-from-mac.sh)"
|
||||
if [ "$mac" != "$box" ]; then
|
||||
if [ "${IGNEUM_TOOLCHAIN_MISMATCH:-}" = ok ]; then bs_log "WARNING: rustc $mac on the Mac, $box on the box (IGNEUM_TOOLCHAIN_MISMATCH=ok)"
|
||||
else bs_die "rustc $mac on the Mac, $box on the box; re-provision with RUST_TOOLCHAIN=$mac or set IGNEUM_TOOLCHAIN_MISMATCH=ok"; fi
|
||||
else bs_log "rustc $box on both sides"; fi
|
||||
if [ -n "$pin" ] && { [ "$mac" != "$pin" ] || [ "$box" != "$pin" ]; } || [ "$mac" != "$box" ]; then
|
||||
if [ "${IGNEUM_TOOLCHAIN_MISMATCH:-}" = ok ]; then bs_log "WARNING: pin ${pin:-none}, rustc $mac on the Mac, $box on the box (IGNEUM_TOOLCHAIN_MISMATCH=ok)"
|
||||
else bs_die "toolchain mismatch: rust-toolchain.toml pins ${pin:-nothing}, the Mac resolves rustc $mac in $BS_CRATE, the box runs $box; align the pin (one commit), 'rustup toolchain install $pin' on the Mac, 'RUST_TOOLCHAIN=$pin infra/build-server/run-from-mac.sh <ip>' for the box, or set IGNEUM_TOOLCHAIN_MISMATCH=ok"; fi
|
||||
else bs_log "rustc $box on both sides${pin:+ (pinned $pin by rust-toolchain.toml)}"; fi
|
||||
}
|
||||
|
||||
# Where am I? Sets BS_KIND (node = a worktree of the fork under vendor/; repo = a crate of the igneum repo), BS_TOP (the git
|
||||
|
|
@ -59,6 +122,7 @@ bs_toolchain_check() {
|
|||
# BS_CRATE (the crate dir, = $PWD), BS_CRATE_REL (relative to BS_WT_ROOT), BS_MIRROR, BS_BRANCH, BS_SHA, BS_REMOTE_WT,
|
||||
# BS_REMOTE_CRATE, and BS_LOCAL_DIRS (every directory of a path dependency, relative to BS_WT_ROOT, from cargo metadata).
|
||||
bs_context() {
|
||||
local d
|
||||
BS_CRATE="$PWD"
|
||||
[ -f "$BS_CRATE/Cargo.toml" ] || bs_die "no Cargo.toml in $BS_CRATE: run from the crate directory (the fork worktree, igneum-pow, app/igneum-app, proving/igneum-prove)"
|
||||
BS_TOP=$(git -C "$BS_CRATE" rev-parse --show-toplevel 2>/dev/null) || bs_die "$BS_CRATE is not inside a git worktree"
|
||||
|
|
@ -115,6 +179,39 @@ print("\n".join(rel(m) for m in sorted(dirs)))
|
|||
print("VENDOR_REPOS " + " ".join(rel(t) for t in sorted(repos)))' "$BS_WT_ROOT" "$BS_TOP" "$BS_KIND") || bs_die "cargo metadata failed in $BS_CRATE"
|
||||
BS_VENDOR_REPOS=$(printf '%s\n' "$BS_LOCAL_DIRS" | sed -n 's/^VENDOR_REPOS //p')
|
||||
BS_LOCAL_DIRS=$(printf '%s\n' "$BS_LOCAL_DIRS" | grep -v '^VENDOR_REPOS ' || true)
|
||||
# Files a crate reaches by include_bytes!/include_str! with a relative path that LEAVES its repository (the shipper, 7 Oct 2026:
|
||||
# the fork's igneum-exec embeds proving/igneum-prove/elf/igneum-prove-program.vk five levels up, and the box build failed with
|
||||
# "couldn't read ...: No such file or directory" because such a file is no path dependency and the overlay never carried it).
|
||||
# Every .rs under the trees that travel is scanned; a path resolving outside BS_TOP but inside the worktree root adds its
|
||||
# directory to the overlay. proving/igneum-prove/elf is added for a fork build whatever the scan finds (the fixed fallback).
|
||||
local extra
|
||||
extra=$(python3 - "$BS_WT_ROOT" "$BS_TOP" $BS_LOCAL_DIRS $BS_VENDOR_REPOS <<'PY2'
|
||||
import os, re, sys
|
||||
root, top, rels = sys.argv[1], sys.argv[2], sys.argv[3:]
|
||||
rx = re.compile(r'include_(?:bytes|str)!\(\s*"((?:\.\./)+[^"]+)"')
|
||||
out = set()
|
||||
for rel in rels:
|
||||
base = os.path.join(root, rel)
|
||||
for dp, dn, fn in os.walk(base):
|
||||
dn[:] = [d for d in dn if d not in ("target", ".git") and not d.startswith("target-")]
|
||||
for f in fn:
|
||||
if not f.endswith(".rs"): continue
|
||||
p = os.path.join(dp, f)
|
||||
try: text = open(p, encoding="utf-8", errors="ignore").read()
|
||||
except OSError: continue
|
||||
for m in rx.finditer(text):
|
||||
a = os.path.normpath(os.path.join(dp, m.group(1)))
|
||||
if (a == top or a.startswith(top + "/")): continue
|
||||
if not a.startswith(root + "/"): continue
|
||||
out.add(os.path.relpath(os.path.dirname(a), root))
|
||||
print("\n".join(sorted(out)))
|
||||
PY2
|
||||
) || extra=""
|
||||
[ "$BS_KIND" = node ] && [ -d "$BS_WT_ROOT/proving/igneum-prove/elf" ] && extra=$(printf '%s\n%s\n' "$extra" "proving/igneum-prove/elf" | grep . | sort -u)
|
||||
for d in $extra; do
|
||||
case " $BS_LOCAL_DIRS " in *" $d "*) ;; *) BS_LOCAL_DIRS="$BS_LOCAL_DIRS
|
||||
$d"; bs_log "included by include_bytes!/include_str! outside the crate's repository: $d" ;; esac
|
||||
done
|
||||
[ -n "$BS_LOCAL_DIRS$BS_VENDOR_REPOS" ] || bs_die "cargo metadata listed no local packages in $BS_CRATE"
|
||||
}
|
||||
|
||||
|
|
@ -239,9 +336,11 @@ bs_remote_run() {
|
|||
label="$label; agent=$agent"
|
||||
BR_DIR="$dir" BR_LABEL="$label" BR_CMD="$cmd" BR_TOOL="${BS_TOOL:-build-remote}" BR_WT="$BS_WT" BR_CRATE="$BS_CRATE_REL" \
|
||||
BR_BRANCH="$BS_BRANCH" BR_SHA="$BS_SHA" BR_AGENT="$agent" BR_KIND="${BR_KIND:-other}" BR_COMMAND="${BR_COMMAND:-}" \
|
||||
BR_TARGET="${BR_TARGET:-}" BR_ARTEFACTS="${BR_ARTEFACTS:-}" BR_SDE="${BR_SDE:-}" \
|
||||
BR_TARGET="${BR_TARGET:-}" BR_ARTEFACTS="${BR_ARTEFACTS:-}" BR_SDE="${BR_SDE:-}" BR_PAIRS_WITH="${BR_PAIRS_WITH:-}" \
|
||||
BR_NICE="${BR_NICE:-0}" BR_CORES="${BR_CORES:-0}" BR_JOBS_CAP="${BR_JOBS_CAP:-0}" BR_PRIORITY="${BR_PRIORITY:-normal}" \
|
||||
BR_ROUTE_PREF="${BR_ROUTE_PREF:-}" BR_ROUTE_BOX="${BR_ROUTE_BOX:-}" BR_ROUTE_SPILLED="${BR_ROUTE_SPILLED:-0}" BR_ROUTE_REASON="${BR_ROUTE_REASON:-}" \
|
||||
bash -c '
|
||||
for v in BR_DIR BR_LABEL BR_CMD BR_TOOL BR_WT BR_CRATE BR_BRANCH BR_SHA BR_AGENT BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS BR_SDE; do
|
||||
for v in BR_DIR BR_LABEL BR_CMD BR_TOOL BR_WT BR_CRATE BR_BRANCH BR_SHA BR_AGENT BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS BR_SDE BR_PAIRS_WITH BR_NICE BR_CORES BR_JOBS_CAP BR_PRIORITY; do
|
||||
printf "export %s=%q\n" "$v" "${!v}"
|
||||
done
|
||||
cat "$0"' "$(dirname "${BASH_SOURCE[0]}")/remote-run.sh" | bs_ssh 'bash -s'
|
||||
|
|
|
|||
|
|
@ -18,9 +18,9 @@
|
|||
# directory per agent worktree, sshd key-only, ufw with 22 and the seed p2p ports.
|
||||
#
|
||||
# Settings (environment, all optional):
|
||||
# RUST_TOOLCHAIN 1.99.0 the Mac's `rustc --version` on 6 October 2026. Neither the repo nor the fork carries a
|
||||
# rust-toolchain file (checked 6 October 2026), so the pin lives here; the fork's Cargo.toml says
|
||||
# rust-version 1.91.0. tools/build-remote.sh compares the two sides and refuses a mismatch.
|
||||
# RUST_TOOLCHAIN 1.99.0 the channel of rust-toolchain.toml at the repo root (run-from-mac.sh passes it; one file pins
|
||||
# the Mac, the box, the PCs and CI since 7 October 2026). tools/build-remote.sh refuses a build
|
||||
# when the pin, the Mac's rustc or the box's rustc differ.
|
||||
# SCCACHE_GB 100 the local disk cache at /srv/sccache
|
||||
# SCCACHE_VERSION (unset) a `cargo install sccache --version` pin; unset = the newest on crates.io
|
||||
# NODE_MAJOR 22
|
||||
|
|
@ -53,9 +53,10 @@ SCCACHE_GB="${SCCACHE_GB:-100}"
|
|||
SCCACHE_VERSION="${SCCACHE_VERSION:-}"
|
||||
NODE_MAJOR="${NODE_MAJOR:-22}"
|
||||
WORKTREES="${WORKTREES:-}"
|
||||
SLOTS="${SLOTS:-2}" # 2 since main's ruling of 6 October 2026 (20:3x UK); remote-run.sh gives 90 jobs alone, 45 beside another
|
||||
SLOTS_GIVEN="${SLOTS:-}"; SLOTS="${SLOTS:-2}" # 2 since main's ruling of 6 October 2026 (20:3x UK); remote-run.sh gives 90 jobs alone, 45 beside another
|
||||
P2P_PORTS="${P2P_PORTS:-26611 26811}"
|
||||
BOX_HOSTNAME="${BOX_HOSTNAME:-igneum-build-1}"
|
||||
case "$BOX_HOSTNAME" in *-2) [ -n "$SLOTS_GIVEN" ] || SLOTS=3 ;; esac # build-2: three slots (main, 7 Oct 2026; the bounded runs take one 32-core band each)
|
||||
WORKERS_HOST="${WORKERS_HOST:-build.igneum.network}" # the dashboard feed's HTTPS name (A record in deSEC, 6 Oct 2026)
|
||||
SSH_PUBKEY="${SSH_PUBKEY:-}"
|
||||
BUILD_USER=build
|
||||
|
|
@ -139,7 +140,7 @@ step_os_check() {
|
|||
APT_PACKAGES=(
|
||||
build-essential clang lld llvm libclang-dev pkg-config libssl-dev cmake protobuf-compiler libprotobuf-dev
|
||||
gcc-mingw-w64-x86-64 g++-mingw-w64-x86-64 binutils-mingw-w64-x86-64 mingw-w64-x86-64-dev mingw-w64-tools
|
||||
git tmux curl ca-certificates xz-utils zstd unzip rsync jq python3 ufw htop file caddy
|
||||
git tmux curl ca-certificates xz-utils zstd unzip rsync jq python3 ufw htop file caddy docker.io
|
||||
# the GitHub Actions runner's .NET runtime needs libicu (bin/installdependencies.sh would install it); the two Python
|
||||
# simulators (sim/finality_v2.py, sim/difficulty/sim.py) need numpy, which ci.yml pip-installs on GitHub's runners
|
||||
libicu74 python3-numpy
|
||||
|
|
@ -218,6 +219,14 @@ step_user() {
|
|||
|
||||
worktree_count() { find /srv/builds -mindepth 1 -maxdepth 1 -type d -not -name '_*' | wc -l | tr -d ' '; }
|
||||
|
||||
# docker (7 October 2026): the glibc proof runs a shipped binary inside ubuntu:20.04 and ubuntu:22.04 on the box (tools/build-remote.sh
|
||||
# --ship's proof); user build may run containers. Nothing else of the box runs in docker.
|
||||
step_docker() {
|
||||
command -v docker >/dev/null 2>&1 || die "docker is not installed (apt docker.io)"
|
||||
systemctl is-active --quiet docker || systemctl enable --now docker >/dev/null 2>&1
|
||||
if id -nG "$BUILD_USER" | tr ' ' '\n' | grep -qx docker; then ok docker "$(docker --version | cut -d, -f1), $BUILD_USER in the docker group"; else usermod -aG docker "$BUILD_USER"; changed docker "$(docker --version | cut -d, -f1), $BUILD_USER added to the docker group (new ssh sessions see it)"; fi
|
||||
}
|
||||
|
||||
step_dirs() {
|
||||
local d any=0
|
||||
for d in /srv/builds /srv/builds/_locks /srv/sccache /srv/artefacts; do
|
||||
|
|
@ -332,6 +341,22 @@ step_node() {
|
|||
changed node "$(/usr/local/bin/node --version) from nodejs.org (sha256 checked)"
|
||||
}
|
||||
|
||||
# zig (main, 7 October 2026): the glibc 2.36 Linux artefacts for Debian 12 seeds and HiveOS rigs come from cargo-zigbuild with zig as
|
||||
# the C/C++ toolchain, as infra/cross/build-linux.sh does on the Mac (tonight's seed took 14 restarts and three minutes down on a
|
||||
# glibc 2.39 native build). The release the Mac uses (0.17.0), from ziglang.org with the sha256 of the official download index.
|
||||
ZIG_VERSION="${ZIG_VERSION:-0.17.0}"
|
||||
ZIG_SHA256="${ZIG_SHA256:-1cbe9df9f27e6b78d14ccbca43b6703a404ef79ef1c463de901d7f088d4e2026}" # zig-x86_64-linux-0.17.0.tar.xz, index.json 7 Oct 2026
|
||||
step_zig() {
|
||||
local have dir tar
|
||||
have=$(/usr/local/bin/zig version 2>/dev/null || true)
|
||||
if [ "$have" = "$ZIG_VERSION" ]; then ok zig "$have"; return; fi
|
||||
dir=/usr/local/lib/zig; tar="zig-x86_64-linux-$ZIG_VERSION.tar.xz"
|
||||
install -d "$dir"
|
||||
( cd "$dir" && curl -fsSLO "https://ziglang.org/download/$ZIG_VERSION/$tar" && echo "$ZIG_SHA256 $tar" | sha256sum -c --quiet - && tar -xJf "$tar" && rm -f "$tar" )
|
||||
ln -sfn "$dir/zig-x86_64-linux-$ZIG_VERSION/zig" /usr/local/bin/zig
|
||||
changed zig "$(/usr/local/bin/zig version) from ziglang.org (sha256 checked) at /usr/local/bin/zig"
|
||||
}
|
||||
|
||||
step_sshd() {
|
||||
local f=/etc/ssh/sshd_config.d/10-igneum-build.conf tmp
|
||||
tmp=$(mktemp)
|
||||
|
|
@ -543,7 +568,9 @@ step_runner() {
|
|||
step_cargo_tools() {
|
||||
local cargo="$BUILD_HOME/.cargo/bin/cargo" any=0
|
||||
if [ ! -x "$BUILD_HOME/.cargo/bin/cargo-audit" ]; then as_build "$cargo install cargo-audit --locked" >/dev/null 2>&1 || as_build "$cargo install cargo-audit" >/dev/null; any=1; fi
|
||||
[ "$any" = 1 ] && changed cargo-tools "$(as_build "$BUILD_HOME/.cargo/bin/cargo-audit --version")" || ok cargo-tools "$(as_build "$BUILD_HOME/.cargo/bin/cargo-audit --version")"
|
||||
# cargo-zigbuild (main, 7 October 2026): the glibc 2.36 Linux artefacts for Debian 12 seeds and HiveOS rigs (tools/build-remote.sh --ship)
|
||||
if [ ! -x "$BUILD_HOME/.cargo/bin/cargo-zigbuild" ]; then as_build "$cargo install cargo-zigbuild --locked" >/dev/null 2>&1 || as_build "$cargo install cargo-zigbuild" >/dev/null; any=1; fi
|
||||
[ "$any" = 1 ] && changed cargo-tools "$(as_build "$BUILD_HOME/.cargo/bin/cargo-audit --version"), $(as_build "$BUILD_HOME/.cargo/bin/cargo-zigbuild --version")" || ok cargo-tools "$(as_build "$BUILD_HOME/.cargo/bin/cargo-audit --version"), $(as_build "$BUILD_HOME/.cargo/bin/cargo-zigbuild --version")"
|
||||
}
|
||||
|
||||
# the night battery (infra/build-server/night): the script and remote-run.sh into /srv/builds/_bin, the two units, the timer
|
||||
|
|
@ -579,6 +606,7 @@ step_summary() {
|
|||
printf 'raid: %s\n' "$(grep -E '^md' /proc/mdstat 2>/dev/null | tr '\n' ';' || echo none)"
|
||||
printf 'rust: %s | %s | targets %s\n' "$(as_build "$BUILD_HOME/.cargo/bin/rustc --version")" "$(as_build "$BUILD_HOME/.cargo/bin/cargo --version")" "$(as_build "$BUILD_HOME/.cargo/bin/rustup target list --installed" | tr '\n' ' ')"
|
||||
printf 'sccache: %s, %s\n' "$(as_build "$BUILD_HOME/.cargo/bin/sccache --version")" "$(cat "$BUILD_HOME/.config/sccache/config" | tr '\n' ' ')"
|
||||
printf 'zig: %s, cargo-zigbuild %s (glibc 2.36 Linux artefacts: tools/build-remote.sh --ship, tools/workers-remote.sh)\n' "$(/usr/local/bin/zig version 2>/dev/null || echo missing)" "$(as_build "$BUILD_HOME/.cargo/bin/cargo-zigbuild --version 2>/dev/null | awk '{ print \$NF }'" || echo missing)"
|
||||
printf 'mingw: %s\n' "$(x86_64-w64-mingw32-gcc-posix --version | head -1)"
|
||||
printf 'clang: %s | lld: %s\n' "$(clang --version | head -1)" "$(ld.lld --version | head -1)"
|
||||
printf 'node: %s | git: %s | tmux: %s\n' "$(/usr/local/bin/node --version)" "$(git --version)" "$(tmux -V)"
|
||||
|
|
@ -603,6 +631,7 @@ do_provision() {
|
|||
step_sysctl
|
||||
step_limits
|
||||
step_user
|
||||
step_docker
|
||||
step_dirs
|
||||
step_mirrors
|
||||
step_rustup
|
||||
|
|
@ -616,6 +645,7 @@ do_provision() {
|
|||
step_ufw
|
||||
step_runner
|
||||
step_cargo_tools
|
||||
step_zig
|
||||
step_night
|
||||
step_summary
|
||||
log "done"
|
||||
|
|
|
|||
|
|
@ -51,6 +51,26 @@ _slots_env="${IGNEUM_BUILD_SLOTS_DIR:-}"; _log_env="${IGNEUM_BUILD_LOG_DIR:-}"
|
|||
[ -f /etc/profile.d/igneum-build.sh ] && . /etc/profile.d/igneum-build.sh
|
||||
[ -n "$_slots_env" ] && IGNEUM_BUILD_SLOTS_DIR="$_slots_env"; [ -n "$_log_env" ] && IGNEUM_BUILD_LOG_DIR="$_log_env"
|
||||
|
||||
# What the clean spares beyond target dirs and stamps: a lane's scratch. The fixed prefixes, plus every glob in the mirror-local
|
||||
# file `.igneum-scratch-spare` (one per line, # comments; the file itself is spared). spare_args <dir> fills SPARE_ARGS with
|
||||
# the `-e <glob>` arguments; it is never empty (the fixed list), so bash 3.2's unbound-empty-array rule cannot bite the self-test.
|
||||
SPARE_FIXED=('attack-*' 'scratch-*' 'target-attack-*' '.build-remote.log' '.igneum-scratch-spare')
|
||||
spare_args() {
|
||||
local p
|
||||
SPARE_ARGS=()
|
||||
for p in "${SPARE_FIXED[@]}"; do SPARE_ARGS+=(-e "$p"); done
|
||||
[ -f "$1/.igneum-scratch-spare" ] || return 0
|
||||
while IFS= read -r p || [ -n "$p" ]; do
|
||||
p="${p%%#*}"; p="${p#"${p%%[![:space:]]*}"}"; p="${p%"${p##*[![:space:]]}"}"
|
||||
[ -n "$p" ] && SPARE_ARGS+=(-e "$p")
|
||||
done < "$1/.igneum-scratch-spare"
|
||||
return 0
|
||||
}
|
||||
# the untracked paths the clean would still take, up to three (empty = the tree is clean); the same excludes as the clean line
|
||||
tree_left() {
|
||||
git -C "$1" clean -nd -e target -e 'target-*' -e '.build-remote-sha-*' -e '.cross-remote-sha-*' -e sccache "${SPARE_ARGS[@]}" | head -3
|
||||
}
|
||||
|
||||
# discard the previous overlay (tracked edits and untracked files; target dirs, the sha stamps and anything ignored are kept),
|
||||
# fetch, then the branch at the commit. Runs in BR_CO_DIR, clones it from BR_CO_MIRROR when it has no .git.
|
||||
checkout_tree() {
|
||||
|
|
@ -68,7 +88,12 @@ checkout_tree() {
|
|||
if [ "$lock_age" -gt 30 ]; then rm -f .git/index.lock; echo "checkout: removed a stale .git/index.lock (${lock_age}s old) in $dir" >&2; fi
|
||||
fi
|
||||
git checkout -q -- . 2>/dev/null || true
|
||||
git clean -qfd -e target -e 'target-*' -e '.build-remote-sha-*' -e '.cross-remote-sha-*' -e sccache
|
||||
# 7 October 2026: the attack rows lost their scratch dirs (attack-f3/, attack-f1-venv/, tools/attack/*/target) to each
|
||||
# other's builds, because this clean ran on the shared mirror before every build from any agent and took every untracked
|
||||
# directory. A lane's scratch is now spared: the fixed prefixes and the mirror-local .igneum-scratch-spare (SPARE_ARGS,
|
||||
# see spare_args above). Never -x here: .git/info/exclude still applies. tools/ci/scratch-spare-check.sh guards this line.
|
||||
spare_args "$dir"
|
||||
git clean -qfd -e target -e 'target-*' -e '.build-remote-sha-*' -e '.cross-remote-sha-*' -e sccache "${SPARE_ARGS[@]}"
|
||||
git fetch -q origin '+refs/heads/*:refs/remotes/origin/*'
|
||||
git checkout -q -B "$branch" "$sha"
|
||||
git reset -q --hard "$sha"
|
||||
|
|
@ -78,11 +103,28 @@ checkout_tree() {
|
|||
# ... at any depth: a repo-kind crate (pool/, igneum-pow/, app/igneum-app/) writes its stamp in its own directory, and the
|
||||
# root-anchored pattern of the first version failed the second build of every such crate (6 October 2026, 18:51 UTC, the
|
||||
# pool build: "tree not clean after reset: ?? pool/.build-remote-sha-target"; the self-test has the subdirectory case now)
|
||||
local left; left=$(git status --porcelain --untracked-files=all | grep -vE '^\?\? (.*/)?(target|target-|sccache|\.build-remote-sha-|\.cross-remote-sha-)' | head -3 || true)
|
||||
# ... and since 7 October 2026 a lane's spared scratch, so the test asks the clean itself what it would still remove
|
||||
# (tree_left: `git clean -nd` with the same excludes; a spared directory is no longer "not clean")
|
||||
local left; left=$(tree_left "$dir" || true)
|
||||
[ -z "$left" ] || { echo "checkout: tree not clean after reset at $dir: $left" >&2; return 1; }
|
||||
set +e
|
||||
}
|
||||
|
||||
if [ "${1:-}" = --self-test-keeper ]; then
|
||||
# the keeper restores a truncated holder line within its interval, and stops at release
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
BR_PID=$$; BR_LABEL="keeper self-test"; got=0; waited=3; SLOTS_DIR="$t"; BR_KEEP_S=1
|
||||
holder_line() { printf 'pid %s since %sZ waited %s s: %s\n' "$BR_PID" "$(date -u +%H:%M:%S)" "$1" "$BR_LABEL"; }
|
||||
holder_line "$waited" > "$t/build-0"
|
||||
keeper_pid=""; keep_line() { local f="$1" w="$2"; ( while kill -0 "$BR_PID" 2>/dev/null; do [ -s "$f" ] || holder_line "$w" > "$f" 2>/dev/null; sleep "${BR_KEEP_S:-20}"; done ) & keeper_pid=$!; }
|
||||
keep_line "$t/build-0" "$waited"
|
||||
: > "$t/build-0"; sleep 2.5
|
||||
grep -q "^pid $$ since .* waited 3 s: keeper self-test$" "$t/build-0" || { echo "keeper self-test: the truncated holder line was NOT restored"; kill "$keeper_pid" 2>/dev/null; exit 1; }
|
||||
kill "$keeper_pid" 2>/dev/null; wait "$keeper_pid" 2>/dev/null; : > "$t/build-0"; sleep 2.5
|
||||
[ ! -s "$t/build-0" ] || { echo "keeper self-test: the keeper kept writing after release"; exit 1; }
|
||||
echo "keeper self-test: a truncated holder line comes back within the interval; nothing is written after release"; exit 0
|
||||
fi
|
||||
|
||||
if [ "${1:-}" = --self-test ]; then
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
git init -q --bare -b master "$t/mirror.git"
|
||||
|
|
@ -95,6 +137,11 @@ if [ "${1:-}" = --self-test ]; then
|
|||
echo edited > "$t/box/a.txt"; echo new > "$t/box/b.txt"; mkdir -p "$t/box/target/release"; echo bin > "$t/box/target/release/x"; echo "$sha1" > "$t/box/.build-remote-sha-target"
|
||||
# a crate in a subdirectory: its stamp and target dir must survive, its untracked overlay file must not
|
||||
mkdir -p "$t/box/sub/target/release"; echo bin > "$t/box/sub/target/release/y"; echo "$sha1" > "$t/box/sub/.build-remote-sha-target"; echo new > "$t/box/sub/c.txt"
|
||||
# a lane's scratch (7 October 2026): a fixed-prefix dir at the root and nested, a dir declared in .igneum-scratch-spare
|
||||
# (comments and blank lines in the file), and an undeclared dir that the clean must still take
|
||||
mkdir -p "$t/box/attack-f3" "$t/box/sub/attack-f1-venv" "$t/box/bs-lane-1" "$t/box/undeclared-1"
|
||||
echo x > "$t/box/attack-f3/x"; echo x > "$t/box/sub/attack-f1-venv/x"; echo x > "$t/box/bs-lane-1/x"; echo x > "$t/box/undeclared-1/x"
|
||||
printf '# the build-server lane\n\n bs-lane-* # trailing comment\n' > "$t/box/.igneum-scratch-spare"
|
||||
: > "$t/box/.git/index.lock" # a run killed mid-git leaves this; the checkout mode removes it once it is older than 30 s
|
||||
touch -t "$(date -d '-2 min' +%Y%m%d%H%M.%S 2>/dev/null || date -v-2M +%Y%m%d%H%M.%S)" "$t/box/.git/index.lock" # backdated two minutes (GNU date, then BSD date)
|
||||
[ $(( $(date +%s) - $(stat -c %Y "$t/box/.git/index.lock" 2>/dev/null || stat -f %m "$t/box/.git/index.lock") )) -gt 30 ] || { echo "self-test: could not backdate the fixture lock"; exit 1; }
|
||||
|
|
@ -111,12 +158,21 @@ if [ "${1:-}" = --self-test ]; then
|
|||
[ -f "$t/box/sub/.build-remote-sha-target" ] || { echo "self-test: a subdirectory crate's sha stamp was cleaned"; exit 1; }
|
||||
[ -f "$t/box/sub/target/release/y" ] || { echo "self-test: a subdirectory crate's target dir was cleaned"; exit 1; }
|
||||
[ ! -e "$t/box/sub/c.txt" ] || { echo "self-test: a subdirectory's untracked overlay file survived"; exit 1; }
|
||||
# a lane's scratch (7 October 2026): fixed prefixes at any depth and a declared glob survive, the spare file survives, an
|
||||
# undeclared dir is removed
|
||||
[ -f "$t/box/attack-f3/x" ] || { echo "self-test: a fixed-prefix scratch dir (attack-*) was cleaned"; exit 1; }
|
||||
[ -f "$t/box/sub/attack-f1-venv/x" ] || { echo "self-test: a nested fixed-prefix scratch dir was cleaned"; exit 1; }
|
||||
[ -f "$t/box/bs-lane-1/x" ] || { echo "self-test: a scratch dir declared in .igneum-scratch-spare was cleaned"; exit 1; }
|
||||
[ -f "$t/box/.igneum-scratch-spare" ] || { echo "self-test: the .igneum-scratch-spare file itself was cleaned"; exit 1; }
|
||||
[ ! -e "$t/box/undeclared-1" ] || { echo "self-test: an undeclared scratch dir survived the clean"; exit 1; }
|
||||
# the known-failed case: an untracked file the overlay left that no rule keeps must fail the check
|
||||
echo stray > "$t/box/sub/stray.txt"
|
||||
if (cd "$t/box" && left=$(git status --porcelain --untracked-files=all | grep -vE '^\?\? (.*/)?(target|target-|sccache|\.build-remote-sha-|\.cross-remote-sha-)' | head -3); [ -n "$left" ]); then :; else echo "self-test: the clean-tree check did NOT fire on a stray untracked file"; exit 1; fi
|
||||
spare_args "$t/box"; left=$(tree_left "$t/box"); [ -n "$left" ] || { echo "self-test: the clean-tree check did NOT fire on a stray untracked file"; exit 1; }
|
||||
rm -f "$t/box/sub/stray.txt"
|
||||
# ... and a spared directory alone must NOT fire it (the check asks the clean, not the status list)
|
||||
left=$(tree_left "$t/box"); [ -z "$left" ] || { echo "self-test: the clean-tree check fired on spared scratch: $left"; exit 1; }
|
||||
[ ! -f "$t/box/.git/index.lock" ] || { echo "self-test: the stale index.lock survived"; exit 1; }
|
||||
echo "self-test: checkout mode lands on the new commit with a clean tree, target dirs and sha stamps kept at any depth, stale index.lock removed, and fires on a stray file"; exit 0
|
||||
echo "self-test: checkout mode lands on the new commit with a clean tree, target dirs and sha stamps kept at any depth, declared and fixed-prefix scratch kept, an undeclared dir removed, stale index.lock removed, and fires on a stray file"; exit 0
|
||||
fi
|
||||
|
||||
if [ "${1:-}" = --self-test-slots ]; then
|
||||
|
|
@ -196,8 +252,11 @@ d = {
|
|||
"agent": e.get('BR_AGENT'), "slot": num(e['BR_SLOT']), "wait_s": num(e['BR_WAIT']), "queued_at": iso(e['BR_T0']),
|
||||
"start": iso(e['BR_START']), "end": iso(e['BR_END']), "secs": num(e['BR_SECS']), "exit": num(e['BR_EXIT']),
|
||||
"compiles": num(e['BR_COMPILES']), "jobs": num(e.get('BR_JOBS')), "measure": (e.get('BR_MEASURE') == '1') or None,
|
||||
"source_date_epoch": num(e.get('BR_SDE')),
|
||||
"source_date_epoch": num(e.get('BR_SDE')), "pairs_with": e.get('BR_PAIRS_WITH') or None,
|
||||
"nice": num(e.get('BR_NICE')) or 0, "cores": (num(e.get('BR_CORES')) or 0) or os.cpu_count(), "priority": e.get('BR_PRIORITY') or "normal",
|
||||
"class": e.get('BR_CLASS') or None, "run_log": e.get('BR_RUN_LOG') or None,
|
||||
"route": ({"preferred": num(e.get('BR_ROUTE_PREF')), "box": num(e.get('BR_ROUTE_BOX')), "spilled": e.get('BR_ROUTE_SPILLED') == '1',
|
||||
"reason": e.get('BR_ROUTE_REASON') or ""} if e.get('BR_ROUTE_BOX') else None),
|
||||
}
|
||||
sc = {k: num(e[v]) for k, v in (("hits", "BR_HITS"), ("misses", "BR_MISSES"), ("hits_total", "BR_HITS_T"), ("misses_total", "BR_MISSES_T"))}
|
||||
sc = {k: v for k, v in sc.items() if v is not None}
|
||||
|
|
@ -280,6 +339,18 @@ else
|
|||
flock -s -w 7200 "$mfd" || give_up "the measurement to end"
|
||||
rm -f "$waitfile"; trap - EXIT
|
||||
fi
|
||||
# scheduling (main, 7 Oct 2026): a gate announces itself (gate-pending-<pid>) and takes the next free slot; a suite, bench or
|
||||
# other non-gate run that has not taken a slot yet yields while any gate-pending marker younger than 15 min exists
|
||||
gatefile=""
|
||||
if [ "${BR_PRIORITY:-normal}" = gate ]; then gatefile="$SLOTS_DIR/gate-pending-$BR_PID"; holder_line 0 > "$gatefile"; trap 'rm -f "$gatefile"' EXIT
|
||||
else
|
||||
yt0=$(date +%s)
|
||||
while pending=$(find "$SLOTS_DIR" -maxdepth 1 -name 'gate-pending-*' -mmin -15 2>/dev/null | head -1) && [ -n "$pending" ]; do
|
||||
[ -f "$waitfile" ] || { echo "build-remote: a gate is queued ($(head -c 120 "$pending")), this ${BR_KIND:-run} yields" >&2; holder_line 0 > "$waitfile"; trap 'rm -f "$waitfile"' EXIT; }
|
||||
[ $(( $(date +%s) - yt0 )) -lt 7200 ] || give_up "the queued gate"
|
||||
sleep 5
|
||||
done
|
||||
fi
|
||||
got=""
|
||||
for k in $(seq 0 $((slots - 1))); do
|
||||
exec {fd}>>"$SLOTS_DIR/build-$k"
|
||||
|
|
@ -308,11 +379,24 @@ else
|
|||
exec {tfd}>&-
|
||||
done
|
||||
if [ "$held" -gt 1 ]; then BR_JOBS=$JOBS_SHARED; else BR_JOBS=$JOBS_ALONE; fi
|
||||
# the bounded classes cap the jobs (suite and bench: 32 unless the caller passed --priority gate)
|
||||
if [ "${BR_JOBS_CAP:-0}" -gt 0 ] && [ "$BR_JOBS" -gt "$BR_JOBS_CAP" ]; then BR_JOBS=$BR_JOBS_CAP; fi
|
||||
export CARGO_BUILD_JOBS="$BR_JOBS" BR_JOBS
|
||||
echo "build-remote: holding build-$got on $BR_HOST (waited $waited s; $held of $slots slots held, CARGO_BUILD_JOBS=$BR_JOBS)" >&2
|
||||
[ -n "$gatefile" ] && { rm -f "$gatefile"; trap - EXIT; }
|
||||
echo "build-remote: holding build-$got on $BR_HOST (waited $waited s; $held of $slots slots held, CARGO_BUILD_JOBS=$BR_JOBS, kind ${BR_KIND:-other}, nice ${BR_NICE:-0}, cores $( [ "${BR_CORES:-0}" = 0 ] && nproc || echo "$BR_CORES"))" >&2
|
||||
fi
|
||||
|
||||
release_slot() { if [ "$got" = measure ]; then : > "$SLOTS_DIR/measure"; else : > "$SLOTS_DIR/build-$got"; fi; }
|
||||
# The holder keeps its own line (the watcher-trust rule, 7 October 2026 10:39Z: two held slots read EMPTY while two suites ran,
|
||||
# because a run from a worktree without last night's append-mode fix still opens a busy sibling's slot file with `>` on every
|
||||
# probe). A keeper re-writes the holder line whenever it finds the file empty, every BR_KEEP_S seconds, until release_slot.
|
||||
keeper_pid=""
|
||||
keep_line() { # <file> <waited>
|
||||
local f="$1" w="$2"
|
||||
( while kill -0 "$BR_PID" 2>/dev/null; do [ -s "$f" ] || holder_line "$w" > "$f" 2>/dev/null; sleep "${BR_KEEP_S:-20}"; done ) &
|
||||
keeper_pid=$!
|
||||
}
|
||||
if [ "$got" = measure ]; then keep_line "$SLOTS_DIR/measure" "$waited"; else keep_line "$SLOTS_DIR/build-$got" "$waited"; fi
|
||||
release_slot() { [ -n "$keeper_pid" ] && { kill "$keeper_pid" 2>/dev/null; wait "$keeper_pid" 2>/dev/null; keeper_pid=""; }; if [ "$got" = measure ]; then : > "$SLOTS_DIR/measure"; else : > "$SLOTS_DIR/build-$got"; fi; }
|
||||
cd "$BR_DIR" || { BR_CLASS=no-dir jsonlog 2 "$got" "$waited" "" "$(date +%s)" "" "" "" "" "" ""; redlog 2 0 no-dir; release_slot; exit 2; }
|
||||
|
||||
# PRE-FLIGHT for a cargo command (the instant-death class, 6 October 2026): the manifest parses and every -p package exists,
|
||||
|
|
@ -362,8 +446,22 @@ t1=$(date +%s)
|
|||
# streams stay where they were for the Mac (stdout to stdout, stderr to stderr), each teed into the run log
|
||||
RUN_LOG_DIR="$LOG_DIR/runs"; mkdir -p "$RUN_LOG_DIR"
|
||||
BR_RUN_LOG="$RUN_LOG_DIR/$BR_HOST-$BR_T0-$BR_PID.log"; export BR_RUN_LOG
|
||||
( eval "$BR_CMD" ) > >(tee -a "$BR_RUN_LOG") 2> >(tee -a "$BR_RUN_LOG" >&2)
|
||||
# the class's nice and core set apply to the command's subshell and everything it starts (renice and taskset on the subshell's own
|
||||
# pid, BASHPID; cores are the LAST N of the box's set, so gates and builds keep the first ones to themselves)
|
||||
# (since the third slot on build-2, 7 Oct 2026: a bounded run takes the band its SLOT owns, counted from the top: slot 0 the last N
|
||||
# cores, slot 1 the N below, slot 2 the N below that, so three bounded runs never share a core; a band below core 0 falls back to
|
||||
# the last N)
|
||||
ncpu=$(nproc); cores_str="0-$((ncpu - 1))"
|
||||
if [ "${BR_CORES:-0}" -gt 0 ] && [ "${BR_CORES}" -lt "$ncpu" ]; then
|
||||
band=0; case "${got:-}" in ''|measure) ;; *) band=$got ;; esac
|
||||
lo=$((ncpu - BR_CORES * (band + 1))); [ "$lo" -ge 0 ] || lo=$((ncpu - BR_CORES))
|
||||
cores_str="$lo-$((lo + BR_CORES - 1))"
|
||||
fi
|
||||
( [ "${BR_NICE:-0}" -gt 0 ] && renice -n "$BR_NICE" -p $BASHPID >/dev/null 2>&1; [ "$cores_str" != "0-$((ncpu - 1))" ] && taskset -cp "$cores_str" $BASHPID >/dev/null 2>&1; eval "$BR_CMD" ) > >(tee -a "$BR_RUN_LOG") 2> >(tee -a "$BR_RUN_LOG" >&2)
|
||||
rc=$?
|
||||
# the keeper stops BEFORE the bare `wait` (which flushes the two tees): a bare wait also waits for the keeper, and the keeper waits
|
||||
# for this script, a deadlock that held build-2's first run 15 minutes after its test had passed (7 Oct 2026, 11:04 to 11:19Z)
|
||||
if [ -n "$keeper_pid" ]; then kill "$keeper_pid" 2>/dev/null; wait "$keeper_pid" 2>/dev/null; keeper_pid=""; fi
|
||||
wait
|
||||
t2=$(date +%s); secs=$(( t2 - t1 ))
|
||||
exec_after=$(stat_field "Compile requests executed"); hits_after=$(stat_field "Cache hits "); misses_after=$(stat_field "Cache misses ")
|
||||
|
|
@ -385,8 +483,8 @@ elif [[ "$BR_CMD" == cargo\ test* ]] && { [[ "$BR_CMD" == *" -- "* ]] || grep -q
|
|||
echo "build-remote: REFUSED after the run: the test filter matched no test in any binary (every 'running 0 tests'); check the name" >&2
|
||||
fi
|
||||
export BR_CLASS
|
||||
printf 'build-remote: RESULT rc=%s secs=%s compiles=%s sccache_hits=%s sccache_misses=%s sccache_hits_total=%s sccache_misses_total=%s jobs=%s load=%s class=%s\n' \
|
||||
"$rc" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-?}" "${misses_after:-?}" "${BR_JOBS:-measure}" "$(cut -d' ' -f1-3 /proc/loadavg)" "${BR_CLASS:-ok}"
|
||||
printf 'build-remote: RESULT rc=%s secs=%s compiles=%s sccache_hits=%s sccache_misses=%s sccache_hits_total=%s sccache_misses_total=%s jobs=%s nice=%s cores=%s load=%s class=%s\n' \
|
||||
"$rc" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-?}" "${misses_after:-?}" "${BR_JOBS:-measure}" "${BR_NICE:-0}" "$( [ "${BR_CORES:-0}" = 0 ] && nproc || echo "$BR_CORES")" "$(cut -d' ' -f1-3 /proc/loadavg)" "${BR_CLASS:-ok}"
|
||||
jsonlog "$rc" "$([ "$got" = measure ] && echo "" || echo "$got")" "$waited" "$t1" "$t2" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-}" "${misses_after:-}"
|
||||
[ "$rc" = 0 ] || redlog "$rc" "$secs" "$BR_CLASS"
|
||||
release_slot
|
||||
|
|
|
|||
|
|
@ -20,11 +20,20 @@ BS_TOOL=run-from-mac
|
|||
# shellcheck source=lib.sh
|
||||
. "$HERE/lib.sh"
|
||||
|
||||
BOX=1; while [ "${1:-}" = --box ]; do BOX="$2"; shift 2; done # --box N: igneum-build-N, host file build-server-N (7 Oct 2026)
|
||||
IP="${1:-}"; shift || true
|
||||
[ -n "$IP" ] || bs_die "usage: run-from-mac.sh <ip> [--wire-only]"
|
||||
[ -n "$IP" ] || bs_die "usage: run-from-mac.sh [--box N] <ip> [--wire-only]"
|
||||
WIRE_ONLY=0; [ "${1:-}" = --wire-only ] && WIRE_ONLY=1
|
||||
# box N is igneum-build-N with its own dashboard feed name build-N.igneum.network (the dashboard lane's collector reads one server
|
||||
# section per box; main adds the A record with the IP)
|
||||
[ "$BOX" = 1 ] || { BOX_HOSTNAME="${BOX_HOSTNAME:-igneum-build-$BOX}"; WORKERS_HOST="${WORKERS_HOST:-build-$BOX.igneum.network}"; export BOX_HOSTNAME WORKERS_HOST; }
|
||||
export BS_BOX="$BOX" # lib.sh's bs_host derives the host file from BS_BOX (box 1: build-server; box N: build-server-N); never set BS_HOST_FILE here (7 Oct 2026: a second suffix, build-server-2-2)
|
||||
HOST_FILE=$(bs_box_file "$BOX")
|
||||
REMOTE="build"; [ "$BOX" = 1 ] || REMOTE="build-$BOX" # one git remote per box in the two repositories
|
||||
# the toolchain pin travels from rust-toolchain.toml unless RUST_TOOLCHAIN is set by hand (one file pins every side, 7 Oct 2026)
|
||||
[ -n "${RUST_TOOLCHAIN:-}" ] || RUST_TOOLCHAIN=$(sed -n 's/^channel *= *"\([^"]*\)".*/\1/p' "$REPO/rust-toolchain.toml" 2>/dev/null | head -1); export RUST_TOOLCHAIN
|
||||
PASS="" # a string, not an array: bash 3.2 (the Mac) treats an empty array as unbound under set -u
|
||||
for v in MODE RUST_TOOLCHAIN SCCACHE_GB SCCACHE_VERSION NODE_MAJOR SLOTS P2P_PORTS BOX_HOSTNAME SSH_PUBKEY; do
|
||||
for v in MODE RUST_TOOLCHAIN SCCACHE_GB SCCACHE_VERSION NODE_MAJOR SLOTS P2P_PORTS BOX_HOSTNAME WORKERS_HOST SSH_PUBKEY; do
|
||||
[ -n "${!v:-}" ] && PASS="$PASS $v=$(printf '%q' "${!v}")"
|
||||
done
|
||||
|
||||
|
|
@ -41,19 +50,19 @@ if [ "$WIRE_ONLY" = 0 ]; then
|
|||
if "${ROOT_SSH[@]}" 'hostname' 2>/dev/null | grep -q '^rescue'; then bs_die "still in the rescue system after provision.sh"; fi
|
||||
fi
|
||||
|
||||
mkdir -p "$(dirname "$BS_HOST_FILE")"
|
||||
printf 'build@%s\n' "$IP" > "$BS_HOST_FILE"
|
||||
bs_log "wrote $BS_HOST_FILE"
|
||||
mkdir -p "$(dirname "$HOST_FILE")"
|
||||
printf 'build@%s\n' "$IP" > "$HOST_FILE"
|
||||
bs_log "wrote $HOST_FILE"
|
||||
bs_host
|
||||
bs_ssh 'hostname; nproc' >/dev/null || bs_die "build@$IP does not answer with $BS_KEY"
|
||||
|
||||
wire_remote() { # repo-dir mirror label
|
||||
local dir="$1" mirror="$2" label="$3" url="$BS_HOST:$2" cur
|
||||
cur=$(git -C "$dir" remote get-url build 2>/dev/null || true)
|
||||
if [ -z "$cur" ]; then git -C "$dir" remote add build "$url"; bs_log "$label: remote build = $url"
|
||||
elif [ "$cur" != "$url" ]; then git -C "$dir" remote set-url build "$url"; bs_log "$label: remote build -> $url"
|
||||
else bs_log "$label: remote build ok"; fi
|
||||
GIT_SSH_COMMAND="$BS_SSH_CMD" git -C "$dir" push -q --force build --all && bs_log "$label: every branch pushed to $mirror ($(git -C "$dir" branch --list | wc -l | tr -d ' ') branches)" || bs_die "$label: push to $mirror failed"
|
||||
cur=$(git -C "$dir" remote get-url "$REMOTE" 2>/dev/null || true)
|
||||
if [ -z "$cur" ]; then git -C "$dir" remote add "$REMOTE" "$url"; bs_log "$label: remote $REMOTE = $url"
|
||||
elif [ "$cur" != "$url" ]; then git -C "$dir" remote set-url "$REMOTE" "$url"; bs_log "$label: remote $REMOTE -> $url"
|
||||
else bs_log "$label: remote $REMOTE ok"; fi
|
||||
GIT_SSH_COMMAND="$BS_SSH_CMD" git -C "$dir" push -q --force "$REMOTE" --all && bs_log "$label: every branch pushed to $mirror ($(git -C "$dir" branch --list | wc -l | tr -d ' ') branches)" || bs_die "$label: push to $mirror failed"
|
||||
}
|
||||
wire_remote "$MAIN_REPO" "$BS_MIRROR_REPO" "igneum"
|
||||
wire_remote "$MAIN_REPO/vendor/igneum-node" "$BS_MIRROR_NODE" "igneum-node (the fork)"
|
||||
|
|
|
|||
|
|
@ -1,11 +1,18 @@
|
|||
#!/usr/bin/env bash
|
||||
# Install or refresh the worker dashboard collector on igneum-build-1 from this Mac.
|
||||
# infra/build-server/workers/install.sh copies tools/workers/{lib,collect}.mjs and the two units, enables the timer
|
||||
# Needs root over ssh (root@<ip> with ~/.ssh/igneum_ed25519); the host ip comes from ~/.config/igneum/build-server (build@<ip>).
|
||||
# infra/build-server/workers/install.sh build-1: copies tools/workers/{lib,collect}.mjs and the two units, enables the timer
|
||||
# infra/build-server/workers/install.sh 2 igneum-build-2 (host line in ~/.config/igneum/build-server-2), 3 for build-3
|
||||
# infra/build-server/workers/install.sh build@<ip> any box by its host line
|
||||
# Needs root over ssh (root@<ip> with ~/.ssh/igneum_ed25519); the host ip comes from ~/.config/igneum/build-server[-N] (build@<ip>).
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; ROOT="$(cd "$HERE/../../.." && pwd)"
|
||||
KEY="${IGNEUM_BUILD_KEY:-$HOME/.ssh/igneum_ed25519}"
|
||||
HOST_LINE="$(head -1 "${IGNEUM_BUILD_HOST_FILE:-$HOME/.config/igneum/build-server}" | tr -d '[:space:]')"
|
||||
case "${1:-}" in
|
||||
"") HOST_LINE="$(head -1 "${IGNEUM_BUILD_HOST_FILE:-$HOME/.config/igneum/build-server}" | tr -d '[:space:]')" ;;
|
||||
[2-9]) HOST_LINE="$(head -1 "$HOME/.config/igneum/build-server-$1" | tr -d '[:space:]')" ;;
|
||||
*@*) HOST_LINE="$1" ;;
|
||||
*) echo "usage: install.sh [N | build@<ip>]" >&2; exit 2 ;;
|
||||
esac
|
||||
IP="${HOST_LINE#*@}"; [ -n "$IP" ] || { echo "no build server in ~/.config/igneum/build-server" >&2; exit 1; }
|
||||
SSH=(ssh -i "$KEY" -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10)
|
||||
"${SSH[@]}" "root@$IP" 'install -d -o build -g build /srv/workers /srv/workers/bin /srv/workers/sources /srv/builds/_log; chown build:build /srv/builds/_log'
|
||||
|
|
|
|||
|
|
@ -12,6 +12,30 @@
|
|||
# tools/build-remote.sh --jobs 48 -- check
|
||||
# tools/build-remote.sh --target-dir target-exp -- build --release another persistent target dir on the box
|
||||
# tools/build-remote.sh --no-fetch -- clippy --all-targets nothing comes back (tests, check, clippy)
|
||||
# tools/build-remote.sh --ship [hive|rig|seed|linux] [--glibc X.Y] anything that SHIPS: cargo zigbuild for
|
||||
# x86_64-unknown-linux-gnu.<glibc> where the glibc comes
|
||||
# from the class (hive/rig 2.31: HiveOS is Ubuntu 20.04
|
||||
# based; seed/linux 2.35: Debian 12 and Ubuntu 22.04;
|
||||
# default seed; --glibc overrides)
|
||||
# (zig as the C/C++ toolchain, as the Mac's
|
||||
# infra/cross/build-linux.sh), artefacts from
|
||||
# target/x86_64-unknown-linux-gnu/release, each checked by
|
||||
# tools/ci/glibc-ceiling-check.sh (need at most <glibc>).
|
||||
# A plain build is native glibc 2.39: the box, the fleet's
|
||||
# Ubuntu 24.04 hosts, never a seed or a rig (7 Oct 2026:
|
||||
# a seed took 14 restarts on a 2.39 binary).
|
||||
# tools/build-remote.sh --priority gate -- test ... a RELEASE GATE (the app gate, the canary cut, the pre-push
|
||||
# self-tests): nice 0, the full core set, a slot ahead of
|
||||
# queued suites and benches
|
||||
# tools/build-remote.sh --plan [--priority gate] -- <cargo args> print the resolved class (kind, nice, cores, jobs) and stop;
|
||||
# no box, no crate needed (the CI check uses it)
|
||||
#
|
||||
# Scheduling (main, 7 October 2026, after a load of 190 on 96 threads: a release join bench and the 0.3.19 app gate starving each
|
||||
# other and "builds" of 16 minutes): every SUITE (cargo test) and BENCH (cargo bench) runs under nice 10 on the last 32 cores with
|
||||
# -j 32 unless the caller passes --priority gate; builds keep the box's own jobs rule (90 alone, 45 beside another slot holder); a
|
||||
# gate runs at nice 0 on the full set and takes a slot ahead of queued suites and benches (remote-run.sh gate-pending marker). The
|
||||
# class travels in the slot label ("; kind=suite nice=10 cores=32") and the JSONL line, so the dashboard's job card shows why a
|
||||
# job is slow. A call without a priority flag defaults to the bounded class for suites and benches: tools/ci/build-kind-default-check.sh.
|
||||
# tools/build-remote.sh --self-test-repro [--full] from a fork worktree: igneum-miner built twice a minute
|
||||
# apart without sccache into one target dir must give one
|
||||
# sha256, and a per-run target path must not (prost's
|
||||
|
|
@ -47,8 +71,11 @@ BS_TOOL=build-remote
|
|||
# shellcheck source=../infra/build-server/lib.sh
|
||||
. "$HERE/../infra/build-server/lib.sh"
|
||||
|
||||
{ # whole-body: bash parses this block entirely before running a line of it, so an edit to this file while a run is in
|
||||
# flight cannot reach the running copy (7 Oct 2026: build-remote.sh was edited mid-run and died on shifted bytes after a 4-min build)
|
||||
|
||||
# JOBS empty = the box decides: 90 alone, 45 beside another slot holder (remote-run.sh, main's ruling 6 Oct 2026)
|
||||
JOBS="${JOBS:-}"; OUT=""; ARTEFACTS=""; TARGET_DIR="target"; FETCH=1; CARGO_ARGS=(); SELFTEST=0; FULL=0
|
||||
JOBS="${JOBS:-}"; OUT=""; ARTEFACTS=""; TARGET_DIR="target"; FETCH=1; CARGO_ARGS=(); SELFTEST=0; FULL=0; SHIP=0; SHIP_CLASS="${SHIP_CLASS:-seed}"; PRIORITY="${PRIORITY:-normal}"; PLAN=0; BOX="${BOX:-}"; GLIBC="${GLIBC:-}"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--jobs) JOBS="$2"; shift 2 ;;
|
||||
|
|
@ -57,6 +84,12 @@ while [ $# -gt 0 ]; do
|
|||
--target-dir) TARGET_DIR="$2"; shift 2 ;;
|
||||
--no-fetch) FETCH=0; shift ;;
|
||||
--self-test-repro) SELFTEST=1; shift ;;
|
||||
--ship) SHIP=1; case "${2:-}" in hive|rig|seed|linux|native) SHIP_CLASS="$2"; shift 2 ;; *) shift ;; esac ;;
|
||||
--priority) PRIORITY="$2"; shift 2 ;;
|
||||
--box) BOX="$2"; BOX_GIVEN=1; shift 2 ;;
|
||||
--gate) PRIORITY=gate; shift ;;
|
||||
--plan) PLAN=1; shift ;;
|
||||
--glibc) GLIBC="$2"; shift 2 ;;
|
||||
--full) FULL=1; shift ;;
|
||||
--) shift; CARGO_ARGS=("$@"); break ;;
|
||||
-h|--help) sed -n '2,32p' "$0"; exit 0 ;;
|
||||
|
|
@ -66,8 +99,42 @@ done
|
|||
[ "${CARGO_ARGS[0]:-}" = cargo ] && CARGO_ARGS=("${CARGO_ARGS[@]:1}")
|
||||
CARGO_ARGS_GIVEN=""; [ -n "${CARGO_ARGS[*]:-}" ] && CARGO_ARGS_GIVEN=1
|
||||
|
||||
bs_host
|
||||
# the scheduling class, from the cargo subcommand and the priority flag alone (no box, no crate): BR_NICE, BR_CORES (the last N of the
|
||||
# box's 96; 0 = the full set), BR_JOBS_CAP (0 = the box's rule), BR_PRIORITY; the kind for the log is refined after bs_context
|
||||
BR_NICE=0; BR_CORES=0; BR_JOBS_CAP=0; BR_PRIORITY="$PRIORITY"; SCHED_CLASS=build
|
||||
case "$PRIORITY" in gate|normal) ;; *) bs_die "--priority takes gate or normal, not '$PRIORITY'" ;; esac
|
||||
case "${CARGO_ARGS[0]:-build}" in
|
||||
test) SCHED_CLASS=suite ;;
|
||||
bench) SCHED_CLASS=bench ;;
|
||||
check|clippy) SCHED_CLASS=check ;;
|
||||
*) SCHED_CLASS=build ;;
|
||||
esac
|
||||
if [ "$PRIORITY" = gate ]; then BR_NICE=0; BR_CORES=0; BR_JOBS_CAP=0
|
||||
elif [ "$SCHED_CLASS" = suite ] || [ "$SCHED_CLASS" = bench ]; then BR_NICE=10; BR_CORES=32; BR_JOBS_CAP=32; fi
|
||||
# an explicit --jobs above the bounded class's cap is clamped (main's rule: bounded unless a priority flag; 7 Oct 2026: two suites
|
||||
# ran at -j 90 on a box at load 190 because their callers passed --jobs 90)
|
||||
if [ "$BR_JOBS_CAP" -gt 0 ] && [ -n "$JOBS" ] && [ "$JOBS" -gt "$BR_JOBS_CAP" ]; then bs_log "--jobs $JOBS clamped to $BR_JOBS_CAP for a $SCHED_CLASS (pass --priority gate for the full set)"; JOBS=$BR_JOBS_CAP; fi
|
||||
export BR_NICE BR_CORES BR_JOBS_CAP BR_PRIORITY
|
||||
if [ "$PLAN" = 1 ]; then
|
||||
k="$SCHED_CLASS"; [ "$PRIORITY" = gate ] && k=gate
|
||||
printf 'kind=%s nice=%s cores=%s jobs=%s priority=%s\n' "$k" "$BR_NICE" "$( [ "$BR_CORES" = 0 ] && echo 96 || echo "$BR_CORES")" "$( [ "$BR_JOBS_CAP" = 0 ] && echo box || echo "$BR_JOBS_CAP")" "$PRIORITY"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# the box: --box N pins it; else the class's PREFERRED box with spill-over (lib.sh bs_route_spill, the project lead 7 Oct 2026: a build or gate
|
||||
# prefers box 1, a suite or bench box 2, a proving crate box 3; a preferred box with no free slot or a 1-minute load above 64 hands
|
||||
# the job to the other box when that one qualifies; the decision line is printed here and lands in the JSONL row as "route")
|
||||
ROUTE_CLASS="$SCHED_CLASS"; [ "$PRIORITY" = gate ] && ROUTE_CLASS=gate
|
||||
if [ -z "$BOX" ]; then bs_route_spill "$ROUTE_CLASS"; BOX=$BS_ROUTE_BOX; else BR_ROUTE_PREF=$BOX BR_ROUTE_BOX=$BOX BR_ROUTE_SPILLED=0 BR_ROUTE_REASON="box $BOX by --box"; export BR_ROUTE_PREF BR_ROUTE_BOX BR_ROUTE_SPILLED BR_ROUTE_REASON; fi
|
||||
bs_host "$BOX"
|
||||
bs_context
|
||||
# a proving crate prefers box 3 unless the caller chose (its builds and suites alike; the same spill-over)
|
||||
if [ "$BS_CRATE_REL" = proving/igneum-prove ] && [ -z "${BOX_GIVEN:-}" ] && [ "$PRIORITY" != gate ]; then bs_route_spill prove; [ "$BS_ROUTE_BOX" != "$BOX" ] && { BOX=$BS_ROUTE_BOX; bs_host "$BOX"; }; fi
|
||||
# box 2 keeps the suites' numbers: everything that lands there runs at the bounded class (nice 10, a 32-core band, -j 32), builds
|
||||
# and gates included (main, 7 Oct 2026); a gate still takes its slot ahead of queued suites
|
||||
if [ "$BOX" = 2 ] && [ "$BR_CORES" = 0 ]; then BR_NICE=10; BR_CORES=32; BR_JOBS_CAP=32; export BR_NICE BR_CORES BR_JOBS_CAP; bs_log "bounded on box 2 (nice 10, a 32-core band, -j 32) so a suite beside it keeps its number"; fi
|
||||
if [ "$BR_JOBS_CAP" -gt 0 ] && [ -n "$JOBS" ] && [ "$JOBS" -gt "$BR_JOBS_CAP" ]; then bs_log "--jobs $JOBS clamped to $BR_JOBS_CAP on box $BOX"; JOBS=$BR_JOBS_CAP; fi
|
||||
bs_log "box $BOX ($BS_HOST) for class $SCHED_CLASS, priority $PRIORITY$( [ "${BR_ROUTE_SPILLED:-0}" = 1 ] && echo ", SPILLED from box $BR_ROUTE_PREF")"
|
||||
|
||||
if [ "$SELFTEST" = 1 ]; then
|
||||
[ "$BS_KIND" = node ] || bs_die "--self-test-repro runs from a fork worktree (igneum-miner and kaspad live there)"
|
||||
|
|
@ -103,24 +170,46 @@ fi
|
|||
exit 0
|
||||
fi
|
||||
|
||||
# --ship: the zig path and the target triple dir for the artefacts
|
||||
SHIP_TARGET=x86_64-unknown-linux-gnu
|
||||
if [ "$SHIP" = 1 ]; then
|
||||
[ -n "$GLIBC" ] || GLIBC=$("$HERE/ci/glibc-ceiling-check.sh" --ceiling-of "$SHIP_CLASS") || bs_die "unknown ship class $SHIP_CLASS"
|
||||
[ "$GLIBC" != native ] || bs_die "--ship native is a plain build: drop --ship"
|
||||
TARGET_SUB="$SHIP_TARGET/release"
|
||||
else TARGET_SUB="release"; fi
|
||||
# defaults per crate
|
||||
case "$BS_KIND:$BS_CRATE_REL" in
|
||||
node:*)
|
||||
[ -n "${CARGO_ARGS[*]:-}" ] || CARGO_ARGS=(build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow)
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/release/igneumd $TARGET_DIR/release/igneum-miner" ;;
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/$TARGET_SUB/igneumd $TARGET_DIR/$TARGET_SUB/igneum-miner" ;;
|
||||
repo:app/igneum-app)
|
||||
[ -n "${CARGO_ARGS[*]:-}" ] || CARGO_ARGS=(build --release)
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/release/igneum-app $TARGET_DIR/release/igneum-ota-sign $TARGET_DIR/release/igneum-prove-verify" ;;
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/$TARGET_SUB/igneum-app $TARGET_DIR/$TARGET_SUB/igneum-ota-sign $TARGET_DIR/$TARGET_SUB/igneum-prove-verify" ;;
|
||||
repo:proving/igneum-prove)
|
||||
[ -n "${CARGO_ARGS[*]:-}" ] || CARGO_ARGS=(build --release)
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/release/igneum-prove-host $TARGET_DIR/release/igneum-prove-export" ;;
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/$TARGET_SUB/igneum-prove-host $TARGET_DIR/$TARGET_SUB/igneum-prove-export" ;;
|
||||
*)
|
||||
[ -n "${CARGO_ARGS[*]:-}" ] || CARGO_ARGS=(build --release) ;;
|
||||
esac
|
||||
case "${CARGO_ARGS[0]}" in build) ;; *) [ -n "${ARTEFACTS_SET:-}" ] || { FETCH=0; ARTEFACTS=""; } ;; esac # test, check, clippy: nothing to fetch
|
||||
if [ "$SHIP" = 1 ]; then
|
||||
[ "${CARGO_ARGS[0]}" = build ] || bs_die "--ship is for cargo build"
|
||||
CARGO_ARGS=(zigbuild "${CARGO_ARGS[@]:1}" --target "$SHIP_TARGET.$GLIBC")
|
||||
BR_TARGET_SHIP="$SHIP_TARGET.$GLIBC"
|
||||
fi
|
||||
[ -n "$OUT" ] || OUT="$BS_CRATE/target-remote"
|
||||
|
||||
bs_log "$BS_KIND crate $BS_WT/$BS_CRATE_REL at $BS_SHA ($BS_BRANCH) -> $BS_HOST:$BS_REMOTE_CRATE; cargo ${CARGO_ARGS[*]} -j ${JOBS:-auto}; target dir $TARGET_DIR"
|
||||
# a fork build pairs with the igneum-pow of the igneum worktree it sits in (the fork's path dependency ../../../../igneum-pow), so the
|
||||
# pairing is said on the first line and recorded (7 Oct 2026, 0.3.17: a fork at 12153428 under a master worktree failed in kaspa-pow
|
||||
# four minutes in, "no associated function chain_program_shadow", because master's igneum-pow predates the release branch's)
|
||||
if [ "$BS_KIND" = node ]; then
|
||||
pair_branch=$(git -C "$BS_WT_ROOT" branch --show-current 2>/dev/null || true); pair_dirty=$(git -C "$BS_WT_ROOT" status --porcelain -- igneum-pow 2>/dev/null || true)
|
||||
# no `[ ... ] && echo` inside an assignment's $( ): its false status is the assignment's status and set -e ends the script (7 Oct 2026, 03:0x UTC)
|
||||
PAIR="igneum $(git -C "$BS_WT_ROOT" rev-parse --short HEAD) (${pair_branch:-detached})${pair_dirty:+ with uncommitted igneum-pow changes}"
|
||||
bs_log "pairs with $PAIR: igneum-pow $(grep -m1 '^version' "$BS_WT_ROOT/igneum-pow/Cargo.toml" | sed 's/.*"\(.*\)".*/\1/')"
|
||||
export BR_PAIRS_WITH="$PAIR"
|
||||
fi
|
||||
bs_toolchain_check
|
||||
t_sync0=$(date +%s)
|
||||
bs_sync_sources
|
||||
|
|
@ -130,12 +219,16 @@ bs_log "sources in place after $(( $(date +%s) - t_sync0 )) s (every changed fil
|
|||
# rerun-if-changed and is never run again by cargo (release-0.3.11 plan: `cargo clean -p kaspa-build-info` first); so when
|
||||
# the commit the box builds differs from the last one built in this target dir, that one crate is cleaned (a relink, seconds)
|
||||
pre=""
|
||||
if [ "$BS_KIND" = node ] && [ "${CARGO_ARGS[0]}" = build ]; then
|
||||
if [ "$BS_KIND" = node ] && { [ "${CARGO_ARGS[0]}" = build ] || [ "${CARGO_ARGS[0]}" = zigbuild ]; }; then
|
||||
pre="[ \"\$(cat '.build-remote-sha-$TARGET_DIR' 2>/dev/null)\" = '$BS_SHA' ] || CARGO_TARGET_DIR='$TARGET_DIR' cargo clean -q --release -p kaspa-build-info 2>/dev/null; "
|
||||
fi
|
||||
cmd="$(bs_repro_env)${pre}CARGO_TARGET_DIR='$TARGET_DIR' cargo $(printf '%q ' "${CARGO_ARGS[@]}")${JOBS:+-j $JOBS} 2>&1 | tee -a '$BS_REMOTE_WT/.build-remote.log'; rc=\${PIPESTATUS[0]}; [ \$rc = 0 ] && echo '$BS_SHA' > '.build-remote-sha-$TARGET_DIR'; ( exit \$rc )" # a subshell exit: the runner reads \$? and still prints its RESULT line
|
||||
label="$BS_WT/$BS_CRATE_REL cargo ${CARGO_ARGS[*]}"
|
||||
BR_KIND=$(bs_kind build-remote "${CARGO_ARGS[0]}"); BR_COMMAND="cargo ${CARGO_ARGS[*]}"; BR_TARGET=x86_64-unknown-linux-gnu
|
||||
BR_KIND=$(bs_kind build-remote "$( [ "${CARGO_ARGS[0]}" = zigbuild ] && echo build || echo "${CARGO_ARGS[0]}")"); BR_COMMAND="cargo ${CARGO_ARGS[*]}"; BR_TARGET="${BR_TARGET_SHIP:-x86_64-unknown-linux-gnu}"
|
||||
[ "$SCHED_CLASS" = bench ] && BR_KIND=bench
|
||||
[ "$PRIORITY" = gate ] && BR_KIND=gate
|
||||
label="$label; kind=$BR_KIND nice=$BR_NICE cores=$( [ "$BR_CORES" = 0 ] && echo 96 || echo "$BR_CORES")"
|
||||
[ "${BR_ROUTE_SPILLED:-0}" = 1 ] && label="$label; spilled from box $BR_ROUTE_PREF"
|
||||
for ((i = 0; i < ${#CARGO_ARGS[@]}; i++)); do [ "${CARGO_ARGS[$i]}" = --target ] && BR_TARGET="${CARGO_ARGS[$((i + 1))]:-}"; done
|
||||
BR_ARTEFACTS=""; [ "$FETCH" = 1 ] && BR_ARTEFACTS="$ARTEFACTS"
|
||||
export BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS
|
||||
|
|
@ -162,6 +255,10 @@ if [ "$FETCH" = 1 ] && [ -n "$ARTEFACTS" ]; then
|
|||
bs_log "artefact $dest: $(bs_size "$dest") bytes, sha256 $(bs_sha256 "$dest"), $(file -b "$dest" | cut -c1-60)"
|
||||
# the commit-string gate (rule of 6 October 2026): a node binary without its commit in its strings fails the run
|
||||
case "$BS_KIND:$(basename "$dest")" in node:igneumd) "$HERE/ci/commit-string-check.sh" "$dest" "$BS_SHA" || bs_die "commit-string gate failed for $a" ;; esac # only kaspad depends on kaspa-build-info
|
||||
# the glibc ceiling of anything that ships (main, 7 Oct 2026): a seed or a rig refuses a binary needing more than 2.36
|
||||
if [ "$SHIP" = 1 ]; then "$HERE/ci/glibc-ceiling-check.sh" "$dest" "$GLIBC" || bs_die "glibc ceiling gate failed for $a"; fi
|
||||
done
|
||||
fi
|
||||
bs_wt_unlock
|
||||
exit 0
|
||||
}
|
||||
|
|
|
|||
|
|
@ -36,6 +36,9 @@ BS_TOOL=cross-remote
|
|||
# shellcheck source=../infra/build-server/lib.sh
|
||||
. "$HERE/../infra/build-server/lib.sh"
|
||||
|
||||
{ # whole-body: bash parses this block entirely before running a line of it, so an edit to this file while a run is in
|
||||
# flight cannot reach the running copy (7 Oct 2026: build-remote.sh was edited mid-run and died on shifted bytes after a 4-min build)
|
||||
|
||||
TARGET=x86_64-pc-windows-gnu
|
||||
# JOBS empty = the box decides: 90 alone, 45 beside another slot holder (remote-run.sh, main's ruling 6 Oct 2026). One line, no
|
||||
# trailing comment: a `#` on this line once swallowed every assignment after it, CARGO_ARGS was never declared and the default
|
||||
|
|
@ -126,3 +129,5 @@ if printf '%s\n' "$dlls" | grep -q 'libstdc++-6.dll'; then
|
|||
fi
|
||||
bs_log "exes in $OUT/$TARGET/release"
|
||||
bs_wt_unlock
|
||||
exit 0
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,7 +7,8 @@
|
|||
# libstdc++ (Ubuntu 22.04) runs them; the glibc ceiling of each binary is printed (a native Ubuntu 24.04 build; the HiveOS
|
||||
# package keeps the Mac's zig build at glibc 2.36 until zig is on the box).
|
||||
#
|
||||
# tools/workers-remote.sh [--out <dir>] from any worktree of the igneum repo; artefacts in <worktree>/infra/cross/out-workers-box/
|
||||
# tools/workers-remote.sh [--class hive|rig|seed|linux|native] [--out <dir>] from any igneum worktree; artefacts in
|
||||
# <worktree>/infra/cross/out-workers-box/ (class rig, glibc 2.31, by default)
|
||||
#
|
||||
# Sources travel as HEAD through the mirror plus the overlay of proto-cuda and proto-opencl (lib.sh); the build takes a remote
|
||||
# slot like every other build. Asked for by main on 6 October 2026 for the class v4 rehearsal (the fleet agent's generator-4 worker).
|
||||
|
|
@ -17,7 +18,10 @@ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|||
BS_TOOL=workers-remote
|
||||
# shellcheck source=../infra/build-server/lib.sh
|
||||
. "$HERE/../infra/build-server/lib.sh"
|
||||
OUT=""; while [ $# -gt 0 ]; do case "$1" in --out) OUT="$2"; shift 2 ;; *) bs_die "unknown argument $1" ;; esac; done
|
||||
|
||||
{ # whole-body: bash parses this block entirely before running a line of it, so an edit to this file while a run is in
|
||||
# flight cannot reach the running copy (7 Oct 2026: build-remote.sh was edited mid-run and died on shifted bytes after a 4-min build)
|
||||
OUT=""; CLASS="${CLASS:-rig}"; while [ $# -gt 0 ]; do case "$1" in --out) OUT="$2"; shift 2 ;; --class) CLASS="$2"; shift 2 ;; *) bs_die "unknown argument $1" ;; esac; done
|
||||
bs_host
|
||||
# the context by hand (bs_context wants a Cargo.toml): the igneum worktree root is the repo; lib.sh reads these
|
||||
BS_TOP=$(git -C "$HERE" rev-parse --show-toplevel); BS_WT_ROOT="$BS_TOP"; BS_KIND=repo; BS_MIRROR="$BS_MIRROR_REPO"; BS_TOP_REL=.
|
||||
|
|
@ -28,13 +32,20 @@ BS_LOCAL_DIRS="proto-cuda proto-opencl"; BS_VENDOR_REPOS=""
|
|||
bs_log "workers from $BS_WT at $BS_SHA ($BS_BRANCH) -> $BS_HOST:$BS_REMOTE_WT (proto-cuda, proto-opencl)"
|
||||
bs_sync_sources
|
||||
PLACEHOLDER=proto-cuda/packs/igneum-devnet-v4-epoch0 # the OpenCL worker's compile-time pack, as build-workers-linux.sh
|
||||
# the workers go to rigs: class rig (2.31, HiveOS is Ubuntu 20.04 based) by default; --class seed|linux gives 2.35, --class native
|
||||
# uses clang (2.39: the box, Ubuntu 24.04 hosts). The table lives in tools/ci/glibc-ceiling-check.sh; GLIBC=X.Y overrides.
|
||||
[ -n "${GLIBC:-}" ] || GLIBC=$("$HERE/ci/glibc-ceiling-check.sh" --ceiling-of "$CLASS") || bs_die "unknown class $CLASS"
|
||||
# zig brings its own libc and libc++ headers and links them statically; the host's /usr/include must stay OUT of the search path
|
||||
# (7 Oct 2026: `-I /usr/include` for CL/cl.h pulled Ubuntu's glibc 2.39 stdlib.h in front of zig's 2.36 one and the link died on
|
||||
# __isoc23_strtol), so the OpenCL headers are reached through a directory that holds only CL/ (a symlink made on the box)
|
||||
if [ "$GLIBC" = native ]; then CXX='clang++ -std=c++17 -static-libstdc++ -static-libgcc'; CC='clang -std=gnu99 -static-libgcc'; else CXX="zig c++ -target x86_64-linux-gnu.$GLIBC -std=c++17"; CC="zig cc -target x86_64-linux-gnu.$GLIBC -std=gnu99"; fi
|
||||
cmd="$(bs_repro_env)"'. /etc/profile.d/igneum-build.sh
|
||||
CUDA=$(ls -d /usr/local/cuda-12.* 2>/dev/null | sort -V | tail -1); [ -n "$CUDA" ] || { echo "no CUDA headers under /usr/local/cuda-12.*: provision.sh step_cuda"; exit 2; }
|
||||
[ -f /usr/include/CL/cl.h ] || { echo "no /usr/include/CL/cl.h: apt opencl-c-headers"; exit 2; }
|
||||
mkdir -p out-workers-box
|
||||
echo "workers-remote: $(clang++ --version | head -1); CUDA headers $CUDA/include; $(git rev-parse --short HEAD)"
|
||||
clang++ -std=c++17 -O2 -Wall -Wextra -I proto-cuda/nvrtc -I "$CUDA/include" -static-libstdc++ -static-libgcc -o out-workers-box/igneum-worker-cuda proto-cuda/nvrtc/worker.cpp -ldl -lpthread || exit 1
|
||||
clang -std=gnu99 -D_GNU_SOURCE -O2 -Wall -Wextra -Wno-format-truncation -DIGNEUM_CL_DYNAMIC -DCL_TARGET_OPENCL_VERSION=120 -I /usr/include -I '"$PLACEHOLDER"' -DIGNEUM_KERNEL_PATH='"'"'"kernel_bound.cl"'"'"' -static-libgcc -o out-workers-box/igneum-worker-opencl proto-opencl/host.c -ldl -lm -lpthread || exit 1
|
||||
mkdir -p out-workers-box "$HOME/.cache/igneum-cl"; ln -sfn /usr/include/CL "$HOME/.cache/igneum-cl/CL"
|
||||
echo "workers-remote: '"$CXX"' ($(zig version 2>/dev/null || clang++ --version | head -1)); CUDA headers $CUDA/include; $(git rev-parse --short HEAD)"
|
||||
'"$CXX"' -O2 -Wall -Wextra -I proto-cuda/nvrtc -I "$CUDA/include" -o out-workers-box/igneum-worker-cuda proto-cuda/nvrtc/worker.cpp -ldl -lpthread || exit 1
|
||||
'"$CC"' -D_GNU_SOURCE -O2 -Wall -Wextra -Wno-format-truncation -DIGNEUM_CL_DYNAMIC -DCL_TARGET_OPENCL_VERSION=120 -I "$HOME/.cache/igneum-cl" -I '"$PLACEHOLDER"' -DIGNEUM_KERNEL_PATH='"'"'"kernel_bound.cl"'"'"' -o out-workers-box/igneum-worker-opencl proto-opencl/host.c -ldl -lm -lpthread || exit 1
|
||||
for b in igneum-worker-cuda igneum-worker-opencl; do printf "workers-remote: %s glibc ceiling %s, needs %s\n" "$b" "$(objdump -T out-workers-box/$b | grep -oE "GLIBC_[0-9.]+" | sort -V | tail -1)" "$(readelf -d out-workers-box/$b | grep -oE "\[lib[^]]+\]" | tr -d "[]" | tr "\n" " ")"; done'
|
||||
BR_KIND=other BR_COMMAND="clang++ worker.cpp; clang host.c" BR_TARGET=x86_64-unknown-linux-gnu BR_ARTEFACTS="out-workers-box/igneum-worker-cuda out-workers-box/igneum-worker-opencl"; export BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS
|
||||
t0=$(date +%s); set +e; bs_remote_run "$BS_REMOTE_WT" "$BS_WT/proto-cuda workers" "$cmd" 2>&1 | tee "/tmp/workers-remote-$$.log"; rc=${PIPESTATUS[0]}; set -e
|
||||
|
|
@ -45,6 +56,9 @@ mkdir -p "$OUT"
|
|||
for b in igneum-worker-cuda igneum-worker-opencl; do
|
||||
bs_rsync -p "$BS_HOST:$BS_REMOTE_WT/out-workers-box/$b" "$OUT/$b" || bs_die "no $b on the box"
|
||||
bs_log "artefact $OUT/$b: $(bs_size "$OUT/$b") bytes, sha256 $(bs_sha256 "$OUT/$b"), $(file -b "$OUT/$b" | cut -c1-50)"
|
||||
[ "$GLIBC" = native ] || "$HERE/ci/glibc-ceiling-check.sh" "$OUT/$b" "$GLIBC" || bs_die "glibc ceiling gate failed for $b"
|
||||
done
|
||||
{ printf 'igneum workers, linux x86_64, built on igneum-build-1 with clang++ (static libstdc++) on %s from %s (%s); run-time libraries dlopen-ed\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$BS_SHA" "$BS_BRANCH"; } > "$OUT/version.txt"
|
||||
{ printf 'igneum workers, linux x86_64, built on igneum-build-1 (glibc %s, static libstdc++) on %s from %s (%s); run-time libraries dlopen-ed\n' "$GLIBC" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$BS_SHA" "$BS_BRANCH"; } > "$OUT/version.txt"
|
||||
bs_wt_unlock
|
||||
exit 0
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue