6f5787ed71
Main, 7 October 2026, 15:07 UK: one global exclusive measure flock across unrelated measurements stalled build-1 at load 120 with free slots (a stopped probe held it 5.5 h; an exclusive waiter queued every new shared taker) and build-2 behind a one-core VDF bench. lease.sh (installed at /srv/builds/_bin/lease by provision.sh and by hand on both boxes) takes one flock per core for a pinned measurement and the quiet file for a whole-box one; remote-run.sh takes quiet shared only for unbounded runs, excludes leased cores from its set, and its keeper refreshes the holder file and calls lease reap (a STOPPED holder of a lease, quiet or a slot for 5 minutes is killed with a line in _log/reaped.log). Keepers close the lock descriptors they inherit (an orphaned sleep held a slot and the worktree lock 20 s past the release; the slot self-test had rotted on that since the worktree lock landed). tools/ci/box-locks-check.sh runs lease.sh --self-test and remote-run.sh --self-test-slots on build-1. provision.sh also carries the 16 libraries headless Chromium needs (installed by hand on both boxes at 14:5x UK) and a headless self-test step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
529 lines
42 KiB
Bash
Executable file
529 lines
42 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# The remote half of tools/build-remote.sh and tools/cross-remote.sh. It runs ON igneum-build-1, fed by lib.sh bs_remote_run
|
|
# over `ssh build@<box> bash -s` with these exports prepended (never run it by hand on the Mac):
|
|
# BR_DIR the crate directory on the box BR_CMD the shell string to run there (cargo ...)
|
|
# BR_LABEL the slot-file label (ends with "; agent=<name>")
|
|
# BR_TOOL build-remote | cross-remote BR_KIND node-linux | node-windows | app | app-windows | prove | suite | check | other
|
|
# BR_WT the worktree name BR_CRATE the crate path relative to the worktree root
|
|
# BR_COMMAND the cargo command as typed BR_TARGET the rust target triple
|
|
# BR_BRANCH BR_SHA BR_AGENT the agent name (IGNEUM_AGENT on the Mac, else the worktree)
|
|
# BR_ARTEFACTS space-separated paths (relative to BR_DIR) the Mac will fetch; empty for test, check, clippy
|
|
#
|
|
# 1. Takes a build slot: flock on $IGNEUM_BUILD_SLOTS_DIR/build-<k> for k below the count in .../slots (the box's own slot
|
|
# files, never the Mac's; 2 since main's ruling of 6 October 2026, 20:3x UK); when every slot is busy it waits up to 2 h on
|
|
# build-0 and exits 75 if it gives up. The holder line is `pid N since HH:MM:SSZ waited S s: <label>`, the format
|
|
# tools/lock/with-lock.sh writes on the Mac. The cargo job count follows the slots: after one second of settling, a build
|
|
# that holds the only taken slot gets CARGO_BUILD_JOBS=90, one that sees the other slot held gets 45 (JOBS_ALONE and
|
|
# JOBS_SHARED), so two builds share the 96 threads without thrashing; a `-j` on the cargo line (--jobs on the Mac) wins.
|
|
# A MEASUREMENT (BR_MEASURE=1: the CPU prover timing, bench rows) takes the `measure` file exclusively and excludes every
|
|
# build, as with-lock.sh's `measure` does on the Mac: builds hold `measure` shared for their whole run, so a measure waits
|
|
# for the running builds and blocks new ones until it ends. Lock files are opened in APPEND mode: the first version opened
|
|
# them with `>` and truncated a busy slot's holder line every time another build probed it (found 6 October 2026, evening).
|
|
# 2. Runs BR_CMD in BR_DIR with sccache, prints one `build-remote: RESULT rc= secs= compiles= sccache_hits_total= ...` line.
|
|
# 3. Appends one JSON line to /srv/builds/_log/builds.jsonl (the worker dashboard reads it; asked for by main on 6 October
|
|
# 2026): on success, on failure and on the slot give-up. UTC ISO 8601 Z times, numbers unquoted, unknown fields omitted,
|
|
# artefacts with bytes and sha256 only when the run succeeded (a failed build would list the previous build's files),
|
|
# the line kept under 4 KB. Since the evening of 6 October 2026 (the project lead: "make sure we are fixing and learning from all the
|
|
# errors here") the line also carries "class" and "run_log":
|
|
# - PRE-FLIGHT before cargo runs: the manifest must parse (cargo metadata --no-deps) and every `-p` package must exist
|
|
# (a workspace member, else cargo pkgid --offline); a refusal is exit 3, class preflight-manifest or preflight-package,
|
|
# and costs a second instead of a slot. The instant-death class: three suite runs on 6 October died in 0 s with exit
|
|
# 101 and no compile, and nothing on the box had kept the reason.
|
|
# - the run's output is kept: the last 400 lines in /srv/builds/_log/runs/<id>.log, so a red row can be read after the
|
|
# agent's terminal is gone.
|
|
# - CLASS of a red run from that output: instant (under 3 s, nothing compiled), compile-error (error[E...] or "could not
|
|
# compile"), link-error, test-failure ("test result: FAILED"), slot-timeout (75), no-dir (2), other.
|
|
# - a `cargo test` with a filter that ran 0 tests everywhere is a wasted round trip: exit 3, class no-test-matched.
|
|
# - every red row is also appended to the shared red-run file (/srv/ci-red/red.jsonl, $IGNEUM_CI_RED_FILE), the file
|
|
# tools/ci/red-watch.mjs posts from; box rows are not posted one by one, the 09:00 UK digest counts them per class.
|
|
# Modes (BR_MODE, default run): `checkout` resets the box's tree and checks the branch out at the commit (lib.sh
|
|
# bs_push_and_checkout: BR_CO_DIR, BR_CO_MIRROR, BR_CO_BRANCH, BR_CO_SHA, BR_CO_WT); `--self-test` (first argument) builds a
|
|
# scratch mirror and clone, dirties the clone the way a build's overlay does, moves the mirror one commit on, and shows the
|
|
# checkout mode lands on the new commit with a clean tree (the 6 October 2026 case: a stale overlay made `git checkout -B`
|
|
# refuse with "local changes would be overwritten"); `--self-test-slots` runs fake builds and a fake measurement against a
|
|
# scratch slots directory: two concurrent builds get 45 jobs each, one alone gets 90, a measure blocks a build and a build
|
|
# blocks a measure, and a probing build leaves a busy slot's holder line intact (IGNEUM_REMOTE_RUN_UNDER_TEST=<script> runs
|
|
# the cases against another copy, which is how the old script was shown to fail them).
|
|
set -uo pipefail
|
|
# the profile sets the box's paths; an IGNEUM_BUILD_SLOTS_DIR or IGNEUM_BUILD_LOG_DIR already in the environment wins (the slot
|
|
# self-test runs against a scratch directory; the first version let the profile reset it and the test took the REAL slot)
|
|
_slots_env="${IGNEUM_BUILD_SLOTS_DIR:-}"; _log_env="${IGNEUM_BUILD_LOG_DIR:-}"
|
|
[ -f /etc/profile.d/igneum-build.sh ] && . /etc/profile.d/igneum-build.sh
|
|
[ -n "$_slots_env" ] && IGNEUM_BUILD_SLOTS_DIR="$_slots_env"; [ -n "$_log_env" ] && IGNEUM_BUILD_LOG_DIR="$_log_env"
|
|
|
|
# What the clean spares beyond target dirs and stamps: a lane's scratch. The fixed prefixes, plus every glob in the mirror-local
|
|
# file `.igneum-scratch-spare` (one per line, # comments; the file itself is spared). spare_args <dir> fills SPARE_ARGS with
|
|
# the `-e <glob>` arguments; it is never empty (the fixed list), so bash 3.2's unbound-empty-array rule cannot bite the self-test.
|
|
SPARE_FIXED=('attack-*' 'scratch-*' 'target-attack-*' '.build-remote.log' '.igneum-scratch-spare')
|
|
spare_args() {
|
|
local p
|
|
SPARE_ARGS=()
|
|
for p in "${SPARE_FIXED[@]}"; do SPARE_ARGS+=(-e "$p"); done
|
|
[ -f "$1/.igneum-scratch-spare" ] || return 0
|
|
while IFS= read -r p || [ -n "$p" ]; do
|
|
p="${p%%#*}"; p="${p#"${p%%[![:space:]]*}"}"; p="${p%"${p##*[![:space:]]}"}"
|
|
[ -n "$p" ] && SPARE_ARGS+=(-e "$p")
|
|
done < "$1/.igneum-scratch-spare"
|
|
return 0
|
|
}
|
|
# the untracked paths the clean would still take, up to three (empty = the tree is clean); the same excludes as the clean line
|
|
tree_left() {
|
|
git -C "$1" clean -nd -e target -e 'target-*' -e '.build-remote-sha-*' -e '.cross-remote-sha-*' -e sccache "${SPARE_ARGS[@]}" | head -3
|
|
}
|
|
|
|
# discard the previous overlay (tracked edits and untracked files; target dirs, the sha stamps and anything ignored are kept),
|
|
# fetch, then the branch at the commit. Runs in BR_CO_DIR, clones it from BR_CO_MIRROR when it has no .git.
|
|
checkout_tree() {
|
|
local dir="$1" mirror="$2" branch="$3" sha="$4" wt="${5:-}"
|
|
set -e
|
|
[ -n "$wt" ] && mkdir -p "$wt"
|
|
if [ ! -d "$dir/.git" ]; then rm -rf "$dir"; git clone -q --no-checkout "$mirror" "$dir"; fi
|
|
cd "$dir"
|
|
# a run killed mid-git (two runs on one worktree, 18:48:56Z the same day) leaves .git/index.lock; stale when it is older
|
|
# than 30 s (an index write takes milliseconds, a checkout of the fork seconds). The first version asked `pgrep -x git`,
|
|
# which said "in use" whenever ANY git ran on the machine: the pre-push hook's own `git push` and any other agent's build
|
|
# kept the lock and the checkout died with "index.lock: File exists" (6 October 2026, 22:2x UK; the fact is the lock's age)
|
|
if [ -f .git/index.lock ]; then
|
|
lock_age=$(( $(date +%s) - $(stat -c %Y .git/index.lock 2>/dev/null || stat -f %m .git/index.lock) ))
|
|
if [ "$lock_age" -gt 30 ]; then rm -f .git/index.lock; echo "checkout: removed a stale .git/index.lock (${lock_age}s old) in $dir" >&2; fi
|
|
fi
|
|
git checkout -q -- . 2>/dev/null || true
|
|
# 7 October 2026: the attack rows lost their scratch dirs (attack-f3/, attack-f1-venv/, tools/attack/*/target) to each
|
|
# other's builds, because this clean ran on the shared mirror before every build from any agent and took every untracked
|
|
# directory. A lane's scratch is now spared: the fixed prefixes and the mirror-local .igneum-scratch-spare (SPARE_ARGS,
|
|
# see spare_args above). Never -x here: .git/info/exclude still applies. tools/ci/scratch-spare-check.sh guards this line.
|
|
spare_args "$dir"
|
|
git clean -qfd -e target -e 'target-*' -e '.build-remote-sha-*' -e '.cross-remote-sha-*' -e sccache "${SPARE_ARGS[@]}"
|
|
git fetch -q origin '+refs/heads/*:refs/remotes/origin/*'
|
|
git checkout -q -B "$branch" "$sha"
|
|
git reset -q --hard "$sha"
|
|
# what may remain untracked: target dirs, the sha stamps of build-remote.sh and cross-remote.sh (kept so a build after the
|
|
# checkout knows whether to clean kaspa-build-info), sccache; the first live run after this mode was added failed on a
|
|
# stamp it had itself kept (6 October 2026, 18:14 UTC: the self-test had no stamp file; it has one now)
|
|
# ... at any depth: a repo-kind crate (pool/, igneum-pow/, app/igneum-app/) writes its stamp in its own directory, and the
|
|
# root-anchored pattern of the first version failed the second build of every such crate (6 October 2026, 18:51 UTC, the
|
|
# pool build: "tree not clean after reset: ?? pool/.build-remote-sha-target"; the self-test has the subdirectory case now)
|
|
# ... and since 7 October 2026 a lane's spared scratch, so the test asks the clean itself what it would still remove
|
|
# (tree_left: `git clean -nd` with the same excludes; a spared directory is no longer "not clean")
|
|
local left; left=$(tree_left "$dir" || true)
|
|
[ -z "$left" ] || { echo "checkout: tree not clean after reset at $dir: $left" >&2; return 1; }
|
|
set +e
|
|
}
|
|
|
|
if [ "${1:-}" = --self-test-keeper ]; then
|
|
# the keeper restores a truncated holder line within its interval, and stops at release
|
|
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
|
BR_PID=$$; BR_LABEL="keeper self-test"; got=0; waited=3; SLOTS_DIR="$t"; BR_KEEP_S=1
|
|
holder_line() { printf 'pid %s since %sZ waited %s s: %s\n' "$BR_PID" "$(date -u +%H:%M:%S)" "$1" "$BR_LABEL"; }
|
|
holder_line "$waited" > "$t/build-0"
|
|
keeper_pid=""; keep_line() { local f="$1" w="$2"; ( while kill -0 "$BR_PID" 2>/dev/null; do [ -s "$f" ] || holder_line "$w" > "$f" 2>/dev/null; sleep "${BR_KEEP_S:-20}"; done ) & keeper_pid=$!; }
|
|
keep_line "$t/build-0" "$waited"
|
|
: > "$t/build-0"; sleep 2.5
|
|
grep -q "^pid $$ since .* waited 3 s: keeper self-test$" "$t/build-0" || { echo "keeper self-test: the truncated holder line was NOT restored"; kill "$keeper_pid" 2>/dev/null; exit 1; }
|
|
kill "$keeper_pid" 2>/dev/null; wait "$keeper_pid" 2>/dev/null; : > "$t/build-0"; sleep 2.5
|
|
[ ! -s "$t/build-0" ] || { echo "keeper self-test: the keeper kept writing after release"; exit 1; }
|
|
echo "keeper self-test: a truncated holder line comes back within the interval; nothing is written after release"; exit 0
|
|
fi
|
|
|
|
if [ "${1:-}" = --self-test ]; then
|
|
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
|
git init -q --bare -b master "$t/mirror.git"
|
|
git init -q -b master "$t/src"; git -C "$t/src" -c user.name=t -c user.email=t@t commit -q --allow-empty -m one
|
|
echo a > "$t/src/a.txt"; git -C "$t/src" add a.txt; git -C "$t/src" -c user.name=t -c user.email=t@t commit -q -m two
|
|
git -C "$t/src" push -q "$t/mirror.git" master
|
|
sha1=$(git -C "$t/src" rev-parse HEAD)
|
|
checkout_tree "$t/box" "$t/mirror.git" master "$sha1" || { echo "self-test: first checkout failed"; exit 1; }
|
|
# the overlay of a build: a tracked file edited, an untracked file added, a target dir that must survive
|
|
echo edited > "$t/box/a.txt"; echo new > "$t/box/b.txt"; mkdir -p "$t/box/target/release"; echo bin > "$t/box/target/release/x"; echo "$sha1" > "$t/box/.build-remote-sha-target"
|
|
# a crate in a subdirectory: its stamp and target dir must survive, its untracked overlay file must not
|
|
mkdir -p "$t/box/sub/target/release"; echo bin > "$t/box/sub/target/release/y"; echo "$sha1" > "$t/box/sub/.build-remote-sha-target"; echo new > "$t/box/sub/c.txt"
|
|
# a lane's scratch (7 October 2026): a fixed-prefix dir at the root and nested, a dir declared in .igneum-scratch-spare
|
|
# (comments and blank lines in the file), and an undeclared dir that the clean must still take
|
|
mkdir -p "$t/box/attack-f3" "$t/box/sub/attack-f1-venv" "$t/box/bs-lane-1" "$t/box/undeclared-1"
|
|
echo x > "$t/box/attack-f3/x"; echo x > "$t/box/sub/attack-f1-venv/x"; echo x > "$t/box/bs-lane-1/x"; echo x > "$t/box/undeclared-1/x"
|
|
printf '# the build-server lane\n\n bs-lane-* # trailing comment\n' > "$t/box/.igneum-scratch-spare"
|
|
: > "$t/box/.git/index.lock" # a run killed mid-git leaves this; the checkout mode removes it once it is older than 30 s
|
|
touch -t "$(date -d '-2 min' +%Y%m%d%H%M.%S 2>/dev/null || date -v-2M +%Y%m%d%H%M.%S)" "$t/box/.git/index.lock" # backdated two minutes (GNU date, then BSD date)
|
|
[ $(( $(date +%s) - $(stat -c %Y "$t/box/.git/index.lock" 2>/dev/null || stat -f %m "$t/box/.git/index.lock") )) -gt 30 ] || { echo "self-test: could not backdate the fixture lock"; exit 1; }
|
|
# the Mac moves on: a new commit that changes a.txt
|
|
echo a2 > "$t/src/a.txt"; git -C "$t/src" -c user.name=t -c user.email=t@t commit -qam three; git -C "$t/src" push -q "$t/mirror.git" master
|
|
sha2=$(git -C "$t/src" rev-parse HEAD)
|
|
if (cd "$t/box" && git fetch -q origin && git checkout -q -B master "$sha2" 2>/dev/null); then echo "self-test: the plain checkout did NOT refuse on the dirty tree (the case no longer reproduces; the reset is still right)"; else echo "self-test: the plain checkout refuses on the dirty tree, as on 6 October"; fi
|
|
checkout_tree "$t/box" "$t/mirror.git" master "$sha2" || { echo "self-test: checkout mode FAILED on the dirty tree"; exit 1; }
|
|
[ "$(git -C "$t/box" rev-parse HEAD)" = "$sha2" ] || { echo "self-test: wrong commit"; exit 1; }
|
|
[ "$(cat "$t/box/a.txt")" = a2 ] || { echo "self-test: tracked edit survived"; exit 1; }
|
|
[ ! -e "$t/box/b.txt" ] || { echo "self-test: untracked overlay file survived"; exit 1; }
|
|
[ -f "$t/box/target/release/x" ] || { echo "self-test: target dir was cleaned"; exit 1; }
|
|
[ -f "$t/box/.build-remote-sha-target" ] || { echo "self-test: the sha stamp was cleaned"; exit 1; }
|
|
[ -f "$t/box/sub/.build-remote-sha-target" ] || { echo "self-test: a subdirectory crate's sha stamp was cleaned"; exit 1; }
|
|
[ -f "$t/box/sub/target/release/y" ] || { echo "self-test: a subdirectory crate's target dir was cleaned"; exit 1; }
|
|
[ ! -e "$t/box/sub/c.txt" ] || { echo "self-test: a subdirectory's untracked overlay file survived"; exit 1; }
|
|
# a lane's scratch (7 October 2026): fixed prefixes at any depth and a declared glob survive, the spare file survives, an
|
|
# undeclared dir is removed
|
|
[ -f "$t/box/attack-f3/x" ] || { echo "self-test: a fixed-prefix scratch dir (attack-*) was cleaned"; exit 1; }
|
|
[ -f "$t/box/sub/attack-f1-venv/x" ] || { echo "self-test: a nested fixed-prefix scratch dir was cleaned"; exit 1; }
|
|
[ -f "$t/box/bs-lane-1/x" ] || { echo "self-test: a scratch dir declared in .igneum-scratch-spare was cleaned"; exit 1; }
|
|
[ -f "$t/box/.igneum-scratch-spare" ] || { echo "self-test: the .igneum-scratch-spare file itself was cleaned"; exit 1; }
|
|
[ ! -e "$t/box/undeclared-1" ] || { echo "self-test: an undeclared scratch dir survived the clean"; exit 1; }
|
|
# the known-failed case: an untracked file the overlay left that no rule keeps must fail the check
|
|
echo stray > "$t/box/sub/stray.txt"
|
|
spare_args "$t/box"; left=$(tree_left "$t/box"); [ -n "$left" ] || { echo "self-test: the clean-tree check did NOT fire on a stray untracked file"; exit 1; }
|
|
rm -f "$t/box/sub/stray.txt"
|
|
# ... and a spared directory alone must NOT fire it (the check asks the clean, not the status list)
|
|
left=$(tree_left "$t/box"); [ -z "$left" ] || { echo "self-test: the clean-tree check fired on spared scratch: $left"; exit 1; }
|
|
[ ! -f "$t/box/.git/index.lock" ] || { echo "self-test: the stale index.lock survived"; exit 1; }
|
|
echo "self-test: checkout mode lands on the new commit with a clean tree, target dirs and sha stamps kept at any depth, declared and fixed-prefix scratch kept, an undeclared dir removed, stale index.lock removed, and fires on a stray file"; exit 0
|
|
fi
|
|
|
|
if [ "${1:-}" = --self-test-slots ]; then
|
|
me="${IGNEUM_REMOTE_RUN_UNDER_TEST:-$0}"
|
|
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
|
mkdir -p "$t/locks" "$t/log" "$t/dir"; echo 2 > "$t/locks/slots"
|
|
fake() { # <name> <seconds> [BR_MEASURE=1]: a fake run that records its start and end epoch and the job count it was given
|
|
local name="$1" secs="$2" measure="${3:-0}"
|
|
IGNEUM_BUILD_SLOTS_DIR="$t/locks" IGNEUM_BUILD_LOG_DIR="$t/log" BR_MEASURE="$measure" BR_CORES="${BR_CORES:-0}" BR_NICE="${BR_NICE:-0}" BR_DIR="$t/dir" BR_CMD="date +%s.%N > '$t/$name.start'; echo JOBS=\${CARGO_BUILD_JOBS:-none} > '$t/$name.jobs'; sleep $secs; date +%s.%N > '$t/$name.end'" \
|
|
BR_LABEL="self-test $name" BR_TOOL=self-test BR_KIND=other BR_WT="wt-$name" BR_CRATE=t BR_BRANCH=t BR_SHA=0 BR_AGENT=self-test BR_COMMAND="fake $name" \
|
|
bash "$me" >"$t/$name.out" 2>&1
|
|
}
|
|
fail() { echo "self-test-slots: FAIL: $*"; exit 1; }
|
|
after() { python3 -c "import sys; sys.exit(0 if float(open(sys.argv[1]).read()) >= float(open(sys.argv[2]).read()) else 1)" "$1" "$2"; }
|
|
# 1. two concurrent builds: 45 jobs each
|
|
fake a 3 & fake b 3 & wait
|
|
[ "$(cat "$t/a.jobs")" = JOBS=45 ] && [ "$(cat "$t/b.jobs")" = JOBS=45 ] || fail "two concurrent builds got $(cat "$t/a.jobs" "$t/b.jobs" | tr '\n' ' ') (want JOBS=45 JOBS=45)"
|
|
# 2. one build alone: 90
|
|
fake c 1
|
|
[ "$(cat "$t/c.jobs")" = JOBS=90 ] || fail "a lone build got $(cat "$t/c.jobs") (want JOBS=90)"
|
|
# 3. a quiet measurement blocks an unbounded build (it starts only after the quiet ended) and lets a bounded suite run beside it
|
|
fake m 3 1 & sleep 0.5; fake d 1 & BR_CORES=1 BR_NICE=10 fake s 1 & wait
|
|
after "$t/d.start" "$t/m.end" || fail "an unbounded build started while a quiet measurement held the box (build start $(cat "$t/d.start"), quiet end $(cat "$t/m.end"))"
|
|
python3 -c "import sys; sys.exit(0 if float(open('$t/s.start').read()) < float(open('$t/m.end').read()) else 1)" || fail "a bounded suite waited for the quiet measurement"
|
|
[ "$(cat "$t/m.jobs")" = JOBS=none ] || fail "a measurement was given a job count"
|
|
grep -q 'owner=self-test' "$t/m.out" "$t/log/builds.jsonl" 2>/dev/null || fail "the quiet holder line lacks its owner"
|
|
# 4. a quiet measurement is REFUSED (exit 73) while a build holds a slot or a core is leased
|
|
fake e 3 & sleep 0.5; fake n 1 1; rc=$?; wait
|
|
[ "$rc" = 73 ] && [ ! -f "$t/n.start" ] || fail "a quiet measurement was not refused while a build ran (rc $rc)"
|
|
( exec 9>>"$t/locks/core-7"; flock 9; sleep 2 ) & sleep 0.3; fake o 1 1; rc=$?; wait
|
|
[ "$rc" = 73 ] || fail "a quiet measurement was not refused while a core was leased (rc $rc)"
|
|
# 4b. a run keeps off leased cores: with core 1 leased, a 2-core bounded run on a 2-core box says so (the exclusion line)
|
|
( exec 9>>"$t/locks/core-$(( $(nproc) - 1 ))"; flock 9; sleep 2 ) & sleep 0.3; BR_CORES=2 BR_NICE=10 fake p 1; wait
|
|
grep -q 'are leased to a measurement; this run keeps to' "$t/p.out" || fail "a run beside a leased core did not exclude it: $(cat "$t/p.out" | tail -3)"
|
|
# 5. a probing build leaves a busy slot's holder line intact
|
|
fake f 3 & sleep 1.2; fake g 1 & sleep 0.3
|
|
grep -q 'self-test f' "$t/locks/build-0" || fail "the holder line of the busy slot build-0 was lost when another build probed it: '$(cat "$t/locks/build-0")'"
|
|
wait
|
|
# 6. the log carries the job count and the measure flag
|
|
grep -q '"jobs":45' "$t/log/builds.jsonl" && grep -q '"measure":true' "$t/log/builds.jsonl" || fail "builds.jsonl lacks jobs or measure fields"
|
|
echo "self-test-slots: two concurrent builds 45 each, a lone build 90, a quiet blocks an unbounded build and not a bounded suite, a quiet is refused beside a slot or a lease, a run keeps off leased cores, a probe keeps the holder line, the log carries jobs and measure"; exit 0
|
|
fi
|
|
|
|
# One run per worktree directory at a time (6 October 2026, 19:51:09 UK: two runs of one worktree started in the same second;
|
|
# one found no crate directory while the other's checkout was replacing the tree, exit 2). The checkout and the run each take
|
|
# the worktree's lock (append mode, held to exit) and wait up to 2 h for it instead of dying; the wait is said on stderr.
|
|
wt_lock() { # <worktree name>
|
|
local name="${1//\//_}"
|
|
[ -n "$name" ] || return 0
|
|
mkdir -p "$IGNEUM_BUILD_SLOTS_DIR" 2>/dev/null || return 0
|
|
exec {WT_FD}>>"$IGNEUM_BUILD_SLOTS_DIR/wt-$name.lock" || return 0 # WT_FD is global: the keeper closes it (an orphaned sleep held it 20 s)
|
|
if ! flock -n "$WT_FD"; then
|
|
echo "build-remote: another run holds worktree $1 on this box, waiting for it (up to 2 h)" >&2
|
|
flock -w 7200 "$WT_FD" || { echo "build-remote: gave up waiting for worktree $1 after 2 h" >&2; exit 75; }
|
|
fi
|
|
}
|
|
if [ "${BR_MODE:-run}" = checkout ]; then
|
|
: "${BR_CO_DIR:?}" "${BR_CO_MIRROR:?}" "${BR_CO_BRANCH:?}" "${BR_CO_SHA:?}"
|
|
wt_lock "${BR_CO_WT:-$(basename "$BR_CO_DIR")}"
|
|
checkout_tree "$BR_CO_DIR" "$BR_CO_MIRROR" "$BR_CO_BRANCH" "$BR_CO_SHA" "${BR_CO_WT:-}"; exit $?
|
|
fi
|
|
|
|
: "${BR_DIR:?}" "${BR_CMD:?}" "${BR_LABEL:?}" "${BR_TOOL:?}" "${BR_KIND:?}"
|
|
BR_HOST=$(hostname); BR_PID=$$; BR_T0=$(date +%s)
|
|
export BR_HOST BR_PID BR_T0
|
|
wt_lock "${BR_WT:-}"
|
|
LOG_DIR="${IGNEUM_BUILD_LOG_DIR:-/srv/builds/_log}"; mkdir -p "$LOG_DIR"
|
|
|
|
# jsonlog <exit> <slot> <wait_s> <start> <end> <secs> <compiles> <hits> <misses> <hits_total> <misses_total>
|
|
jsonlog() {
|
|
BR_EXIT="$1" BR_SLOT="$2" BR_WAIT="$3" BR_START="$4" BR_END="$5" BR_SECS="$6" BR_COMPILES="$7" \
|
|
BR_HITS="$8" BR_MISSES="$9" BR_HITS_T="${10}" BR_MISSES_T="${11}" BR_LOG="$LOG_DIR/builds.jsonl" python3 - <<'PY' || echo "build-remote: WARNING the JSONL log line was not written" >&2
|
|
import hashlib, json, os, time
|
|
e = os.environ
|
|
def iso(t):
|
|
return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(int(t))) if t else None
|
|
def num(v):
|
|
try: return int(v)
|
|
except (TypeError, ValueError): return None
|
|
d = {
|
|
"v": 1, "id": f"{e['BR_HOST']}-{e['BR_T0']}-{e['BR_PID']}", "host": e['BR_HOST'], "tool": e['BR_TOOL'],
|
|
"worktree": e.get('BR_WT'), "crate": e.get('BR_CRATE'), "kind": e['BR_KIND'], "command": e.get('BR_COMMAND'),
|
|
"target": e.get('BR_TARGET'), "branch": e.get('BR_BRANCH'), "sha": e.get('BR_SHA'), "label": e['BR_LABEL'],
|
|
"agent": e.get('BR_AGENT'), "slot": num(e['BR_SLOT']), "wait_s": num(e['BR_WAIT']), "queued_at": iso(e['BR_T0']),
|
|
"start": iso(e['BR_START']), "end": iso(e['BR_END']), "secs": num(e['BR_SECS']), "exit": num(e['BR_EXIT']),
|
|
"compiles": num(e['BR_COMPILES']), "jobs": num(e.get('BR_JOBS')), "measure": (e.get('BR_MEASURE') == '1') or None,
|
|
"source_date_epoch": num(e.get('BR_SDE')), "pairs_with": e.get('BR_PAIRS_WITH') or None,
|
|
"nice": num(e.get('BR_NICE')) or 0, "cores": (num(e.get('BR_CORES')) or 0) or os.cpu_count(), "priority": e.get('BR_PRIORITY') or "normal",
|
|
"class": e.get('BR_CLASS') or None, "run_log": e.get('BR_RUN_LOG') or None,
|
|
"route": ({"preferred": num(e.get('BR_ROUTE_PREF')), "box": num(e.get('BR_ROUTE_BOX')), "spilled": e.get('BR_ROUTE_SPILLED') == '1',
|
|
"reason": e.get('BR_ROUTE_REASON') or ""} if e.get('BR_ROUTE_BOX') else None),
|
|
}
|
|
sc = {k: num(e[v]) for k, v in (("hits", "BR_HITS"), ("misses", "BR_MISSES"), ("hits_total", "BR_HITS_T"), ("misses_total", "BR_MISSES_T"))}
|
|
sc = {k: v for k, v in sc.items() if v is not None}
|
|
if sc: d["sccache"] = sc
|
|
try:
|
|
d["load_end"] = [float(x) for x in open('/proc/loadavg').read().split()[:3]]
|
|
except OSError:
|
|
pass
|
|
arts = []
|
|
if d["exit"] == 0:
|
|
for p in e.get('BR_ARTEFACTS', '').split():
|
|
fp = os.path.join(e['BR_DIR'], p)
|
|
if os.path.isfile(fp):
|
|
h = hashlib.sha256()
|
|
with open(fp, 'rb') as f:
|
|
for chunk in iter(lambda: f.read(1 << 20), b''):
|
|
h.update(chunk)
|
|
arts.append({"path": p, "bytes": os.path.getsize(fp), "sha256": h.hexdigest()})
|
|
d["artefacts"] = arts
|
|
d = {k: v for k, v in d.items() if v is not None and v != ""}
|
|
line = json.dumps(d, separators=(',', ':'))
|
|
if len(line) > 4000:
|
|
for k in ("command", "label"):
|
|
if k in d: d[k] = d[k][:200]
|
|
line = json.dumps(d, separators=(',', ':'))
|
|
with open(e['BR_LOG'], 'a') as f:
|
|
f.write(line + "\n")
|
|
PY
|
|
}
|
|
|
|
# redlog <exit> <secs> <class>: one line in the shape tools/ci/red-watch.mjs reads (source "box"; the digest counts it)
|
|
redlog() {
|
|
local f="${IGNEUM_CI_RED_FILE:-/srv/ci-red/red.jsonl}"
|
|
[ -w "$f" ] || [ -w "$(dirname "$f")" ] || { echo "build-remote: note: $f is not writable, the red row is in builds.jsonl only" >&2; return 0; }
|
|
BR_EXIT="$1" BR_SECS="$2" BR_CLASS="$3" BR_RED="$f" python3 - <<'PY' || echo "build-remote: WARNING the red-run line was not written" >&2
|
|
import json, os, time
|
|
e = os.environ
|
|
line = {
|
|
"source": "box", "run_id": f"{e['BR_HOST']}-{e['BR_T0']}-{e['BR_PID']}", "attempt": 1, "workflow": f"box:{e['BR_KIND']}",
|
|
"branch": e.get('BR_BRANCH') or '', "sha": (e.get('BR_SHA') or '')[:7], "event": e.get('BR_TOOL') or '',
|
|
"url": "", "at": time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), "title": (e.get('BR_COMMAND') or '')[:100],
|
|
"failed": [{"job": f"{e.get('BR_WT') or ''}/{e.get('BR_CRATE') or ''}", "conclusion": "failure",
|
|
"step": f"{e['BR_CLASS']}: exit {e['BR_EXIT']} after {e['BR_SECS'] or 0} s"}],
|
|
"class": e['BR_CLASS'], "agent": e.get('BR_AGENT') or '', "run_log": e.get('BR_RUN_LOG') or '', "note": "",
|
|
}
|
|
with open(e['BR_RED'], 'a') as f:
|
|
f.write(json.dumps(line, separators=(',', ':')) + "\n")
|
|
PY
|
|
}
|
|
|
|
SLOTS_DIR="$IGNEUM_BUILD_SLOTS_DIR"
|
|
slots=$(cat "$SLOTS_DIR/slots" 2>/dev/null || echo 1); [ "$slots" -ge 1 ] 2>/dev/null || slots=1
|
|
JOBS_ALONE="${JOBS_ALONE:-90}"; JOBS_SHARED="${JOBS_SHARED:-45}"
|
|
holder_line() { printf 'pid %s since %sZ waited %s s: %s\n' "$BR_PID" "$(date -u +%H:%M:%S)" "$1" "$BR_LABEL"; }
|
|
give_up() { # <what>
|
|
echo "build-remote: gave up waiting for $1 after 2 h" >&2
|
|
BR_CLASS=slot-timeout jsonlog 75 0 $(( $(date +%s) - BR_T0 )) "" "$(date +%s)" "" "" "" "" "" ""
|
|
redlog 75 $(( $(date +%s) - BR_T0 )) slot-timeout
|
|
rm -f "$waitfile"; exit 75
|
|
}
|
|
waitfile="$SLOTS_DIR/wait-$BR_PID"
|
|
# The measure file is RETIRED (main, 7 October 2026, 15:07 UK: one global exclusive flock across unrelated measurements stalled
|
|
# build-1 at load 120 with free slots). A measurement that pins cores takes a lease on those cores only (lease.sh, installed at
|
|
# /srv/builds/_bin/lease), and this runner keeps its command off leased cores (see cores_str below). A WHOLE-BOX quiet measurement
|
|
# (BR_MEASURE=1) is its own class: refused with exit 73 while any slot or core lease is held, capped at 20 minutes, holder line with
|
|
# the owner in $SLOTS_DIR/quiet. An unbounded run (nice 0, the full core set) takes quiet shared and waits for it; a bounded run
|
|
# (BR_CORES > 0: suites, benches, everything on box 2) never takes it.
|
|
held_cores() { local f c out=""; for f in "$SLOTS_DIR"/core-*; do [ -e "$f" ] || continue; c=${f##*/core-}; exec {cfd}>>"$f"; if ! flock -n "$cfd"; then out="$out $c"; fi; exec {cfd}>&-; done; echo "${out# }"; }
|
|
any_slot_held() { local k f; for ((k = 0; k < slots; k++)); do f="$SLOTS_DIR/build-$k"; [ -e "$f" ] || continue; exec {sfd}>>"$f"; if ! flock -n "$sfd"; then exec {sfd}>&-; return 0; fi; exec {sfd}>&-; done; return 1; }
|
|
# append mode: opening a lock file must never truncate the holder line another run wrote into it
|
|
exec {mfd}>>"$SLOTS_DIR/quiet"
|
|
if [ "${BR_MEASURE:-0}" = 1 ]; then
|
|
if any_slot_held; then echo "build-remote: QUIET REFUSED: a build slot is held ($(for f in "$SLOTS_DIR"/build-*; do head -c 100 "$f" 2>/dev/null; done | tr '\n' ' ')); a whole-box measurement needs an idle box" >&2; BR_CLASS=quiet-refused jsonlog 73 0 0 "" "$(date +%s)" "" "" "" "" "" ""; exit 73; fi
|
|
lc=$(held_cores); if [ -n "$lc" ]; then echo "build-remote: QUIET REFUSED: cores $(echo $lc | tr ' ' ',') are leased; a whole-box measurement needs an idle box" >&2; BR_CLASS=quiet-refused jsonlog 73 0 0 "" "$(date +%s)" "" "" "" "" "" ""; exit 73; fi
|
|
if ! flock -n "$mfd"; then echo "build-remote: QUIET REFUSED: another quiet measurement holds the box: $(head -c 160 "$SLOTS_DIR/quiet")" >&2; BR_CLASS=quiet-refused jsonlog 73 0 0 "" "$(date +%s)" "" "" "" "" "" ""; exit 73; fi
|
|
[ -n "${BR_AGENT:-}" ] && [ "$BR_AGENT" != unknown ] || { echo "build-remote: QUIET REFUSED: a whole-box measurement needs a named owner (IGNEUM_AGENT)" >&2; exit 73; }
|
|
waited=$(( $(date +%s) - BR_T0 )); got=measure
|
|
BR_LABEL="quiet (cap 1200 s): $BR_LABEL; owner=$BR_AGENT"; holder_line "$waited" > "$SLOTS_DIR/quiet"
|
|
echo "build-remote: holding the box QUIET on $BR_HOST (waited $waited s; owner $BR_AGENT; cap 20 min; unbounded builds wait, bounded suites run beside it on their bands)" >&2
|
|
BR_CMD="timeout --signal TERM --kill-after 30 1200 bash -c $(printf '%q' "$BR_CMD")"
|
|
else
|
|
if [ "${BR_CORES:-0}" = 0 ] && ! flock -s -n "$mfd"; then
|
|
echo "build-remote: a quiet measurement holds the box, waiting (up to 20 min): $(head -c 160 "$SLOTS_DIR/quiet" 2>/dev/null)" >&2
|
|
holder_line 0 > "$waitfile"; trap 'rm -f "$waitfile"' EXIT
|
|
flock -s -w 1500 "$mfd" || give_up "the quiet measurement to end"
|
|
rm -f "$waitfile"; trap - EXIT
|
|
fi
|
|
# scheduling (main, 7 Oct 2026): a gate announces itself (gate-pending-<pid>) and takes the next free slot; a suite, bench or
|
|
# other non-gate run that has not taken a slot yet yields while any gate-pending marker younger than 15 min exists
|
|
gatefile=""
|
|
if [ "${BR_PRIORITY:-normal}" = gate ]; then gatefile="$SLOTS_DIR/gate-pending-$BR_PID"; holder_line 0 > "$gatefile"; trap 'rm -f "$gatefile"' EXIT
|
|
else
|
|
yt0=$(date +%s)
|
|
while pending=$(find "$SLOTS_DIR" -maxdepth 1 -name 'gate-pending-*' -mmin -15 2>/dev/null | head -1) && [ -n "$pending" ]; do
|
|
[ -f "$waitfile" ] || { echo "build-remote: a gate is queued ($(head -c 120 "$pending")), this ${BR_KIND:-run} yields" >&2; holder_line 0 > "$waitfile"; trap 'rm -f "$waitfile"' EXIT; }
|
|
[ $(( $(date +%s) - yt0 )) -lt 7200 ] || give_up "the queued gate"
|
|
sleep 5
|
|
done
|
|
fi
|
|
got=""
|
|
for k in $(seq 0 $((slots - 1))); do
|
|
exec {fd}>>"$SLOTS_DIR/build-$k"
|
|
if flock -n "$fd"; then got=$k; break; fi
|
|
exec {fd}>&-
|
|
done
|
|
if [ -z "$got" ]; then
|
|
echo "build-remote: all $slots slot(s) busy, waiting (up to 2 h) for build-0: $(head -c 160 "$SLOTS_DIR/build-0" 2>/dev/null)" >&2
|
|
# the queue is visible while it waits (the worker dashboard reads wait-* files; asked for on 6 October 2026): the same line
|
|
# format as a slot file, removed the moment the slot is taken or the wait is given up
|
|
holder_line 0 > "$waitfile"; trap 'rm -f "$waitfile"' EXIT
|
|
exec {fd}>>"$SLOTS_DIR/build-0"
|
|
flock -w 7200 "$fd" || give_up "a slot"
|
|
rm -f "$waitfile"; trap - EXIT
|
|
got=0
|
|
fi
|
|
waited=$(( $(date +%s) - BR_T0 ))
|
|
holder_line "$waited" > "$SLOTS_DIR/build-$got"
|
|
# the job count: let a build that started in the same second take its slot, then count the slots held (this one included)
|
|
sleep 1
|
|
held=0
|
|
for k in $(seq 0 $((slots - 1))); do
|
|
if [ "$k" = "$got" ]; then held=$((held + 1)); continue; fi
|
|
exec {tfd}>>"$SLOTS_DIR/build-$k"
|
|
if flock -n "$tfd"; then flock -u "$tfd"; else held=$((held + 1)); fi
|
|
exec {tfd}>&-
|
|
done
|
|
if [ "$held" -gt 1 ]; then BR_JOBS=$JOBS_SHARED; else BR_JOBS=$JOBS_ALONE; fi
|
|
# the bounded classes cap the jobs (suite and bench: 32 unless the caller passed --priority gate)
|
|
if [ "${BR_JOBS_CAP:-0}" -gt 0 ] && [ "$BR_JOBS" -gt "$BR_JOBS_CAP" ]; then BR_JOBS=$BR_JOBS_CAP; fi
|
|
export CARGO_BUILD_JOBS="$BR_JOBS" BR_JOBS
|
|
[ -n "$gatefile" ] && { rm -f "$gatefile"; trap - EXIT; }
|
|
echo "build-remote: holding build-$got on $BR_HOST (waited $waited s; $held of $slots slots held, CARGO_BUILD_JOBS=$BR_JOBS, kind ${BR_KIND:-other}, nice ${BR_NICE:-0}, cores $( [ "${BR_CORES:-0}" = 0 ] && nproc || echo "$BR_CORES"))" >&2
|
|
fi
|
|
|
|
# The holder keeps its own line (the watcher-trust rule, 7 October 2026 10:39Z: two held slots read EMPTY while two suites ran,
|
|
# because a run from a worktree without last night's append-mode fix still opens a busy sibling's slot file with `>` on every
|
|
# probe). A keeper re-writes the holder line whenever it finds the file empty, every BR_KEEP_S seconds, until release_slot.
|
|
keeper_pid=""
|
|
keep_line() { # <file> <waited>; the keeper also refreshes the file's mtime (a stopped holder's file goes stale) and reaps stopped
|
|
# holders of any lease, quiet or slot after 5 minutes through lease.sh (/srv/builds/_bin/lease reap, one line each in reaped.log)
|
|
local f="$1" w="$2"
|
|
( exec {mfd}>&- {fd}>&- 2>/dev/null; [ -n "${WT_FD:-}" ] && exec {WT_FD}>&-; while kill -0 "$BR_PID" 2>/dev/null; do [ -s "$f" ] || holder_line "$w" > "$f" 2>/dev/null; touch "$f" 2>/dev/null
|
|
[ -x /srv/builds/_bin/lease ] && IGNEUM_BUILD_SLOTS_DIR="$SLOTS_DIR" IGNEUM_BUILD_LOG_DIR="$LOG_DIR" /srv/builds/_bin/lease reap >/dev/null 2>&1
|
|
sleep "${BR_KEEP_S:-20}"; done ) &
|
|
keeper_pid=$!
|
|
}
|
|
if [ "$got" = measure ]; then keep_line "$SLOTS_DIR/quiet" "$waited"; else keep_line "$SLOTS_DIR/build-$got" "$waited"; fi
|
|
release_slot() { [ -n "$keeper_pid" ] && { pkill -P "$keeper_pid" 2>/dev/null; kill "$keeper_pid" 2>/dev/null; wait "$keeper_pid" 2>/dev/null; keeper_pid=""; }; if [ "$got" = measure ]; then : > "$SLOTS_DIR/quiet"; else : > "$SLOTS_DIR/build-$got"; fi; }
|
|
cd "$BR_DIR" || { BR_CLASS=no-dir jsonlog 2 "$got" "$waited" "" "$(date +%s)" "" "" "" "" "" ""; redlog 2 0 no-dir; release_slot; exit 2; }
|
|
|
|
# PRE-FLIGHT for a cargo command (the instant-death class, 6 October 2026): the manifest parses and every -p package exists,
|
|
# answered in about a second from the workspace metadata (no network), before any compile time is spent
|
|
if [[ "$BR_CMD" =~ ^cargo[[:space:]] ]]; then
|
|
pf_class=""; pf_msg=""
|
|
sub=$(printf '%s\n' "$BR_CMD" | awk '{print $2}')
|
|
if ! cargo --list 2>/dev/null | awk 'NR>1 {print $1}' | grep -qx -- "$sub"; then
|
|
pf_class=preflight-subcommand; pf_msg="cargo has no '$sub' subcommand on this box (cargo audit at 21:17 UK on 6 October died this way: install it in provision.sh)"
|
|
elif ! pf_meta=$(cargo metadata --no-deps --format-version 1 2>&1 >/tmp/br-meta-$BR_PID.json); then
|
|
pf_class=preflight-manifest; pf_msg="$pf_meta"
|
|
else
|
|
members=$(python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print("\n".join(p["name"] for p in d["packages"]))' "/tmp/br-meta-$BR_PID.json" 2>/dev/null || true)
|
|
for pkg in $(printf '%s\n' "$BR_CMD" | grep -oE '(^|[[:space:]])(-p|--package)[[:space:]=]+[A-Za-z0-9_.@:-]+' | awk '{print $NF}' | sed -E 's/^(-p|--package)=?//'); do
|
|
grep -qx "$pkg" <<<"$members" && continue
|
|
cargo pkgid --offline -p "$pkg" >/dev/null 2>&1 && continue
|
|
pf_class=preflight-package; pf_msg="package '$pkg' is not in this workspace (and not a dependency): the -p argument names nothing"; break
|
|
done
|
|
# --features pkg/feat (or -F): the feature must exist on that member (the shipper's prove build died in 0 s twice on
|
|
# 6 October, 19:22 and 19:51 UK, with a feature name; nothing kept the message)
|
|
if [ -z "$pf_class" ]; then
|
|
for spec in $(printf '%s\n' "$BR_CMD" | grep -oE '(^|[[:space:]])(-F|--features)[[:space:]=]+[A-Za-z0-9_./@:,-]+' | awk '{print $NF}' | sed -E 's/^(-F|--features)=?//' | tr ',' '\n'); do
|
|
case "$spec" in */*) fpkg="${spec%%/*}"; feat="${spec#*/}" ;; *) continue ;; esac
|
|
has=$(python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); m=[p for p in d["packages"] if p["name"]==sys.argv[2]]; print("member" if not m else ("yes" if sys.argv[3] in m[0]["features"] else "no"))' "/tmp/br-meta-$BR_PID.json" "$fpkg" "$feat" 2>/dev/null || echo yes)
|
|
if [ "$has" = no ]; then pf_class=preflight-feature; pf_msg="package '$fpkg' has no feature '$feat'"; break; fi
|
|
done
|
|
fi
|
|
fi
|
|
rm -f "/tmp/br-meta-$BR_PID.json"
|
|
if [ -n "$pf_class" ]; then
|
|
echo "build-remote: PRE-FLIGHT REFUSED ($pf_class): $pf_msg" >&2
|
|
printf 'build-remote: RESULT rc=3 secs=0 compiles=0 class=%s\n' "$pf_class"
|
|
BR_CLASS=$pf_class jsonlog 3 "$([ "$got" = measure ] && echo "" || echo "$got")" "$waited" "$(date +%s)" "$(date +%s)" 0 0 "" "" "" ""
|
|
redlog 3 0 "$pf_class"; release_slot; exit 3
|
|
fi
|
|
fi
|
|
# reproducible builds (main, 6 October 2026, the 0.3.14 repro): the commit's author time as SOURCE_DATE_EPOCH (mimalloc's
|
|
# __DATE__/__TIME__), UTC; the target dir is fixed per target by the caller (prost's generated code embeds OUT_DIR)
|
|
if [ -n "${BR_SDE:-}" ]; then export SOURCE_DATE_EPOCH="$BR_SDE" TZ=UTC; echo "build-remote: SOURCE_DATE_EPOCH=$BR_SDE TZ=UTC" >&2; fi
|
|
sccache --start-server >/dev/null 2>&1 || true
|
|
stat_field() { sccache --show-stats 2>/dev/null | awk -v key="$1" 'index($0, key) == 1 { print $NF; exit }'; }
|
|
exec_before=$(stat_field "Compile requests executed"); hits_before=$(stat_field "Cache hits "); misses_before=$(stat_field "Cache misses ")
|
|
t1=$(date +%s)
|
|
# in a subshell: a command string that carries `set -e` or `exit` ends only the subshell, never this runner (6 October 2026,
|
|
# workers-remote.sh: both workers built, then the leaked set -e killed the runner before its RESULT line, reported as rc 101)
|
|
# the output is kept on the box (the last 400 lines) so a red row can be read after the agent's terminal is gone; both
|
|
# streams stay where they were for the Mac (stdout to stdout, stderr to stderr), each teed into the run log
|
|
RUN_LOG_DIR="$LOG_DIR/runs"; mkdir -p "$RUN_LOG_DIR"
|
|
BR_RUN_LOG="$RUN_LOG_DIR/$BR_HOST-$BR_T0-$BR_PID.log"; export BR_RUN_LOG
|
|
# the class's nice and core set apply to the command's subshell and everything it starts (renice and taskset on the subshell's own
|
|
# pid, BASHPID; cores are the LAST N of the box's set, so gates and builds keep the first ones to themselves)
|
|
# (since the third slot on build-2, 7 Oct 2026: a bounded run takes the band its SLOT owns, counted from the top: slot 0 the last N
|
|
# cores, slot 1 the N below, slot 2 the N below that, so three bounded runs never share a core; a band below core 0 falls back to
|
|
# the last N)
|
|
ncpu=$(nproc); cores_str="0-$((ncpu - 1))"
|
|
if [ "${BR_CORES:-0}" -gt 0 ] && [ "${BR_CORES}" -lt "$ncpu" ]; then
|
|
band=0; case "${got:-}" in ''|measure) ;; *) band=$got ;; esac
|
|
lo=$((ncpu - BR_CORES * (band + 1))); [ "$lo" -ge 0 ] || lo=$((ncpu - BR_CORES))
|
|
cores_str="$lo-$((lo + BR_CORES - 1))"
|
|
fi
|
|
# leased cores (lease.sh: a pinned measurement's core-<n> flocks) are taken out of this run's set; a set that would be empty keeps
|
|
# its cores (the measurement is told by its own lease line); the exclusion is said once
|
|
if [ "$got" != measure ]; then
|
|
leased=$(held_cores)
|
|
if [ -n "$leased" ]; then
|
|
kept=$(python3 -c '
|
|
import sys
|
|
def expand(s):
|
|
out=set()
|
|
for part in s.split(","):
|
|
a,_,b=part.partition("-"); a=int(a); b=int(b) if b else a; out.update(range(a,b+1))
|
|
return out
|
|
mine=expand(sys.argv[1]); leased=set(int(x) for x in sys.argv[2].split()); keep=sorted(mine-leased)
|
|
if not keep: print(sys.argv[1]); sys.exit()
|
|
runs=[];
|
|
for c in keep:
|
|
if runs and runs[-1][1]==c-1: runs[-1][1]=c
|
|
else: runs.append([c,c])
|
|
print(",".join(f"{a}-{b}" if a!=b else f"{a}" for a,b in runs))' "$cores_str" "$leased")
|
|
[ "$kept" != "$cores_str" ] && echo "build-remote: cores $(echo $leased | tr ' ' ',') are leased to a measurement; this run keeps to $kept" >&2
|
|
cores_str="$kept"
|
|
fi
|
|
fi
|
|
( [ "${BR_NICE:-0}" -gt 0 ] && renice -n "$BR_NICE" -p $BASHPID >/dev/null 2>&1; [ "$cores_str" != "0-$((ncpu - 1))" ] && taskset -cp "$cores_str" $BASHPID >/dev/null 2>&1; eval "$BR_CMD" ) > >(tee -a "$BR_RUN_LOG") 2> >(tee -a "$BR_RUN_LOG" >&2)
|
|
rc=$?
|
|
# the keeper stops BEFORE the bare `wait` (which flushes the two tees): a bare wait also waits for the keeper, and the keeper waits
|
|
# for this script, a deadlock that held build-2's first run 15 minutes after its test had passed (7 Oct 2026, 11:04 to 11:19Z)
|
|
if [ -n "$keeper_pid" ]; then kill "$keeper_pid" 2>/dev/null; wait "$keeper_pid" 2>/dev/null; keeper_pid=""; fi
|
|
wait
|
|
t2=$(date +%s); secs=$(( t2 - t1 ))
|
|
exec_after=$(stat_field "Compile requests executed"); hits_after=$(stat_field "Cache hits "); misses_after=$(stat_field "Cache misses ")
|
|
compiles=$(( ${exec_after:-0} - ${exec_before:-0} )); hits=$(( ${hits_after:-0} - ${hits_before:-0} )); misses=$(( ${misses_after:-0} - ${misses_before:-0} ))
|
|
tail -n 400 "$BR_RUN_LOG" > "$BR_RUN_LOG.tmp" 2>/dev/null && mv -f "$BR_RUN_LOG.tmp" "$BR_RUN_LOG"
|
|
# the class of the run, from its exit, its duration and its output
|
|
BR_CLASS=""
|
|
if [ "$rc" != 0 ]; then
|
|
# what the output says first (a failing test in a tiny crate also runs in under 3 s); instant is the rest
|
|
if grep -qE '^(error\[E[0-9]+\]|error: could not compile)' "$BR_RUN_LOG"; then BR_CLASS=compile-error
|
|
elif grep -qE 'error: linking with|undefined reference to' "$BR_RUN_LOG"; then BR_CLASS=link-error
|
|
elif grep -q 'test result: FAILED' "$BR_RUN_LOG"; then BR_CLASS=test-failure
|
|
elif [ "$secs" -le 2 ] && [ "${compiles:-0}" -le 0 ]; then BR_CLASS=instant
|
|
else BR_CLASS=other; fi
|
|
elif [[ "$BR_CMD" == cargo\ test* ]] && { [[ "$BR_CMD" == *" -- "* ]] || grep -qE -- '--(lib|tests|bins|all-targets)[[:space:]]+[^-[:space:]]' <<<"$BR_CMD"; } \
|
|
&& grep -q '^running 0 tests' "$BR_RUN_LOG" && ! grep -qE '^running [1-9][0-9]* tests?' "$BR_RUN_LOG"; then
|
|
# a filter that matched no test anywhere: the run was a wasted round trip, and the agent would have read "ok"
|
|
BR_CLASS=no-test-matched; rc=3
|
|
echo "build-remote: REFUSED after the run: the test filter matched no test in any binary (every 'running 0 tests'); check the name" >&2
|
|
fi
|
|
export BR_CLASS
|
|
printf 'build-remote: RESULT rc=%s secs=%s compiles=%s sccache_hits=%s sccache_misses=%s sccache_hits_total=%s sccache_misses_total=%s jobs=%s nice=%s cores=%s load=%s class=%s\n' \
|
|
"$rc" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-?}" "${misses_after:-?}" "${BR_JOBS:-measure}" "${BR_NICE:-0}" "$( [ "${BR_CORES:-0}" = 0 ] && nproc || echo "$BR_CORES")" "$(cut -d' ' -f1-3 /proc/loadavg)" "${BR_CLASS:-ok}"
|
|
jsonlog "$rc" "$([ "$got" = measure ] && echo "" || echo "$got")" "$waited" "$t1" "$t2" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-}" "${misses_after:-}"
|
|
[ "$rc" = 0 ] || redlog "$rc" "$secs" "$BR_CLASS"
|
|
release_slot
|
|
exit "$rc"
|