b2262e5dc6
The box's 34 red rows of 6 October classified (docs/analysis/ci-failures-2026-10-06.md section 6): 22 iterations, 12 in three real classes (instant deaths with nothing kept, a shared worktree directory, an unread dry run). remote-run.sh now: pre-flight (subcommand, manifest, -p package, --features) refuses in a second with exit 3 and a class; the last 400 lines of every run kept in /srv/builds/_log/runs; a class on every row (compile-error, link-error, test-failure, instant, no-test-matched, slot-timeout, no-dir, preflight-*); a cargo test whose filter matched no test exits 3; a per-worktree lock in checkout and run mode; every red row appended to /srv/ci-red/red.jsonl as source box. red-watch.mjs never posts a box row alone and sends one digest a day (counts per class with each class's guard); the timer runs tick. Shared group cired on the box so the runner and build append to one file. Shown in a sandbox on the box: pass, failing test, empty filter, bad package, bad feature, missing subcommand, compile error, broken manifest, two concurrent runs of one worktree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
393 lines
29 KiB
Bash
Executable file
393 lines
29 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# The remote half of tools/build-remote.sh and tools/cross-remote.sh. It runs ON igneum-build-1, fed by lib.sh bs_remote_run
|
|
# over `ssh build@<box> bash -s` with these exports prepended (never run it by hand on the Mac):
|
|
# BR_DIR the crate directory on the box BR_CMD the shell string to run there (cargo ...)
|
|
# BR_LABEL the slot-file label (ends with "; agent=<name>")
|
|
# BR_TOOL build-remote | cross-remote BR_KIND node-linux | node-windows | app | app-windows | prove | suite | check | other
|
|
# BR_WT the worktree name BR_CRATE the crate path relative to the worktree root
|
|
# BR_COMMAND the cargo command as typed BR_TARGET the rust target triple
|
|
# BR_BRANCH BR_SHA BR_AGENT the agent name (IGNEUM_AGENT on the Mac, else the worktree)
|
|
# BR_ARTEFACTS space-separated paths (relative to BR_DIR) the Mac will fetch; empty for test, check, clippy
|
|
#
|
|
# 1. Takes a build slot: flock on $IGNEUM_BUILD_SLOTS_DIR/build-<k> for k below the count in .../slots (the box's own slot
|
|
# files, never the Mac's; 2 since main's ruling of 6 October 2026, 20:3x UK); when every slot is busy it waits up to 2 h on
|
|
# build-0 and exits 75 if it gives up. The holder line is `pid N since HH:MM:SSZ waited S s: <label>`, the format
|
|
# tools/lock/with-lock.sh writes on the Mac. The cargo job count follows the slots: after one second of settling, a build
|
|
# that holds the only taken slot gets CARGO_BUILD_JOBS=90, one that sees the other slot held gets 45 (JOBS_ALONE and
|
|
# JOBS_SHARED), so two builds share the 96 threads without thrashing; a `-j` on the cargo line (--jobs on the Mac) wins.
|
|
# A MEASUREMENT (BR_MEASURE=1: the CPU prover timing, bench rows) takes the `measure` file exclusively and excludes every
|
|
# build, as with-lock.sh's `measure` does on the Mac: builds hold `measure` shared for their whole run, so a measure waits
|
|
# for the running builds and blocks new ones until it ends. Lock files are opened in APPEND mode: the first version opened
|
|
# them with `>` and truncated a busy slot's holder line every time another build probed it (found 6 October 2026, evening).
|
|
# 2. Runs BR_CMD in BR_DIR with sccache, prints one `build-remote: RESULT rc= secs= compiles= sccache_hits_total= ...` line.
|
|
# 3. Appends one JSON line to /srv/builds/_log/builds.jsonl (the worker dashboard reads it; asked for by main on 6 October
|
|
# 2026): on success, on failure and on the slot give-up. UTC ISO 8601 Z times, numbers unquoted, unknown fields omitted,
|
|
# artefacts with bytes and sha256 only when the run succeeded (a failed build would list the previous build's files),
|
|
# the line kept under 4 KB. Since the evening of 6 October 2026 (the project lead: "make sure we are fixing and learning from all the
|
|
# errors here") the line also carries "class" and "run_log":
|
|
# - PRE-FLIGHT before cargo runs: the manifest must parse (cargo metadata --no-deps) and every `-p` package must exist
|
|
# (a workspace member, else cargo pkgid --offline); a refusal is exit 3, class preflight-manifest or preflight-package,
|
|
# and costs a second instead of a slot. The instant-death class: three suite runs on 6 October died in 0 s with exit
|
|
# 101 and no compile, and nothing on the box had kept the reason.
|
|
# - the run's output is kept: the last 400 lines in /srv/builds/_log/runs/<id>.log, so a red row can be read after the
|
|
# agent's terminal is gone.
|
|
# - CLASS of a red run from that output: instant (under 3 s, nothing compiled), compile-error (error[E...] or "could not
|
|
# compile"), link-error, test-failure ("test result: FAILED"), slot-timeout (75), no-dir (2), other.
|
|
# - a `cargo test` with a filter that ran 0 tests everywhere is a wasted round trip: exit 3, class no-test-matched.
|
|
# - every red row is also appended to the shared red-run file (/srv/ci-red/red.jsonl, $IGNEUM_CI_RED_FILE), the file
|
|
# tools/ci/red-watch.mjs posts from; box rows are not posted one by one, the 09:00 UK digest counts them per class.
|
|
# Modes (BR_MODE, default run): `checkout` resets the box's tree and checks the branch out at the commit (lib.sh
|
|
# bs_push_and_checkout: BR_CO_DIR, BR_CO_MIRROR, BR_CO_BRANCH, BR_CO_SHA, BR_CO_WT); `--self-test` (first argument) builds a
|
|
# scratch mirror and clone, dirties the clone the way a build's overlay does, moves the mirror one commit on, and shows the
|
|
# checkout mode lands on the new commit with a clean tree (the 6 October 2026 case: a stale overlay made `git checkout -B`
|
|
# refuse with "local changes would be overwritten"); `--self-test-slots` runs fake builds and a fake measurement against a
|
|
# scratch slots directory: two concurrent builds get 45 jobs each, one alone gets 90, a measure blocks a build and a build
|
|
# blocks a measure, and a probing build leaves a busy slot's holder line intact (IGNEUM_REMOTE_RUN_UNDER_TEST=<script> runs
|
|
# the cases against another copy, which is how the old script was shown to fail them).
|
|
set -uo pipefail
|
|
# the profile sets the box's paths; an IGNEUM_BUILD_SLOTS_DIR or IGNEUM_BUILD_LOG_DIR already in the environment wins (the slot
|
|
# self-test runs against a scratch directory; the first version let the profile reset it and the test took the REAL slot)
|
|
_slots_env="${IGNEUM_BUILD_SLOTS_DIR:-}"; _log_env="${IGNEUM_BUILD_LOG_DIR:-}"
|
|
[ -f /etc/profile.d/igneum-build.sh ] && . /etc/profile.d/igneum-build.sh
|
|
[ -n "$_slots_env" ] && IGNEUM_BUILD_SLOTS_DIR="$_slots_env"; [ -n "$_log_env" ] && IGNEUM_BUILD_LOG_DIR="$_log_env"
|
|
|
|
# discard the previous overlay (tracked edits and untracked files; target dirs, the sha stamps and anything ignored are kept),
|
|
# fetch, then the branch at the commit. Runs in BR_CO_DIR, clones it from BR_CO_MIRROR when it has no .git.
|
|
checkout_tree() {
|
|
local dir="$1" mirror="$2" branch="$3" sha="$4" wt="${5:-}"
|
|
set -e
|
|
[ -n "$wt" ] && mkdir -p "$wt"
|
|
if [ ! -d "$dir/.git" ]; then rm -rf "$dir"; git clone -q --no-checkout "$mirror" "$dir"; fi
|
|
cd "$dir"
|
|
# a run killed mid-git (two runs on one worktree, 18:48:56Z the same day) leaves .git/index.lock; stale when it is older
|
|
# than 30 s (an index write takes milliseconds, a checkout of the fork seconds). The first version asked `pgrep -x git`,
|
|
# which said "in use" whenever ANY git ran on the machine: the pre-push hook's own `git push` and any other agent's build
|
|
# kept the lock and the checkout died with "index.lock: File exists" (6 October 2026, 22:2x UK; the fact is the lock's age)
|
|
if [ -f .git/index.lock ]; then
|
|
lock_age=$(( $(date +%s) - $(stat -c %Y .git/index.lock 2>/dev/null || stat -f %m .git/index.lock) ))
|
|
if [ "$lock_age" -gt 30 ]; then rm -f .git/index.lock; echo "checkout: removed a stale .git/index.lock (${lock_age}s old) in $dir" >&2; fi
|
|
fi
|
|
git checkout -q -- . 2>/dev/null || true
|
|
git clean -qfd -e target -e 'target-*' -e '.build-remote-sha-*' -e '.cross-remote-sha-*' -e sccache
|
|
git fetch -q origin '+refs/heads/*:refs/remotes/origin/*'
|
|
git checkout -q -B "$branch" "$sha"
|
|
git reset -q --hard "$sha"
|
|
# what may remain untracked: target dirs, the sha stamps of build-remote.sh and cross-remote.sh (kept so a build after the
|
|
# checkout knows whether to clean kaspa-build-info), sccache; the first live run after this mode was added failed on a
|
|
# stamp it had itself kept (6 October 2026, 18:14 UTC: the self-test had no stamp file; it has one now)
|
|
# ... at any depth: a repo-kind crate (pool/, igneum-pow/, app/igneum-app/) writes its stamp in its own directory, and the
|
|
# root-anchored pattern of the first version failed the second build of every such crate (6 October 2026, 18:51 UTC, the
|
|
# pool build: "tree not clean after reset: ?? pool/.build-remote-sha-target"; the self-test has the subdirectory case now)
|
|
local left; left=$(git status --porcelain --untracked-files=all | grep -vE '^\?\? (.*/)?(target|target-|sccache|\.build-remote-sha-|\.cross-remote-sha-)' | head -3 || true)
|
|
[ -z "$left" ] || { echo "checkout: tree not clean after reset at $dir: $left" >&2; return 1; }
|
|
set +e
|
|
}
|
|
|
|
if [ "${1:-}" = --self-test ]; then
|
|
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
|
git init -q --bare -b master "$t/mirror.git"
|
|
git init -q -b master "$t/src"; git -C "$t/src" -c user.name=t -c user.email=t@t commit -q --allow-empty -m one
|
|
echo a > "$t/src/a.txt"; git -C "$t/src" add a.txt; git -C "$t/src" -c user.name=t -c user.email=t@t commit -q -m two
|
|
git -C "$t/src" push -q "$t/mirror.git" master
|
|
sha1=$(git -C "$t/src" rev-parse HEAD)
|
|
checkout_tree "$t/box" "$t/mirror.git" master "$sha1" || { echo "self-test: first checkout failed"; exit 1; }
|
|
# the overlay of a build: a tracked file edited, an untracked file added, a target dir that must survive
|
|
echo edited > "$t/box/a.txt"; echo new > "$t/box/b.txt"; mkdir -p "$t/box/target/release"; echo bin > "$t/box/target/release/x"; echo "$sha1" > "$t/box/.build-remote-sha-target"
|
|
# a crate in a subdirectory: its stamp and target dir must survive, its untracked overlay file must not
|
|
mkdir -p "$t/box/sub/target/release"; echo bin > "$t/box/sub/target/release/y"; echo "$sha1" > "$t/box/sub/.build-remote-sha-target"; echo new > "$t/box/sub/c.txt"
|
|
: > "$t/box/.git/index.lock" # a run killed mid-git leaves this; the checkout mode removes it once it is older than 30 s
|
|
touch -t "$(date -d '-2 min' +%Y%m%d%H%M.%S 2>/dev/null || date -v-2M +%Y%m%d%H%M.%S)" "$t/box/.git/index.lock" # backdated two minutes (GNU date, then BSD date)
|
|
[ $(( $(date +%s) - $(stat -c %Y "$t/box/.git/index.lock" 2>/dev/null || stat -f %m "$t/box/.git/index.lock") )) -gt 30 ] || { echo "self-test: could not backdate the fixture lock"; exit 1; }
|
|
# the Mac moves on: a new commit that changes a.txt
|
|
echo a2 > "$t/src/a.txt"; git -C "$t/src" -c user.name=t -c user.email=t@t commit -qam three; git -C "$t/src" push -q "$t/mirror.git" master
|
|
sha2=$(git -C "$t/src" rev-parse HEAD)
|
|
if (cd "$t/box" && git fetch -q origin && git checkout -q -B master "$sha2" 2>/dev/null); then echo "self-test: the plain checkout did NOT refuse on the dirty tree (the case no longer reproduces; the reset is still right)"; else echo "self-test: the plain checkout refuses on the dirty tree, as on 6 October"; fi
|
|
checkout_tree "$t/box" "$t/mirror.git" master "$sha2" || { echo "self-test: checkout mode FAILED on the dirty tree"; exit 1; }
|
|
[ "$(git -C "$t/box" rev-parse HEAD)" = "$sha2" ] || { echo "self-test: wrong commit"; exit 1; }
|
|
[ "$(cat "$t/box/a.txt")" = a2 ] || { echo "self-test: tracked edit survived"; exit 1; }
|
|
[ ! -e "$t/box/b.txt" ] || { echo "self-test: untracked overlay file survived"; exit 1; }
|
|
[ -f "$t/box/target/release/x" ] || { echo "self-test: target dir was cleaned"; exit 1; }
|
|
[ -f "$t/box/.build-remote-sha-target" ] || { echo "self-test: the sha stamp was cleaned"; exit 1; }
|
|
[ -f "$t/box/sub/.build-remote-sha-target" ] || { echo "self-test: a subdirectory crate's sha stamp was cleaned"; exit 1; }
|
|
[ -f "$t/box/sub/target/release/y" ] || { echo "self-test: a subdirectory crate's target dir was cleaned"; exit 1; }
|
|
[ ! -e "$t/box/sub/c.txt" ] || { echo "self-test: a subdirectory's untracked overlay file survived"; exit 1; }
|
|
# the known-failed case: an untracked file the overlay left that no rule keeps must fail the check
|
|
echo stray > "$t/box/sub/stray.txt"
|
|
if (cd "$t/box" && left=$(git status --porcelain --untracked-files=all | grep -vE '^\?\? (.*/)?(target|target-|sccache|\.build-remote-sha-|\.cross-remote-sha-)' | head -3); [ -n "$left" ]); then :; else echo "self-test: the clean-tree check did NOT fire on a stray untracked file"; exit 1; fi
|
|
rm -f "$t/box/sub/stray.txt"
|
|
[ ! -f "$t/box/.git/index.lock" ] || { echo "self-test: the stale index.lock survived"; exit 1; }
|
|
echo "self-test: checkout mode lands on the new commit with a clean tree, target dirs and sha stamps kept at any depth, stale index.lock removed, and fires on a stray file"; exit 0
|
|
fi
|
|
|
|
if [ "${1:-}" = --self-test-slots ]; then
|
|
me="${IGNEUM_REMOTE_RUN_UNDER_TEST:-$0}"
|
|
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
|
mkdir -p "$t/locks" "$t/log" "$t/dir"; echo 2 > "$t/locks/slots"
|
|
fake() { # <name> <seconds> [BR_MEASURE=1]: a fake run that records its start and end epoch and the job count it was given
|
|
local name="$1" secs="$2" measure="${3:-0}"
|
|
IGNEUM_BUILD_SLOTS_DIR="$t/locks" IGNEUM_BUILD_LOG_DIR="$t/log" BR_MEASURE="$measure" BR_DIR="$t/dir" BR_CMD="date +%s.%N > '$t/$name.start'; echo JOBS=\${CARGO_BUILD_JOBS:-none} > '$t/$name.jobs'; sleep $secs; date +%s.%N > '$t/$name.end'" \
|
|
BR_LABEL="self-test $name" BR_TOOL=self-test BR_KIND=other BR_WT=t BR_CRATE=t BR_BRANCH=t BR_SHA=0 BR_AGENT=self-test BR_COMMAND="fake $name" \
|
|
bash "$me" >"$t/$name.out" 2>&1
|
|
}
|
|
fail() { echo "self-test-slots: FAIL: $*"; exit 1; }
|
|
after() { python3 -c "import sys; sys.exit(0 if float(open(sys.argv[1]).read()) >= float(open(sys.argv[2]).read()) else 1)" "$1" "$2"; }
|
|
# 1. two concurrent builds: 45 jobs each
|
|
fake a 3 & fake b 3 & wait
|
|
[ "$(cat "$t/a.jobs")" = JOBS=45 ] && [ "$(cat "$t/b.jobs")" = JOBS=45 ] || fail "two concurrent builds got $(cat "$t/a.jobs" "$t/b.jobs" | tr '\n' ' ') (want JOBS=45 JOBS=45)"
|
|
# 2. one build alone: 90
|
|
fake c 1
|
|
[ "$(cat "$t/c.jobs")" = JOBS=90 ] || fail "a lone build got $(cat "$t/c.jobs") (want JOBS=90)"
|
|
# 3. a measure blocks a build: the build starts only after the measure ended
|
|
fake m 3 1 & sleep 0.5; fake d 1 & wait
|
|
after "$t/d.start" "$t/m.end" || fail "a build started while a measurement held the box (build start $(cat "$t/d.start"), measure end $(cat "$t/m.end"))"
|
|
[ "$(cat "$t/m.jobs")" = JOBS=none ] || fail "a measurement was given a job count"
|
|
# 4. a build blocks a measure: the measure starts only after the build ended
|
|
fake e 3 & sleep 0.5; fake n 1 1 & wait
|
|
after "$t/n.start" "$t/e.end" || fail "a measurement started while a build ran (measure start $(cat "$t/n.start"), build end $(cat "$t/e.end"))"
|
|
# 5. a probing build leaves a busy slot's holder line intact
|
|
fake f 3 & sleep 1.2; fake g 1 & sleep 0.3
|
|
grep -q 'self-test f' "$t/locks/build-0" || fail "the holder line of the busy slot build-0 was lost when another build probed it: '$(cat "$t/locks/build-0")'"
|
|
wait
|
|
# 6. the log carries the job count and the measure flag
|
|
grep -q '"jobs":45' "$t/log/builds.jsonl" && grep -q '"measure":true' "$t/log/builds.jsonl" || fail "builds.jsonl lacks jobs or measure fields"
|
|
echo "self-test-slots: two concurrent builds 45 each, a lone build 90, a measure blocks a build, a build blocks a measure, a probe keeps the holder line, the log carries jobs and measure"; exit 0
|
|
fi
|
|
|
|
# One run per worktree directory at a time (6 October 2026, 19:51:09 UK: two runs of one worktree started in the same second;
|
|
# one found no crate directory while the other's checkout was replacing the tree, exit 2). The checkout and the run each take
|
|
# the worktree's lock (append mode, held to exit) and wait up to 2 h for it instead of dying; the wait is said on stderr.
|
|
wt_lock() { # <worktree name>
|
|
local name="${1//\//_}" fd
|
|
[ -n "$name" ] || return 0
|
|
mkdir -p "$IGNEUM_BUILD_SLOTS_DIR" 2>/dev/null || return 0
|
|
exec {fd}>>"$IGNEUM_BUILD_SLOTS_DIR/wt-$name.lock" || return 0
|
|
if ! flock -n "$fd"; then
|
|
echo "build-remote: another run holds worktree $1 on this box, waiting for it (up to 2 h)" >&2
|
|
flock -w 7200 "$fd" || { echo "build-remote: gave up waiting for worktree $1 after 2 h" >&2; exit 75; }
|
|
fi
|
|
}
|
|
if [ "${BR_MODE:-run}" = checkout ]; then
|
|
: "${BR_CO_DIR:?}" "${BR_CO_MIRROR:?}" "${BR_CO_BRANCH:?}" "${BR_CO_SHA:?}"
|
|
wt_lock "${BR_CO_WT:-$(basename "$BR_CO_DIR")}"
|
|
checkout_tree "$BR_CO_DIR" "$BR_CO_MIRROR" "$BR_CO_BRANCH" "$BR_CO_SHA" "${BR_CO_WT:-}"; exit $?
|
|
fi
|
|
|
|
: "${BR_DIR:?}" "${BR_CMD:?}" "${BR_LABEL:?}" "${BR_TOOL:?}" "${BR_KIND:?}"
|
|
BR_HOST=$(hostname); BR_PID=$$; BR_T0=$(date +%s)
|
|
export BR_HOST BR_PID BR_T0
|
|
wt_lock "${BR_WT:-}"
|
|
LOG_DIR="${IGNEUM_BUILD_LOG_DIR:-/srv/builds/_log}"; mkdir -p "$LOG_DIR"
|
|
|
|
# jsonlog <exit> <slot> <wait_s> <start> <end> <secs> <compiles> <hits> <misses> <hits_total> <misses_total>
|
|
jsonlog() {
|
|
BR_EXIT="$1" BR_SLOT="$2" BR_WAIT="$3" BR_START="$4" BR_END="$5" BR_SECS="$6" BR_COMPILES="$7" \
|
|
BR_HITS="$8" BR_MISSES="$9" BR_HITS_T="${10}" BR_MISSES_T="${11}" BR_LOG="$LOG_DIR/builds.jsonl" python3 - <<'PY' || echo "build-remote: WARNING the JSONL log line was not written" >&2
|
|
import hashlib, json, os, time
|
|
e = os.environ
|
|
def iso(t):
|
|
return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(int(t))) if t else None
|
|
def num(v):
|
|
try: return int(v)
|
|
except (TypeError, ValueError): return None
|
|
d = {
|
|
"v": 1, "id": f"{e['BR_HOST']}-{e['BR_T0']}-{e['BR_PID']}", "host": e['BR_HOST'], "tool": e['BR_TOOL'],
|
|
"worktree": e.get('BR_WT'), "crate": e.get('BR_CRATE'), "kind": e['BR_KIND'], "command": e.get('BR_COMMAND'),
|
|
"target": e.get('BR_TARGET'), "branch": e.get('BR_BRANCH'), "sha": e.get('BR_SHA'), "label": e['BR_LABEL'],
|
|
"agent": e.get('BR_AGENT'), "slot": num(e['BR_SLOT']), "wait_s": num(e['BR_WAIT']), "queued_at": iso(e['BR_T0']),
|
|
"start": iso(e['BR_START']), "end": iso(e['BR_END']), "secs": num(e['BR_SECS']), "exit": num(e['BR_EXIT']),
|
|
"compiles": num(e['BR_COMPILES']), "jobs": num(e.get('BR_JOBS')), "measure": (e.get('BR_MEASURE') == '1') or None,
|
|
"source_date_epoch": num(e.get('BR_SDE')),
|
|
"class": e.get('BR_CLASS') or None, "run_log": e.get('BR_RUN_LOG') or None,
|
|
}
|
|
sc = {k: num(e[v]) for k, v in (("hits", "BR_HITS"), ("misses", "BR_MISSES"), ("hits_total", "BR_HITS_T"), ("misses_total", "BR_MISSES_T"))}
|
|
sc = {k: v for k, v in sc.items() if v is not None}
|
|
if sc: d["sccache"] = sc
|
|
try:
|
|
d["load_end"] = [float(x) for x in open('/proc/loadavg').read().split()[:3]]
|
|
except OSError:
|
|
pass
|
|
arts = []
|
|
if d["exit"] == 0:
|
|
for p in e.get('BR_ARTEFACTS', '').split():
|
|
fp = os.path.join(e['BR_DIR'], p)
|
|
if os.path.isfile(fp):
|
|
h = hashlib.sha256()
|
|
with open(fp, 'rb') as f:
|
|
for chunk in iter(lambda: f.read(1 << 20), b''):
|
|
h.update(chunk)
|
|
arts.append({"path": p, "bytes": os.path.getsize(fp), "sha256": h.hexdigest()})
|
|
d["artefacts"] = arts
|
|
d = {k: v for k, v in d.items() if v is not None and v != ""}
|
|
line = json.dumps(d, separators=(',', ':'))
|
|
if len(line) > 4000:
|
|
for k in ("command", "label"):
|
|
if k in d: d[k] = d[k][:200]
|
|
line = json.dumps(d, separators=(',', ':'))
|
|
with open(e['BR_LOG'], 'a') as f:
|
|
f.write(line + "\n")
|
|
PY
|
|
}
|
|
|
|
# redlog <exit> <secs> <class>: one line in the shape tools/ci/red-watch.mjs reads (source "box"; the digest counts it)
|
|
redlog() {
|
|
local f="${IGNEUM_CI_RED_FILE:-/srv/ci-red/red.jsonl}"
|
|
[ -w "$f" ] || [ -w "$(dirname "$f")" ] || { echo "build-remote: note: $f is not writable, the red row is in builds.jsonl only" >&2; return 0; }
|
|
BR_EXIT="$1" BR_SECS="$2" BR_CLASS="$3" BR_RED="$f" python3 - <<'PY' || echo "build-remote: WARNING the red-run line was not written" >&2
|
|
import json, os, time
|
|
e = os.environ
|
|
line = {
|
|
"source": "box", "run_id": f"{e['BR_HOST']}-{e['BR_T0']}-{e['BR_PID']}", "attempt": 1, "workflow": f"box:{e['BR_KIND']}",
|
|
"branch": e.get('BR_BRANCH') or '', "sha": (e.get('BR_SHA') or '')[:7], "event": e.get('BR_TOOL') or '',
|
|
"url": "", "at": time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), "title": (e.get('BR_COMMAND') or '')[:100],
|
|
"failed": [{"job": f"{e.get('BR_WT') or ''}/{e.get('BR_CRATE') or ''}", "conclusion": "failure",
|
|
"step": f"{e['BR_CLASS']}: exit {e['BR_EXIT']} after {e['BR_SECS'] or 0} s"}],
|
|
"class": e['BR_CLASS'], "agent": e.get('BR_AGENT') or '', "run_log": e.get('BR_RUN_LOG') or '', "note": "",
|
|
}
|
|
with open(e['BR_RED'], 'a') as f:
|
|
f.write(json.dumps(line, separators=(',', ':')) + "\n")
|
|
PY
|
|
}
|
|
|
|
SLOTS_DIR="$IGNEUM_BUILD_SLOTS_DIR"
|
|
slots=$(cat "$SLOTS_DIR/slots" 2>/dev/null || echo 1); [ "$slots" -ge 1 ] 2>/dev/null || slots=1
|
|
JOBS_ALONE="${JOBS_ALONE:-90}"; JOBS_SHARED="${JOBS_SHARED:-45}"
|
|
holder_line() { printf 'pid %s since %sZ waited %s s: %s\n' "$BR_PID" "$(date -u +%H:%M:%S)" "$1" "$BR_LABEL"; }
|
|
give_up() { # <what>
|
|
echo "build-remote: gave up waiting for $1 after 2 h" >&2
|
|
BR_CLASS=slot-timeout jsonlog 75 0 $(( $(date +%s) - BR_T0 )) "" "$(date +%s)" "" "" "" "" "" ""
|
|
redlog 75 $(( $(date +%s) - BR_T0 )) slot-timeout
|
|
rm -f "$waitfile"; exit 75
|
|
}
|
|
waitfile="$SLOTS_DIR/wait-$BR_PID"
|
|
# append mode: opening a lock file must never truncate the holder line another run wrote into it
|
|
exec {mfd}>>"$SLOTS_DIR/measure"
|
|
if [ "${BR_MEASURE:-0}" = 1 ]; then
|
|
# a measurement: the measure file exclusively; every running build holds it shared, so this waits for them and blocks new ones
|
|
if ! flock -n "$mfd"; then
|
|
echo "build-remote: measure waits for the running build(s) (up to 2 h): $(for f in "$SLOTS_DIR"/build-*; do head -c 120 "$f" 2>/dev/null; done | tr '\n' ' ')" >&2
|
|
holder_line 0 > "$waitfile"; trap 'rm -f "$waitfile"' EXIT
|
|
flock -w 7200 "$mfd" || give_up "the measure hold"
|
|
rm -f "$waitfile"; trap - EXIT
|
|
fi
|
|
waited=$(( $(date +%s) - BR_T0 )); got=measure
|
|
holder_line "$waited" > "$SLOTS_DIR/measure"
|
|
echo "build-remote: holding measure on $BR_HOST (waited $waited s; builds are excluded until this run ends)" >&2
|
|
else
|
|
# a build: the measure file shared (a running measurement blocks us), then one exclusive slot
|
|
if ! flock -s -n "$mfd"; then
|
|
echo "build-remote: a measurement holds the box, waiting (up to 2 h): $(head -c 160 "$SLOTS_DIR/measure" 2>/dev/null)" >&2
|
|
holder_line 0 > "$waitfile"; trap 'rm -f "$waitfile"' EXIT
|
|
flock -s -w 7200 "$mfd" || give_up "the measurement to end"
|
|
rm -f "$waitfile"; trap - EXIT
|
|
fi
|
|
got=""
|
|
for k in $(seq 0 $((slots - 1))); do
|
|
exec {fd}>>"$SLOTS_DIR/build-$k"
|
|
if flock -n "$fd"; then got=$k; break; fi
|
|
exec {fd}>&-
|
|
done
|
|
if [ -z "$got" ]; then
|
|
echo "build-remote: all $slots slot(s) busy, waiting (up to 2 h) for build-0: $(head -c 160 "$SLOTS_DIR/build-0" 2>/dev/null)" >&2
|
|
# the queue is visible while it waits (the worker dashboard reads wait-* files; asked for on 6 October 2026): the same line
|
|
# format as a slot file, removed the moment the slot is taken or the wait is given up
|
|
holder_line 0 > "$waitfile"; trap 'rm -f "$waitfile"' EXIT
|
|
exec {fd}>>"$SLOTS_DIR/build-0"
|
|
flock -w 7200 "$fd" || give_up "a slot"
|
|
rm -f "$waitfile"; trap - EXIT
|
|
got=0
|
|
fi
|
|
waited=$(( $(date +%s) - BR_T0 ))
|
|
holder_line "$waited" > "$SLOTS_DIR/build-$got"
|
|
# the job count: let a build that started in the same second take its slot, then count the slots held (this one included)
|
|
sleep 1
|
|
held=0
|
|
for k in $(seq 0 $((slots - 1))); do
|
|
if [ "$k" = "$got" ]; then held=$((held + 1)); continue; fi
|
|
exec {tfd}>>"$SLOTS_DIR/build-$k"
|
|
if flock -n "$tfd"; then flock -u "$tfd"; else held=$((held + 1)); fi
|
|
exec {tfd}>&-
|
|
done
|
|
if [ "$held" -gt 1 ]; then BR_JOBS=$JOBS_SHARED; else BR_JOBS=$JOBS_ALONE; fi
|
|
export CARGO_BUILD_JOBS="$BR_JOBS" BR_JOBS
|
|
echo "build-remote: holding build-$got on $BR_HOST (waited $waited s; $held of $slots slots held, CARGO_BUILD_JOBS=$BR_JOBS)" >&2
|
|
fi
|
|
|
|
release_slot() { if [ "$got" = measure ]; then : > "$SLOTS_DIR/measure"; else : > "$SLOTS_DIR/build-$got"; fi; }
|
|
cd "$BR_DIR" || { BR_CLASS=no-dir jsonlog 2 "$got" "$waited" "" "$(date +%s)" "" "" "" "" "" ""; redlog 2 0 no-dir; release_slot; exit 2; }
|
|
|
|
# PRE-FLIGHT for a cargo command (the instant-death class, 6 October 2026): the manifest parses and every -p package exists,
|
|
# answered in about a second from the workspace metadata (no network), before any compile time is spent
|
|
if [[ "$BR_CMD" =~ ^cargo[[:space:]] ]]; then
|
|
pf_class=""; pf_msg=""
|
|
sub=$(printf '%s\n' "$BR_CMD" | awk '{print $2}')
|
|
if ! cargo --list 2>/dev/null | awk 'NR>1 {print $1}' | grep -qx -- "$sub"; then
|
|
pf_class=preflight-subcommand; pf_msg="cargo has no '$sub' subcommand on this box (cargo audit at 21:17 UK on 6 October died this way: install it in provision.sh)"
|
|
elif ! pf_meta=$(cargo metadata --no-deps --format-version 1 2>&1 >/tmp/br-meta-$BR_PID.json); then
|
|
pf_class=preflight-manifest; pf_msg="$pf_meta"
|
|
else
|
|
members=$(python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print("\n".join(p["name"] for p in d["packages"]))' "/tmp/br-meta-$BR_PID.json" 2>/dev/null || true)
|
|
for pkg in $(printf '%s\n' "$BR_CMD" | grep -oE '(^|[[:space:]])(-p|--package)[[:space:]=]+[A-Za-z0-9_.@:-]+' | awk '{print $NF}' | sed -E 's/^(-p|--package)=?//'); do
|
|
grep -qx "$pkg" <<<"$members" && continue
|
|
cargo pkgid --offline -p "$pkg" >/dev/null 2>&1 && continue
|
|
pf_class=preflight-package; pf_msg="package '$pkg' is not in this workspace (and not a dependency): the -p argument names nothing"; break
|
|
done
|
|
# --features pkg/feat (or -F): the feature must exist on that member (the shipper's prove build died in 0 s twice on
|
|
# 6 October, 19:22 and 19:51 UK, with a feature name; nothing kept the message)
|
|
if [ -z "$pf_class" ]; then
|
|
for spec in $(printf '%s\n' "$BR_CMD" | grep -oE '(^|[[:space:]])(-F|--features)[[:space:]=]+[A-Za-z0-9_./@:,-]+' | awk '{print $NF}' | sed -E 's/^(-F|--features)=?//' | tr ',' '\n'); do
|
|
case "$spec" in */*) fpkg="${spec%%/*}"; feat="${spec#*/}" ;; *) continue ;; esac
|
|
has=$(python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); m=[p for p in d["packages"] if p["name"]==sys.argv[2]]; print("member" if not m else ("yes" if sys.argv[3] in m[0]["features"] else "no"))' "/tmp/br-meta-$BR_PID.json" "$fpkg" "$feat" 2>/dev/null || echo yes)
|
|
if [ "$has" = no ]; then pf_class=preflight-feature; pf_msg="package '$fpkg' has no feature '$feat'"; break; fi
|
|
done
|
|
fi
|
|
fi
|
|
rm -f "/tmp/br-meta-$BR_PID.json"
|
|
if [ -n "$pf_class" ]; then
|
|
echo "build-remote: PRE-FLIGHT REFUSED ($pf_class): $pf_msg" >&2
|
|
printf 'build-remote: RESULT rc=3 secs=0 compiles=0 class=%s\n' "$pf_class"
|
|
BR_CLASS=$pf_class jsonlog 3 "$([ "$got" = measure ] && echo "" || echo "$got")" "$waited" "$(date +%s)" "$(date +%s)" 0 0 "" "" "" ""
|
|
redlog 3 0 "$pf_class"; release_slot; exit 3
|
|
fi
|
|
fi
|
|
# reproducible builds (main, 6 October 2026, the 0.3.14 repro): the commit's author time as SOURCE_DATE_EPOCH (mimalloc's
|
|
# __DATE__/__TIME__), UTC; the target dir is fixed per target by the caller (prost's generated code embeds OUT_DIR)
|
|
if [ -n "${BR_SDE:-}" ]; then export SOURCE_DATE_EPOCH="$BR_SDE" TZ=UTC; echo "build-remote: SOURCE_DATE_EPOCH=$BR_SDE TZ=UTC" >&2; fi
|
|
sccache --start-server >/dev/null 2>&1 || true
|
|
stat_field() { sccache --show-stats 2>/dev/null | awk -v key="$1" 'index($0, key) == 1 { print $NF; exit }'; }
|
|
exec_before=$(stat_field "Compile requests executed"); hits_before=$(stat_field "Cache hits "); misses_before=$(stat_field "Cache misses ")
|
|
t1=$(date +%s)
|
|
# in a subshell: a command string that carries `set -e` or `exit` ends only the subshell, never this runner (6 October 2026,
|
|
# workers-remote.sh: both workers built, then the leaked set -e killed the runner before its RESULT line, reported as rc 101)
|
|
# the output is kept on the box (the last 400 lines) so a red row can be read after the agent's terminal is gone; both
|
|
# streams stay where they were for the Mac (stdout to stdout, stderr to stderr), each teed into the run log
|
|
RUN_LOG_DIR="$LOG_DIR/runs"; mkdir -p "$RUN_LOG_DIR"
|
|
BR_RUN_LOG="$RUN_LOG_DIR/$BR_HOST-$BR_T0-$BR_PID.log"; export BR_RUN_LOG
|
|
( eval "$BR_CMD" ) > >(tee -a "$BR_RUN_LOG") 2> >(tee -a "$BR_RUN_LOG" >&2)
|
|
rc=$?
|
|
wait
|
|
t2=$(date +%s); secs=$(( t2 - t1 ))
|
|
exec_after=$(stat_field "Compile requests executed"); hits_after=$(stat_field "Cache hits "); misses_after=$(stat_field "Cache misses ")
|
|
compiles=$(( ${exec_after:-0} - ${exec_before:-0} )); hits=$(( ${hits_after:-0} - ${hits_before:-0} )); misses=$(( ${misses_after:-0} - ${misses_before:-0} ))
|
|
tail -n 400 "$BR_RUN_LOG" > "$BR_RUN_LOG.tmp" 2>/dev/null && mv -f "$BR_RUN_LOG.tmp" "$BR_RUN_LOG"
|
|
# the class of the run, from its exit, its duration and its output
|
|
BR_CLASS=""
|
|
if [ "$rc" != 0 ]; then
|
|
# what the output says first (a failing test in a tiny crate also runs in under 3 s); instant is the rest
|
|
if grep -qE '^(error\[E[0-9]+\]|error: could not compile)' "$BR_RUN_LOG"; then BR_CLASS=compile-error
|
|
elif grep -qE 'error: linking with|undefined reference to' "$BR_RUN_LOG"; then BR_CLASS=link-error
|
|
elif grep -q 'test result: FAILED' "$BR_RUN_LOG"; then BR_CLASS=test-failure
|
|
elif [ "$secs" -le 2 ] && [ "${compiles:-0}" -le 0 ]; then BR_CLASS=instant
|
|
else BR_CLASS=other; fi
|
|
elif [[ "$BR_CMD" == cargo\ test* ]] && { [[ "$BR_CMD" == *" -- "* ]] || grep -qE -- '--(lib|tests|bins|all-targets)[[:space:]]+[^-[:space:]]' <<<"$BR_CMD"; } \
|
|
&& grep -q '^running 0 tests' "$BR_RUN_LOG" && ! grep -qE '^running [1-9][0-9]* tests?' "$BR_RUN_LOG"; then
|
|
# a filter that matched no test anywhere: the run was a wasted round trip, and the agent would have read "ok"
|
|
BR_CLASS=no-test-matched; rc=3
|
|
echo "build-remote: REFUSED after the run: the test filter matched no test in any binary (every 'running 0 tests'); check the name" >&2
|
|
fi
|
|
export BR_CLASS
|
|
printf 'build-remote: RESULT rc=%s secs=%s compiles=%s sccache_hits=%s sccache_misses=%s sccache_hits_total=%s sccache_misses_total=%s jobs=%s load=%s class=%s\n' \
|
|
"$rc" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-?}" "${misses_after:-?}" "${BR_JOBS:-measure}" "$(cut -d' ' -f1-3 /proc/loadavg)" "${BR_CLASS:-ok}"
|
|
jsonlog "$rc" "$([ "$got" = measure ] && echo "" || echo "$got")" "$waited" "$t1" "$t2" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-}" "${misses_after:-}"
|
|
[ "$rc" = 0 ] || redlog "$rc" "$secs" "$BR_CLASS"
|
|
release_slot
|
|
exit "$rc"
|