igneum/tools/ci/prover-socket-check.sh
igneum-labs de28429f91 Kill by exact command line or pid file, never by a name: tools/ci/kill-by-name-check.sh in the gate; the 36 pgrep/pkill literals in the tree fixed
The fleet's 22:09 UK incident (a Mac-side pkill -f <log file name> matched nothing, the roll-everything script lived on and wiped a held box) and the day's two pgrep self-matches are one class. The check flags pgrep -f / pkill -f with a plain literal (every one on a line), any pgrep/pkill on a file-name shape, and ps | grep with a literal; it allows the bracket form, -x, -F pidfile, kill $(cat pidfile), a variable and a full path; 11 banned and 16 allowed shapes in its self-test; 0.15 s over the tree. The 25 pkill -f sp1-gpu-server inside bash -c bodies (which matched the calling bash) are pkill -x; the other 11 literals take the bracket form; prover-socket-check accepts both. Row R in the record; the CLAUDE.md rule names the check and covers pkill and file names.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 22:10:31 +00:00

25 lines
2.3 KiB
Bash
Executable file

#!/usr/bin/env bash
# The root-socket class (5 October 2026, 20:00Z): a PC 2 job ran igneum-prove-host as root inside WSL2, which started
# an sp1-gpu-server whose socket /tmp/sp1-cuda-0.sock stayed root-owned after the job; the live prover (the app's user)
# then failed every shard with "CudaClientError: Connect(PermissionDenied)" until the socket was gone. Rule: every
# playbook that runs the prover host as root on a shared card kills the server AND unlinks its socket (at the start
# and at the end), or runs as the app's user. This check fails CI when a script runs `igneum-prove-host` under a
# `-u root` WSL session without both lines. With file arguments it checks those files only (the jobs publisher runs it
# on the script being published, packaging/ota/publish-jobs.sh add --kind run; a PC job never passes CI before it runs).
set -euo pipefail
cd "$(dirname "$0")/../.."
fail=0
# the publisher's test writes a known-bad root prover script on purpose, to prove the publish refuses it
ALLOW='^(packaging/ota/test-publish-jobs\.sh|tools/amd-prove/pc1-cpu-prove(-sp)?\.ps1)$' # pc1-cpu-prove: the CPU path starts no GPU server (its author's allow entry, 5 October 2026)
list_files() { if [ $# -gt 0 ]; then printf '%s\n' "$@"; else git ls-files 'tools/**' 'relay/playbooks/**' 'proving/**' 'packaging/**' | grep -E '\.(sh|ps1|mjs)$'; fi; }
while IFS= read -r f; do
[[ "$f" =~ $ALLOW ]] && continue
# a playbook that only runs `--mode id` or `--mode verify` starts no GPU server; the prove modes do
if grep -qE 'igneum-prove-host' "$f" && grep -qE -- '--mode (compressed|chain|aggregate|block|shard|all)\b' "$f" && grep -qE -- '-u root' "$f"; then
# -x on the binary name (kill-by-name rule, 6 October 2026: `pkill -x sp1-gpu-server` inside a `bash -c '...'` matched the calling bash's own command line)
if ! grep -qE 'pkill -(x|f) (\[s\]p1|sp1)-gpu-server' "$f"; then echo "prover-socket: $f runs the prover host as root without killing sp1-gpu-server"; fail=1; fi
if ! grep -qE 'rm -f /tmp/sp1-cuda-' "$f"; then echo "prover-socket: $f runs the prover host as root without unlinking /tmp/sp1-cuda-*.sock"; fail=1; fi
fi
done < <(list_files "$@")
[ "$fail" = 0 ] && echo "prover-socket: every root prover playbook kills the GPU server and unlinks its socket"
exit $fail