a0eeb725f9
Thirteen failure emails between 15:26 and 16:53 UK. The classes and what closes them: - the box-locks check on a hosted runner (10 runs): closed bybd6fcb88and165e8b35earlier - windows-ci's stale payload-inputs pin (3 runs): closed on master by 4b4e1bc1; update-return's dispatches still carry e69e8a39 - three hosted site jobs on master hung in the tree gate for over two hours (no timeout-minutes): ci.yml now carries site 15, changes 10, pow 60, sims 45, the overlap sweep runs under a 10-minute wall clock where GNU timeout exists, and tools/ci/workflow-timeouts-check.sh fails a job without a budget (self-test: a job without the key, a wrong budget) - a branch merged with no ci run of its own (era-vdf61421005, 16:31 UK): master's igneum-pow suite went red and five docs-only merges landed green over it because their runs skip the compile job. tools/ci/ci-state.mjs reads the runs API through gh (a commit's newest run, master's last COMPILED run, a branch's last red); merge-to-master.sh pushes an unrun branch for a run, waits for a queued one printing the clock, refuses a red one and refuses any merge onto a red master except the declared fix (--fixes-master); the pre-push hook refuses a push to master whose commit, or whose merge's branch parent, has no green run on that exact sha; a feature-branch push prints the branch's previous red first. Self-tests with a fake gh in all three. - ci-red.yml fires on failure, cancelled and timed_out and hands the conclusion to red-watch.mjs, whose line names the kind (CI red, CI cancelled, CI timed out); the self-test reads the workflow file for the three conclusions - tools/ci/retry-once.sh: one retry before red for the box-locks check, the scene parity check and the live public API check (each keeps its own skip line on a runner without the resource) GitHub's branch protection cannot be applied: the organisation is on the free plan and the repository is private (the API answers 403, "Upgrade to GitHub Pro or make this repository public"), so the two scripts are the enforcement; the rule is one line in CLAUDE.md under the CI block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
53 lines
3.6 KiB
Bash
Executable file
53 lines
3.6 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Every job in .github/workflows carries timeout-minutes, and the named jobs carry their budgets (7 October 2026, 15:13 to
|
|
# 17:20 UK: three hosted `site` jobs on master hung in the tree gate for over two hours each, with GitHub's six-hour default
|
|
# as the only stop; a fourth would have been a failure email at 21:13 UK). The budgets, from the measured times on
|
|
# 7 October: the tree gate on a hosted runner 229 s plus a 40 s Playwright install, so `site` 15; the classifier `changes`
|
|
# a 7 s API call, so 10; the box's igneum-pow suite 45 s to 2 min 40 s, so `pow` 60; the two simulators 2 x 120 s, so `sims` 45.
|
|
#
|
|
# tools/ci/workflow-timeouts-check.sh # exit 1 naming each job without a timeout or with the wrong budget
|
|
# tools/ci/workflow-timeouts-check.sh --self-test # a fixture job without the key fails; a wrong budget fails; the tree passes
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")/../.."
|
|
|
|
# <file> -> lines "job timeout" for every job (timeout "-" when missing). A job is a key at four spaces under `jobs:`;
|
|
# its timeout-minutes is the key at six spaces before the next job.
|
|
jobs_of() {
|
|
awk '
|
|
/^jobs:/ { injobs = 1; next }
|
|
injobs && /^[^ ]/ { injobs = 0 }
|
|
injobs && /^ [A-Za-z0-9_-]+:/ { if (job != "") print job, (t == "" ? "-" : t); job = $1; sub(":", "", job); t = ""; next }
|
|
injobs && job != "" && /^ timeout-minutes:/ { t = $2 }
|
|
END { if (job != "") print job, (t == "" ? "-" : t) }
|
|
' "$1"
|
|
}
|
|
check_tree() { # <dir with workflows> -> exit 1 with one line per wrong job
|
|
local f rc=0 job t want
|
|
for f in "$1"/*.yml "$1"/*.yaml; do
|
|
[ -f "$f" ] || continue
|
|
while read -r job t; do
|
|
if [ "$t" = "-" ]; then echo "workflow-timeouts: $(basename "$f") job \`$job\` has no timeout-minutes" >&2; rc=1; continue; fi
|
|
want=""
|
|
case "$(basename "$f"):$job" in
|
|
ci.yml:site) want=15 ;; ci.yml:changes) want=10 ;; ci.yml:pow) want=60 ;; ci.yml:sims) want=45 ;;
|
|
esac
|
|
if [ -n "$want" ] && [ "$t" != "$want" ]; then echo "workflow-timeouts: $(basename "$f") job \`$job\` has timeout-minutes $t, the budget is $want" >&2; rc=1; fi
|
|
done < <(jobs_of "$f")
|
|
done
|
|
return $rc
|
|
}
|
|
if [ "${1:-}" = --self-test ]; then
|
|
fails=0; d=$(mktemp -d); mkdir -p "$d/bad" "$d/budget"
|
|
printf 'name: x\non: push\njobs:\n a:\n runs-on: ubuntu-latest\n timeout-minutes: 5\n steps: []\n b:\n runs-on: ubuntu-latest\n steps: []\n' > "$d/bad/ci.yml"
|
|
out=$(check_tree "$d/bad" 2>&1) && { echo "self-test failed: a job without timeout-minutes passed"; fails=1; }
|
|
case "$out" in *'job `b` has no timeout-minutes'*) ;; *) echo "self-test failed: the job without a timeout was not named: $out"; fails=1 ;; esac
|
|
case "$out" in *'job `a`'*) echo "self-test failed: a job with a timeout was named: $out"; fails=1 ;; esac
|
|
printf 'name: ci\non: push\njobs:\n site:\n runs-on: ubuntu-latest\n timeout-minutes: 360\n steps: []\n' > "$d/budget/ci.yml"
|
|
out=$(check_tree "$d/budget" 2>&1) && { echo "self-test failed: site at 360 minutes passed"; fails=1; }
|
|
case "$out" in *'has timeout-minutes 360, the budget is 15'*) ;; *) echo "self-test failed: the wrong budget was not named: $out"; fails=1 ;; esac
|
|
rm -rf "$d"
|
|
check_tree .github/workflows || { echo "self-test failed: the tree's workflows do not pass"; fails=1; }
|
|
[ "$fails" = 0 ] && echo "self-test passed: a job without timeout-minutes fails, a wrong budget fails, the tree's workflows pass (site 15, changes 10, pow 60, sims 45)"
|
|
exit $fails
|
|
fi
|
|
check_tree .github/workflows && echo "workflow-timeouts: every job in .github/workflows carries timeout-minutes (site 15, changes 10, pow 60, sims 45)"
|