a0eeb725f9
Thirteen failure emails between 15:26 and 16:53 UK. The classes and what closes them: - the box-locks check on a hosted runner (10 runs): closed bybd6fcb88and165e8b35earlier - windows-ci's stale payload-inputs pin (3 runs): closed on master by 4b4e1bc1; update-return's dispatches still carry e69e8a39 - three hosted site jobs on master hung in the tree gate for over two hours (no timeout-minutes): ci.yml now carries site 15, changes 10, pow 60, sims 45, the overlap sweep runs under a 10-minute wall clock where GNU timeout exists, and tools/ci/workflow-timeouts-check.sh fails a job without a budget (self-test: a job without the key, a wrong budget) - a branch merged with no ci run of its own (era-vdf61421005, 16:31 UK): master's igneum-pow suite went red and five docs-only merges landed green over it because their runs skip the compile job. tools/ci/ci-state.mjs reads the runs API through gh (a commit's newest run, master's last COMPILED run, a branch's last red); merge-to-master.sh pushes an unrun branch for a run, waits for a queued one printing the clock, refuses a red one and refuses any merge onto a red master except the declared fix (--fixes-master); the pre-push hook refuses a push to master whose commit, or whose merge's branch parent, has no green run on that exact sha; a feature-branch push prints the branch's previous red first. Self-tests with a fake gh in all three. - ci-red.yml fires on failure, cancelled and timed_out and hands the conclusion to red-watch.mjs, whose line names the kind (CI red, CI cancelled, CI timed out); the self-test reads the workflow file for the three conclusions - tools/ci/retry-once.sh: one retry before red for the box-locks check, the scene parity check and the live public API check (each keeps its own skip line on a runner without the resource) GitHub's branch protection cannot be applied: the organisation is on the free plan and the repository is private (the API answers 403, "Upgrade to GitHub Pro or make this repository public"), so the two scripts are the enforcement; the rule is one line in CLAUDE.md under the CI block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
37 lines
2.6 KiB
Bash
Executable file
37 lines
2.6 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Run a check that can red on box state or the network twice before calling it red (standing rule, 7 October 2026, 17:2x UK:
|
|
# a check that talks to a build box or a live host gets one retry and, on a runner without the resource, a clear skip line
|
|
# from the check itself). The first failure is printed as a line, the command runs again after a pause, and only the second
|
|
# failure is the check's verdict. A check that passes first time runs once.
|
|
#
|
|
# tools/ci/retry-once.sh <name> <command...> # exit = the last run's exit code; RETRY_ONCE_PAUSE seconds between (default 5)
|
|
# tools/ci/retry-once.sh --self-test # a command that fails once then passes is ok; one that fails twice is red;
|
|
# # a command that passes runs exactly once
|
|
set -uo pipefail
|
|
PAUSE="${RETRY_ONCE_PAUSE:-5}"
|
|
if [ "${1:-}" = --self-test ]; then
|
|
fails=0; d=$(mktemp -d)
|
|
# fails once then passes: the marker file is the memory
|
|
flaky="$d/flaky.sh"; printf '#!/usr/bin/env bash\nif [ -f "%s/seen" ]; then exit 0; fi; touch "%s/seen"; echo first-try-failed; exit 7\n' "$d" "$d" > "$flaky"; chmod +x "$flaky"
|
|
out=$(RETRY_ONCE_PAUSE=0 bash "$0" flaky "$flaky" 2>&1); rc=$?
|
|
[ "$rc" = 0 ] || { echo "self-test failed: a command that fails once then passes was red (exit $rc): $out"; fails=1; }
|
|
case "$out" in *"retry-once: flaky failed once (exit 7)"*) ;; *) echo "self-test failed: the first failure was not printed: $out"; fails=1 ;; esac
|
|
# fails twice: red with the second exit code
|
|
out=$(RETRY_ONCE_PAUSE=0 bash "$0" dead bash -c 'echo nope; exit 3' 2>&1); rc=$?
|
|
[ "$rc" = 3 ] || { echo "self-test failed: a command that fails twice was not red with its exit code (got $rc): $out"; fails=1; }
|
|
[ "$(printf '%s\n' "$out" | grep -c '^nope$')" = 2 ] || { echo "self-test failed: the command did not run exactly twice: $out"; fails=1; }
|
|
# passes: runs once
|
|
c="$d/count"; : > "$c"
|
|
RETRY_ONCE_PAUSE=0 bash "$0" fine bash -c "echo x >> '$c'" >/dev/null 2>&1 || { echo "self-test failed: a passing command was red"; fails=1; }
|
|
[ "$(wc -l < "$c" | tr -d ' ')" = 1 ] || { echo "self-test failed: a passing command ran $(wc -l < "$c") times"; fails=1; }
|
|
rm -rf "$d"
|
|
[ "$fails" = 0 ] && echo "self-test passed: one retry after a first failure, red only on the second, a passing command runs once"
|
|
exit $fails
|
|
fi
|
|
[ $# -ge 2 ] || { echo "usage: tools/ci/retry-once.sh <name> <command...> | --self-test" >&2; exit 2; }
|
|
NAME="$1"; shift
|
|
"$@"; rc=$?
|
|
[ "$rc" = 0 ] && exit 0
|
|
echo "retry-once: $NAME failed once (exit $rc); second try in ${PAUSE}s (a box or network check gets one retry before it is red)"
|
|
sleep "$PAUSE"
|
|
"$@"
|