b3c16fa068
Thirteen failure emails between 15:26 and 16:53 UK. The classes and what closes them: - the box-locks check on a hosted runner (10 runs): closed bycbf57294andb1d8e4b5earlier - windows-ci's stale payload-inputs pin (3 runs): closed on master by 958037d3; update-return's dispatches still carry e69e8a39 - three hosted site jobs on master hung in the tree gate for over two hours (no timeout-minutes): ci.yml now carries site 15, changes 10, pow 60, sims 45, the overlap sweep runs under a 10-minute wall clock where GNU timeout exists, and tools/ci/workflow-timeouts-check.sh fails a job without a budget (self-test: a job without the key, a wrong budget) - a branch merged with no ci run of its own (era-vdfe792962b, 16:31 UK): master's igneum-pow suite went red and five docs-only merges landed green over it because their runs skip the compile job. tools/ci/ci-state.mjs reads the runs API through gh (a commit's newest run, master's last COMPILED run, a branch's last red); merge-to-master.sh pushes an unrun branch for a run, waits for a queued one printing the clock, refuses a red one and refuses any merge onto a red master except the declared fix (--fixes-master); the pre-push hook refuses a push to master whose commit, or whose merge's branch parent, has no green run on that exact sha; a feature-branch push prints the branch's previous red first. Self-tests with a fake gh in all three. - ci-red.yml fires on failure, cancelled and timed_out and hands the conclusion to red-watch.mjs, whose line names the kind (CI red, CI cancelled, CI timed out); the self-test reads the workflow file for the three conclusions - tools/ci/retry-once.sh: one retry before red for the box-locks check, the scene parity check and the live public API check (each keeps its own skip line on a runner without the resource) GitHub's branch protection cannot be applied: the organisation is on the free plan and the repository is private (the API answers 403, "Upgrade to GitHub Pro or make this repository public"), so the two scripts are the enforcement; the rule is one line in CLAUDE.md under the CI block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
48 lines
3.4 KiB
YAML
48 lines
3.4 KiB
YAML
# The red watcher as its own workflow, on workflow_run, so the copy on master watches EVERY branch's ci run whatever
|
|
# ci.yml that branch carries: GitHub runs a workflow_run workflow from the default branch only, and the branch's own
|
|
# ci.yml never enters it (7 October 2026: the inline `red` job of ci.yml was conditioned on master and release-*, and
|
|
# a feature branch would have waited for a merge of master before its reds were posted at all).
|
|
#
|
|
# One line per failed, cancelled or timed-out run (tools/ci/red-watch.mjs record, idempotent per run attempt) to /srv/ci-red/red.jsonl on the
|
|
# box; the box's igneum-ci-red.timer posts each new line once to the hidden updates channel, naming the branch, the
|
|
# commit, the red check and the pushing author. Runs on the box's own runner (not a GitHub-hosted machine: the billing
|
|
# block of 6 October 2026, 18:37Z to 20:10Z, failed every hosted job at start and nobody was told). Never blocks a
|
|
# release: it reads the run, writes one line, and ends.
|
|
name: ci-red
|
|
on:
|
|
workflow_run:
|
|
workflows: [ci]
|
|
types: [completed]
|
|
jobs:
|
|
red:
|
|
name: red watcher (every branch; one line per failed run, with the branch, commit, red check and pushing author, to the updates channel and the box file)
|
|
# failure, and since 7 October 2026 (17:2x UK) cancelled and timed_out too: a job that hangs into its timeout-minutes or a run
|
|
# someone cancels is a run that never answered, and a lane reads it like a red (tools/ci/red-watch.mjs names the kind)
|
|
if: ${{ github.event.workflow_run.conclusion == 'failure' || github.event.workflow_run.conclusion == 'cancelled' || github.event.workflow_run.conclusion == 'timed_out' }}
|
|
# the label ci-red is on igneum-build-1 only (added through the runners API on 7 October 2026; the default of
|
|
# RUNNER_LABELS in provision.sh carries it): the record file and the poster (igneum-ci-red.timer, the webhook file)
|
|
# live on that box, and the pool label igneum-build-1 is shared with igneum-build-2 since the same day
|
|
runs-on: [self-hosted, linux, x64, ci-red]
|
|
timeout-minutes: 5
|
|
permissions:
|
|
actions: read # the failed run's jobs API (the first real red run, 21:19Z on 6 October: the default token answered 403 and the line carried no step)
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
sparse-checkout: tools/ci
|
|
- name: record the failed run (one line, the branch, the commit, the failed jobs and their first failed step from the run's own API, the pushing author)
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
RED_WATCH_RUN_ID: ${{ github.event.workflow_run.id }}
|
|
RED_WATCH_ATTEMPT: ${{ github.event.workflow_run.run_attempt }}
|
|
RED_WATCH_WORKFLOW: ${{ github.event.workflow_run.name }}
|
|
RED_WATCH_CONCLUSION: ${{ github.event.workflow_run.conclusion }}
|
|
RED_WATCH_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
|
RED_WATCH_SHA: ${{ github.event.workflow_run.head_sha }}
|
|
RED_WATCH_EVENT: ${{ github.event.workflow_run.event }}
|
|
RED_WATCH_URL: ${{ github.event.workflow_run.html_url }}
|
|
RED_WATCH_ACTOR: ${{ github.event.workflow_run.actor.login }}
|
|
RED_WATCH_TITLE: ${{ github.event.workflow_run.head_commit.message }}
|
|
RED_WATCH_AUTHOR: ${{ github.event.workflow_run.head_commit.author.name }}
|
|
run: node tools/ci/red-watch.mjs record --file /srv/ci-red/red.jsonl
|