Thirteen failure emails between 15:26 and 16:53 UK. The classes and what closes them: - the box-locks check on a hosted runner (10 runs): closed bybd6fcb88and165e8b35earlier - windows-ci's stale payload-inputs pin (3 runs): closed on master by 4b4e1bc1; update-return's dispatches still carry e69e8a39 - three hosted site jobs on master hung in the tree gate for over two hours (no timeout-minutes): ci.yml now carries site 15, changes 10, pow 60, sims 45, the overlap sweep runs under a 10-minute wall clock where GNU timeout exists, and tools/ci/workflow-timeouts-check.sh fails a job without a budget (self-test: a job without the key, a wrong budget) - a branch merged with no ci run of its own (era-vdf61421005, 16:31 UK): master's igneum-pow suite went red and five docs-only merges landed green over it because their runs skip the compile job. tools/ci/ci-state.mjs reads the runs API through gh (a commit's newest run, master's last COMPILED run, a branch's last red); merge-to-master.sh pushes an unrun branch for a run, waits for a queued one printing the clock, refuses a red one and refuses any merge onto a red master except the declared fix (--fixes-master); the pre-push hook refuses a push to master whose commit, or whose merge's branch parent, has no green run on that exact sha; a feature-branch push prints the branch's previous red first. Self-tests with a fake gh in all three. - ci-red.yml fires on failure, cancelled and timed_out and hands the conclusion to red-watch.mjs, whose line names the kind (CI red, CI cancelled, CI timed out); the self-test reads the workflow file for the three conclusions - tools/ci/retry-once.sh: one retry before red for the box-locks check, the scene parity check and the live public API check (each keeps its own skip line on a runner without the resource) GitHub's branch protection cannot be applied: the organisation is on the free plan and the repository is private (the API answers 403, "Upgrade to GitHub Pro or make this repository public"), so the two scripts are the enforcement; the rule is one line in CLAUDE.md under the CI block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
128 lines
8.2 KiB
YAML
128 lines
8.2 KiB
YAML
# CI on every push and pull request (private repository, free runner minutes).
|
|
#
|
|
# What runs: the lottery-hash crate's tests (igneum-pow, release profile), the census tool's build, the two Python
|
|
# simulators' --quick modes (each under two minutes), and the tree gate: every fast check in ONE script,
|
|
# tools/ci/pre-push.sh (site build and link check, the ledger sentences, the identity grep of the public export list
|
|
# and the served site, Windows-valid paths, workflow shell syntax, the copied-sources and playbook classes, the unit
|
|
# tests, the no-secrets check). The pre-push hook runs the SAME script before a push to master or release-*, so the
|
|
# local gate and CI cannot drift (6 October 2026: 131 red `ci` runs in three days, 92 of them on master, every one a
|
|
# tree check that would have failed on the pushing machine in under 25 s; docs/analysis/ci-failures-2026-10-06.md).
|
|
#
|
|
# Where it runs: `pow` and `sims` go to the self-hosted pool (label igneum-build-1: the runners on igneum-build-1 and, since
|
|
# 7 October 2026, igneum-build-2, which carries that label too; rustc pinned, sccache read-only) and only when the push
|
|
# touched code (the `changes` job; a docs-only push skips them)
|
|
# when the repository variable IGNEUM_CI_RUNNER is `box`, else to ubuntu-latest (docs/plans/ci-self-hosted.md; GitHub
|
|
# has no fallback in runs-on, the variable is the switch). The `site` job stays on GitHub's machines. The red watcher
|
|
# is its own workflow, .github/workflows/ci-red.yml (workflow_run, so the copy on master watches every branch's run
|
|
# whatever ci.yml that branch carries): one line per failed run, naming the branch, the commit, the red check and the
|
|
# pushing author, to the hidden updates channel and to /srv/ci-red/red.jsonl (tools/ci/red-watch.mjs;
|
|
# infra/build-server/ci-red), so nobody opens the Actions page to learn a branch is red (the inline `red` job here
|
|
# watched master and release-* only until 7 October 2026, when eight red runs on ca3-v4-node went unseen).
|
|
#
|
|
# What does not run, on purpose: the node fork (vendor/igneum-node*, a rusty-kaspa fork of about 500 crates with
|
|
# rocksdb, blst and the execution layer) is gitignored here and too big for the free runners today (a cold build is
|
|
# 20 to 55 minutes on 2 to 8 vCPU, docs/bench-log.md). The workflow builds igneum-pow only; the fork's own tests run
|
|
# on the Mac and the seed node (infra/seed-nodes, infra/fast-time).
|
|
name: ci
|
|
on:
|
|
push:
|
|
pull_request:
|
|
jobs:
|
|
changes:
|
|
# What the push touched (tools/ci/docs-only-check.sh): a push of documents only (docs/, site/, *.md) skips the two
|
|
# compile-or-compute jobs below, which read none of those paths, so the self-hosted queue carries only runs that can
|
|
# change their result (7 October 2026: 31 runs queued on one runner, most of them status-document pushes). The tree
|
|
# gate (the `site` job) runs on ubuntu-latest for every push. A pull request, a new branch or a force push answers
|
|
# code=true (no `before` to compare from), as does any error reading the compare API: when in doubt, run.
|
|
name: what the push touched (docs-only runs skip the Rust and simulator jobs)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10 # a 7 s API call; every job carries a budget (tools/ci/workflow-timeouts-check.sh)
|
|
outputs:
|
|
code: ${{ steps.classify.outputs.code }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
sparse-checkout: tools/ci
|
|
- id: classify
|
|
env:
|
|
GH_TOKEN: ${{ github.token }}
|
|
BEFORE: ${{ github.event.before }}
|
|
AFTER: ${{ github.sha }}
|
|
REPO: ${{ github.repository }}
|
|
EVENT: ${{ github.event_name }}
|
|
run: |
|
|
if [ "$EVENT" != push ] || [ -z "$BEFORE" ] || [ "$BEFORE" = 0000000000000000000000000000000000000000 ]; then
|
|
echo "code=true" >> "$GITHUB_OUTPUT"; echo "no base to compare from ($EVENT): the compile jobs run"; exit 0
|
|
fi
|
|
files="$(gh api "repos/$REPO/compare/$BEFORE...$AFTER" --paginate --jq '.files[].filename' 2>/dev/null || true)"
|
|
line="$(printf '%s\n' "$files" | bash tools/ci/docs-only-check.sh)"
|
|
echo "$line" >> "$GITHUB_OUTPUT"
|
|
echo "$line: $(printf '%s\n' "$files" | grep -c .) changed path(s) between ${BEFORE:0:8} and ${AFTER:0:8}"
|
|
pow:
|
|
name: igneum-pow tests, igneum-census build
|
|
needs: changes
|
|
if: ${{ needs.changes.outputs.code == 'true' }}
|
|
runs-on: ${{ vars.IGNEUM_CI_RUNNER == 'box' && fromJSON('["self-hosted", "linux", "x64", "igneum-build-1"]') || 'ubuntu-latest' }}
|
|
timeout-minutes: 60 # the box's suite ran 45 s to 2 min 40 s on 7 October 2026; a hosted fallback compiles cold
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: toolchain
|
|
run: rustc --version && cargo --version
|
|
- name: igneum-pow tests (release)
|
|
working-directory: igneum-pow
|
|
run: cargo test --release
|
|
- name: pack loader seed rule (packfile.h on a known-good and a known-mismatched pack)
|
|
run: bash proto-cuda/nvrtc/emu/packfile-test.sh
|
|
- name: igneum-census build (release)
|
|
working-directory: igneum-census
|
|
run: cargo build --release
|
|
sims:
|
|
name: simulators, quick modes
|
|
needs: changes
|
|
# master and release-* pushes, and pull requests into them, only (main, 7 October 2026: every code push cost two box jobs and the
|
|
# queue read 22); a feature-branch code push runs the igneum-pow tests alone. tools/ci/sims-branch-check.sh holds this rule.
|
|
if: ${{ needs.changes.outputs.code == 'true' && ((github.event_name == 'push' && (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release-'))) || (github.event_name == 'pull_request' && (github.base_ref == 'master' || startsWith(github.base_ref, 'release-')))) }}
|
|
runs-on: ${{ vars.IGNEUM_CI_RUNNER == 'box' && fromJSON('["self-hosted", "linux", "x64", "igneum-build-1"]') || 'ubuntu-latest' }}
|
|
timeout-minutes: 45 # two simulators under 120 s each by their own timeout, plus a hosted fallback's pip install
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-python@v5
|
|
if: vars.IGNEUM_CI_RUNNER != 'box' # the box has python3 and numpy from provision.sh
|
|
with:
|
|
python-version: '3.12'
|
|
- run: python3 -m pip install --quiet numpy
|
|
if: vars.IGNEUM_CI_RUNNER != 'box'
|
|
- name: finality_v2.py --quick (under two minutes)
|
|
working-directory: sim
|
|
run: time timeout 120 python3 finality_v2.py --quick > finality_quick.md
|
|
- name: difficulty/sim.py --quick (under two minutes)
|
|
working-directory: sim/difficulty
|
|
run: time timeout 120 python3 sim.py --quick > difficulty_quick.md
|
|
- uses: actions/upload-artifact@v4
|
|
with:
|
|
name: sim-quick-output
|
|
path: |
|
|
sim/finality_quick.md
|
|
sim/difficulty/difficulty_quick.md
|
|
site:
|
|
name: site build, link check, identity grep
|
|
runs-on: ubuntu-latest
|
|
# 15: the gate took 229 s on a hosted runner on 7 October 2026 plus a 40 s Playwright install; the same day three
|
|
# hosted site jobs on master hung in the gate for over two hours each with no budget, and GitHub's six-hour default
|
|
# would have ended each as a failure email. A hung job is a red the watcher posts (ci-red.yml fires on timed_out).
|
|
timeout-minutes: 15
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '22'
|
|
- name: a headless Chromium for the text-overlap sweep (Playwright outside the tree; the gate finds it through IGNEUM_PLAYWRIGHT_DIR)
|
|
run: |
|
|
mkdir -p /tmp/pw && cd /tmp/pw && npm init -y >/dev/null && npm i --no-audit --no-fund playwright@1.56 | tail -1
|
|
npx playwright install --with-deps chromium | tail -1
|
|
echo "IGNEUM_PLAYWRIGHT_DIR=/tmp/pw" >> "$GITHUB_ENV"
|
|
- name: the tree gate, tools/ci/pre-push.sh --ci (the same script the pre-push hook runs; one line per check, a red check prints its output)
|
|
run: bash tools/ci/pre-push.sh --ci
|
|
- name: public stats API answers with the documented fields (the live site; master only, the endpoints exist there after the merge)
|
|
if: github.ref == 'refs/heads/master'
|
|
run: bash tools/ci/retry-once.sh public-api node tools/ci/public-api-check.mjs https://igneum.network # a live host: one retry before red
|