igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity
igneum/tools/ci/ci-state.mjs
igneum-labs a0eeb725f9 CI steward (7 October 2026, 17:3x UK): master takes only CI-passed commits; every job has a budget; the watcher reads cancelled and timed-out runs; box and network checks retry once 
Thirteen failure emails between 15:26 and 16:53 UK. The classes and what closes them:
- the box-locks check on a hosted runner (10 runs): closed by bd6fcb88 and 165e8b35 earlier
- windows-ci's stale payload-inputs pin (3 runs): closed on master by 4b4e1bc1; update-return's dispatches still carry e69e8a39
- three hosted site jobs on master hung in the tree gate for over two hours (no timeout-minutes): ci.yml now carries
  site 15, changes 10, pow 60, sims 45, the overlap sweep runs under a 10-minute wall clock where GNU timeout exists, and
  tools/ci/workflow-timeouts-check.sh fails a job without a budget (self-test: a job without the key, a wrong budget)
- a branch merged with no ci run of its own (era-vdf 61421005, 16:31 UK): master's igneum-pow suite went red and five
  docs-only merges landed green over it because their runs skip the compile job. tools/ci/ci-state.mjs reads the runs
  API through gh (a commit's newest run, master's last COMPILED run, a branch's last red); merge-to-master.sh pushes an
  unrun branch for a run, waits for a queued one printing the clock, refuses a red one and refuses any merge onto a red
  master except the declared fix (--fixes-master); the pre-push hook refuses a push to master whose commit, or whose
  merge's branch parent, has no green run on that exact sha; a feature-branch push prints the branch's previous red
  first. Self-tests with a fake gh in all three.
- ci-red.yml fires on failure, cancelled and timed_out and hands the conclusion to red-watch.mjs, whose line names the
  kind (CI red, CI cancelled, CI timed out); the self-test reads the workflow file for the three conclusions
- tools/ci/retry-once.sh: one retry before red for the box-locks check, the scene parity check and the live public API
  check (each keeps its own skip line on a runner without the resource)

GitHub's branch protection cannot be applied: the organisation is on the free plan and the repository is private (the
API answers 403, "Upgrade to GitHub Pro or make this repository public"), so the two scripts are the enforcement; the
rule is one line in CLAUDE.md under the CI block.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 16:37:13 +00:00

152 lines
12 KiB
JavaScript
Executable file
Raw Blame History

#!/usr/bin/env node
// What CI says about a commit or a branch, read from the runs API through gh (the Mac's gh login; the repository is
// igneum-network/igneum unless IGNEUM_REPO says otherwise). The merge tool and the pre-push hook read these lines, so a
// merge lands on master only when the branch's own ci run is green on the exact commit being merged (standing rule,
// 7 October 2026, 17:2x UK: era-vdf's tip 0e2d6b1c was merged with no ci run at all and master's igneum-pow suite went
// red for 40 minutes, hidden by docs-only merges whose runs skip the compile job).
// Node 22, standard library only; gh does the HTTP.
//
// node tools/ci/ci-state.mjs <sha> one line: "<state> <run id> <url> <detail>"
// state: success | failure | cancelled | timed_out | pending | none | unknown
// (the NEWEST ci run on that exact commit; a re-run replaces the first attempt)
// node tools/ci/ci-state.mjs --master-code the verdict of master's newest completed ci run whose compile job (igneum-pow)
// RAN: a docs-only push skips it and its green hides a red suite
// node tools/ci/ci-state.mjs --branch-red <branch> prints "previous CI red on <branch>: ..." when the branch's newest completed ci
// run is red and no newer run is green; prints nothing otherwise; exit 0 always
// node tools/ci/ci-state.mjs --self-test a fake gh on PATH answers every case: success, failure, pending, none, newest
// wins, the docs-only skip walked past, the branch line, a gh error = unknown
//
// Exit 0 for every answered query (callers read the first word); 2 on usage; the self-test exits 1 on a failure.
import { spawnSync } from 'node:child_process';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
const REPO = process.env.IGNEUM_REPO || 'igneum-network/igneum';
const FIELDS = 'databaseId,status,conclusion,headSha,url,createdAt,event';
const RED = new Set(['failure', 'cancelled', 'timed_out', 'startup_failure', 'action_required']);
function gh(args) {
const env = { ...process.env, PATH: `${process.env.PATH || ''}:/opt/homebrew/bin:/usr/local/bin` };
const r = spawnSync('gh', args, { encoding: 'utf8', env, timeout: 60000 });
if (r.error) throw new Error(`gh: ${r.error.message}`);
if (r.status !== 0) throw new Error(`gh ${args.slice(0, 2).join(' ')}: exit ${r.status}: ${(r.stderr || '').trim().slice(0, 200)}`);
return JSON.parse(r.stdout || 'null');
}
function fullSha(sha) {
if (/^[0-9a-f]{40}$/.test(sha)) return sha; // gh's --commit filter matches the full sha only (an abbreviation answers nothing)
const r = spawnSync('git', ['rev-parse', '--verify', `${sha}^{commit}`], { encoding: 'utf8' });
if (r.status !== 0) throw new Error(`${sha} is not a commit here`);
return r.stdout.trim();
}
export const newest = (runs) => [...(runs || [])].sort((a, b) => Date.parse(b.createdAt) - Date.parse(a.createdAt))[0];
export function verdictOf(run) {
if (!run) return 'none';
if (run.status !== 'completed') return 'pending';
return run.conclusion || 'pending';
}
export function firstRed(jobs) {
for (const j of jobs || []) {
if (j.conclusion === 'success' || j.conclusion === 'skipped' || j.conclusion == null) continue;
const s = (j.steps || []).find((x) => x.conclusion && x.conclusion !== 'success' && x.conclusion !== 'skipped');
return `${j.name.replace(/,.*$/, '')} at "${s ? s.name : '(no step)'}"`;
}
return '';
}
const london = (iso) => new Intl.DateTimeFormat('en-GB', { timeZone: 'Europe/London', hour: '2-digit', minute: '2-digit', hour12: false }).format(new Date(iso)) + ' UK';
const runsForSha = (sha) => gh(['run', 'list', '--repo', REPO, '--workflow', 'ci', '--commit', sha, '--limit', '10', '--json', FIELDS]) || [];
const runsForBranch = (branch, limit = 12) => gh(['run', 'list', '--repo', REPO, '--workflow', 'ci', '--branch', branch, '--limit', String(limit), '--json', FIELDS]) || [];
const jobsOf = (id) => (gh(['run', 'view', String(id), '--repo', REPO, '--json', 'jobs']) || {}).jobs || [];
export function stateOfSha(sha) {
const run = newest(runsForSha(fullSha(sha)));
const state = verdictOf(run);
if (!run) return `none - - no ci run on ${sha.slice(0, 8)} yet`;
let detail = state === 'pending' ? `${run.status} since ${london(run.createdAt)}` : `${run.event} run at ${london(run.createdAt)}`;
if (RED.has(state)) { const red = firstRed(jobsOf(run.databaseId)); if (red) detail += `: ${red}`; }
return `${state} ${run.databaseId} ${run.url} ${detail}`;
}
export function masterCodeState() {
const runs = [...runsForBranch('master', 12)].sort((a, b) => Date.parse(b.createdAt) - Date.parse(a.createdAt));
for (const run of runs) {
if (run.status !== 'completed') continue;
const jobs = jobsOf(run.databaseId);
const pow = jobs.find((j) => /^igneum-pow/.test(j.name));
if (!pow || pow.conclusion === 'skipped') continue; // a docs-only push: its green says nothing about the suite
const red = RED.has(run.conclusion) ? firstRed(jobs) : '';
return `${run.conclusion} ${run.databaseId} ${run.url} master @${run.headSha.slice(0, 8)} at ${london(run.createdAt)}, compile job ${pow.conclusion}${red ? `: ${red}` : ''}`;
}
return 'none - - no completed master ci run with the compile job among the last 12';
}
export function branchRedLine(branch) {
const runs = [...runsForBranch(branch, 12)].sort((a, b) => Date.parse(b.createdAt) - Date.parse(a.createdAt));
const done = runs.find((r) => r.status === 'completed');
if (!done || !RED.has(done.conclusion)) return '';
const red = firstRed(jobsOf(done.databaseId));
return `previous CI red on ${branch}: @${done.headSha.slice(0, 7)} ${done.conclusion} at ${london(done.createdAt)}${red ? `: ${red}` : ''} ${done.url} (no newer green run on this branch; this push gets its own run, read it)`;
}
async function selfTest() {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ci-state-'));
const fake = path.join(dir, 'gh');
// the fake gh answers `run list` from list-<commit or branch>.json and `run view <id>` from view-<id>.json; FAKE_GH_FAIL=1 fails
fs.writeFileSync(fake, `#!/usr/bin/env bash
[ "\${FAKE_GH_FAIL:-0}" = 1 ] && { echo "HTTP 502 from the fake" >&2; exit 1; }
key=""; prev=""
for a in "$@"; do case "$prev" in --commit|--branch) key="$a" ;; esac; prev="$a"; done
case "$1 $2" in
"run list") f="$FAKE_GH_DIR/list-$key.json"; [ -f "$f" ] && cat "$f" || echo '[]' ;;
"run view") f="$FAKE_GH_DIR/view-$3.json"; [ -f "$f" ] && cat "$f" || echo '{"jobs":[]}' ;;
*) echo "fake gh: unknown $*" >&2; exit 1 ;;
esac
`, { mode: 0o755 });
const sha = (c) => c.repeat(40);
const run = (id, status, conclusion, created, headSha = sha('a')) => ({ databaseId: id, status, conclusion, headSha, url: `https://github.com/x/y/actions/runs/${id}`, createdAt: created, event: 'push' });
const w = (name, obj) => fs.writeFileSync(path.join(dir, name), JSON.stringify(obj));
w(`list-${sha('a')}.json`, [run(1, 'completed', 'success', '2026-10-07T15:00:00Z')]);
w(`list-${sha('b')}.json`, [run(2, 'completed', 'failure', '2026-10-07T15:00:00Z', sha('b'))]);
w('view-2.json', { jobs: [{ name: 'site build, link check', conclusion: 'success', steps: [] }, { name: 'igneum-pow tests, igneum-census build', conclusion: 'failure', steps: [{ name: 'toolchain', conclusion: 'success' }, { name: 'igneum-pow tests (release)', conclusion: 'failure' }] }] });
w(`list-${sha('c')}.json`, [run(3, 'in_progress', null, '2026-10-07T15:00:00Z', sha('c'))]);
w(`list-${sha('d')}.json`, [run(4, 'completed', 'failure', '2026-10-07T14:00:00Z', sha('d')), run(5, 'completed', 'success', '2026-10-07T15:00:00Z', sha('d'))]); // the re-run wins
// master: the newest run is a docs-only green (compile job skipped); the one before it ran the compile job and is red
w('list-master.json', [run(10, 'completed', 'success', '2026-10-07T16:00:00Z', sha('1')), run(11, 'completed', 'failure', '2026-10-07T15:30:00Z', sha('2')), run(12, 'completed', 'success', '2026-10-07T15:00:00Z', sha('3'))]);
w('view-10.json', { jobs: [{ name: 'igneum-pow tests, igneum-census build', conclusion: 'skipped', steps: [] }, { name: 'site build', conclusion: 'success', steps: [] }] });
w('view-11.json', { jobs: [{ name: 'igneum-pow tests, igneum-census build', conclusion: 'failure', steps: [{ name: 'igneum-pow tests (release)', conclusion: 'failure' }] }] });
// branches: x red newest; y green newest; z pending newest over a red
w('list-x.json', [run(20, 'completed', 'failure', '2026-10-07T15:00:00Z', sha('e'))]);
w('view-20.json', { jobs: [{ name: 'site build, link check, identity grep', conclusion: 'failure', steps: [{ name: 'the tree gate', conclusion: 'failure' }] }] });
w('list-y.json', [run(21, 'completed', 'success', '2026-10-07T15:00:00Z'), run(22, 'completed', 'failure', '2026-10-07T14:00:00Z')]);
w('list-z.json', [run(23, 'queued', null, '2026-10-07T15:10:00Z'), run(24, 'completed', 'cancelled', '2026-10-07T15:00:00Z', sha('f'))]);
const me = new URL(import.meta.url).pathname;
const ask = (args, extraEnv = {}) => {
const r = spawnSync(process.execPath, [me, ...args], { encoding: 'utf8', env: { ...process.env, PATH: `${dir}:${process.env.PATH}`, FAKE_GH_DIR: dir, ...extraEnv } });
return { out: (r.stdout || '').trim(), code: r.status, err: (r.stderr || '').trim() };
};
const fails = [];
const expect = (name, got, re) => { if (!re.test(got.out) || got.code !== 0) fails.push(`${name}: got exit ${got.code} "${got.out}" ${got.err}`); };
expect('success', ask([sha('a')]), /^success 1 https:\/\/github.com\/x\/y\/actions\/runs\/1 push run at /);
expect('failure names the red step', ask([sha('b')]), /^failure 2 \S+ push run at \d\d:\d\d UK: igneum-pow tests at "igneum-pow tests \(release\)"$/);
expect('pending', ask([sha('c')]), /^pending 3 \S+ in_progress since /);
expect('newest wins', ask([sha('d')]), /^success 5 /);
expect('none', ask([sha('9')]), /^none - - no ci run on 99999999 yet$/);
expect('gh error is unknown', ask([sha('a')], { FAKE_GH_FAIL: '1' }), /^unknown - - gh run list: exit 1: HTTP 502 from the fake/);
expect('master-code walks past the docs-only green', ask(['--master-code']), /^failure 11 \S+ master @22222222 at \d\d:\d\d UK, compile job failure: igneum-pow tests at "igneum-pow tests \(release\)"$/);
expect('branch red line', ask(['--branch-red', 'x']), /^previous CI red on x: @eeeeeee failure at \d\d:\d\d UK: site build at "the tree gate" https:\/\/github.com\/x\/y\/actions\/runs\/20 \(no newer green/);
const y = ask(['--branch-red', 'y']); if (y.out !== '' || y.code !== 0) fails.push(`branch green prints nothing: "${y.out}" exit ${y.code}`);
expect('branch pending over a cancelled run still names the red', ask(['--branch-red', 'z']), /^previous CI red on z: @fffffff cancelled at /);
const noArg = ask([]); if (noArg.code !== 2) fails.push(`usage: exit ${noArg.code}`);
fs.rmSync(dir, { recursive: true, force: true });
if (fails.length) { for (const f of fails) console.error(`self-test failed: ${f}`); process.exit(1); }
console.log('self-test passed: a commit answers success, failure (with the red step), pending, none; the newest run wins; a gh error is unknown; --master-code walks past a docs-only green to the run that compiled; --branch-red names the newest completed red and stays silent on green');
}
const args = process.argv.slice(2);
try {
if (args[0] === '--self-test') await selfTest();
else if (args[0] === '--master-code') console.log(masterCodeState());
else if (args[0] === '--branch-red') { if (!args[1]) { console.error('usage: ci-state.mjs --branch-red <branch>'); process.exit(2); } console.log(branchRedLine(args[1])); }
else if (args[0] && !args[0].startsWith('-')) console.log(stateOfSha(args[0]));
else { console.error('usage: tools/ci/ci-state.mjs <sha> | --master-code | --branch-red <branch> | --self-test'); process.exit(2); }
} catch (e) {
console.log(`unknown - - ${e.message.replace(/\s+/g, ' ').slice(0, 300)}`);
}
Reference in a new issue View git blame Copy permalink