igneum/docs/analysis/ci-failures-2026-10-06.md

CI failures, 4 to 6 October 2026: every non-green run classified, the fixes, the guards

Written 6 October 2026, 22:0x UK, from gh run list (430 runs, every run since the first workflow run at 09:57Z on 4 October) and gh run view --log-failed on one run per class. Times UTC (UK was UTC+1). This document is an operations record and is listed in tools/ci/export-exclude.txt: it is not exported to the public mirror.

1. The totals

Workflow	Runs	Green	Failed	Cancelled
ci	336	205	131	0
windows-ci	94	57	20	17
total	430	262	151	17

126 of the 168 non-green runs were on master. Master was red without a break from 18:37Z to the end of the evening (20:23Z, run 37526027569) across three causes in a row: the billing block, the copied-sources check, the identity grep.

2. Every failure by root cause

One line per class. "Guard" is what now stops the class before it reaches master.

Class	Runs	First	Last	Cause	Fix	Guard
A1 identity grep	56	04 13:26Z	05 01:46Z	A simulator's run log committed under sim/difficulty/records carried the Mac's home path in its first line; 43 master pushes in twelve hours each failed the same step	The log was scrubbed; .log files joined the generic scrub (mirror e18256d)	The pre-push gate runs the identity grep locally before any push to master or release-* (tools/ci/pre-push.sh --hook), so the hit lands on the pushing machine, not on master
A2 identity grep	17	05 02:17Z	05 10:36Z	vmmap dumps under docs/benchmarks/memory-floods-2026-10-04/vmmap/ carried a local time offset on their Date/Time lines	The dumps were re-stamped to UTC	Same gate
A3 identity grep	1	06 20:23Z	06 20:23Z	docs/analysis/horizon/polish.md, a research review of internal tooling, quotes the overlay network's product name, the intake key's variable name and the identity guard's own regexes as text; docs/analysis is in the export list, so the grep is right to flag it	The document is listed in tools/ci/export-exclude.txt; the identity check and the mirror's sync.sh both prune that list, so the guard's purpose (nothing the public reads carries these strings) is intact and the research text is untouched	The exclusion list, plus the gate; identity-check.sh --self-test shows an excluded path may quote the patterns and an exported one may not
B1 hosted runner refused	32	06 18:37Z	06 20:05Z	"The job was not started because recent account payments have failed or your spending limit needs to be increased": every GitHub-hosted job of every run failed at start with zero steps, 26 master runs and 6 release-0.3.15 runs, and nothing told anyone	Cleared on the billing page by 20:08Z	The red job runs on the box's own runner after any failed master or release-* run and posts one line per run (section 4); the pow and sims jobs can move to the box with one repository variable (section 5)
C1 copied-sources	1	06 18:00Z	06 18:00Z	tools/workers/collect.mjs mentioned rsync and cargo in a comment; the check read comments	The check skips comment lines	The gate
C2 copied-sources	12	06 20:08Z	06 20:19Z	infra/build-server/repro/rebuild-on-box.sh clones sources and runs cargo without touch; 10 master runs and 2 release-0.3.15 runs	0f0abc6 (box-work 2bd3bec): the repro script re-stamps its clones. Release-0.3.15 still carries the old script at c25a3ca and will fail this step again until it takes master (or cherry-picks 2bd3bec)	The gate
D no-foreign-tree-writes	2	06 18:20Z	06 18:24Z	The new check's warning pipeline (`grep	grep -v	sed
E Windows checkout	3	04 13:53Z	06 20:15Z	Nine screenshot files under docs/plans/site-ui-3-shots/ carried a colon from an address; git on windows-latest refuses the path, so actions/checkout died and with it every Windows build of the tree (the 0.3.15 installer waited on it)	61f46cc renamed the nine files	tools/ci/windows-paths-check.sh: colon and the other forbidden characters, trailing dot or space, reserved device names, over 240 characters; as the pre-commit hook on the staged paths and in the gate on every tracked path
F1 site build	5	04 13:46Z	04 13:53Z	site/scrub-bench.sh failed on docs/bench-log.md and build.mjs threw from the execSync	Fixed in the bench log the same afternoon	The gate builds the site in a temporary copy before the push
F2 site build	2	05 16:42Z	05 16:43Z	Conflict markers left in site/journey.json; build.mjs parsed it as JSON	Resolved by hand; no-conflict-markers.sh and the first pre-push hook (fb076de) followed	The gate's first check, on every push
G link check	1	05 09:28Z	05 09:28Z	/#wallet linked from every page with no such id (release-0.3.6)	Anchor added	The gate
H windows payload inputs	4	05 16:30Z	06 18:15Z	The signed inputs manifest on the downloads host pinned one node commit and packaging/windows/node-source.pin in the tree another: the Mac had pushed new inputs without committing the pin, or committed a pin without pushing inputs	Each time, the pin and the inputs were brought level	Not a tree check and not in the gate: the shipper's push-inputs.sh writes the pin and the commit must carry it; the red job now reports the mismatch within a minute instead of the next person opening the Actions page
I windows installer	1	04 10:32Z	04 10:32Z	The runner image's Inno Setup was older than 6.3	The workflow installs Inno Setup when the image's is too old	Resolved in the workflow
J windows engine	2	04 13:53Z	04 13:56Z	Rust that did not compile pushed to master (expected identifier, found keyword let)	Fixed in the next push	A compile is not a 25-second check; the owner's merge rule (CLAUDE.md, "CI red is stop-the-line") covers it: the merger fixes or reverts inside 15 minutes
K igneum-census	1	05 23:18Z	05 23:18Z	igneum-pow's Instr, Program and a layout argument changed; igneum-census was not rebuilt (release-0.3.11)	Updated with the crate	As J
L prover-socket	1	06 08:49Z	06 08:49Z	tools/proving-v1/pc2-agg-cost.ps1 ran the prover host as root without killing sp1-gpu-server (release-0.3.12)	The playbook was fixed	The gate
M public API check	1	06 15:12Z	06 15:12Z	The live observer was 969 s stale when the master-only live check ran	The observer recovered; the hands moved to the box that evening	A live check stays in CI only, master only; it is not a tree fact and not in the gate
N no-secrets	2	06 15:56Z	06 16:23Z	A 64-hex test vector next to private_key in app/igneum-wallet/src/vault.rs (wallet-0.1.5)	Allow-listed as a test value	The gate
P swallowed defaults line (no CI run: a silent class)	0	06 19:5xZ	06 21:xxZ	A comment appended to a line of shell assignments in tools/build-remote.sh and then tools/cross-remote.sh turned every assignment after the # into comment text; bash -n and shellcheck are silent on it; the default cross-build never ran and its chain kept the previous exes from about 20:40 to 22:00 UK	36e4ee7 on master: the lines split; tools/ci/defaults-line-check.sh with its self-test	In ci.yml at 36e4ee7 and in the gate from this change, so it runs on the pushing machine before the push
O1 windows-ci cancelled	16	04 10:42Z	06 18:12Z	concurrency: cancel-in-progress on windows.yml: a newer master push superseded the run. Not a failure	None needed	None; they are listed because gh run list counts them as non-green
O2 hosted runner not acquired	8	05 19:26Z	05 20:54Z	"The job was not acquired by Runner of type hosted even after multiple attempts" on release-0.3.10 (7) and master (1): GitHub capacity, retried by hand	Re-run	The red job reports it; the box runner for pow and sims (section 5) takes those jobs off the hosted pool

Sum: 168 runs, plus class P, which never reached CI because nothing checked for it. Classes A1 to A3, C1, C2, D, E, F1, F2, G, L and N are 102 runs (61 percent), every one a tree check that finishes in under 25 s on the pushing machine. B1 and O2 are 40 runs (24 percent) of GitHub-side refusals that nobody saw until the Actions page was opened. O1 is 16 runs (10 percent) of expected cancellations. H, I, J, K and M are the remaining 10.

3. The gate: one script, local and CI (tools/ci/pre-push.sh)

Every fast tree check CI runs is in one script. The site job of ci.yml calls tools/ci/pre-push.sh --ci; the pre-push hook calls tools/ci/pre-push.sh --hook. The two cannot drift because there is one list. A check added to ci.yml alone is the wrong place; it goes in the script.

Mode	When	What
--hook on a push to master or release-*	installed by tools/ci/install-hooks.sh into the shared hooks directory (one set for every worktree)	all 31 checks; a red check refuses the push and prints its output
--hook on any other ref	same	the two structural checks only (conflict markers, Windows paths)
--ci	the site job	all 31 checks, with the site built in place
default	by hand in any worktree	all 31 checks
--self-test	in the gate itself	a known failure is RED and fails the gate; a known success is ok; master and release-* select the full gate, other refs the light one

Measured 6 October 2026, 21:5x UK, on the Mac: 30 checks, GREEN, 25 s (no-secrets 11 s, identity grep 3 s, the rest under 2 s each). The hook never writes into the worktree: the site is built in a temporary copy with SITE_DOWNLOADS_OFFLINE=1 (063bbca: the earlier hook built in place and rewrote the downloads snapshot in five worktrees); git status before and after the full gate is identical.

Checks that joined CI through the gate and were not in ci.yml before: the ledger sentence check (ledger-text-check.mjs), the workflow shell parse (check-workflow-shell.mjs), the Windows paths check, and the three self-tests (identity, Windows paths, the red watcher). The swallowed-defaults check (36e4ee7) is in the gate too.

4. The red watcher (tools/ci/red-watch.mjs, infra/build-server/ci-red/)

A red job in ci.yml and windows.yml runs only when a master or release-* run has a failed job. It runs on the box's own runner (igneum-build-1), not on a GitHub-hosted machine, because the hosted pool is the thing that was refused in B1 and O2. It appends one JSON line for the run (id, workflow, branch, commit, title, the failed jobs and each one's first failed step from the run's own API, the URL) to /srv/ci-red/red.jsonl, idempotent per run attempt. On the box, igneum-ci-red.timer runs the poster every minute as build: each line not yet posted goes once to the hidden updates channel through DISCORD_WEBHOOK_UPDATES in /srv/discord-hooks/env, then its run id is recorded in /srv/discord-hooks/ci-red-posted.json. The orchestrator reads the file (ssh build@<box> cat /srv/ci-red/red.jsonl) or the channel. No URL is ever printed; a missing key is logged by name.

Shown on 6 October 2026: the self-test (one line however often record runs; the dry run sends nothing; a missing key is named, never a URL; one live send per run; a webhook error keeps the run pending). The poster is installed and active on the box (22:55 CEST, "nothing to post (0 recorded)"). Open: the updates channel has no webhook yet, so the first real red run will land in red.jsonl and the poster will log the missing key until DISCORD_WEBHOOK_UPDATES is added to ~/.config/igneum/discord on the Mac and infra/build-server/discord-hooks/install.sh is re-run. The Actions-side trigger has not fired on a real red run yet (master was made green in the same change); the first red master or release-* run is its known-failed case.

5. Where CI runs, and why ci takes about three minutes

Job	Where today	Time on ubuntu-latest	Time on the box (measured 6 October)	Note
pow (igneum-pow cargo test --release, packfile test, igneum-census build)	ubuntu-latest	2 min 30 s to 3 min, cold every run (no cache action)	42 s cold as the runner user (99 tests), sccache read-only hits after the first build	the long pole
sims (two Python simulators, --quick)	ubuntu-latest	about 1 min with setup-python and pip	python3 and numpy are on the box from provision.sh
site (the gate)	ubuntu-latest	under 1 min	not moved: the live public API check belongs on a neutral egress
red	the box's runner		seconds	only after a failed master or release-* run

The workflow now reads the repository variable IGNEUM_CI_RUNNER: box sends pow and sims to [self-hosted, linux, x64, igneum-build-1], anything else keeps ubuntu-latest (docs/plans/ci-self-hosted.md: GitHub has no fallback in runs-on, so a variable is the switch; gh variable set IGNEUM_CI_RUNNER --body box as igneum-labs, gh variable delete IGNEUM_CI_RUNNER to come back). Recommendation: flip it. The two compile-or-compute jobs are what GitHub's minutes and the billing block were spent on, the box compiles the agents' own pinned rustc 1.99.0, and a ci run drops from about three minutes to about one. The hosted runner then serves only the gate and the Windows pipeline (MSVC, WebView2, Inno Setup, PowerShell 5.1, which a Linux box cannot provide). The flip is main's call after the 0.3.15 cut, per build-server.md section 7.1.

6. What belongs on another branch

Branch	One-line change
release-0.3.15	take master (or cherry-pick 2bd3bec): infra/build-server/repro/rebuild-on-box.sh re-stamps its clones, else the copied-sources step fails again at the next push. Its own tools/ci/windows-paths-check.sh (c25a3ca) is superseded by master's: on merge keep master's file and drop the extra ci.yml step, the gate runs it