Merge remote-tracking branch 'origin/master' into ca3-coord
This commit is contained in:
igneum-labs
commit
bd542bf76b
10 changed files with 309 additions and 32 deletions
16
.github/workflows/ci.yml
vendored
16
.github/workflows/ci.yml
vendored
|
|
@ -11,9 +11,10 @@
|
|||
# Where it runs: `pow` and `sims` go to the box's runner (igneum-build-1, rustc pinned, sccache read-only, 48 jobs)
|
||||
# when the repository variable IGNEUM_CI_RUNNER is `box`, else to ubuntu-latest (docs/plans/ci-self-hosted.md; GitHub
|
||||
# has no fallback in runs-on, the variable is the switch). The `site` job stays on GitHub's machines. The `red` job
|
||||
# runs on the box after any failed master or release-* run and records the failure for the watcher
|
||||
# (tools/ci/red-watch.mjs; infra/build-server/ci-red): one line per run to the hidden updates channel and to
|
||||
# /srv/ci-red/red.jsonl, so nobody opens the Actions page to learn master is red.
|
||||
# runs on the box after any failed run on ANY branch and records the failure for the watcher
|
||||
# (tools/ci/red-watch.mjs; infra/build-server/ci-red): one line per run, naming the branch, the commit, the red check
|
||||
# and the pushing author, to the hidden updates channel and to /srv/ci-red/red.jsonl, so nobody opens the Actions page
|
||||
# to learn a branch is red (master and release-* only until 7 October 2026, when eight red runs on ca3-v4-node went unseen).
|
||||
#
|
||||
# What does not run, on purpose: the node fork (vendor/igneum-node*, a rusty-kaspa fork of about 500 crates with
|
||||
# rocksdb, blst and the execution layer) is gitignored here and too big for the free runners today (a cold build is
|
||||
|
|
@ -77,14 +78,14 @@ jobs:
|
|||
run: node tools/ci/public-api-check.mjs https://igneum.network
|
||||
|
||||
red:
|
||||
# Runs only when a master or release-* run has a failed job, on the box's own runner (not a GitHub-hosted machine:
|
||||
# Runs when a run on any branch has a failed job, on the box's own runner (not a GitHub-hosted machine:
|
||||
# the billing block of 6 October 2026, 18:37Z to 20:10Z, failed every hosted job at start and nobody was told).
|
||||
# tools/ci/red-watch.mjs record appends ONE line for this run to /srv/ci-red/red.jsonl (idempotent per run attempt);
|
||||
# the box's igneum-ci-red.timer posts each new line once to the hidden updates channel. Never blocks a release:
|
||||
# it reads the run, writes one line, and ends.
|
||||
name: red watcher (master and release-* only; one line per failed run to the updates channel and the box file)
|
||||
name: red watcher (every branch; one line per failed run, with the branch, commit, red check and pushing author, to the updates channel and the box file)
|
||||
needs: [pow, sims, site]
|
||||
if: ${{ failure() && (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release-')) }}
|
||||
if: ${{ failure() }}
|
||||
runs-on: [self-hosted, linux, x64, igneum-build-1]
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
|
|
@ -94,8 +95,9 @@ jobs:
|
|||
- uses: actions/checkout@v4
|
||||
with:
|
||||
sparse-checkout: tools/ci
|
||||
- name: record this run (one line, the failed jobs and their first failed step, from the run's own API)
|
||||
- name: record this run (one line, the branch, the commit, the failed jobs and their first failed step from the run's own API, the pushing author)
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
RED_WATCH_TITLE: ${{ github.event.head_commit.message }}
|
||||
RED_WATCH_AUTHOR: ${{ github.event.head_commit.author.name }}
|
||||
run: node tools/ci/red-watch.mjs record --file /srv/ci-red/red.jsonl
|
||||
|
|
|
|||
|
|
@ -357,3 +357,23 @@ service. The layer tracks the repo branch `master`; until this work merges, the
|
|||
tree, so install writes a drop-in pinning `CAP_REPO_BRANCH=box-capacity` and removes it once `master` carries
|
||||
`infra/build-server/capacity/run.sh` (self-healing after the merge). `--no-start` enables without starting;
|
||||
`--smoke <job>` installs then runs one job's 10-minute smoke and prints the summary.
|
||||
|
||||
## 7. Spill-over between the boxes (7 October 2026, 15:0x UK)
|
||||
|
||||
the project lead's reading at 15:02 UK: build-1 at load 139 / 114 / 90 with both slots held and a queue (the horizon lane waited 1 h 40 min)
|
||||
while build-2 read 4.5 / 27 / 46 with both slots free. The class router (section 6) pinned each class to its box with no
|
||||
spill-over. Now (lib.sh `bs_route_spill`, master from this commit):
|
||||
|
||||
- The class is a PREFERENCE: a build, check or gate prefers box 1, a suite, bench or attack row box 2, a proving crate box 3.
|
||||
- Before a run, the preferred box is read with one ssh (free slots of its slot count, 1-minute load). It takes the job when it
|
||||
has a free slot and its load is at or under 64 (`BS_SPILL_LOAD`). Otherwise the other box (1 and 2 swap; 3 falls to 1) takes it
|
||||
when THAT one qualifies; when neither does, the job queues on its own box. A box without a host file is never chosen; an
|
||||
unreachable box reads as "down" and is skipped.
|
||||
- The decision is the first route line of the run ("route: class suite prefers box 2; box 2 (free=0 slots=3 load1=70) is full or
|
||||
over load 64: spilled to box 1 (free=1 slots=2 load1=20)"), the slot label carries "; spilled from box N", and the JSONL row
|
||||
carries `"route": {"preferred", "box", "spilled", "reason"}` for the dashboard's job card.
|
||||
- build-2 has THREE slots (its `slots` file reads 3 since 14:0x UK; provision.sh defaults a `-2` hostname to 3). Everything that
|
||||
lands on box 2 runs at the bounded class, nice 10 on a 32-core band with -j 32, builds and gates included, so a suite beside
|
||||
them keeps its number; since this commit a bounded run takes the band its slot owns (slot 0 the last 32 cores, slot 1 the 32
|
||||
below, slot 2 the 32 below that), so three bounded runs never share a core. A gate still takes its slot ahead of queued suites.
|
||||
- `--box N` still pins. Self-test: tools/ci/route-spill-check.sh (thirteen cases through `BS_ROUTE_STATE_<n>`, no ssh), in the gate.
|
||||
|
|
|
|||
|
|
@ -22,13 +22,55 @@ BS_HOST_FILE="${IGNEUM_BUILD_HOST_FILE:-$HOME/.config/igneum/build-server}" #
|
|||
# caller passes --box: gates, builds, checks, cross-builds, the workers, the hands and the observer stay on box 1; suites, benches and
|
||||
# the attack rows go to box 2; proving and aggregation CPU work, the second prover's shadow runner and the pool's fast-time NETWORK go
|
||||
# to box 3. A class whose box has no host file yet falls back to box 1, and the log line says so. No box mines, ever (README.md).
|
||||
# Since 7 October 2026 15:xx UK the class is a preference with spill-over (bs_route_spill below): a full or overloaded box hands
|
||||
# the job to the other one.
|
||||
bs_box_file() { case "${1:-1}" in 1) echo "$BS_HOST_FILE" ;; *) echo "${BS_HOST_FILE}-$1" ;; esac; }
|
||||
bs_route() { # <class: gate|build|check|suite|bench|prove|attack|other> -> the box number, falling back to 1
|
||||
bs_route() { # <class: gate|build|check|suite|bench|prove|attack|other> -> the PREFERRED box number, falling back to 1
|
||||
local want=1
|
||||
case "$1" in suite|bench|attack) want=2 ;; prove|shadow|fasttime) want=3 ;; esac
|
||||
if [ "$want" != 1 ] && [ ! -s "$(bs_box_file "$want")" ]; then bs_log "class $1 routes to box $want, which has no host file yet ($(bs_box_file "$want")): box 1"; want=1; fi
|
||||
if [ "$want" != 1 ] && [ ! -s "$(bs_box_file "$want")" ]; then bs_log "class $1 prefers box $want, which has no host file yet ($(bs_box_file "$want")): box 1"; want=1; fi
|
||||
echo "$want"
|
||||
}
|
||||
# Spill-over (the project lead, 7 October 2026, 15:02 UK: build-1 at load 139 with a queue of 1 h 40 min while build-2 read 4.5 with both
|
||||
# slots free). The class is a PREFERENCE, not a pin: a job goes to its class's box unless that box has no free slot or its 1-minute
|
||||
# load is above BS_SPILL_LOAD (64), in which case it goes to the other box when THAT one has a free slot under the same load
|
||||
# line; when neither qualifies it queues on its own box. The alternate of box 1 is box 2, of box 2 box 1, of box 3 box 1; a box
|
||||
# without a host file is never chosen. The decision is one line on the Mac (bs_log) and travels to the box in BR_ROUTE_* for the
|
||||
# JSONL row ("route": preferred, box, spilled, reason), so the dashboard shows it per job. A box is read with one ssh
|
||||
# (bs_box_state: free slots of the slot count, load1); BS_ROUTE_STATE_<n> in the environment replaces the ssh for the self-test
|
||||
# (tools/ci/route-spill-check.sh), "down" standing for an unreachable box.
|
||||
BS_SPILL_LOAD="${BS_SPILL_LOAD:-64}"
|
||||
bs_box_state() { # <box> -> "free=<n> slots=<n> load1=<x>" | "absent" | "down"
|
||||
local b="$1" v f h
|
||||
v=$(eval "printf '%s' \"\${BS_ROUTE_STATE_$b:-}\""); if [ -n "$v" ]; then printf '%s' "$v"; return 0; fi
|
||||
f=$(bs_box_file "$b"); [ -s "$f" ] || { printf 'absent'; return 0; }
|
||||
h="$(head -1 "$f" | tr -d '[:space:]')"
|
||||
ssh "${BS_SSH_OPTS[@]}" -o ConnectTimeout=8 "$h" 'd=/srv/builds/_locks; n=$(cat $d/slots 2>/dev/null || echo 1); free=0; k=0; while [ $k -lt $n ]; do exec 9>>$d/build-$k; if flock -n 9; then free=$((free+1)); fi; exec 9>&-; k=$((k+1)); done; printf "free=%s slots=%s load1=%s" $free $n "$(cut -d" " -f1 /proc/loadavg)"' 2>/dev/null || printf 'down'
|
||||
}
|
||||
bs_state_ok() { # <state> -> 0 when the box can take a job now (a free slot, load1 at or under the line)
|
||||
local st="$1" free load
|
||||
case "$st" in free=*) ;; *) return 1 ;; esac
|
||||
free=${st#free=}; free=${free%% *}; load=${st##*load1=}
|
||||
[ "$free" -gt 0 ] 2>/dev/null || return 1
|
||||
awk -v l="$load" -v m="$BS_SPILL_LOAD" 'BEGIN { exit !(l + 0 <= m + 0) }'
|
||||
}
|
||||
bs_route_spill() { # <class> [priority] -> the box number; sets BS_ROUTE_PREF, BS_ROUTE_BOX, BS_ROUTE_SPILLED, BS_ROUTE_REASON
|
||||
local class="$1" pref alt ps as
|
||||
pref=$(bs_route "$class" 2>/dev/null)
|
||||
case "$pref" in 1) alt=2 ;; 2) alt=1 ;; *) alt=1 ;; esac
|
||||
BS_ROUTE_PREF=$pref; BS_ROUTE_BOX=$pref; BS_ROUTE_SPILLED=0
|
||||
ps=$(bs_box_state "$pref")
|
||||
if bs_state_ok "$ps"; then BS_ROUTE_REASON="box $pref ($ps) takes it"
|
||||
else
|
||||
as=$(bs_box_state "$alt")
|
||||
if bs_state_ok "$as"; then BS_ROUTE_BOX=$alt; BS_ROUTE_SPILLED=1; BS_ROUTE_REASON="box $pref ($ps) is full or over load $BS_SPILL_LOAD: spilled to box $alt ($as)"
|
||||
else BS_ROUTE_REASON="box $pref ($ps) and box $alt ($as) are both full or over load $BS_SPILL_LOAD: queued on box $pref"; fi
|
||||
fi
|
||||
bs_log "route: class $class prefers box $pref; $BS_ROUTE_REASON"
|
||||
BR_ROUTE_PREF=$BS_ROUTE_PREF BR_ROUTE_BOX=$BS_ROUTE_BOX BR_ROUTE_SPILLED=$BS_ROUTE_SPILLED BR_ROUTE_REASON=$BS_ROUTE_REASON
|
||||
export BR_ROUTE_PREF BR_ROUTE_BOX BR_ROUTE_SPILLED BR_ROUTE_REASON
|
||||
echo "$BS_ROUTE_BOX"
|
||||
}
|
||||
BS_ROOT_REMOTE=/srv/builds
|
||||
BS_MIRROR_REPO=/srv/igneum.git
|
||||
BS_MIRROR_NODE=/srv/igneum-node.git
|
||||
|
|
@ -296,6 +338,7 @@ bs_remote_run() {
|
|||
BR_BRANCH="$BS_BRANCH" BR_SHA="$BS_SHA" BR_AGENT="$agent" BR_KIND="${BR_KIND:-other}" BR_COMMAND="${BR_COMMAND:-}" \
|
||||
BR_TARGET="${BR_TARGET:-}" BR_ARTEFACTS="${BR_ARTEFACTS:-}" BR_SDE="${BR_SDE:-}" BR_PAIRS_WITH="${BR_PAIRS_WITH:-}" \
|
||||
BR_NICE="${BR_NICE:-0}" BR_CORES="${BR_CORES:-0}" BR_JOBS_CAP="${BR_JOBS_CAP:-0}" BR_PRIORITY="${BR_PRIORITY:-normal}" \
|
||||
BR_ROUTE_PREF="${BR_ROUTE_PREF:-}" BR_ROUTE_BOX="${BR_ROUTE_BOX:-}" BR_ROUTE_SPILLED="${BR_ROUTE_SPILLED:-0}" BR_ROUTE_REASON="${BR_ROUTE_REASON:-}" \
|
||||
bash -c '
|
||||
for v in BR_DIR BR_LABEL BR_CMD BR_TOOL BR_WT BR_CRATE BR_BRANCH BR_SHA BR_AGENT BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS BR_SDE BR_PAIRS_WITH BR_NICE BR_CORES BR_JOBS_CAP BR_PRIORITY; do
|
||||
printf "export %s=%q\n" "$v" "${!v}"
|
||||
|
|
|
|||
|
|
@ -53,9 +53,10 @@ SCCACHE_GB="${SCCACHE_GB:-100}"
|
|||
SCCACHE_VERSION="${SCCACHE_VERSION:-}"
|
||||
NODE_MAJOR="${NODE_MAJOR:-22}"
|
||||
WORKTREES="${WORKTREES:-}"
|
||||
SLOTS="${SLOTS:-2}" # 2 since main's ruling of 6 October 2026 (20:3x UK); remote-run.sh gives 90 jobs alone, 45 beside another
|
||||
SLOTS_GIVEN="${SLOTS:-}"; SLOTS="${SLOTS:-2}" # 2 since main's ruling of 6 October 2026 (20:3x UK); remote-run.sh gives 90 jobs alone, 45 beside another
|
||||
P2P_PORTS="${P2P_PORTS:-26611 26811}"
|
||||
BOX_HOSTNAME="${BOX_HOSTNAME:-igneum-build-1}"
|
||||
case "$BOX_HOSTNAME" in *-2) [ -n "$SLOTS_GIVEN" ] || SLOTS=3 ;; esac # build-2: three slots (main, 7 Oct 2026; the bounded runs take one 32-core band each)
|
||||
WORKERS_HOST="${WORKERS_HOST:-build.igneum.network}" # the dashboard feed's HTTPS name (A record in deSEC, 6 Oct 2026)
|
||||
SSH_PUBKEY="${SSH_PUBKEY:-}"
|
||||
BUILD_USER=build
|
||||
|
|
|
|||
|
|
@ -255,6 +255,8 @@ d = {
|
|||
"source_date_epoch": num(e.get('BR_SDE')), "pairs_with": e.get('BR_PAIRS_WITH') or None,
|
||||
"nice": num(e.get('BR_NICE')) or 0, "cores": (num(e.get('BR_CORES')) or 0) or os.cpu_count(), "priority": e.get('BR_PRIORITY') or "normal",
|
||||
"class": e.get('BR_CLASS') or None, "run_log": e.get('BR_RUN_LOG') or None,
|
||||
"route": ({"preferred": num(e.get('BR_ROUTE_PREF')), "box": num(e.get('BR_ROUTE_BOX')), "spilled": e.get('BR_ROUTE_SPILLED') == '1',
|
||||
"reason": e.get('BR_ROUTE_REASON') or ""} if e.get('BR_ROUTE_BOX') else None),
|
||||
}
|
||||
sc = {k: num(e[v]) for k, v in (("hits", "BR_HITS"), ("misses", "BR_MISSES"), ("hits_total", "BR_HITS_T"), ("misses_total", "BR_MISSES_T"))}
|
||||
sc = {k: v for k, v in sc.items() if v is not None}
|
||||
|
|
@ -446,8 +448,15 @@ RUN_LOG_DIR="$LOG_DIR/runs"; mkdir -p "$RUN_LOG_DIR"
|
|||
BR_RUN_LOG="$RUN_LOG_DIR/$BR_HOST-$BR_T0-$BR_PID.log"; export BR_RUN_LOG
|
||||
# the class's nice and core set apply to the command's subshell and everything it starts (renice and taskset on the subshell's own
|
||||
# pid, BASHPID; cores are the LAST N of the box's set, so gates and builds keep the first ones to themselves)
|
||||
# (since the third slot on build-2, 7 Oct 2026: a bounded run takes the band its SLOT owns, counted from the top: slot 0 the last N
|
||||
# cores, slot 1 the N below, slot 2 the N below that, so three bounded runs never share a core; a band below core 0 falls back to
|
||||
# the last N)
|
||||
ncpu=$(nproc); cores_str="0-$((ncpu - 1))"
|
||||
if [ "${BR_CORES:-0}" -gt 0 ] && [ "${BR_CORES}" -lt "$ncpu" ]; then cores_str="$((ncpu - BR_CORES))-$((ncpu - 1))"; fi
|
||||
if [ "${BR_CORES:-0}" -gt 0 ] && [ "${BR_CORES}" -lt "$ncpu" ]; then
|
||||
band=0; case "${got:-}" in ''|measure) ;; *) band=$got ;; esac
|
||||
lo=$((ncpu - BR_CORES * (band + 1))); [ "$lo" -ge 0 ] || lo=$((ncpu - BR_CORES))
|
||||
cores_str="$lo-$((lo + BR_CORES - 1))"
|
||||
fi
|
||||
( [ "${BR_NICE:-0}" -gt 0 ] && renice -n "$BR_NICE" -p $BASHPID >/dev/null 2>&1; [ "$cores_str" != "0-$((ncpu - 1))" ] && taskset -cp "$cores_str" $BASHPID >/dev/null 2>&1; eval "$BR_CMD" ) > >(tee -a "$BR_RUN_LOG") 2> >(tee -a "$BR_RUN_LOG" >&2)
|
||||
rc=$?
|
||||
# the keeper stops BEFORE the bare `wait` (which flushes the two tees): a bare wait also waits for the keeper, and the keeper waits
|
||||
|
|
|
|||
132
infra/fast-time/lib/redact-keys.mjs
Normal file
132
infra/fast-time/lib/redact-keys.mjs
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
#!/usr/bin/env node
|
||||
// Harness summaries never carry a raw key. Every fast-time runner that writes a summary JSON writes it through
|
||||
// writeSummary(): a 64-hex value under a key-shaped field (a name ending in key, keys, token, secret, password or
|
||||
// passphrase, or anything nested under one) is shortened to its first 8 hex characters and an ellipsis, and the
|
||||
// serialised text is then checked with the same rule as tools/ci/no-secrets-check.sh before it touches the disk.
|
||||
// A summary that would fail the no-secrets gate is refused here, at the source, with the line named.
|
||||
//
|
||||
// The class (7 October 2026, 11:26 to 12:47 UK): fork-gate.mjs wrote the two miners' vote-key hashes into
|
||||
// joint_sinks.*.key and split_sinks.*.key of three gate summaries under docs/plans/counter-asic-3-gate; eight CI runs
|
||||
// on ca3-v4-node went red on "no secret file names and no 64-hex secrets in the tree" and no pushing lane saw it,
|
||||
// because the feature-branch hook ran only the structural checks and the red watcher posted master and release-* only.
|
||||
//
|
||||
// import { writeSummary } from './lib/redact-keys.mjs'; writeSummary(OUT, summary);
|
||||
// node infra/fast-time/lib/redact-keys.mjs --self-test a key field is shortened, a hash field is left alone,
|
||||
// a raw key in the text fails the writer's own check
|
||||
// node infra/fast-time/lib/redact-keys.mjs --check <file>... exit 1 if any file carries a raw key line (the hits named)
|
||||
import { mkdirSync, writeFileSync, readFileSync, existsSync, rmSync, mkdtempSync } from 'node:fs';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
|
||||
// the content rule of tools/ci/no-secrets-check.sh, verbatim in spirit: a 64-hex value (0x optional) assigned to a name
|
||||
// ending in token, key, secret, password or passphrase
|
||||
export const RAW_KEY_LINE = /(token|key|secret|password|passphrase)["']?\s*[:=]\s*["']?(0x)?[0-9a-fA-F]{64}(?![0-9a-fA-F])/i;
|
||||
export const KEY_FIELD = /(key|keys|token|secret|password|passphrase)$/i;
|
||||
const HEX64 = /^(0x)?[0-9a-fA-F]{64}$/;
|
||||
|
||||
export function shortHex(v) {
|
||||
if (typeof v !== 'string' || !HEX64.test(v)) return v;
|
||||
const head = v.startsWith('0x') ? 10 : 8;
|
||||
return v.slice(0, head) + '…';
|
||||
}
|
||||
|
||||
// A copy of `value` with every 64-hex string under a key-shaped field shortened. Nothing else changes: hashes, digests
|
||||
// and ids under other names stay whole. The input object is never mutated (the runner keeps comparing live keys).
|
||||
export function redactKeys(value, underKeyField = false) {
|
||||
if (Array.isArray(value)) return value.map((v) => redactKeys(v, underKeyField));
|
||||
if (value && typeof value === 'object') {
|
||||
const out = {};
|
||||
for (const [k, v] of Object.entries(value)) out[k] = redactKeys(v, underKeyField || KEY_FIELD.test(k));
|
||||
return out;
|
||||
}
|
||||
return underKeyField ? shortHex(value) : value;
|
||||
}
|
||||
|
||||
// The line numbers (1-based) of `text` that the no-secrets gate would flag.
|
||||
export function rawKeyLines(text) {
|
||||
return text.split('\n').map((l, i) => (RAW_KEY_LINE.test(l) ? i + 1 : 0)).filter(Boolean);
|
||||
}
|
||||
|
||||
// The text a summary is written as: redacted, then checked. Throws when a raw key survives (a key inside a free-text
|
||||
// field such as a quoted log line), so the runner fails before the tree does.
|
||||
export function summaryText(summary) {
|
||||
const text = JSON.stringify(redactKeys(summary), null, 2);
|
||||
const lines = rawKeyLines(text);
|
||||
if (lines.length) throw new Error(`summary carries a raw 64-hex key at line${lines.length > 1 ? 's' : ''} ${lines.join(', ')}: the no-secrets gate would refuse it; shorten it with redactKeys or drop the field`);
|
||||
return text;
|
||||
}
|
||||
|
||||
export function writeSummary(file, summary) {
|
||||
mkdirSync(dirname(file), { recursive: true });
|
||||
const text = summaryText(summary);
|
||||
writeFileSync(file, text);
|
||||
return text;
|
||||
}
|
||||
|
||||
const hex = (c) => c.repeat(64);
|
||||
|
||||
function selfTest() {
|
||||
const fails = [];
|
||||
const summary = {
|
||||
pass: true, keys: { a: hex('a'), b: hex('b') },
|
||||
joint_sinks: { a: { hash: hex('c'), blocks: 1, key: hex('a') }, b: { hash: hex('d'), blocks: 2, key: '0x' + hex('b') } },
|
||||
samples: [{ t: 1, a: { sink: hex('c').slice(0, 16), key: hex('a') } }],
|
||||
vote_key: hex('e'), signingKey: hex('f'), a_token: hex('1'), digest: hex('2'), exec_restart_hash: hex('3'), id32: 'a'.repeat(32),
|
||||
};
|
||||
const before = JSON.stringify(summary);
|
||||
const out = redactKeys(summary);
|
||||
if (JSON.stringify(summary) !== before) fails.push('redactKeys mutated its input');
|
||||
if (out.keys.a !== 'aaaaaaaa…' || out.keys.b !== 'bbbbbbbb…') fails.push(`keys.* not shortened: ${out.keys.a} ${out.keys.b}`);
|
||||
if (out.joint_sinks.a.key !== 'aaaaaaaa…') fails.push(`joint_sinks.a.key not shortened: ${out.joint_sinks.a.key}`);
|
||||
if (out.joint_sinks.b.key !== '0xbbbbbbbb…') fails.push(`a 0x key kept its prefix wrong: ${out.joint_sinks.b.key}`);
|
||||
if (out.samples[0].a.key !== 'aaaaaaaa…') fails.push('a key inside an array element was not shortened');
|
||||
if (out.vote_key !== 'eeeeeeee…' || out.signingKey !== 'ffffffff…' || out.a_token !== '11111111…') fails.push('a name ending in key/Key/token was not shortened');
|
||||
if (out.joint_sinks.a.hash !== hex('c') || out.digest !== hex('2') || out.exec_restart_hash !== hex('3')) fails.push('a hash, digest or *_hash field was changed');
|
||||
if (out.id32 !== 'a'.repeat(32) || out.joint_sinks.a.blocks !== 1) fails.push('a non-key value was changed');
|
||||
let text;
|
||||
try { text = summaryText(summary); } catch (e) { fails.push(`summaryText refused a redactable summary: ${e.message}`); }
|
||||
if (text && /[0-9a-f]{64}/.test(text.replace(new RegExp(`"(hash|digest|exec_restart_hash)": "(0x)?[0-9a-f]{64}"`, 'g'), ''))) fails.push('a raw 64-hex key survived in the written text');
|
||||
if (text && rawKeyLines(text).length) fails.push('the written text would fail the no-secrets rule');
|
||||
// the writer's own check: a raw key line in the text fails, under each shape the gate catches
|
||||
for (const t of [`{\n "key": "${hex('9')}"\n}`, `KEY=0x${hex('8')}`, `x-igneum-key: ${hex('7')}`, `const signingKey = "${hex('6')}";`]) {
|
||||
if (!rawKeyLines(t).length) fails.push(`rawKeyLines missed: ${t.slice(0, 30)}`);
|
||||
}
|
||||
if (rawKeyLines(`{\n "hash": "${hex('5')}",\n "id": "${'4'.repeat(32)}"\n}`).length) fails.push('rawKeyLines flagged a hash or a 32-hex id');
|
||||
// a key hiding in free text under a field the redactor does not know: summaryText refuses it, and names the line
|
||||
let refused = false;
|
||||
try { summaryText({ ok: true, refusal_example: `Fork choice: refused block by key: ${hex('9')}` }); } catch (e) { refused = /line 3/.test(e.message); }
|
||||
if (!refused) fails.push('summaryText did not refuse (or did not name the line of) a raw key inside a free-text field');
|
||||
// writeSummary writes the redacted text, and --check on a raw file fails
|
||||
const dir = mkdtempSync(join(tmpdir(), 'redact-keys-'));
|
||||
try {
|
||||
const good = join(dir, 'nested', 'good.json'); writeSummary(good, summary);
|
||||
if (!existsSync(good) || rawKeyLines(readFileSync(good, 'utf8')).length) fails.push('writeSummary wrote a raw key or nothing');
|
||||
if (JSON.parse(readFileSync(good, 'utf8')).keys.a !== 'aaaaaaaa…') fails.push('the written file does not parse back to the shortened key');
|
||||
const bad = join(dir, 'bad.json'); writeFileSync(bad, JSON.stringify({ keys: { a: hex('a') }, sink: { key: hex('a') } }, null, 2));
|
||||
if (checkFiles([good]).length) fails.push('--check flagged the redacted file');
|
||||
const hits = checkFiles([bad]);
|
||||
if (hits.length !== 1 || !/bad\.json:\d+/.test(hits[0])) fails.push(`--check did not name the raw file and line: ${hits.join(' | ')}`);
|
||||
if (hits.some((h) => /[0-9a-f]{64}/.test(h))) fails.push('--check printed a 64-hex value');
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
if (fails.length) { for (const f of fails) console.error(`self-test failed: ${f}`); process.exit(1); }
|
||||
console.log('self-test passed: key-shaped fields shorten to 8 hex and an ellipsis, hashes stay whole, the input is not mutated, a raw key in the text fails the writer\'s own check with its line, --check names a raw file without printing the key');
|
||||
}
|
||||
|
||||
export function checkFiles(files) {
|
||||
const hits = [];
|
||||
for (const f of files) {
|
||||
if (!existsSync(f)) { hits.push(`${f}: missing`); continue; }
|
||||
for (const n of rawKeyLines(readFileSync(f, 'utf8'))) hits.push(`${f}:${n}: a raw 64-hex key under a key-shaped field`);
|
||||
}
|
||||
return hits;
|
||||
}
|
||||
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
const args = process.argv.slice(2);
|
||||
if (args[0] === '--self-test') selfTest();
|
||||
else if (args[0] === '--check') {
|
||||
const hits = checkFiles(args.slice(1));
|
||||
if (hits.length) { for (const h of hits) console.error(h); process.exit(1); }
|
||||
console.log(`redact-keys: ${args.length - 1} file(s), no raw key`);
|
||||
} else { console.error('usage: redact-keys.mjs --self-test | --check <file>...'); process.exit(2); }
|
||||
}
|
||||
|
|
@ -121,15 +121,20 @@ if [ "$PLAN" = 1 ]; then
|
|||
exit 0
|
||||
fi
|
||||
|
||||
# the box: --box N, else the route by class (suite and bench to box 2, prove to box 3, the rest to box 1; lib.sh bs_route)
|
||||
if [ -z "$BOX" ]; then
|
||||
case "$SCHED_CLASS:$PRIORITY" in *:gate) BOX=1 ;; suite:*|bench:*) BOX=$(bs_route "$SCHED_CLASS") ;; *) BOX=1 ;; esac
|
||||
fi
|
||||
# the box: --box N pins it; else the class's PREFERRED box with spill-over (lib.sh bs_route_spill, the project lead 7 Oct 2026: a build or gate
|
||||
# prefers box 1, a suite or bench box 2, a proving crate box 3; a preferred box with no free slot or a 1-minute load above 64 hands
|
||||
# the job to the other box when that one qualifies; the decision line is printed here and lands in the JSONL row as "route")
|
||||
ROUTE_CLASS="$SCHED_CLASS"; [ "$PRIORITY" = gate ] && ROUTE_CLASS=gate
|
||||
if [ -z "$BOX" ]; then bs_route_spill "$ROUTE_CLASS"; BOX=$BS_ROUTE_BOX; else BR_ROUTE_PREF=$BOX BR_ROUTE_BOX=$BOX BR_ROUTE_SPILLED=0 BR_ROUTE_REASON="box $BOX by --box"; export BR_ROUTE_PREF BR_ROUTE_BOX BR_ROUTE_SPILLED BR_ROUTE_REASON; fi
|
||||
bs_host "$BOX"
|
||||
bs_context
|
||||
# a proving crate routes to box 3 unless the caller chose (its builds and suites alike)
|
||||
if [ "$BS_CRATE_REL" = proving/igneum-prove ] && [ -z "${BOX_GIVEN:-}" ] && [ "$PRIORITY" != gate ]; then b=$(bs_route prove); [ "$b" != "$BOX" ] && { BOX=$b; bs_host "$BOX"; }; fi
|
||||
bs_log "box $BOX ($BS_HOST) for class $SCHED_CLASS, priority $PRIORITY"
|
||||
# a proving crate prefers box 3 unless the caller chose (its builds and suites alike; the same spill-over)
|
||||
if [ "$BS_CRATE_REL" = proving/igneum-prove ] && [ -z "${BOX_GIVEN:-}" ] && [ "$PRIORITY" != gate ]; then bs_route_spill prove; [ "$BS_ROUTE_BOX" != "$BOX" ] && { BOX=$BS_ROUTE_BOX; bs_host "$BOX"; }; fi
|
||||
# box 2 keeps the suites' numbers: everything that lands there runs at the bounded class (nice 10, a 32-core band, -j 32), builds
|
||||
# and gates included (main, 7 Oct 2026); a gate still takes its slot ahead of queued suites
|
||||
if [ "$BOX" = 2 ] && [ "$BR_CORES" = 0 ]; then BR_NICE=10; BR_CORES=32; BR_JOBS_CAP=32; export BR_NICE BR_CORES BR_JOBS_CAP; bs_log "bounded on box 2 (nice 10, a 32-core band, -j 32) so a suite beside it keeps its number"; fi
|
||||
if [ "$BR_JOBS_CAP" -gt 0 ] && [ -n "$JOBS" ] && [ "$JOBS" -gt "$BR_JOBS_CAP" ]; then bs_log "--jobs $JOBS clamped to $BR_JOBS_CAP on box $BOX"; JOBS=$BR_JOBS_CAP; fi
|
||||
bs_log "box $BOX ($BS_HOST) for class $SCHED_CLASS, priority $PRIORITY$( [ "${BR_ROUTE_SPILLED:-0}" = 1 ] && echo ", SPILLED from box $BR_ROUTE_PREF")"
|
||||
|
||||
if [ "$SELFTEST" = 1 ]; then
|
||||
[ "$BS_KIND" = node ] || bs_die "--self-test-repro runs from a fork worktree (igneum-miner and kaspad live there)"
|
||||
|
|
@ -223,6 +228,7 @@ BR_KIND=$(bs_kind build-remote "$( [ "${CARGO_ARGS[0]}" = zigbuild ] && echo bui
|
|||
[ "$SCHED_CLASS" = bench ] && BR_KIND=bench
|
||||
[ "$PRIORITY" = gate ] && BR_KIND=gate
|
||||
label="$label; kind=$BR_KIND nice=$BR_NICE cores=$( [ "$BR_CORES" = 0 ] && echo 96 || echo "$BR_CORES")"
|
||||
[ "${BR_ROUTE_SPILLED:-0}" = 1 ] && label="$label; spilled from box $BR_ROUTE_PREF"
|
||||
for ((i = 0; i < ${#CARGO_ARGS[@]}; i++)); do [ "${CARGO_ARGS[$i]}" = --target ] && BR_TARGET="${CARGO_ARGS[$((i + 1))]:-}"; done
|
||||
BR_ARTEFACTS=""; [ "$FETCH" = 1 ] && BR_ARTEFACTS="$ARTEFACTS"
|
||||
export BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS
|
||||
|
|
|
|||
|
|
@ -5,10 +5,11 @@
|
|||
#
|
||||
# tools/ci/pre-push.sh # the full gate over this working tree (any worktree; reads the tree, writes nothing in it)
|
||||
# tools/ci/pre-push.sh --ci # the same, with the site built in place (a CI checkout is disposable)
|
||||
# tools/ci/pre-push.sh --hook # from .git/hooks/pre-push: full gate for master and release-*, the two structural
|
||||
# # checks (conflict markers, Windows paths) for every other ref
|
||||
# tools/ci/pre-push.sh --self-test # the runner fires on a known failure, passes a known success, and the hook picks the
|
||||
# # right gate from the ref lines
|
||||
# tools/ci/pre-push.sh --hook # from .git/hooks/pre-push: full gate for master and release-*; the light gate for every
|
||||
# # other ref: the two structural checks (conflict markers, Windows paths) and the two
|
||||
# # never-push classes (the no-secrets check, the identity grep), about 20 s on the Mac
|
||||
# tools/ci/pre-push.sh --self-test # the runner fires on a known failure, passes a known success, the hook picks the
|
||||
# # right gate from the ref lines, and the light gate carries the never-push classes
|
||||
# tools/ci/pre-push.sh --list # the check names, one per line
|
||||
#
|
||||
# What is NOT here, on purpose: the three compile-or-compute jobs (igneum-pow tests, igneum-census, the simulators) and
|
||||
|
|
@ -51,12 +52,20 @@ structural_checks() {
|
|||
run "every tracked path is valid on Windows (colon, trailing dot, reserved names, length)" bash tools/ci/windows-paths-check.sh
|
||||
}
|
||||
|
||||
never_push_checks() {
|
||||
# The two never-push classes, on EVERY ref (7 October 2026: three gate summaries on ca3-v4-node carried a 64-hex key
|
||||
# through eight red CI runs in 80 minutes; the feature-branch hook ran only the structural checks, so no lane saw it).
|
||||
# A secret or an identity leak must not reach the remote on any branch; together about 20 s on the Mac.
|
||||
run "identity grep of the public export list and the served site" bash tools/ci/identity-check.sh
|
||||
run "no secret file names and no 64-hex secrets in the tree" bash -c 'bash tools/ci/no-secrets-check.sh --self-test && bash tools/ci/no-secrets-check.sh'
|
||||
}
|
||||
|
||||
tree_checks() {
|
||||
run "site build (in a temporary copy locally, in place in CI)" site_build
|
||||
run "internal link check of site/*.html" node tools/ci/link-check.mjs
|
||||
run "every served page carries the full header (six items, Download, burger)" node tools/ci/site-nav-check.mjs
|
||||
run "ledger sentences present verbatim on their public pages" node tools/ci/ledger-text-check.mjs
|
||||
run "identity grep of the public export list and the served site" bash tools/ci/identity-check.sh
|
||||
never_push_checks
|
||||
run "shell inside .github/workflows parses (bash -n, the PowerShell 5.1 rule)" node tools/ci/check-workflow-shell.mjs
|
||||
run "PowerShell drive-reference check (\$name: in a double-quoted string)" bash tools/ci/ps-drive-ref-check.sh
|
||||
run "copied sources are re-stamped before a build" bash tools/ci/copied-sources-check.sh
|
||||
|
|
@ -77,6 +86,7 @@ tree_checks() {
|
|||
run "the remote checkout's clean spares a lane's scratch (.igneum-scratch-spare, the fixed prefixes, never -x; the lost-scratch class)" bash -c 'bash tools/ci/scratch-spare-check.sh --self-test && bash tools/ci/scratch-spare-check.sh'
|
||||
run "long-running tools keep their body in one parsed block (the edited-while-running class)" bash -c 'bash tools/ci/whole-body-check.sh --self-test && bash tools/ci/whole-body-check.sh'
|
||||
run "build-remote without a priority flag bounds suites and benches (nice 10, 32 cores); a gate runs unbounded" bash tools/ci/build-kind-default-check.sh
|
||||
run "the class router is a preference with spill-over (a held or overloaded box hands the job to the other one)" bash tools/ci/route-spill-check.sh
|
||||
run "no shell assignment hides behind a trailing comment (the swallowed-defaults class)" bash -c 'bash tools/ci/defaults-line-check.sh --self-test && bash tools/ci/defaults-line-check.sh'
|
||||
run "no script kills or finds a process by a plain name or a file name (pgrep/pkill -f literals, ps | grep)" bash -c 'bash tools/ci/kill-by-name-check.sh --self-test && bash tools/ci/kill-by-name-check.sh'
|
||||
run "the identity check's own self-test (excluded research path passes, exported leak fails)" bash tools/ci/identity-check.sh --self-test
|
||||
|
|
@ -89,10 +99,10 @@ tree_checks() {
|
|||
run "ship tool self-test" node tools/ship-app.mjs --self-test
|
||||
run "relay unit tests" node --test relay/test/parse.test.mjs relay/test/auth.test.mjs relay/test/wake.test.mjs relay/test/ember.test.mjs
|
||||
run "miner app notice strip and update card tests" node --test app/igneum-app/ui/notices.test.mjs app/igneum-app/ui/update-card.test.mjs app/igneum-app/ui/view.test.mjs app/igneum-app/ui/tune-line.test.mjs
|
||||
run "no secret file names and no 64-hex secrets in the tree" bash -c 'bash tools/ci/no-secrets-check.sh --self-test && bash tools/ci/no-secrets-check.sh'
|
||||
run "launch gates: every row with its check, the handoff text clean (self-test, then the tree)" bash -c 'node tools/ci/launch-gates-check.mjs --self-test && node tools/ci/launch-gates-check.mjs'
|
||||
run "income per tier: the public table equals its inputs, the schedule arithmetic" bash -c 'node tools/launch/income-tiers.mjs --check && node --test tools/launch/income-tiers.test.mjs'
|
||||
run "hash-origin report: a known-finished day and a known-failed day" node --test tools/observer/hash-origin.test.mjs
|
||||
run "harness summaries never carry a raw 64-hex key (the writer's own redaction and check)" node infra/fast-time/lib/redact-keys.mjs --self-test
|
||||
}
|
||||
|
||||
gated_refs() {
|
||||
|
|
@ -125,7 +135,12 @@ case "$MODE" in
|
|||
[ "$(printf 'refs/heads/x 1 refs/heads/release-0.3.15 2\n' | gated_refs)" = full ] || { echo "self-test failed: a push to release-0.3.15 did not select the full gate"; fails=1; }
|
||||
[ "$(printf 'refs/heads/x 1 refs/heads/x 2\n' | gated_refs)" = light ] || { echo "self-test failed: a push to a feature branch selected the full gate"; fails=1; }
|
||||
[ "$(printf '' | gated_refs)" = light ] || { echo "self-test failed: a push with no refs selected the full gate"; fails=1; }
|
||||
[ "$fails" = 0 ] && echo "self-test passed: a failing check is RED and fails the gate, a passing one is ok; master and release-* select the full gate, other refs the light one"
|
||||
# the light gate carries the two never-push classes beside the structural checks, and the full gate runs them too
|
||||
declare -f never_push_checks | grep -q 'tools/ci/no-secrets-check.sh' || { echo "self-test failed: the never-push checks do not run the no-secrets check"; fails=1; }
|
||||
declare -f never_push_checks | grep -q 'tools/ci/identity-check.sh' || { echo "self-test failed: the never-push checks do not run the identity grep"; fails=1; }
|
||||
grep -qE '^\s+structural_checks; never_push_checks; finish "feature branch"' "$0" || { echo "self-test failed: the hook's light gate does not run the never-push checks"; fails=1; }
|
||||
declare -f tree_checks | grep -q 'never_push_checks' || { echo "self-test failed: the full gate does not run the never-push checks"; fails=1; }
|
||||
[ "$fails" = 0 ] && echo "self-test passed: a failing check is RED and fails the gate, a passing one is ok; master and release-* select the full gate, other refs the light one (structural checks, no-secrets, identity grep)"
|
||||
exit $fails ;;
|
||||
list)
|
||||
grep -E '^\s+run "' "$0" | sed -E 's/^\s+run "([^"]+)".*/\1/' ;;
|
||||
|
|
@ -135,8 +150,8 @@ case "$MODE" in
|
|||
echo "pre-push gate: a push to master or release-*, the full gate (the same checks CI runs):"
|
||||
structural_checks; tree_checks; finish "push to master or release-*"
|
||||
else
|
||||
echo "pre-push gate: a feature branch, the two structural checks:"
|
||||
structural_checks; finish "feature branch"
|
||||
echo "pre-push gate: a feature branch, the light gate (the two structural checks, the no-secrets check, the identity grep):"
|
||||
structural_checks; never_push_checks; finish "feature branch"
|
||||
fi ;;
|
||||
ci|local)
|
||||
[ "$MODE" = ci ] && echo "pre-push gate in CI (the same script as the local hook):" || echo "pre-push gate over this working tree:"
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
#!/usr/bin/env node
|
||||
// The red-master watcher. One line per failed master or release-* run, so nobody opens the Actions page to learn CI is red.
|
||||
// The red watcher. One line per failed run on ANY branch (master and release-* only until 7 October 2026: eight red runs on
|
||||
// ca3-v4-node went unseen that morning), naming the branch, the commit, the red check and the pushing author, so nobody
|
||||
// opens the Actions page to learn a branch is red.
|
||||
// Node 22, standard library only.
|
||||
//
|
||||
// node tools/ci/red-watch.mjs record --file <red.jsonl> in the workflow's `red` job (runs on igneum-build-1 after a
|
||||
|
|
@ -41,7 +43,7 @@ const STATE_FILE = process.env.IGNEUM_CI_RED_STATE || path.join(os.homedir(), '.
|
|||
const WEBHOOK_KEY = 'DISCORD_WEBHOOK_UPDATES';
|
||||
// what stops each class now (named in the digest so the line teaches, not just counts); docs/analysis/ci-failures-2026-10-06.md
|
||||
export const GUARDS = {
|
||||
'ci': 'the pre-push gate (tools/ci/pre-push.sh, the same checks CI runs, before any push to master or release-*)',
|
||||
'ci': 'the pre-push gate (tools/ci/pre-push.sh: the full gate before a push to master or release-*, the never-push checks before a push to any branch)',
|
||||
'instant': 'pre-flight in remote-run.sh (manifest, -p package, feature, subcommand checked in a second) and the kept run log',
|
||||
'preflight-manifest': 'refused before the slot: the manifest did not parse',
|
||||
'preflight-package': 'refused before the slot: the -p package does not exist',
|
||||
|
|
@ -68,6 +70,7 @@ export function runFromEnv(env = process.env) {
|
|||
return {
|
||||
run_id: String(env.GITHUB_RUN_ID), attempt: Number(env.GITHUB_RUN_ATTEMPT || 1), workflow: env.GITHUB_WORKFLOW,
|
||||
branch: env.GITHUB_REF_NAME, sha: env.GITHUB_SHA.slice(0, 7), event: env.GITHUB_EVENT_NAME || '',
|
||||
actor: env.GITHUB_ACTOR || '', author: env.RED_WATCH_AUTHOR || '', // who pushed (the GitHub login), who the head commit names
|
||||
url: `${server}/${env.GITHUB_REPOSITORY}/actions/runs/${env.GITHUB_RUN_ID}`, at: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
|
@ -115,7 +118,8 @@ export async function record(file, env = process.env, fetchImpl = fetch, title =
|
|||
export function formatLine(l) {
|
||||
const where = l.failed.length ? l.failed.map((f) => `${f.job.replace(/,.*$/, '')} at "${f.step}"`).join('; ') : (l.note || 'no step detail');
|
||||
const title = l.title ? ` "${l.title}"` : '';
|
||||
return `CI red: ${l.workflow} on ${l.branch} @${l.sha}${title}: ${where} ${l.url}`;
|
||||
const who = l.actor ? ` pushed by ${l.actor}${l.author && l.author !== l.actor ? ` (commit by ${l.author})` : ''};` : '';
|
||||
return `CI red: ${l.workflow} on ${l.branch} @${l.sha}${title}:${who} ${where} ${l.url}`;
|
||||
}
|
||||
|
||||
function readCredentials(file) {
|
||||
|
|
@ -208,7 +212,8 @@ async function selfTest() {
|
|||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'red-watch-'));
|
||||
const file = path.join(dir, 'red.jsonl'); const stateFile = path.join(dir, 'posted.json'); const credFile = path.join(dir, 'discord');
|
||||
const env = { GITHUB_RUN_ID: '424242', GITHUB_RUN_ATTEMPT: '1', GITHUB_REPOSITORY: 'igneum-network/igneum', GITHUB_REF_NAME: 'master',
|
||||
GITHUB_SHA: '0f0abc6deadbeef', GITHUB_WORKFLOW: 'ci', GITHUB_TOKEN: 'x', RED_WATCH_TITLE: 'Merge box-work 2bd3bec: the repro script re-stamps its clones' };
|
||||
GITHUB_SHA: '0f0abc6deadbeef', GITHUB_WORKFLOW: 'ci', GITHUB_TOKEN: 'x', RED_WATCH_TITLE: 'Merge box-work 2bd3bec: the repro script re-stamps its clones',
|
||||
GITHUB_ACTOR: 'igneum-labs', RED_WATCH_AUTHOR: 'igneum-labs' };
|
||||
const jobs = { jobs: [
|
||||
{ name: 'site build, link check, identity grep', conclusion: 'failure', steps: [{ name: 'site build', conclusion: 'success' }, { name: 'identity grep of the public export list', conclusion: 'failure' }] },
|
||||
{ name: 'igneum-pow tests, igneum-census build', conclusion: 'success', steps: [] },
|
||||
|
|
@ -226,6 +231,13 @@ async function selfTest() {
|
|||
if (lines[0].failed[1].step !== '(job never started: runner or billing)') fails.push('record: a job with no steps was not named as never started');
|
||||
const text = formatLine(lines[0]);
|
||||
if (!/^CI red: ci on master @0f0abc6 "Merge box-work/.test(text) || !text.includes('actions/runs/424242')) fails.push(`format: ${text}`);
|
||||
if (!text.includes('pushed by igneum-labs (commit by igneum-labs);') || !text.includes('site build at "identity grep of the public export list"')) fails.push(`format: the line does not name the pushing author and the red check: ${text}`);
|
||||
// a feature branch is recorded and formatted like master (every branch since 7 October 2026)
|
||||
const envFeature = { ...env, GITHUB_RUN_ID: '424299', GITHUB_REF_NAME: 'ca3-v4-node', RED_WATCH_AUTHOR: 'igneum-labs' };
|
||||
const fileFeature = path.join(dir, 'feature.jsonl');
|
||||
await record(fileFeature, envFeature, fakeFetch);
|
||||
const textFeature = formatLine(readLines(fileFeature)[0]);
|
||||
if (!/^CI red: ci on ca3-v4-node @0f0abc6 "Merge box-work[^"]*": pushed by igneum-labs; site build at/.test(textFeature)) fails.push(`format on a feature branch: ${textFeature}`);
|
||||
// post, dry run: prints, sends nothing, marks nothing
|
||||
let printed = []; const log = (s) => printed.push(s);
|
||||
const sends = []; const hookFetch = async (url, init) => { sends.push({ url, body: JSON.parse(init.body) }); return { ok: true, status: 204 }; };
|
||||
|
|
@ -274,7 +286,7 @@ async function selfTest() {
|
|||
if (!d3.sent) fails.push('digest: not sent the next day');
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
if (fails.length) { for (const f of fails) console.error(`self-test failed: ${f}`); process.exit(1); }
|
||||
console.log('self-test passed: one line per run however often record runs; the dry run sends nothing; a missing key is named, never a URL; one live send per run; a webhook error keeps the run pending; a box row is counted, never posted alone; the digest goes once per London day, at or after 09:00');
|
||||
console.log('self-test passed: one line per run however often record runs, on any branch, naming the pushing author; the dry run sends nothing; a missing key is named, never a URL; one live send per run; a webhook error keeps the run pending; a box row is counted, never posted alone; the digest goes once per London day, at or after 09:00');
|
||||
}
|
||||
|
||||
const cmd = args[0];
|
||||
|
|
|
|||
37
tools/ci/route-spill-check.sh
Executable file
37
tools/ci/route-spill-check.sh
Executable file
|
|
@ -0,0 +1,37 @@
|
|||
#!/usr/bin/env bash
|
||||
# The spill-over router of tools/build-remote.sh (lib.sh bs_route_spill; the project lead, 7 October 2026, 15:02 UK: build-1 at load 139 with a
|
||||
# queue of 1 h 40 min while build-2 sat at 4.5 with free slots). The class is a preference: a job goes to its class's box unless
|
||||
# that box has no free slot or its 1-minute load is above 64, in which case it goes to the other box when that one qualifies, else
|
||||
# it queues on its own box. This check feeds the router fixed box states through BS_ROUTE_STATE_<n> (no ssh) and host files in a
|
||||
# scratch directory, and compares the chosen box and the spilled flag for the known cases: a held build-1 selects build-2, a free
|
||||
# build-1 selects build-1, an overloaded build-1 spills, a held build-2 sends a suite to build-1, two full boxes queue on the
|
||||
# class's own box, a box without a host file is never chosen, a build-1 that is down spills.
|
||||
#
|
||||
# tools/ci/route-spill-check.sh # exit 1 with the case that resolved wrongly (the check IS its self-test)
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/../.."
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
printf 'build@10.0.0.1\n' > "$t/hosts"; printf 'build@10.0.0.2\n' > "$t/hosts-2" # box 3 absent on purpose
|
||||
export IGNEUM_BUILD_HOST_FILE="$t/hosts"
|
||||
fail=0
|
||||
expect() { # <case> <class> <want box> <want spilled> ; the states come from the environment
|
||||
local name="$1" class="$2" wbox="$3" wsp="$4" out
|
||||
out=$(bash -c '. infra/build-server/lib.sh; bs_route_spill "$1" >/dev/null 2>&1; printf "%s %s" "$BS_ROUTE_BOX" "$BS_ROUTE_SPILLED"' _ "$class" 2>/dev/null)
|
||||
if [ "$out" = "$wbox $wsp" ]; then echo "route-spill: $name: $class -> box $wbox spilled=$wsp"
|
||||
else echo "route-spill: $name: $class resolved to '$out', expected 'box $wbox spilled=$wsp'" >&2; fail=1; fi
|
||||
}
|
||||
BS_ROUTE_STATE_1="free=0 slots=2 load1=30" BS_ROUTE_STATE_2="free=3 slots=3 load1=5" expect "a held build-1 selects build-2" build 2 1
|
||||
BS_ROUTE_STATE_1="free=1 slots=2 load1=20" BS_ROUTE_STATE_2="free=3 slots=3 load1=5" expect "a free build-1 selects build-1" build 1 0
|
||||
BS_ROUTE_STATE_1="free=2 slots=2 load1=70" BS_ROUTE_STATE_2="free=3 slots=3 load1=5" expect "an overloaded build-1 spills" build 2 1
|
||||
BS_ROUTE_STATE_1="free=0 slots=2 load1=30" BS_ROUTE_STATE_2="free=3 slots=3 load1=5" expect "a gate spills like a build" gate 2 1
|
||||
BS_ROUTE_STATE_1="free=1 slots=2 load1=5" BS_ROUTE_STATE_2="free=0 slots=3 load1=10" expect "a held build-2 sends a suite to build-1" suite 1 1
|
||||
BS_ROUTE_STATE_1="free=1 slots=2 load1=5" BS_ROUTE_STATE_2="free=1 slots=3 load1=90" expect "an overloaded build-2 sends a bench away" bench 1 1
|
||||
BS_ROUTE_STATE_1="free=2 slots=2 load1=5" BS_ROUTE_STATE_2="free=2 slots=3 load1=10" expect "a free build-2 keeps its suite" suite 2 0
|
||||
BS_ROUTE_STATE_1="free=0 slots=2 load1=80" BS_ROUTE_STATE_2="free=0 slots=3 load1=90" expect "two full boxes queue a build on build-1" build 1 0
|
||||
BS_ROUTE_STATE_1="free=0 slots=2 load1=80" BS_ROUTE_STATE_2="free=0 slots=3 load1=90" expect "two full boxes queue a suite on build-2" suite 2 0
|
||||
BS_ROUTE_STATE_1="down" BS_ROUTE_STATE_2="free=3 slots=3 load1=5" expect "a build-1 that is down spills" build 2 1
|
||||
BS_ROUTE_STATE_1="free=1 slots=2 load1=5" BS_ROUTE_STATE_2="free=3 slots=3 load1=5" expect "a proving class with no box 3 prefers box 1" prove 1 0
|
||||
BS_ROUTE_STATE_1="free=0 slots=2 load1=5" BS_ROUTE_STATE_2="free=3 slots=3 load1=5" expect "a proving class spills to box 2 when box 1 is held" prove 2 1
|
||||
BS_ROUTE_STATE_1="free=1 slots=2 load1=64" BS_ROUTE_STATE_2="free=3 slots=3 load1=5" expect "load exactly 64 is under the line" build 1 0
|
||||
[ "$fail" = 0 ] && echo "route-spill: the class is a preference; a full or overloaded box hands the job to the other one"
|
||||
exit $fail
|
||||
Loading…
Reference in a new issue