21143c979c
docs/plans/miner-faults.md: MF-1 to MF-7, each with its rule, test and gate line.
- MF-1/MF-2: a worker starts and is judged only when the node is READY (synced and igneum_getExecStatus reports an
executed tip; execrpc::probe every 5 s off the engine thread); the node watchdog never counts the catch-up (settled
once read synced; 30 min cap before that; any RPC answer is a sign of life); the watchdog restarts on a ladder 10 s,
30 s, 2 min, 5 min, then every 5 min for ever (watchdog::RETRY_LADDER_S); the faulted state and the one-restart
budget are gone (tools/ci/permanent-fault-check.sh in the gate); a node-caused restart resets the ladder at sync.
- MF-3: the hot-plug pass starts a recovered or revived card's worker (unchanged rule, now in the register).
- MF-4: the status clock starts at ready (program loaded), loading bounded by 300 s; a self-test failure holds the
card 30 min with the reason on its row, released on a driver change; a crash loop climbs the ladder; the pack is
exported once a minute for every card (a refused pack forces one).
- MF-5: the app reads template_wait=, template_ms=, identities_active= from the 0.3.20 miner's STATUS; waiting on
the node is never the card's fault; the row says node slow; every node-wait label clears on the first rate.
- MF-6: a miners hold belongs to the job that took it and releases when that job is gone or at its own cap.
- MF-7: the engine owns every igneum-miner it started: an untracked one on this engine's node RPC is killed at start,
after every stop and every minute, one line and one fault report per kill; a restart kills the old process first.
- Every fault line posts one FAULT line to the log intake (label fault-<id8>, app and node version, 60/h cap).
- The signed cards job kind (per card enabled, identities, power_pct; refused for a card the machine lacks; applied
through the app's own card path, persisted, read back): packaging/ota/publish-jobs.sh add --kind cards.
- LG-4 as a job: relay/playbooks/first-share.ps1 and tools/fleet/first-share-gate.mjs (no Windows box yet).
- tools/reliability: the fault injector with one step per class (catch-up, card-appears, own-restart, zero-ladder,
no-status, node-silent, one-card-fails, orphan-miner); fake-worker.mjs lists devices and fails self-tests on command.
- master's build tooling (97255a4e) and release-0.3.20's igneum-pow taken into the worktree for the box routes.
Box: app 198 + 27 + 8 tests green on igneum-build-2; the tree gate green (33 checks).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
134 lines
10 KiB
Bash
Executable file
134 lines
10 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# The one gate. Every fast tree check CI runs, in one script, so the local gate and CI can never drift: the `site` job
|
|
# of .github/workflows/ci.yml calls `tools/ci/pre-push.sh --ci`, and the pre-push hook (tools/ci/install-hooks.sh) calls
|
|
# `tools/ci/pre-push.sh --hook` before any push to master or a release-* branch and refuses the push on red.
|
|
#
|
|
# tools/ci/pre-push.sh # the full gate over this working tree (any worktree; reads the tree, writes nothing in it)
|
|
# tools/ci/pre-push.sh --ci # the same, with the site built in place (a CI checkout is disposable)
|
|
# tools/ci/pre-push.sh --hook # from .git/hooks/pre-push: full gate for master and release-*, the two structural
|
|
# # checks (conflict markers, Windows paths) for every other ref
|
|
# tools/ci/pre-push.sh --self-test # the runner fires on a known failure, passes a known success, and the hook picks the
|
|
# # right gate from the ref lines
|
|
# tools/ci/pre-push.sh --list # the check names, one per line
|
|
#
|
|
# What is NOT here, on purpose: the three compile-or-compute jobs (igneum-pow tests, igneum-census, the simulators) and
|
|
# the live public API check (master only, a network call), which stay separate steps in ci.yml. Everything here finished
|
|
# in under 25 s on the Mac on 6 October 2026 (no-secrets 10 s, everything else under 3 s each).
|
|
#
|
|
# Local mode never writes into the worktree: the site is built in a temporary copy with SITE_DOWNLOADS_OFFLINE=1
|
|
# (063bbca, 6 October 2026: a hook that built in place rewrote the downloads snapshot in five worktrees). The link, ledger
|
|
# and identity checks then read the committed pages; CI builds in place and checks the rebuilt pages, the one difference.
|
|
set -uo pipefail
|
|
cd "$(git rev-parse --show-toplevel)" || exit 1
|
|
# git hands a hook its own repository through the environment (GIT_DIR, GIT_INDEX_FILE, GIT_PREFIX, ...). Left in place,
|
|
# every nested git inside the self-tests (remote-run.sh builds a mirror and pushes into it) would act on THIS repository
|
|
# and fire this hook again inside the fixture: the first master push through the gate died that way (6 October 2026).
|
|
unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_PREFIX GIT_COMMON_DIR GIT_OBJECT_DIRECTORY GIT_ALTERNATE_OBJECT_DIRECTORIES GIT_QUARANTINE_PATH GIT_PUSH_OPTION_COUNT
|
|
MODE="${1:-local}"; MODE="${MODE#--}"
|
|
RED=0; N=0; LOG="$(mktemp)"; trap 'rm -rf "$LOG" "${SITE_TMP:-}"' EXIT
|
|
T0=$(date +%s)
|
|
|
|
run() {
|
|
# run <name> <command...>: one line per check; the output of a red check is shown in full
|
|
local name="$1"; shift; N=$((N + 1))
|
|
local s=$(date +%s)
|
|
if "$@" >"$LOG" 2>&1; then
|
|
printf ' ok %3ds %s\n' "$(( $(date +%s) - s ))" "$name"
|
|
else
|
|
printf ' RED %3ds %s\n' "$(( $(date +%s) - s ))" "$name"; sed 's/^/ /' "$LOG" | cut -c1-240; RED=1
|
|
fi
|
|
}
|
|
run_quiet() { "$@" >/dev/null 2>&1; }
|
|
|
|
site_build() {
|
|
if [ "$MODE" = ci ]; then node site/build.mjs; return; fi
|
|
SITE_TMP="$(mktemp -d)"; cp -R site "$SITE_TMP/site"
|
|
(cd "$SITE_TMP/site" && SITE_DOWNLOADS_OFFLINE=1 node build.mjs)
|
|
}
|
|
|
|
structural_checks() {
|
|
run "no conflict markers in tracked files" bash tools/ci/no-conflict-markers.sh
|
|
run "every tracked path is valid on Windows (colon, trailing dot, reserved names, length)" bash tools/ci/windows-paths-check.sh
|
|
}
|
|
|
|
tree_checks() {
|
|
run "site build (in a temporary copy locally, in place in CI)" site_build
|
|
run "internal link check of site/*.html" node tools/ci/link-check.mjs
|
|
run "ledger sentences present verbatim on their public pages" node tools/ci/ledger-text-check.mjs
|
|
run "identity grep of the public export list and the served site" bash tools/ci/identity-check.sh
|
|
run "shell inside .github/workflows parses (bash -n, the PowerShell 5.1 rule)" node tools/ci/check-workflow-shell.mjs
|
|
run "PowerShell drive-reference check (\$name: in a double-quoted string)" bash tools/ci/ps-drive-ref-check.sh
|
|
run "copied sources are re-stamped before a build" bash tools/ci/copied-sources-check.sh
|
|
run "override params files parse with no duplicate key" bash tools/ci/override-json-check.sh
|
|
run "second-engine playbooks log to a file and end their tree (C35)" bash tools/ci/second-engine-check.sh
|
|
run "no playbook quits, pauses or resumes the installed app" bash -c 'bash tools/ci/playbook-quit-check.sh --self-test && bash tools/ci/playbook-quit-check.sh'
|
|
run "no script writes into another worktree or walks Projects" bash -c 'bash tools/ci/no-foreign-tree-writes.sh --self-test && bash tools/ci/no-foreign-tree-writes.sh'
|
|
run "the signer is never piped into head" bash tools/ci/signer-pipe-check.sh
|
|
run "bash bodies in PowerShell job scripts pass bash -n" bash -c 'bash tools/ci/bash-body-check.sh --self-test && bash tools/ci/bash-body-check.sh'
|
|
run "run jobs test their fetched kit before use" bash -c 'bash tools/ci/kit-path-check.sh --self-test && bash tools/ci/kit-path-check.sh'
|
|
run "every Windows spawn of the app runs with a hidden console" bash -c 'node tools/ci/windows-spawn-check.mjs --self-test && node tools/ci/windows-spawn-check.mjs'
|
|
run "pinned guest programs match their manifest" bash tools/ci/pinned-guests-check.sh
|
|
run "no permanent miner fault in the engine (miner-faults.md MF-2)" bash -c 'bash tools/ci/permanent-fault-check.sh --self-test && bash tools/ci/permanent-fault-check.sh'
|
|
run "root prover playbooks kill the GPU server and unlink its socket" bash tools/ci/prover-socket-check.sh
|
|
run "commit-string gate self-test" bash tools/ci/commit-string-check.sh --self-test
|
|
run "build server remote checkout self-test" bash infra/build-server/remote-run.sh --self-test
|
|
run "no shell assignment hides behind a trailing comment (the swallowed-defaults class)" bash -c 'bash tools/ci/defaults-line-check.sh --self-test && bash tools/ci/defaults-line-check.sh'
|
|
run "the identity check's own self-test (excluded research path passes, exported leak fails)" bash tools/ci/identity-check.sh --self-test
|
|
run "the Windows paths check's own self-test" bash tools/ci/windows-paths-check.sh --self-test
|
|
run "the red watcher's own self-test (one line per run, posted once)" node tools/ci/red-watch.mjs --self-test
|
|
run "faucet unit tests" node --test site/api/faucet.test.mjs
|
|
run "explorer, emission and public stats unit tests" node --test site/lib/explorer.test.mjs site/lib/emission.test.mjs site/api/public-stats.test.mjs
|
|
run "ship tool self-test" node tools/ship-app.mjs --self-test
|
|
run "relay unit tests" node --test relay/test/parse.test.mjs relay/test/auth.test.mjs relay/test/wake.test.mjs relay/test/ember.test.mjs
|
|
run "miner app notice strip and update card tests" node --test app/igneum-app/ui/notices.test.mjs app/igneum-app/ui/update-card.test.mjs app/igneum-app/ui/view.test.mjs app/igneum-app/ui/tune-line.test.mjs
|
|
run "no secret file names and no 64-hex secrets in the tree" bash -c 'bash tools/ci/no-secrets-check.sh --self-test && bash tools/ci/no-secrets-check.sh'
|
|
}
|
|
|
|
gated_refs() {
|
|
# stdin: the pre-push hook's lines "<local ref> <local sha> <remote ref> <remote sha>". Prints "full" when any remote
|
|
# ref is master or release-*, else "light".
|
|
local lref lsha rref rsha full=0
|
|
while read -r lref lsha rref rsha; do
|
|
case "$rref" in refs/heads/master|refs/heads/release-*) full=1 ;; esac
|
|
done
|
|
[ "$full" = 1 ] && echo full || echo light
|
|
}
|
|
|
|
finish() {
|
|
local what="$1" secs=$(( $(date +%s) - T0 ))
|
|
if [ "$RED" = 0 ]; then echo "pre-push gate ($what): GREEN, $N checks in ${secs}s"; exit 0; fi
|
|
echo "pre-push gate ($what): RED after $N checks in ${secs}s. Fix it before pushing (every check above is one CI runs; the same script runs there)." >&2
|
|
exit 1
|
|
}
|
|
|
|
case "$MODE" in
|
|
self-test)
|
|
fails=0
|
|
st="$(mktemp)" # run in this shell, not a $(...) subshell, so RED is visible here
|
|
run "known failure" false >"$st" 2>&1; out="$(cat "$st")"; case "$out" in *"RED"*"known failure"*) ;; *) echo "self-test failed: a failing check was not reported RED"; fails=1 ;; esac
|
|
[ "$RED" = 1 ] || { echo "self-test failed: a failing check did not set RED"; fails=1; }
|
|
RED=0
|
|
run "known success" true >"$st" 2>&1; out="$(cat "$st")"; rm -f "$st"; case "$out" in *"ok"*"known success"*) ;; *) echo "self-test failed: a passing check was not reported ok"; fails=1 ;; esac
|
|
[ "$RED" = 0 ] || { echo "self-test failed: a passing check set RED"; fails=1; }
|
|
[ "$(printf 'refs/heads/x 1 refs/heads/master 2\n' | gated_refs)" = full ] || { echo "self-test failed: a push to master did not select the full gate"; fails=1; }
|
|
[ "$(printf 'refs/heads/x 1 refs/heads/release-0.3.15 2\n' | gated_refs)" = full ] || { echo "self-test failed: a push to release-0.3.15 did not select the full gate"; fails=1; }
|
|
[ "$(printf 'refs/heads/x 1 refs/heads/x 2\n' | gated_refs)" = light ] || { echo "self-test failed: a push to a feature branch selected the full gate"; fails=1; }
|
|
[ "$(printf '' | gated_refs)" = light ] || { echo "self-test failed: a push with no refs selected the full gate"; fails=1; }
|
|
[ "$fails" = 0 ] && echo "self-test passed: a failing check is RED and fails the gate, a passing one is ok; master and release-* select the full gate, other refs the light one"
|
|
exit $fails ;;
|
|
list)
|
|
grep -E '^\s+run "' "$0" | sed -E 's/^\s+run "([^"]+)".*/\1/' ;;
|
|
hook)
|
|
which="$(gated_refs)"
|
|
if [ "$which" = full ]; then
|
|
echo "pre-push gate: a push to master or release-*, the full gate (the same checks CI runs):"
|
|
structural_checks; tree_checks; finish "push to master or release-*"
|
|
else
|
|
echo "pre-push gate: a feature branch, the two structural checks:"
|
|
structural_checks; finish "feature branch"
|
|
fi ;;
|
|
ci|local)
|
|
[ "$MODE" = ci ] && echo "pre-push gate in CI (the same script as the local hook):" || echo "pre-push gate over this working tree:"
|
|
structural_checks; tree_checks; finish "$MODE" ;;
|
|
*) echo "usage: tools/ci/pre-push.sh [--ci|--hook|--self-test|--list]" >&2; exit 2 ;;
|
|
esac
|