b2262e5dc6
The box's 34 red rows of 6 October classified (docs/analysis/ci-failures-2026-10-06.md section 6): 22 iterations, 12 in three real classes (instant deaths with nothing kept, a shared worktree directory, an unread dry run). remote-run.sh now: pre-flight (subcommand, manifest, -p package, --features) refuses in a second with exit 3 and a class; the last 400 lines of every run kept in /srv/builds/_log/runs; a class on every row (compile-error, link-error, test-failure, instant, no-test-matched, slot-timeout, no-dir, preflight-*); a cargo test whose filter matched no test exits 3; a per-worktree lock in checkout and run mode; every red row appended to /srv/ci-red/red.jsonl as source box. red-watch.mjs never posts a box row alone and sends one digest a day (counts per class with each class's guard); the timer runs tick. Shared group cired on the box so the runner and build append to one file. Shown in a sandbox on the box: pass, failing test, empty filter, bad package, bad feature, missing subcommand, compile error, broken manifest, two concurrent runs of one worktree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
26 lines
2.9 KiB
Bash
26 lines
2.9 KiB
Bash
#!/usr/bin/env bash
|
|
# Install or refresh the CI red watcher's poster on igneum-build-1 from this Mac.
|
|
# infra/build-server/ci-red/install.sh (re-run whenever tools/ci/red-watch.mjs or the units change)
|
|
# Copies tools/ci/red-watch.mjs to /srv/discord-hooks/bin (beside the Discord scheduler, which already holds the webhook
|
|
# file), creates /srv/ci-red with the shared group `cired` (runner and build; the workflow's `red` job appends CI rows as
|
|
# runner, remote-run.sh appends box rows as build, the file is 664 in a 2775 directory), installs the two units and enables
|
|
# the timer (one `tick` a minute: post new CI rows, then the 09:00 London digest). No secret moves here: the poster
|
|
# reads the webhook file the Discord scheduler's install.sh already placed (DISCORD_WEBHOOK_UPDATES is the key it needs;
|
|
# until that key is in ~/.config/igneum/discord and that install.sh is re-run, every pass logs the missing key by name
|
|
# and the lines wait in red.jsonl). Needs root over ssh (root@<ip> with ~/.ssh/igneum_ed25519); the host ip comes from
|
|
# ~/.config/igneum/build-server (build@<ip>).
|
|
set -euo pipefail
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; ROOT="$(cd "$HERE/../../.." && pwd)"
|
|
KEY="${IGNEUM_BUILD_KEY:-$HOME/.ssh/igneum_ed25519}"
|
|
HOST_LINE="$(head -1 "${IGNEUM_BUILD_HOST_FILE:-$HOME/.config/igneum/build-server}" | tr -d '[:space:]')"
|
|
IP="${HOST_LINE#*@}"; [ -n "$IP" ] || { echo "no build server in ~/.config/igneum/build-server" >&2; exit 1; }
|
|
node "$ROOT/tools/ci/red-watch.mjs" --self-test >/dev/null || { echo "red-watch.mjs fails its own self-test; not installing" >&2; exit 1; }
|
|
|
|
SSH=(ssh -i "$KEY" -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10)
|
|
"${SSH[@]}" "root@$IP" 'install -d -o build -g build -m 750 /srv/discord-hooks /srv/discord-hooks/bin; getent group cired >/dev/null || groupadd cired; usermod -aG cired runner; usermod -aG cired build; install -d -o root -g cired -m 2775 /srv/ci-red; [ -f /srv/ci-red/red.jsonl ] || install -o root -g cired -m 664 /dev/null /srv/ci-red/red.jsonl; chgrp cired /srv/ci-red/red.jsonl; chmod 664 /srv/ci-red/red.jsonl'
|
|
scp -q -i "$KEY" "$ROOT/tools/ci/red-watch.mjs" "build@$IP:/srv/discord-hooks/bin/"
|
|
scp -q -i "$KEY" "$HERE/igneum-ci-red.service" "$HERE/igneum-ci-red.timer" "root@$IP:/etc/systemd/system/"
|
|
"${SSH[@]}" "root@$IP" 'systemctl daemon-reload && systemctl enable --now igneum-ci-red.timer >/dev/null 2>&1; systemctl is-active igneum-ci-red.timer; systemctl list-timers igneum-ci-red.timer --no-pager | sed -n 2p'
|
|
# one dry pass as the unit's user: what would be posted, names only, never a URL
|
|
"${SSH[@]}" "build@$IP" 'IGNEUM_DISCORD_ENV=/srv/discord-hooks/env IGNEUM_CI_RED_STATE=/srv/discord-hooks/ci-red-posted.json /usr/local/bin/node /srv/discord-hooks/bin/red-watch.mjs post --file /srv/ci-red/red.jsonl'
|
|
echo "installed; the poster runs at :30 every minute: journalctl -u igneum-ci-red -n 20; the record file: ssh build@$IP cat /srv/ci-red/red.jsonl"
|