igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity

remote-run.sh: a stale index.lock is one older than 30 s, not one with no git anywhere on the machine; the self-test backdates its fixture lock

Browse source 
pgrep -x git over the whole machine kept the lock whenever any git ran (the pre-push hook's own git push, another agent's build) and the checkout died with "index.lock: File exists". The fact is the lock's age. Class Q in docs/analysis/ci-failures-2026-10-06.md with the GIT_DIR leak of the previous commit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-06 21:11:30 +00:00
parent 2761700603
commit dc4e9dd5b7
2 changed files with 12 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
docs/analysis/ci-failures-2026-10-06.md
View file

@ -40,6 +40,7 @@ One line per class. "Guard" is what now stops the class before it reaches master
| M public API check | 1 | 06 15:12Z | 06 15:12Z | The live observer was 969 s stale when the master-only live check ran | The observer recovered; the hands moved to the box that evening | A live check stays in CI only, master only; it is not a tree fact and not in the gate |
| N no-secrets | 2 | 06 15:56Z | 06 16:23Z | A 64-hex test vector next to `private_key` in `app/igneum-wallet/src/vault.rs` (wallet-0.1.5) | Allow-listed as a test value | The gate |
| P swallowed defaults line (no CI run: a silent class) | 0 | 06 19:5xZ | 06 21:xxZ | A comment appended to a line of shell assignments in `tools/build-remote.sh` and then `tools/cross-remote.sh` turned every assignment after the `#` into comment text; `bash -n` and shellcheck are silent on it; the default cross-build never ran and its chain kept the previous exes from about 20:40 to 22:00 UK | 36e4ee7 on master: the lines split; `tools/ci/defaults-line-check.sh` with its self-test | In ci.yml at 36e4ee7 and in the gate from this change, so it runs on the pushing machine before the push |
| Q gate checks that read the machine, not the fact (found by the gate itself, 22:1x to 22:3x UK) | 0 | 06 21:1xZ | 06 21:3xZ | Two pushes of this change to master were refused by the new hook: (1) git hands a hook `GIT_DIR`, and `remote-run.sh --self-test`'s nested `git init`, commits and reset then acted on THIS repository's worktree: it set `core.bare`, moved the local `master` to three fixture commits and broke the main checkout for ten minutes (restored from the reflog: master back to 36e4ee7, `core.bare false`; nothing was pushed, nothing lost); (2) the same self-test judged a stale `index.lock` by `pgrep -x git` over the whole machine, so the hook's own `git push` (or any other agent's git) made the fixture's checkout die with "index.lock: File exists" | The gate unsets `GIT_DIR` and the other hook variables before any check; the staleness test is the lock's age (over 30 s), and the self-test backdates its fixture lock | The gate itself: every self-test now runs inside a real hook before every master push, with a `git push` alive beside it |
| O1 windows-ci cancelled | 16 | 04 10:42Z | 06 18:12Z | `concurrency: cancel-in-progress` on windows.yml: a newer master push superseded the run. Not a failure | None needed | None; they are listed because `gh run list` counts them as non-green |
| O2 hosted runner not acquired | 8 | 05 19:26Z | 05 20:54Z | "The job was not acquired by Runner of type hosted even after multiple attempts" on release-0.3.10 (7) and master (1): GitHub capacity, retried by hand | Re-run | The `red` job reports it; the box runner for `pow` and `sims` (section 5) takes those jobs off the hosted pool |

14
infra/build-server/remote-run.sh
View file

@ -47,8 +47,14 @@ checkout_tree() {
[ -n "$wt" ] && mkdir -p "$wt"
if [ ! -d "$dir/.git" ]; then rm -rf "$dir"; git clone -q --no-checkout "$mirror" "$dir"; fi
cd "$dir"
# a run killed mid-git (two runs on one worktree, 18:48:56Z the same day) leaves .git/index.lock; stale when no git runs here
if [ -f .git/index.lock ] && ! pgrep -x git >/dev/null 2>&1; then rm -f .git/index.lock; echo "checkout: removed a stale .git/index.lock in $dir" >&2; fi
# a run killed mid-git (two runs on one worktree, 18:48:56Z the same day) leaves .git/index.lock; stale when it is older
# than 30 s (an index write takes milliseconds, a checkout of the fork seconds). The first version asked `pgrep -x git`,
# which said "in use" whenever ANY git ran on the machine: the pre-push hook's own `git push` and any other agent's build
# kept the lock and the checkout died with "index.lock: File exists" (6 October 2026, 22:2x UK; the fact is the lock's age)
if [ -f .git/index.lock ]; then
lock_age=$(( $(date +%s) - $(stat -c %Y .git/index.lock 2>/dev/null || stat -f %m .git/index.lock) ))
if [ "$lock_age" -gt 30 ]; then rm -f .git/index.lock; echo "checkout: removed a stale .git/index.lock (${lock_age}s old) in $dir" >&2; fi
fi
git checkout -q -- . 2>/dev/null || true
git clean -qfd -e target -e 'target-*' -e '.build-remote-sha-*' -e '.cross-remote-sha-*' -e sccache
git fetch -q origin '+refs/heads/*:refs/remotes/origin/*'
@ -77,7 +83,9 @@ if [ "${1:-}" = --self-test ]; then
echo edited > "$t/box/a.txt"; echo new > "$t/box/b.txt"; mkdir -p "$t/box/target/release"; echo bin > "$t/box/target/release/x"; echo "$sha1" > "$t/box/.build-remote-sha-target"
# a crate in a subdirectory: its stamp and target dir must survive, its untracked overlay file must not
mkdir -p "$t/box/sub/target/release"; echo bin > "$t/box/sub/target/release/y"; echo "$sha1" > "$t/box/sub/.build-remote-sha-target"; echo new > "$t/box/sub/c.txt"
: > "$t/box/.git/index.lock" # a run killed mid-git leaves this; the checkout mode removes it when no git runs here
: > "$t/box/.git/index.lock" # a run killed mid-git leaves this; the checkout mode removes it once it is older than 30 s
touch -t "$(date -d '-2 min' +%Y%m%d%H%M.%S 2>/dev/null || date -v-2M +%Y%m%d%H%M.%S)" "$t/box/.git/index.lock" # backdated two minutes (GNU date, then BSD date)
[ $(( $(date +%s) - $(stat -c %Y "$t/box/.git/index.lock" 2>/dev/null || stat -f %m "$t/box/.git/index.lock") )) -gt 30 ] || { echo "self-test: could not backdate the fixture lock"; exit 1; }
# the Mac moves on: a new commit that changes a.txt
echo a2 > "$t/src/a.txt"; git -C "$t/src" -c user.name=t -c user.email=t@t commit -qam three; git -C "$t/src" push -q "$t/mirror.git" master
sha2=$(git -C "$t/src" rev-parse HEAD)