fleet: every stop goes through a pid file; pkill and killall are gone from the live tree

The founder's word (20:21 BST 8 Oct 2026): pkill and killall refuse by construction on every box and pod (exit 97). lib/pidkill.sh
(kill_pidfile, kill_children by parent pid, kill_sock_owner through ss -xlp) is the shared form; box-prover.py finds the SP1 server by
the pid owning its socket; lib/box.py, box-kill.sh, kill-node.sh, the rig scripts (box-ember, box-rig, box-matrix, box-floor-v5),
lib/standing.py and publish-2-move.py stop by pid files; box-dn3.sh writes node, miner-loop and miner pids and uses pidkill;
fleet-puller.sh and dn3-kill.sh call fleet-stop.sh. Seventeen devnet-2-era scripts that only ever stopped by name move to
tools/fleet/retired/ with a README.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-08 20:16:18 +00:00
parent 71ea298e14
commit 0c245a6083
33 changed files with 265 additions and 32 deletions

77
tools/fleet/box-dn3.sh Normal file
View file

@ -0,0 +1,77 @@
#!/usr/bin/env bash
# A Devnet 3 box (7 October 2026, the founder's clock: a fresh chain from genesis on 0.3.22, network igneum-devnet-3, every activation at 0,
# NO override file). Modelled on box-dn2.sh. The node runs with NET_ARGS (default "--devnet --devnet-suffix=3": own handshake magic,
# own data directory $F/$APPDIR; a live-devnet or Devnet 2 peer refuses it at the handshake), peered with SEED (comma list); one miner on
# card 0 with the box's vote key; PROVER=1 adds the segment prover loop (CHAIN_NAME igneum-devnet-3). NODE_BIN names the igneumd.
# FRESH=1 wipes $F/$APPDIR (a new genesis); UNSYNCED=1 adds --enable-unsynced-mining (the genesis boxes: a fresh chain's nodes start
# unsynced and must mine anyway). RPC_PORT, P2P_PORT, EVM_PORT: other ports on a box whose live node holds 26610/26611/26790
# (a standing box running a second node for Devnet 3: 36610/36611/36790). Nothing here touches the live devnet's node or miner.
set -uo pipefail
mkdir -p /root/fleet/pids; echo $$ > /root/fleet/pids/loop.pid # 15:3xZ 8 Oct 2026: kills by pid file only (main): loop.pid, node.pid, miner.pid under /root/fleet/pids
F=/root/fleet; OUT=$F/out; mkdir -p $F/in $OUT $F/$APPDIR $F/mine/packs; exec >> $OUT/dn3.log 2>&1
stamp() { date -u +%Y-%m-%dT%H:%M:%SZ; }
LABEL="${LABEL:-dn3}"; WALLET="${WALLET:-0x1919191919191919191919191919191919191919}"; SEED="${SEED:-}"; NODE_BIN="${NODE_BIN:-$F/in/igneumd-0322}"
APPDIR="${APPDIR:-dn3}"; PACK="${PACK:-dn3}"; CHAIN_NAME="${CHAIN_NAME:-igneum-devnet-3}" # 15:4xZ 8 Oct 2026: devnet-4 takes APPDIR=dn4 PACK=dn4 CHAIN_NAME=igneum-devnet-4 NET_ARGS="--devnet --devnet-suffix=4"
PROVER="${PROVER:-0}"; UNSYNCED="${UNSYNCED:-}"; RPC_PORT="${RPC_PORT:-26610}"; P2P_PORT="${P2P_PORT:-26611}"; EVM_PORT="${EVM_PORT:-26790}"; JSON_PORT="${JSON_PORT:-$((RPC_PORT+2080))}"; MINE="${MINE:-1}"
NET_ARGS="${NET_ARGS:---devnet --devnet-suffix=3}"; P2P_LISTEN="${P2P_LISTEN:-0.0.0.0:$P2P_PORT}"; MINER_ONLY="${MINER_ONLY:-0}"; ANNOUNCE="${ANNOUNCE:-}"; export -n MINER_ONLY # 21:3xZ 7 Oct 2026: never in the loop's environment (the puller restarts from it; MINER_ONLY=1 there would leave the node down at the next move) # MINER_ONLY=1: the node stays, only the miner loop restarts (with --announce ip:port when ANNOUNCE is set: the peer directory)
JSONF="--rpclisten-json=127.0.0.1:$JSON_PORT"; case " $NET_ARGS ${EXTRA_ARGS:-} " in *rpclisten-json*) JSONF="";; esac # 13:3xZ 8 Oct: the pool boxes already carry it in NET_ARGS (dn3-pool-b: "cannot be used multiple times")
MINER_BIN="${MINER_BIN:-$F/in/igneum-miner-0322}" # the 0.3.22 miner (sub-version 3 draw); the hive package's 0.3.20 miner is the live devnet's and never mines Devnet 3 (16:5xZ: every block BlockInvalid)
[ -x "$NODE_BIN" ] || { echo "RESULT dn3_failed $(stamp) no node binary at $NODE_BIN"; exit 2; }
[ "${MINE:-1}" = 1 ] && { [ -x "$MINER_BIN" ] || { echo "RESULT dn3_failed $(stamp) no 0.3.22 miner at $MINER_BIN"; exit 2; }; }
# the pack-id gate (main through the shipper, 7 Oct 2026 17:0xZ): BEFORE the miner starts, the worker's pack and the node's engine must come
# from the same igneum-pow pin. Tonight's shape (the shipper, 17:05Z): BY CONSTRUCTION, the pack the worker hashes is EXPORTED on this box by the
# paired 0.3.22 miner (MINER_BIN's sha equals PAIR_MINER_SHA16, default 07246920fd9fe895 = igneum-pow 017e7037, the node's pin), never a copied kit;
# a different miner sha or a pre-shipped pack directory is UNREADABLE and holds the miner. The id-equality read over RPC (a785001687d8688a against
# the kit's id) replaces it from the next cut when the node lane names the method (NODE_ID_CMD / PACK_ID_CMD).
pack_gate() {
local msha want; msha=$(sha256sum "$MINER_BIN" | cut -c1-16); want="${PAIR_MINER_SHA16:-07246920fd9fe895 c29f33bbbd284a12 dfdc6883aa79a76f fb147dd1754cbfc0 cfa9f5ca382e0efc} $(cat $F/in/pair-miners.txt 2>/dev/null | tr '\n' ' ')" # 8 Oct 2026 09:4xZ: the list is also a FILE the puller appends every move's miner sha to (the 10:05 move: 18 miners refused for a sha only in a hand-edited default) # the two paired 0.3.22 miners (hands / node-lane builds, both igneum-pow 017e7037)
if [ -n "${NODE_ID_CMD:-}" ] && [ -n "${PACK_ID_CMD:-}" ]; then
local pid nid; pid=$(eval "$PACK_ID_CMD" 2>/dev/null); nid=$(eval "$NODE_ID_CMD" 2>/dev/null)
[ -n "$pid" ] && [ -n "$nid" ] || { echo "RESULT dn3_pack_gate $(stamp) UNREADABLE pack_id=${pid:-none} node_id=${nid:-none}"; return 1; }
[ "$pid" = "$nid" ] || { echo "RESULT dn3_pack_gate $(stamp) REFUSED pack_id=$pid node_id=$nid"; return 1; }
echo "RESULT dn3_pack_gate $(stamp) PASS by id pack_id=$pid node_id=$nid"; return 0
fi
case " $want " in *" $msha "*) ;; *) echo "RESULT dn3_pack_gate $(stamp) UNREADABLE miner=$msha is not a paired miner ($want); a pre-shipped kit or another miner"; return 1;; esac
[ -e $F/mine/packs/$PACK ] && { echo "RESULT dn3_pack_gate $(stamp) UNREADABLE packs/$PACK already present before this export (a copied kit?)"; return 1; }
echo "RESULT dn3_pack_gate $(stamp) PASS by construction (miner $msha = pair, pack exported here by it, generator v4 sub-version 3)"; return 0
}
echo "RESULT dn3_start $(stamp) label=$LABEL node=$(sha256sum $NODE_BIN | cut -c1-16) miner=$(sha256sum $MINER_BIN 2>/dev/null | cut -c1-16) seed=${SEED:-none} prover=$PROVER mine=$MINE net=\"$NET_ARGS\" ports=$RPC_PORT/$P2P_PORT/$EVM_PORT"
command -v curl >/dev/null || { apt-get update -qq >/dev/null 2>&1; apt-get install -y -qq curl ca-certificates python3 >/dev/null 2>&1; }
if [ ! -x /opt/igneum/pkg/bin/igneum-miner ]; then
read -r PKG_PATH PKG_SHA PKG_VER <<< "$(curl -fsSL -m 30 https://dl.igneum.network/dl/public/igneum-downloads.json | python3 -c 'import sys,json; d=json.load(sys.stdin)["files"]["miner-hive"]; print(d["path"], d["sha256"], d["version"])')"
curl -fsSL -o $F/pkg.tgz "https://dl.igneum.network$PKG_PATH" && echo "$PKG_SHA $F/pkg.tgz" | sha256sum -c - >/dev/null && mkdir -p /opt/igneum/pkg && tar -C /opt/igneum/pkg --strip-components=1 -xzf $F/pkg.tgz || { echo "RESULT dn3_failed package"; exit 2; }
fi
B=/opt/igneum/pkg/bin
# the Devnet 3 node and its miner only (anchored on the dn3 appdir and this RPC port), never the live node beside it
# kills by pid file only (founder 8 Oct 2026, by construction: pkill and killall are shimmed to exit 97 on every box)
pidkill() { local P; P=$(cat "$F/pids/$1.pid" 2>/dev/null); [ -n "$P" ] && kill -0 "$P" 2>/dev/null && { kill -TERM "$P" 2>/dev/null; sleep 2; kill -0 "$P" 2>/dev/null && kill -9 "$P" 2>/dev/null; }; return 0; }
[ "$MINER_ONLY" = 1 ] || pidkill node; pidkill miner; sleep 2
[ "${FRESH:-0}" = 1 ] && { rm -rf $F/$APPDIR; mkdir -p $F/$APPDIR; [ -s $F/$APPDIR-node.log ] && mv $F/$APPDIR-node.log $F/$APPDIR-node.prev.log; echo "RESULT dn3_fresh $(stamp) appdir wiped for the genesis"; }
PEER=""; for sd in ${SEED//,/ }; do PEER="$PEER --addpeer=$sd"; done
UNS=""; [ "$UNSYNCED" = 1 ] && UNS="--enable-unsynced-mining"
if [ "$MINER_ONLY" != 1 ]; then
echo "=== dn3 node start $(stamp) $(sha256sum $NODE_BIN | cut -c1-16)" >> $F/$APPDIR-node.log
IGNEUM_PROOF_PROGRAM_IDS="${PROGRAM_IDS:-}" IGNEUM_PROOF_VERIFIER="${HOST_VERIFIER:-}" setsid nohup $NODE_BIN $NET_ARGS --appdir=$F/$APPDIR --rpclisten=0.0.0.0:$RPC_PORT --evm-rpclisten=127.0.0.1:$EVM_PORT $JSONF --listen=$P2P_LISTEN $PEER $UNS --unsaferpc --nodnsseed --disable-upnp --nologfiles --yes </dev/null >> $F/$APPDIR-node.log 2>&1 &
echo $! > $F/pids/node.pid
sleep 10
echo "RESULT dn3_node $(stamp) pid=$(pgrep -f '[/]root/fleet/in/igneumd-[0-9a-f]+ .* --appdir=/root/fleet/$APPDIR ' | head -1) version=$(grep -oE 'igneumd/[0-9]+\.[0-9]+\.[0-9]+-[0-9a-f]+' $F/$APPDIR-node.log | tail -1) digest=$(grep -o 'digest: [0-9a-f]*' $F/$APPDIR-node.log | tail -1 | awk '{print substr($2,1,16)}') network=$(grep -oiE 'igneum-devnet-[0-9][^ ,]*' $F/$APPDIR-node.log | head -1) genesis=$(grep -oiE 'genesis[^\n]{0,120}' $F/$APPDIR-node.log | grep -oE '[0-9a-f]{64}' | head -1 | cut -c1-16)"
for i in $(seq 1 30); do w="$($B/igneum-miner watch 1 grpc://127.0.0.1:$RPC_PORT 2>/dev/null | grep -o 'blocks=[0-9]*.*synced=[a-z]*' | tail -1)"; [ -n "$w" ] && break; sleep 5; done
echo "RESULT dn3_watch $(stamp) $(printf '%s' "$w" | sed -E 's/difficulty=[0-9.]* sink=[0-9a-f]* //')"
fi
rm -rf $F/mine/packs/$PACK.prev; [ -e $F/mine/packs/$PACK ] && mv $F/mine/packs/$PACK $F/mine/packs/$PACK.prev # the previous run's export, moved aside so the gate sees a clean slot
if [ "$MINE" = 1 ] && ! pack_gate; then echo "RESULT dn3_miner_refused $(stamp) the pack-id gate refused the place; node runs, miner off"; MINE=0; fi
if [ "$MINE" = 1 ]; then
cd $F/mine && rm -rf packs/$PACK && $MINER_BIN export-pack grpc://127.0.0.1:$RPC_PORT packs/$PACK > $OUT/dn3-export-pack.log 2>&1
( echo $BASHPID > $F/pids/miner-loop.pid; while :; do $MINER_BIN mine grpc://127.0.0.1:$RPC_PORT 1 100000000 "$LABEL" --exec-rpc http://127.0.0.1:$EVM_PORT --worker $WORKER_BIN --worker-args "--device 0 --pack packs/$PACK" --prepare-packs packs/$PACK-prepare --exit-on-seed-change ${ANNOUNCE:+--announce $ANNOUNCE} --evm-address "$WALLET" --payout-label "$LABEL" --status-secs 30 >> $OUT/dn3-miner.log 2>&1 & echo $! > $F/pids/miner.pid; wait $(cat $F/pids/miner.pid); sleep 5; done ) </dev/null >/dev/null 2>&1 &
echo "RESULT dn3_miner_started $(stamp) miner=$(sha256sum $MINER_BIN | cut -c1-16) announce=${ANNOUNCE:-none}"
# the engine's program id as the paired miner prints it ("class v4 program id <16 hex>", the node lane 17:0xZ): a785001687d8688a on sub-version 3,
# 1a4230699a6b9c60 is the 0.3.20 kit (known-failed); logged as the gate's id line, refused on the known-failed value
# the worker form prints no id (the node lane, main.rs 1471): a second one-thread CPU miner beside the worker for 20 s prints "class v4 program id <16 hex>"
timeout 60 $MINER_BIN mine grpc://127.0.0.1:$RPC_PORT 1 20 idread --engine igneum-pow --no-vote --stall-secs 0 > $OUT/dn3-idread.log 2>&1 </dev/null
pidl=$(grep -oE 'program id [0-9a-f]{16}' $OUT/dn3-idread.log | tail -n 1 | awk '{print $3}')
if [ "$pidl" = "1a4230699a6b9c60" ] || [ "$pidl" = "a785001687d8688a" ]; then pidkill miner; echo "RESULT dn3_program_id $(stamp) REFUSED $pidl is the shared devnet's id (edc4fa84 seeds), not Devnet 3's; miner stopped"; else echo "RESULT dn3_program_id $(stamp) ${pidl:-unread} (want ${WANT_PROGRAM_ID:-fce15bf61030be57}, the epoch-0 id of genesis 4020cb43; the id changes with the epoch seed every 3,600 DAA)"; fi
fi
if [ "$PROVER" = 1 ] && [ -x /opt/igneum-floor/bin/igneum-prove-host ]; then
cd $F && LABEL="$LABEL" WALLET="$WALLET" EXPORT_FROM=0 CHAIN_NAME=$CHAIN_NAME MINER=none RUN_HOURS=48 EVM="http://127.0.0.1:$EVM_PORT" GRPC="grpc://127.0.0.1:$RPC_PORT" setsid nohup python3 -u $F/in/box-prover.py </dev/null >> $OUT/dn3-prover-launch.log 2>&1 &
echo "RESULT dn3_prover_started $(stamp)"
fi

View file

@ -1,4 +1,5 @@
#!/usr/bin/env bash
. /root/fleet/in/pidkill.sh # kills by pid only (founder 8 Oct 2026)
# Ember Tune's two-knob ladder on a rented NVIDIA card (docs/plans/ember-tune.md, branch ember-tune): the miner runs
# throughout; the power ladder 100, 90, 80, 70, 60, 50% of the default limit at the unlocked clock (clamped at the
# card's reported minimum), then the clock ladder 90, 80, 70, 60% of the maximum graphics clock at the power the
@ -15,7 +16,7 @@ stamp() { date -u +%Y-%m-%dT%H:%M:%SZ; }
say() { echo "$(stamp) $*"; }
q() { nvidia-smi --query-gpu="$1" --format=csv,noheader,nounits -i 0 2>/dev/null | head -1 | tr -d ' '; }
NAME="$(q name)"; DRV="$(q driver_version)"; PDEF="$(q power.default_limit)"; PMIN="$(q power.min_limit)"; PMAX="$(q power.max_limit)"; CMAX="$(q clocks.max.graphics)"
pkill -x sp1-gpu-server 2>/dev/null; rm -f /tmp/sp1-cuda-*.sock
. /root/fleet/in/pidkill.sh; kill_sock_owner 'sp1-cuda-[0-9]*\.sock'
echo "RESULT start $(stamp) card=$NAME driver=$DRV power_default_w=$PDEF min_w=$PMIN max_w=$PMAX clock_max_mhz=$CMAX"
# can we set anything?
nvidia-smi -i 0 -pl "$PDEF" >/dev/null 2>&1 && PL_OK=1 || PL_OK=0
@ -27,7 +28,7 @@ rm -rf packs/devnet; $B/igneum-miner export-pack grpc://127.0.0.1:26610 packs/de
nohup $B/igneum-miner mine grpc://127.0.0.1:26610 1 100000000 "$LABEL" --worker $B/igneum-worker-cuda --worker-args "--device 0 --pack packs/devnet" \
--prepare-packs packs/prepare --exit-on-seed-change --evm-address "$WALLET" --payout-label "$LABEL" --status-secs 10 > $OUT/ember-miner.log 2>&1 &
MPID=$!; cd $F
cleanup() { nvidia-smi -i 0 -rgc >/dev/null 2>&1; [ "$PL_OK" = 1 ] && nvidia-smi -i 0 -pl "$PDEF" >/dev/null 2>&1; kill $MPID 2>/dev/null; pkill -f '[/]opt/igneum/pkg/bin/igneum-worker-cuda' 2>/dev/null; }
cleanup() { nvidia-smi -i 0 -rgc >/dev/null 2>&1; [ "$PL_OK" = 1 ] && nvidia-smi -i 0 -pl "$PDEF" >/dev/null 2>&1; kill_children $MPID; kill $MPID 2>/dev/null; }
trap cleanup EXIT
say "miner warming 90 s"; sleep 90
STEPS=$OUT/ember-steps.jsonl; : > $STEPS
@ -38,7 +39,7 @@ step() { # <label> <power_pct> <limit_w> <clock_cap_mhz or 0>
sleep "$SETTLE"
local n0; n0="$(grep -c 'STATUS' $OUT/ember-miner.log)"
( while :; do nvidia-smi --query-gpu=power.draw,clocks.sm,clocks.mem,temperature.gpu --format=csv,noheader,nounits -i 0 2>/dev/null; sleep 1; done ) > $OUT/ember-samp-$lab.csv & local sp=$!
sleep "$HOLD"; pkill -P $sp 2>/dev/null; kill $sp 2>/dev/null; sleep 1; kill -9 $sp 2>/dev/null
sleep "$HOLD"; kill_children $sp; kill $sp 2>/dev/null; sleep 1; kill -9 $sp 2>/dev/null
local n1; n1="$(grep -c 'STATUS' $OUT/ember-miner.log)"
local mhs; mhs="$(grep STATUS $OUT/ember-miner.log | grep -o ' now=[0-9.]*' | tail -n $((n1 - n0 > 0 ? n1 - n0 : 1)) | cut -d= -f2 | awk '{s+=$1; n++} END {if (n) printf "%.2f", s/n; else print 0}')"
local w gclk mclk tmax; read -r w gclk mclk tmax <<< "$(awk -F', *' '{w+=$1; g+=$2; m+=$3; if ($4+0 > t) t=$4+0; n++} END {if (n) printf "%.1f %.0f %.0f %d", w/n, g/n, m/n, t; else print "0 0 0 0"}' $OUT/ember-samp-$lab.csv)"

View file

@ -1,4 +1,5 @@
#!/usr/bin/env bash
. /root/fleet/in/pidkill.sh # kills by pid only (founder 8 Oct 2026)
# The prover-floor agent's known-failed case (6 October 2026, 13:05Z): rebuild sp1-gpu-server from patch v5 (the panic
# hook that turns a failed device allocation into "FLOOR abort ..." and exit 70) and re-run the 2^27 compressed point
# alone on a card it cannot fit; report the host's failing line, the server's FLOOR abort line and the seconds.
@ -19,10 +20,10 @@ echo "RESULT v5_build_exit $rc time_s=$(( $(date +%s) - t0 ))"
[ $rc -eq 0 ] || { grep -n -A6 '^error' $FLOOR/logs/build-server-v5.log | head -30; echo "RESULT v5_failed build"; exit 2; }
mkdir -p $FLOOR/home-v5/.sp1/bin && cp $FLOOR/target/release/sp1-gpu-server $FLOOR/home-v5/.sp1/bin/ && chmod +x $FLOOR/home-v5/.sp1/bin/sp1-gpu-server
echo "RESULT v5_server sha256=$(sha256sum $FLOOR/home-v5/.sp1/bin/sp1-gpu-server | cut -c1-16) bytes=$(stat -c %s $FLOOR/home-v5/.sp1/bin/sp1-gpu-server)"
pkill -9 -x sp1-gpu-server; rm -f /tmp/sp1-cuda-*.sock; sleep 2
kill_sock_owner 'sp1-cuda-[0-9]*\.sock'; sleep 2
t1=$(date +%s)
env HOME=$FLOOR/home-v5 SP1_PROVER=cuda RUST_LOG=off SP1_GPU_FLOOR_LOG=1 SP1_GPU_ELEMENT_THRESHOLD=134217728 timeout 900 $HOST $FLOOR/prove/proving/fixtures/fees-v1-shards2.json --mode compressed --shard 0 --out $OUT/v5-point.json > $OUT/v5-point.log 2>&1; rc=$?
echo "RESULT v5_point rc=$rc wall_s=$(( $(date +%s) - t1 ))"
grep -n -E 'FLOOR abort|panicked|Error|error|RESULT' $OUT/v5-point.log | head -12 | cut -c1-300
pkill -9 -x sp1-gpu-server; rm -f /tmp/sp1-cuda-*.sock
kill_sock_owner 'sp1-cuda-[0-9]*\.sock'
echo "RESULT v5_done $(stamp)"

View file

@ -1,15 +1,11 @@
#!/usr/bin/env bash
# Stops every stage process on a box (the matrix, Ember, the prover loop, their miners, workers, samplers and SP1
# servers); never the node. Run as a FILE (bash in/box-kill.sh) so that no pattern below can match the shell that runs
# it (an inline `pkill -f '[i]gneum-prove-host'` killed its own ssh shell on 6 October 2026, 12:38Z).
for i in 1 2; do
pkill -9 -f '[b]ash in/box-matrix.sh' ; pkill -9 -f '[b]ash in/box-ember.sh'; pkill -9 -f '[b]ash in/box-prover.sh'; pkill -9 -f '[p]ython3 -u /root/fleet/in/box-prover.py'
pkill -9 -x igneum-miner; pkill -9 -f '[/]opt/igneum/pkg/bin/igneum-worker-cuda'; pkill -9 -x sp1-gpu-server; pkill -9 -f '[/]opt/igneum-floor/(bin|target|prove)/.*igneum-prove-(host|export)' # pkill -x cannot match a name over 15 characters (igneum-worker-cuda, igneum-prove-host)
pkill -9 -f '[n]vidia-smi --query'
sleep 2
done
rm -f /tmp/sp1-cuda-*.sock
# Stops every stage process on a box through pid files only (the founder, 8 Oct 2026: pkill and killall exit 97 on every box);
# never the node. The stage scripts write their pids under /root/fleet/pids (matrix, ember, prover, prover-loop, miner-loop, miner,
# sampler, sp1-server); a child the scripts spawned is reached through its parent's pid (kill_children walks by parent pid).
. /root/fleet/in/pidkill.sh
for n in matrix ember prover-loop prover miner-loop miner sampler; do p=$(cat /root/fleet/pids/$n.pid 2>/dev/null); [ -n "$p" ] && kill_children "$p" -9; kill_pidfile $n -9 0; done
kill_sock_owner 'sp1-cuda-[0-9]*\.sock'
nvidia-smi -i 0 -rgc >/dev/null 2>&1
d="$(nvidia-smi --query-gpu=power.default_limit --format=csv,noheader,nounits -i 0 | tr -d ' ' | cut -d. -f1)"; [ -n "$d" ] && nvidia-smi -i 0 -pl "$d" >/dev/null 2>&1
cd /root/fleet/out 2>/dev/null && for f in matrix.log ember.log prover.log prover-launch.log rows.jsonl; do [ -f "$f" ] && mv "$f" "$f.$(date +%s).old"; done
echo "count=$(pgrep -c -f '[b]ash in/box-|^python3 -u /root/fleet/in/box-prover')+$(pgrep -c -x igneum-miner)+$(pgrep -c -f '[/]opt/igneum/pkg/bin/igneum-worker-cuda')+$(pgrep -c -x sp1-gpu-server)+$(pgrep -c -f '[/]opt/igneum-floor/(bin|target|prove)/.*igneum-prove-host') mem=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0 | tr -d ' ')"
echo "stopped by pid files: $(ls /root/fleet/pids 2>/dev/null | tr '\n' ' ') mem=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0 2>/dev/null)"

View file

@ -1,4 +1,5 @@
#!/usr/bin/env bash
. /root/fleet/in/pidkill.sh # kills by pid only (founder 8 Oct 2026)
# Phase 1 on one rented card: the memory matrix of docs/analysis/prover-floor.md on the real card. Every point is one
# igneum-prove-host run against a fresh sp1-gpu-server (killed and its socket unlinked around every point), with an
# nvidia-smi sampler at 1 s (memory.used, power.draw, utilization); peak = max memory.used, base = the reading just
@ -29,10 +30,10 @@ SAMP=""
# one nvidia-smi call a second in a loop: `nvidia-smi -l 1` buffers its file output in 4 KB chunks and ignored SIGTERM
# on the 3080 box (12:19Z), which hung the first matrix in sampler_stop
sampler_start() { ( while :; do nvidia-smi --query-gpu=timestamp,memory.used,power.draw,utilization.gpu --format=csv,noheader,nounits -i 0 2>/dev/null; sleep 1; done ) > "$1" & SAMP=$!; }
sampler_stop() { [ -n "$SAMP" ] && { pkill -P $SAMP 2>/dev/null; kill $SAMP 2>/dev/null; sleep 1; kill -9 $SAMP 2>/dev/null; }; SAMP=""; }
sampler_stop() { [ -n "$SAMP" ] && { kill_children $SAMP; kill $SAMP 2>/dev/null; sleep 1; kill -9 $SAMP 2>/dev/null; }; SAMP=""; }
peak_of() { awk -F', *' 'NR>0 {if ($2+0 > m) m=$2+0} END {print m+0}' "$1"; }
mean_col() { awk -F', *' -v c="$2" '{s+=$c; n++} END {if (n) printf "%.1f", s/n; else print 0}' "$1"; }
kill_server() { pkill -x sp1-gpu-server 2>/dev/null; sleep 2; pkill -9 -x sp1-gpu-server 2>/dev/null; rm -f /tmp/sp1-cuda-*.sock; }
kill_server() { kill_sock_owner 'sp1-cuda-[0-9]*\.sock'; }
idle_mib() { sleep 3; q memory.used; }
# 1. idle
@ -55,7 +56,7 @@ miner_start() {
--prepare-packs packs/prepare --exit-on-seed-change --evm-address "$WALLET" --payout-label "$LABEL" --status-secs 10 > $OUT/miner.log 2>&1 &
MPID=$!; cd $F
}
miner_stop() { [ -n "$MPID" ] && { kill $MPID 2>/dev/null; sleep 2; kill -9 $MPID 2>/dev/null; }; pkill -f '[/]opt/igneum/pkg/bin/igneum-worker-cuda' 2>/dev/null; pkill -x igneum-miner 2>/dev/null; MPID=""; sleep 3; }
miner_stop() { [ -n "$MPID" ] && { kill_children $MPID; kill $MPID 2>/dev/null; sleep 2; kill -9 $MPID 2>/dev/null; }; MPID=""; sleep 3; }
miner_rate() { # mean of the STATUS now= values in the last N lines
grep STATUS $OUT/miner.log | grep -o ' now=[0-9.]*' | tail -n "${1:-10}" | cut -d= -f2 | awk '{s+=$1; n++} END {if (n) printf "%.2f", s/n; else print 0}'
}

View file

@ -190,7 +190,18 @@ def fnv1a(s):
return h
def kill_server():
if CARD: subprocess.run(f"rm -f /tmp/sp1-cuda-{DEV}.sock", shell=True) # a rig: never another card's server
else: subprocess.run("pkill -x sp1-gpu-server; sleep 1; rm -f /tmp/sp1-cuda-*.sock", shell=True)
else:
# kills by pid only (founder 8 Oct 2026, by construction): the server is found through the pid that owns its unix socket
# (ss -xlp on /tmp/sp1-cuda-*.sock), written to /root/fleet/pids/sp1-server.pid, then signalled by that pid; never by name
o=subprocess.run("ss -xlp 2>/dev/null | grep -E '/tmp/sp1-cuda-[0-9]*\\.sock' | grep -oE 'pid=[0-9]+' | cut -d= -f2 | sort -u", shell=True, capture_output=True, text=True).stdout.split()
os.makedirs("/root/fleet/pids", exist_ok=True)
for s in o:
try:
pid=int(s); open("/root/fleet/pids/sp1-server.pid","w").write(f"{pid}\n"); os.kill(pid, 15); time.sleep(1)
try: os.kill(pid, 9)
except ProcessLookupError: pass
except Exception: pass
subprocess.run("rm -f /tmp/sp1-cuda-*.sock", shell=True)
MPROC = None
def miner_start():
global MPROC
@ -204,7 +215,7 @@ def miner_start():
def miner_stop():
global MPROC
if MPROC: MPROC.terminate(); time.sleep(2); MPROC.kill(); MPROC = None
subprocess.run(f"pkill -f '[/]opt/igneum/pkg/bin/igneum-worker-cuda --device {DEV} '", shell=True)
pass # the worker is the miner's child; MPROC.terminate() above ends it (no kill by name: founder 8 Oct 2026)
def miner_rate(n=6):
try:
vals = [float(l.split(" now=")[1].split()[0]) for l in open(f"{OUT}/prover-miner.log").read().split("\n") if "STATUS" in l and " now=" in l][-n:]

View file

@ -1,4 +1,5 @@
#!/usr/bin/env bash
. /root/fleet/in/pidkill.sh # kills by pid only (founder 8 Oct 2026)
# Phase 3 on an 8-card rig (RunPod pod, a container: no systemd, so the rig installer's unit steps are exercised with
# --preflight-only and --dry-run and the units' own scripts are run by hand). After box-setup.sh (node, workers, the
# patched server for the rig's arch, the cuda host):
@ -35,7 +36,7 @@ for d in $(seq 0 $((N-1))); do
done
cd $F; sleep 60
( while :; do nvidia-smi --query-gpu=index,power.draw,memory.used,utilization.gpu --format=csv,noheader,nounits; sleep 1; done ) > $OUT/rig-samp-mine.csv & SP=$!
sleep "$MINE_SECS"; pkill -P $SP; kill $SP 2>/dev/null
sleep "$MINE_SECS"; kill_children $SP; kill $SP 2>/dev/null
sum=0
for d in $(seq 0 $((N-1))); do
r=$(grep STATUS $OUT/rig-miner-$d.log | grep -o ' now=[0-9.]*' | tail -n 10 | cut -d= -f2 | awk '{s+=$1; n++} END {if (n) printf "%.2f", s/n; else print 0}')
@ -43,10 +44,10 @@ for d in $(seq 0 $((N-1))); do
echo "RESULT miner card=$d mhs=$r watts=$w"; sum=$(awk -v a=$sum -v b=$r 'BEGIN {print a+b}')
done
echo "RESULT rig_miners $(stamp) cards=$N sum_mhs=$sum watts=$(awk -F', *' '{s+=$2; n++} END {printf "%.0f", s/n*'$N'}' $OUT/rig-samp-mine.csv)"
for p in "${pids[@]}"; do kill $p 2>/dev/null; done; pkill -f '[/]opt/igneum/pkg/bin/igneum-worker-cuda'; sleep 3
for p in "${pids[@]}"; do kill_children $p; kill $p 2>/dev/null; done; sleep 3
fi
# C + D. seven core-only provers and one compressing card, in parallel
pkill -9 -x sp1-gpu-server; rm -f /tmp/sp1-cuda-*.sock
kill_sock_owner 'sp1-cuda-[0-9]*\.sock'
prove_loop() { # <device> <mode> <threshold> <seconds>
local d=$1 mode=$2 thr=$3 secs=$4 t0=$(date +%s) n=0 tot=0
while [ $(( $(date +%s) - t0 )) -lt $secs ]; do
@ -63,9 +64,9 @@ lp=()
for d in $(seq 0 $((N-2))); do prove_loop $d core 33554432 "$PROVE_SECS" & lp+=($!); done
prove_loop $((N-1)) compressed 67108864 "$PROVE_SECS" & lp+=($!)
for p in "${lp[@]}"; do wait $p; done
pkill -P $SP; kill $SP 2>/dev/null
kill_children $SP; kill $SP 2>/dev/null
echo "RESULT rig_prove $(stamp) host_ram_used_mb_max=$(awk '{if ($2>m) m=$2} END {print m}' $OUT/rig-samp-prove.csv) core_cards=$((N-1)) core_per_min=$(grep 'RESULT prove_loop' $OUT/rig.log | grep 'mode=core' | grep -o 'per_min=[0-9.]*' | cut -d= -f2 | awk '{s+=$1} END {printf "%.2f", s}') compressed_per_min=$(grep 'RESULT prove_loop' $OUT/rig.log | grep 'mode=compressed' | grep -o 'per_min=[0-9.]*' | cut -d= -f2)"
pkill -9 -x sp1-gpu-server; rm -f /tmp/sp1-cuda-*.sock
kill_sock_owner 'sp1-cuda-[0-9]*\.sock'
# E. the hand-off cost by file
cf=$(ls -S $F/mine/*.bin $OUT/*.bin 2>/dev/null | head -1); [ -z "$cf" ] && cf=$(find / -name 'block-*-core.bin' -size +1M 2>/dev/null | head -1)
if [ -n "$cf" ]; then
@ -80,5 +81,5 @@ t0=$(date +%s)
env HOME=$FLOOR/home IGNEUM_CUDA_DEVICE=$((N-1)) SP1_PROVER=cuda RUST_LOG=off timeout 1800 $HOST --mode chain --chain "$list" --prover "$WALLET" --save-shards --out $OUT/rig-chain.json > $OUT/rig-chain.log 2>&1; rc=$?
echo "RESULT chain rc=$rc wall_s=$(( $(date +%s) - t0 )) blocks=$(echo "$list" | tr ',' '\n' | wc -l) $(grep -E '^RESULT chain' $OUT/rig-chain.log | tail -1 | cut -c1-200)"
else echo "RESULT chain skipped: no consecutive live fixtures yet (needs the exec layer)"; fi
pkill -9 -x sp1-gpu-server; rm -f /tmp/sp1-cuda-*.sock
kill_sock_owner 'sp1-cuda-[0-9]*\.sock'
echo "RESULT rig_done $(stamp)"

3
tools/fleet/dn3-kill.sh Normal file
View file

@ -0,0 +1,3 @@
#!/usr/bin/env bash
# kills by pid file only (founder 8 Oct 2026): the old name-pattern kills are gone; fleet-stop.sh all stops node, miner and loop through /root/fleet/pids.
bash /root/fleet/in/fleet-stop.sh all

111
tools/fleet/fleet-puller.sh Normal file
View file

@ -0,0 +1,111 @@
#!/usr/bin/env bash
# Igneum fleet puller (main's order 21:5x BST 7 Oct 2026): the box fetches a SIGNED move file from the fleet file host on a
# one-minute timer, verifies the signature against the fleet key this box already trusts (its authorized_keys), downloads the
# named binary pair, checks every sha, and at the named minute (UTC epoch) restarts its Devnet 3 node and miner together from
# the running box-dn3.sh's own environment, then reads the version string and digest back and posts the line to the intake.
# A proxy outage never blocks a digest-moving release again: nothing here needs ssh. Env: BASE (file host prefix), LABEL,
# INTAKE_URL + INTAKE_KEY (the report-only intake key that every miner package carries). State under /root/fleet/move.
set -u
F=/root/fleet; M=$F/move; mkdir -p $M; cd $M
LABEL="${LABEL:?}"; BASE="${BASE:?}"; INTAKE_URL="${INTAKE_URL:-}"; INTAKE_KEY="${INTAKE_KEY:-}"
stamp(){ date -u +%Y-%m-%dT%H:%M:%SZ; }
log(){ echo "$(stamp) $*" >> $M/puller.log; }
post(){ # post "<title>" "<body>"
echo "$(stamp) $1 | $2" >> $M/readback.log
[ -n "$INTAKE_URL" ] && [ -n "$INTAKE_KEY" ] && python3 - "$1" "$2" "$LABEL" "$INTAKE_URL" "$INTAKE_KEY" <<'PY' >/dev/null 2>&1
import sys, json, urllib.request
t,b,l,u,k=sys.argv[1:]
req=urllib.request.Request(u, data=json.dumps({"from":"fleet:"+l,"kind":"note","title":t[:300],"body":b[:4000]}).encode(), headers={"Content-Type":"application/json","x-igneum-key":k}, method="POST")
try: urllib.request.urlopen(req, timeout=20).read()
except Exception as e: print(e)
PY
true; }
awk '{print "igneum-fleet-move " $1, $2}' /root/.ssh/authorized_keys > $M/allowed_signers
jq_(){ python3 -c 'import sys,json; d=json.load(open(sys.argv[1])); v=d
for k in sys.argv[2].split("."): v=v[k] if isinstance(v,dict) else v[int(k)]
print(v if not isinstance(v,(list,dict)) else json.dumps(v))' "$@" 2>/dev/null; }
log "puller start label=$LABEL base=$BASE"
while :; do
if curl -fsS -m 25 -o $M/move.json.tmp "$BASE/move.json" && curl -fsS -m 25 -o $M/move.json.sig.tmp "$BASE/move.json.sig"; then
if ssh-keygen -Y verify -f $M/allowed_signers -I igneum-fleet-move -n igneum-fleet-move -s $M/move.json.sig.tmp < $M/move.json.tmp >/dev/null 2>&1; then
mv -f $M/move.json.tmp $M/move.json; mv -f $M/move.json.sig.tmp $M/move.json.sig
else log "BAD SIGNATURE on move.json, ignored"; rm -f $M/move.json.tmp $M/move.json.sig.tmp; sleep 60; continue; fi
else [ -f $M/move.json ] || { sleep 60; continue; }; fi
id=$(jq_ $M/move.json id); at=$(jq_ $M/move.json at_epoch); [ -n "$id" ] || { log "move.json without id"; sleep 60; continue; }
# a staggered move (8 Oct 2026, the 0.3.25 re-execution from genesis): "delays": {"<label>": <seconds>} adds to at_epoch for that box; absent = 0
dly=$(jq_ $M/move.json delays.$LABEL 2>/dev/null); case "$dly" in ''|*[!0-9]*) dly=0;; esac
if [ "${at:-0}" -gt 0 ] 2>/dev/null; then at=$((at+dly)); fi
if [ -e $M/applied-$id ]; then sleep 60; continue; fi
pair=hands; for l in $(jq_ $M/move.json node_lane_labels | tr -d '[]",'); do [ "$l" = "$LABEL" ] && pair=node_lane; done
url=$(jq_ $M/move.json pairs.$pair.url); tsha=$(jq_ $M/move.json pairs.$pair.sha); dsha=$(jq_ $M/move.json pairs.$pair.igneumd_sha); msha=$(jq_ $M/move.json pairs.$pair.miner_sha)
if [ ! -e $M/fetched-$id ]; then
mkdir -p $M/$id && curl -fsS -m 600 -o $M/$id/pair.tgz "$url" || { log "fetch failed $url"; sleep 60; continue; }
echo "$tsha $M/$id/pair.tgz" | sha256sum -c - >/dev/null 2>&1 || { log "TARBALL SHA MISMATCH $id"; rm -f $M/$id/pair.tgz; sleep 60; continue; }
tar -xzf $M/$id/pair.tgz -C $M/$id && [ "$(sha256sum $M/$id/igneumd | cut -c1-64)" = "$dsha" ] && [ "$(sha256sum $M/$id/igneum-miner | cut -c1-64)" = "$msha" ] || { log "BINARY SHA MISMATCH $id"; rm -rf $M/$id; sleep 60; continue; }
chmod +x $M/$id/igneumd $M/$id/igneum-miner; touch $M/fetched-$id
# the pack gate's pair list by file: this move's miner sha (16 hex) appended once, so box-dn3.sh accepts the pair at the minute without a hand edit
grep -qx "${msha:0:16}" $F/in/pair-miners.txt 2>/dev/null || echo "${msha:0:16}" >> $F/in/pair-miners.txt
post "FLEET MOVE $id FETCHED $LABEL" "pair=$pair igneumd=${dsha:0:16} miner=${msha:0:16} at_epoch=$at"
log "fetched $id pair=$pair"
fi
p0=$(pgrep -of '[b]ash in/box-dn3.sh'); [ -n "$p0" ] && tr '\0' '\n' < /proc/$p0/environ | grep -E '^[A-Z_][A-Z0-9_]*=' | grep -vE '^(PWD|OLDPWD|SHLVL|_|TERM|SHELL|LS_COLORS|HOSTNAME|SSH_[A-Z_]+|LANG|LC_[A-Z_]+|LOGNAME|USER|MAIL|MINER_ONLY)=' > $M/env-last.tmp && [ -s $M/env-last.tmp ] && mv -f $M/env-last.tmp $M/env-last
now=$(date +%s)
if [ "${at:-0}" -le 0 ] 2>/dev/null; then sleep 60; continue; fi
if [ "$now" -lt "$at" ]; then d=$((at-now)); [ $d -gt 60 ] && d=60; sleep $d; continue; fi
# 12:1xZ 8 Oct 2026 (shipper): a box still re-walking at the minute waits for its own restart until its state reads right. move.json may carry
# "require_root": {"height":"0x6687","root":"0x3f53a7b9"}: the box's own EVM (EVM_PORT from env-last, default 26790) must answer that root
# at that height before this box applies; otherwise HELD (posted once every 10 minutes) and re-checked each minute. No EVM answer = held too.
# 15:0xZ 8 Oct 2026 (node lane, the acaf08b0 move): "require_replay_done": true holds a box whose executor has not reached its sink since
# the node's last start (the log's last of "replaying from genesis"/"no snapshot to resume" vs "records up to the tip N are continuous")
if [ "$(jq_ $M/move.json require_replay_done)" = "True" ] || [ "$(jq_ $M/move.json require_replay_done)" = "true" ]; then
last=$(grep -anE 'replaying from genesis|no snapshot to resume from|records up to the tip [0-9]+ are continuous' $F/dn3-node.log 2>/dev/null | tail -n 1)
case "$last" in *"are continuous"*) ;; *) hl=$M/held-replay-$id-stamp; if [ ! -f $hl ] || [ $(( $(date +%s) - $(stat -c %Y $hl) )) -ge 900 ]; then post "FLEET MOVE $id HELD $LABEL" "replay not done: $(echo "$last" | cut -c1-120)"; touch $hl; fi; sleep 60; continue;; esac
fi
rh=$(jq_ $M/move.json require_root.height); rr=$(jq_ $M/move.json require_root.root)
if [ -n "$rh" ] && [ -n "$rr" ]; then
ep=$(grep -oE '^EVM_PORT=.*' $M/env-last 2>/dev/null | head -n 1 | cut -d= -f2 | tr -d "'\""); ep=${ep:-26790}
blk=$(curl -s -m 8 -X POST -H 'content-type: application/json' --data "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"eth_getBlockByNumber\",\"params\":[\"$rh\",false]}" http://127.0.0.1:$ep 2>/dev/null)
got=$(echo "$blk" | grep -oE '"stateRoot":"0x[0-9a-f]+"' | cut -d'"' -f4); gh=$(echo "$blk" | grep -oE '"hash":"0x[0-9a-f]+"' | head -n 1 | cut -d'"' -f4); rhash=$(jq_ $M/move.json require_root.hash)
if [ "${got:0:${#rr}}" != "$rr" ] || { [ -n "$rhash" ] && [ "${gh:2:${#rhash}}" != "$rhash" ]; }; then
hl=$M/held-$id-stamp; if [ ! -f $hl ] || [ $(( $(date +%s) - $(stat -c %Y $hl) )) -ge 600 ]; then post "FLEET MOVE $id HELD $LABEL" "block $rh reads hash ${gh:2:8} root ${got:-none}, wanted ${rhash:-any}/$rr (re-walk not done); re-checked each minute"; touch $hl; fi
sleep 60; continue
fi
fi
# THE MINUTE: node and miner together, from the running box-dn3.sh's own environment
pid=$(cat /root/fleet/pids/loop.pid 2>/dev/null); [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null || pid=$(pgrep -of '[b]ash in/box-dn3.sh')
# the restart environment: the running box-dn3.sh's (never MINER_ONLY: a loop restarted alone carries it, and it would leave the node down), else env-last (written by every hand start since 21:4xZ 7 Oct 2026)
if [ -n "$pid" ]; then tr '\0' '\n' < /proc/$pid/environ | grep -E '^[A-Z_][A-Z0-9_]*=' | grep -vE '^(PWD|OLDPWD|SHLVL|_|TERM|SHELL|LS_COLORS|HOSTNAME|SSH_[A-Z_]+|LANG|LC_[A-Z_]+|LOGNAME|USER|MAIL|MINER_ONLY)=' > $M/env-$id; else log "no running box-dn3.sh; using last env"; [ -f $M/env-last ] && grep -vE '^MINER_ONLY=' $M/env-last > $M/env-$id; fi
[ -s $M/env-$id ] || { post "FLEET MOVE $id FAILED $LABEL" "no box-dn3.sh environment to restart from"; touch $M/applied-$id; sleep 60; continue; }
cp $M/env-$id $M/env-last
RPC=$(grep -E '^RPC_PORT=' $M/env-$id | cut -d= -f2); RPC=${RPC:-26610}
bash $F/in/fleet-stop.sh all >/dev/null 2>&1; sleep 2
true
cd $F/in && NB=$(grep -oE '^NODE_BIN=.*' $M/env-last | cut -d= -f2 | tr -d "'\"" | xargs -n1 basename 2>/dev/null); NB=${NB:-igneumd-0322}; MB=$(grep -oE '^MINER_BIN=.*' $M/env-last | cut -d= -f2 | tr -d "'\"" | xargs -n1 basename 2>/dev/null); MB=${MB:-igneum-miner-0322}; mv -f $NB $NB.pre-$id 2>/dev/null; mv -f $MB $MB.pre-$id 2>/dev/null
cp $M/$id/igneumd $NB && cp $M/$id/igneum-miner $MB && chmod +x $NB $MB; cd $M
APPDIR_LOG=$(grep -oE '^APPDIR=.*' $M/env-last 2>/dev/null | head -n 1 | cut -d= -f2 | tr -d "'\""); APPDIR_LOG=${APPDIR_LOG:-dn3} # 17:1xZ 8 Oct: the marker and the read-back follow the env's APPDIR (devnet-4 logs to dn4-node.log; the 16:50Z APPLIED lines read version= digest= empty for this)
mk="=== fleet move $id $(date -u +%T)"; echo "$mk" >> $F/$APPDIR_LOG-node.log
# every value quoted before sourcing (09:05Z 8 Oct 2026: a bare NET_ARGS=--devnet --devnet-suffix=3 line ran "--devnet-suffix=3" as a command and nine boxes came up with no node)
python3 - "$M/env-$id" <<'PY' > $M/env-$id.quoted
import sys, shlex
for line in open(sys.argv[1]):
line=line.rstrip("\n")
if "=" not in line or not line.split("=",1)[0].replace("_","").isalnum(): continue
k,v=line.split("=",1); v=v.strip()
if len(v)>=2 and v[0]==v[-1] and v[0] in "'\"": v=v[1:-1]
print(f"{k}={shlex.quote(v)}")
PY
(cd $F && set -a && . $M/env-$id.quoted && set +a && setsid nohup bash in/box-dn3.sh </dev/null >/dev/null 2>&1 &)
want_v=$(jq_ $M/move.json want_version); want_d=$(jq_ $M/move.json want_digest); rb=""; APPDIR_LOG=$(grep -oE '^APPDIR=.*' $M/env-last 2>/dev/null | head -n 1 | cut -d= -f2 | tr -d "'\""); APPDIR_LOG=${APPDIR_LOG:-dn3} # 16:0xZ 8 Oct: devnet-4 logs to dn4-node.log
for i in $(seq 1 18); do sleep 10
v=$(awk -v m="$mk" '$0==m{f=1} f' $F/${APPDIR_LOG:-dn3}-node.log | grep -oE 'igneumd/[0-9]+\.[0-9]+\.[0-9]+-[0-9a-f]+' | tail -n 1); d=$(awk -v m="$mk" '$0==m{f=1} f' $F/${APPDIR_LOG:-dn3}-node.log | grep -o 'digest: [0-9a-f]*' | tail -n 1 | cut -c9-24)
w=$(/opt/igneum/pkg/bin/igneum-miner watch 1 grpc://127.0.0.1:$RPC 2>/dev/null | grep -oE 'blocks=[0-9]+|peers=[0-9]+|synced=[a-z]+' | tr '\n' ' ')
[ -n "$v" ] && [ -n "$d" ] && case "$w" in *synced=true*) [[ "$w" != *peers=0* ]] && break;; esac
done
m=$(pgrep -fc "igneum-miner(-0322)? mine grpc://127.0.0.1:$RPC "); ok=MISMATCH; [ "$v" = "$want_v" ] && [ "$d" = "$want_d" ] && ok=MATCH
fp=$(grep -oE 'igneum-pow fingerprint [0-9a-f]{16}[^ ]*' $F/dn3-node.log 2>/dev/null | tail -n 1); fpb=$(grep -aoE 'IGNEUM_POW_FINGERPRINT=[0-9a-f]{16}' $M/$id/igneumd 2>/dev/null | head -n 1); fp="${fp:-igneum-pow fingerprint none} binary ${fpb:-IGNEUM_POW_FINGERPRINT=none}" # 12:5xZ 8 Oct (shipper): the freeze's crate fingerprint from the node's start line; another value is a stop
post "FLEET MOVE $id APPLIED $LABEL $ok" "version=$v digest=$d $w miner_procs=$m want=$want_v/$want_d rpc=$RPC ${fp:-igneum-pow fingerprint none}"
touch $M/applied-$id; log "applied $id $ok $v $d $w"
# 12:2xZ 8 Oct (node lane): bans persist in the node's DB; once this box's state at the reference block reads equal, unban every address it holds
if [ -n "$rh" ] && [ -n "$rr" ]; then ( cd /root/fleet && EVM_PORT=$(grep -oE '^EVM_PORT=.*' $M/env-last | cut -d= -f2 | tr -d "'\"") RPC_PORT=$RPC setsid nohup bash in/unban-all.sh "$rh" "$rr" "$rhash" </dev/null >> $M/unban.out 2>&1 & ); fi
sleep 60
done

13
tools/fleet/fleet-stop.sh Executable file
View file

@ -0,0 +1,13 @@
#!/usr/bin/env bash
# 15:3xZ 8 Oct 2026 (main: kills by pid file only, never by name): stops the box's Devnet 3 pieces by the pids box-dn3.sh wrote under
# /root/fleet/pids: fleet-stop.sh miner | node | loop | all. The miner process itself is the miner-loop's child (read from the loop pid).
P=/root/fleet/pids; what="${1:-all}"
kp(){ f=$P/$1.pid; [ -s $f ] || return 0; pid=$(cat $f); kill -0 $pid 2>/dev/null || { rm -f $f; return 0; }; kill ${2:--TERM} $pid 2>/dev/null; sleep ${3:-1}; kill -0 $pid 2>/dev/null && kill -9 $pid 2>/dev/null; rm -f $f; echo "stopped $1 pid $pid"; }
kids(){ [ -s $P/$1.pid ] && pgrep -P $(cat $P/$1.pid) 2>/dev/null; }
case "$what" in
loop) kp loop -9 0;;
miner) for c in $(kids miner-loop); do kill -9 $c 2>/dev/null; for g in $(pgrep -P $c 2>/dev/null); do kill -9 $g 2>/dev/null; done; done; kp miner-loop -9 0;;
node) kp node -TERM 6;;
all) kp loop -9 0; for c in $(kids miner-loop); do kill -9 $c 2>/dev/null; for g in $(pgrep -P $c 2>/dev/null); do kill -9 $g 2>/dev/null; done; done; kp miner-loop -9 0; kp node -TERM 6;;
esac
echo "left: node=$(pgrep -fc '[a]ppdir=/root/fleet/dn') miners=$(pgrep -fc '[i]gneum-miner(-0322)? mine') loops=$(pgrep -fc '^bash in/box-dn3.sh')"

3
tools/fleet/kill-node.sh Normal file
View file

@ -0,0 +1,3 @@
#!/usr/bin/env bash
# kill-node.sh <sha>: stops the box's node through its pid file only (founder 8 Oct 2026); the sha argument is kept for the callers' form and read back, not matched.
. /root/fleet/in/pidkill.sh; kill_pidfile node -TERM 4; true

View file

@ -78,7 +78,7 @@ class Box:
return f"{F}/in/{name}"
# ---- node ----
def stop_node(self):
self.run(f"pkill -f '[/]root/fleet/in/igneumd|[/]opt/igneum/pkg/bin/igneumd'; sleep 3; pkill -9 -f '[/]root/fleet/in/igneumd|[/]opt/igneum/pkg/bin/igneumd' 2>/dev/null; true", 40)
self.run(f"bash {F}/in/fleet-stop.sh node >/dev/null 2>&1; . {F}/in/pidkill.sh; kill_pidfile node -TERM 3; true", 40) # by pid file only (founder 8 Oct 2026)
def start_node(self, binary, override_local, peers, devnet_suffix=None, verifier=None, extra=(), appdir=None, log=None, unsynced_mining=False):
"""Writes the override file, kills the old node (anchored on its path), starts the new one; returns the digest it prints."""
appdir = appdir or (f"{F}/dn2" if devnet_suffix else f"{F}/node"); log = log or (f"{F}/dn2-node.log" if devnet_suffix else f"{F}/node.log")
@ -132,11 +132,11 @@ class Box:
log = log or f"{F}/out/miner-{device}.log"
self.check(f"cd {F}/mine 2>/dev/null || mkdir -p {F}/mine/packs && cd {F}/mine; [ -d packs/{pack} ] || {B}/igneum-miner export-pack {grpc} packs/{pack} >/dev/null 2>&1; setsid nohup {B}/igneum-miner mine {grpc} 1 100000000 {shlex.quote(label)} --worker {B}/igneum-worker-cuda --worker-args '--device {device} --pack packs/{pack}' --prepare-packs packs/prepare-{device} --exit-on-seed-change --evm-address {wallet} --payout-label {shlex.quote(label)} --status-secs 30 </dev/null >> {log} 2>&1 & echo $!", 90)
return int(self.run("pgrep -x igneum-miner | tail -1", 20)[1].strip() or 0)
def stop_miners(self): self.run("pkill -x igneum-miner; pkill -f '[/]opt/igneum/pkg/bin/igneum-worker-cuda'; true", 30)
def stop_miners(self): self.run(f"bash {F}/in/fleet-stop.sh miner >/dev/null 2>&1; . {F}/in/pidkill.sh; kill_pidfile miner; true", 30) # by pid file only
def start_prover(self, label, wallet, hub_peer="", threshold="", miner="keep"):
self.check(f"cd {F} && chmod +x in/box-prover.sh && LABEL={shlex.quote(label)} WALLET={wallet} HUB_PEER={hub_peer} THRESHOLD={threshold} MINER={miner} setsid nohup in/box-prover.sh </dev/null >/dev/null 2>&1 & echo started", 60)
def stop_all(self):
self.run(f"bash {F}/in/box-kill.sh >/dev/null 2>&1; pkill -f '[p]ython3 -u /root/fleet/in/box-prover.py'; pkill -x igneum-miner; pkill -f '[/]opt/igneum/pkg/bin/igneum-worker-cuda'; pkill -x sp1-gpu-server; true", 60)
self.run(f"bash {F}/in/box-kill.sh >/dev/null 2>&1; true", 60) # box-kill.sh stops by pid files
# ---- provider ----
def destroy(self):
import sys; sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

View file

@ -0,0 +1,9 @@
#!/usr/bin/env bash
# Kills by pid only (the founder, 8 Oct 2026, by construction: pkill and killall exit 97 on every box and pod).
# kill_pidfile NAME [SIG] [WAIT] the pid in $F/pids/NAME.pid (TERM, then KILL after WAIT s); removes a stale file
# kill_children PID [SIG] every descendant of PID by pid (pgrep -P walks the tree by parent pid, not by name), deepest first
# kill_sock_owner GLOB the pid that owns a listening unix socket matching GLOB (ss -xlp), written to $F/pids/sp1-server.pid, then killed
F=${F:-/root/fleet}; mkdir -p $F/pids
kill_pidfile() { local f=$F/pids/$1.pid p; [ -s "$f" ] || return 0; p=$(cat "$f"); kill -0 "$p" 2>/dev/null || { rm -f "$f"; return 0; }; kill ${2:--TERM} "$p" 2>/dev/null; sleep ${3:-2}; kill -0 "$p" 2>/dev/null && kill -9 "$p" 2>/dev/null; rm -f "$f"; return 0; }
kill_children() { local c; for c in $(pgrep -P "$1" 2>/dev/null); do kill_children "$c" "$2"; kill ${2:--TERM} "$c" 2>/dev/null; done; return 0; }
kill_sock_owner() { local p; for p in $(ss -xlp 2>/dev/null | grep -E -- "$1" | grep -oE 'pid=[0-9]+' | cut -d= -f2 | sort -u); do echo "$p" > $F/pids/sp1-server.pid; kill -TERM "$p" 2>/dev/null; sleep 1; kill -0 "$p" 2>/dev/null && kill -9 "$p" 2>/dev/null; done; rm -f /tmp/sp1-cuda-*.sock; return 0; }

View file

@ -84,7 +84,7 @@ def update(label):
"""The version follow: the manifest's hive package onto the box, the node pointer rewritten, the supervisor restarts it."""
m = manifest(); b = Box.from_registry(label)
if not m: raise SshError("no manifest")
rc, out, err = b.run(f"set -e; cd {F}; curl -fsSL -m 600 -o pkg-{m['version']}.tgz https://dl.igneum.network{m['path']}; echo '{m['sha256']} pkg-{m['version']}.tgz' | sha256sum -c - >/dev/null; rm -rf /opt/igneum/pkg.new; mkdir -p /opt/igneum/pkg.new; tar -C /opt/igneum/pkg.new --strip-components=1 -xzf pkg-{m['version']}.tgz; rm -rf /opt/igneum/pkg.prev; mv /opt/igneum/pkg /opt/igneum/pkg.prev; mv /opt/igneum/pkg.new /opt/igneum/pkg; echo /opt/igneum/pkg/bin/igneumd > {F}/standing.node; pkill -9 -f '[/]opt/igneum/pkg/bin/igneumd' 2>/dev/null; pkill -9 -f '[/]opt/igneum/pkg.prev/bin/igneumd' 2>/dev/null; echo updated-to-{m['version']}", 900)
rc, out, err = b.run(f"set -e; cd {F}; curl -fsSL -m 600 -o pkg-{m['version']}.tgz https://dl.igneum.network{m['path']}; echo '{m['sha256']} pkg-{m['version']}.tgz' | sha256sum -c - >/dev/null; rm -rf /opt/igneum/pkg.new; mkdir -p /opt/igneum/pkg.new; tar -C /opt/igneum/pkg.new --strip-components=1 -xzf pkg-{m['version']}.tgz; rm -rf /opt/igneum/pkg.prev; mv /opt/igneum/pkg /opt/igneum/pkg.prev; mv /opt/igneum/pkg.new /opt/igneum/pkg; echo /opt/igneum/pkg/bin/igneumd > {F}/standing.node; bash /root/fleet/in/fleet-stop.sh node >/dev/null 2>&1; bash /root/fleet/in/fleet-stop.sh node >/dev/null 2>&1; echo updated-to-{m['version']}", 900)
iid, _ = Registry.find(label); Registry.patch(iid, version_wanted=m["version"], updated_at=now()); return out.strip()
def rerent(label):
"""Same shape again on the same provider; the old row is marked destroyed. Returns the new label or None."""

View file

@ -9,7 +9,7 @@ live=[r["label"] for r in standing.roster() if r["role"]=="live"]
def move(l):
try:
b=Box.from_registry(l); b.put([F16], "/root/fleet/override-16.json")
rc,out,err=b.run("cd /root/fleet && cp override.json override-13.json && cp override-16.json override.json && python3 -c \"import json; d=json.load(open('/root/fleet/override.json')); print('fields', len(d))\" && pkill -9 -f '[/](opt/igneum/pkg/bin|root/fleet/in)/igneumd(-0313|-[0-9a-f]{16})? --devnet --appdir=' ; echo killed-for-restart $(date -u +%T)", 60)
rc,out,err=b.run("cd /root/fleet && cp override.json override-13.json && cp override-16.json override.json && python3 -c \"import json; d=json.load(open('/root/fleet/override.json')); print('fields', len(d))\" && bash /root/fleet/in/fleet-stop.sh node >/dev/null 2>&1; ; echo killed-for-restart $(date -u +%T)", 60)
return l, out.replace("\n"," ").strip()+(" ERR "+err[:60] if err.strip() else "")
except Exception as e: return l, "EXC "+str(e)[:60]
print(st(), "moving", len(live), "boxes", flush=True)

View file

@ -0,0 +1,6 @@
# Retired fleet scripts (8 Oct 2026)
Devnet 2 and 0.3.1x-era scripts that stop processes by name (pkill -f / pkill -x). Since the founder's word of 20:21 BST 8 Oct 2026
every box and pod refuses pkill and killall by construction (exit 97), so none of these can run as written; they are kept for the
record only. The live stop forms are tools/fleet/fleet-stop.sh (pid files under /root/fleet/pids), tools/fleet/lib/pidkill.sh
(kill_pidfile, kill_children, kill_sock_owner) and tools/fleet/box-kill.sh.