igneum/infra/cloud-devnet/experiments/collect.sh
igneum-labs 4574890602 Cloud devnet: private-network mode (4 zone networks, 4 gateways), 12 nodes up, first latency measurement
A new Hetzner account is capped at 10 primary IPs (IPv4 and IPv6 both count), 20 shared vCPUs, 8 dedicated
vCPUs and no Arm, and a network cannot span zones. So: one private network per zone (10.20.<zone>.0/24),
the lowest-index node of each zone keeps a public IPv4 and is its gateway (NAT, MSS clamp, one DNAT port
27000+index per private node, persisted as igneum-nat.service), every other node has no public address.
nodes.tsv gains access, pub and port columns; lib resolves same-zone vs cross-zone dial addresses and jumps
ssh through the gateway for private nodes. All nodes.tsv loops read on fd 3 (a backgrounded ssh drained the
file). TYPE_BY_INDEX puts nodes 8 to 11 on ccx13; node 12 is the last shared one the account allows.
create.sh prints the plan's cost from the live API. provision.sh install takes node names and skips binaries
whose sha256 matches. Binaries copied from the seed's staged v4 build (same sources), no vCPU for a builder.

Results 2026-10-04: RTT matrix (hel1-fsn1 35 ms, ash-sin 289 ms) and a 10-minute propagation window of 644
blocks: p50 343 ms, p90 497 ms, p99 666 ms across 12 nodes in 5 locations.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-04 10:41:23 +00:00

38 lines
2.4 KiB
Bash
Executable file

#!/usr/bin/env bash
# Pull logs and RPC samples from every node into results/<date>/nodes/<name>/ and write summary.md plus a
# bench-log entry template (results/<date>/bench-log-entry.md). APPEND_BENCH_LOG=1 also appends that entry to
# docs/bench-log.md (the log is append-only; edit the numbers in the entry file first if a run looked odd).
# ./experiments/collect.sh [date] default today (UTC)
. "$(dirname "$0")/../lib/common.sh"
require_nodes
out="$(results_dir_for "${1:-}")"; mkdir -p "$out/nodes"
log "collecting into $out"
cp "$NODES_FILE" "$out/nodes.tsv"
[ -f "$BIN_DIR/version.txt" ] && cp "$BIN_DIR/version.txt" "$BIN_DIR/src.stamp" "$out/" 2>/dev/null
while IFS=$'\t' read -r -u 3 name idx reg ip access pub port; do
(
d="$out/nodes/$name"; mkdir -p "$d"
nssh "$ip" "journalctl -u igneumd --no-pager -o short-iso --since -24h | tail -c 30000000" > "$d/igneumd.log" 2>/dev/null
nssh "$ip" "journalctl -u igneum-miner --no-pager -o short-iso --since -24h | tail -c 5000000" > "$d/miner.log" 2>/dev/null
for f in blocks chain samples; do nscp "$SSH_USER@$ip:/var/log/igneum/$f.tsv" "$d/$f.tsv" 2>/dev/null || true; done
nssh "$ip" "chronyc tracking 2>/dev/null; echo; free -m; echo; df -h / | tail -1; echo; cat /etc/igneum/node.env" > "$d/host.txt" 2>/dev/null
for m in getInfo getBlockDagInfo getConnectedPeerInfo getFinalityWeights; do
nssh "$ip" "python3 /opt/igneum/bin/wrpc.py call $m" > "$d/$m.json" 2>/dev/null || true
done
nssh "$ip" "python3 /opt/igneum/bin/wrpc.py call getFinalityCheckpoints '{\"last\": 50}'" > "$d/getFinalityCheckpoints.json" 2>/dev/null || true
printf '[%s] %s blocks lines, %s chain lines, %s samples\n' "$name" "$(grep -c . "$d/blocks.tsv" 2>/dev/null || echo 0)" "$(grep -c . "$d/chain.tsv" 2>/dev/null || echo 0)" "$(grep -c . "$d/samples.tsv" 2>/dev/null || echo 0)"
) &
while [ "$(jobs -r | wc -l)" -ge 10 ]; do sleep 1; done
done 3< "$NODES_FILE"
wait
python3 "$HERE/experiments/analyze.py" summary "$out" "$NODES_FILE" | tee "$out/summary.md"
python3 "$HERE/experiments/analyze.py" bench-entry "$out" "$NODES_FILE" > "$out/bench-log-entry.md"
log "wrote $out/summary.md and $out/bench-log-entry.md"
if [ "${APPEND_BENCH_LOG:-0}" = 1 ]; then
{ printf '\n'; cat "$out/bench-log-entry.md"; } >> "$REPO/docs/bench-log.md"
log "appended the entry to docs/bench-log.md (review it before committing)"
else
log "APPEND_BENCH_LOG=1 appends the entry to docs/bench-log.md"
fi