Block rate on Devnet 2: the plan file (runs A 10 bps and B 1 bps on the 38 wave cards, the readings, the per-tier payout arithmetic) and bps-collect.py; the remote-build tooling from master
This commit is contained in:
igneum-josh
parent
90dd8e27e6
commit
697297dfce
8 changed files with 1072 additions and 0 deletions
34
docs/analysis/block-rate-devnet2.md
Normal file
34
docs/analysis/block-rate-devnet2.md
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# Block rate on Devnet 2: 10 blocks per second against 1, on the rented fleet
|
||||
|
||||
6 October 2026, Josh's experiment (through the coordinator, 17:5xZ): "Devnet 2 at a higher block rate for solo miners
|
||||
(Kaspa's answer)". Branch `gpu-fleet`; the node profile on the fork branch `devnet2-bps` (1279a1d6 on release-0.3.14-node
|
||||
4c6b129d): a suffixed devnet started with `IGNEUMD_DEVNET_BPS=10` (or 5) runs `BlockrateParams::new::<10>()` with the
|
||||
TenBps subsidy and target time, Kaspa's Crescendo-style constants for that rate (k 124, merge depth, sample rates,
|
||||
mergeset limit, parents, coinbase maturity from `consensus/core/src/config/bps.rs`); the shared devnet never reads the
|
||||
variable, so the live digest and rules are untouched. Built on igneum-build-1 (`tools/build-remote.sh`). Numbers land
|
||||
below as they are measured; every figure carries its source file under `~/Desktop/fleet/bps/`.
|
||||
|
||||
## The runs
|
||||
|
||||
| Run | Chain | Miners | Length | What is read |
|
||||
|---|---|---|---|---|
|
||||
| A | igneum-devnet-2, fresh genesis, 10 bps | the 38 wave pods plus the 4 Devnet 2 boxes (42 cards, about 2.0 GH/s) | 60 min | blocks per second achieved, blue and red blocks per minute (orphan rate), blue score growth, max reorg depth, checkpoint lock delay and weight at this voter count, p2p bytes per node per minute, node CPU and RSS, exec lag (height minus executed tip), payout intervals per miner from the coinbase records |
|
||||
| B | the same boxes, fresh genesis, 1 bps | the same | 30 min | the same (the control) |
|
||||
|
||||
Payout interval per tier, from each run's measured block rate and the chain's hash: a 4070 at 28 MH/s, a 5090 at
|
||||
128 MH/s, an 8x 4090 rig at 459 MH/s, at tonight's network hash and extrapolated to 1, 10 and 100 TH/s networks
|
||||
(interval = blocks per second x miner share, the arithmetic in `tools/fleet/bps-collect.py`).
|
||||
|
||||
## Measured
|
||||
|
||||
RUN_A
|
||||
|
||||
RUN_B
|
||||
|
||||
## Per tier
|
||||
|
||||
TIER_TABLE
|
||||
|
||||
## The recommendation for mainnet's rate
|
||||
|
||||
RECOMMENDATION
|
||||
181
infra/build-server/lib.sh
Executable file
181
infra/build-server/lib.sh
Executable file
|
|
@ -0,0 +1,181 @@
|
|||
#!/usr/bin/env bash
|
||||
# Shared by infra/build-server/run-from-mac.sh, tools/build-remote.sh and tools/cross-remote.sh. Source it, do not run it.
|
||||
# Everything that talks to igneum-build-1 from the Mac goes through here: the host line, the ssh options (the ops key
|
||||
# ~/.ssh/igneum_ed25519, a shared control socket so one build is one ssh session), the mirror push, the remote checkout
|
||||
# and the source overlay (rsync by checksum, changed files re-stamped: the copied-sources rule of 5 October 2026).
|
||||
#
|
||||
# Layout on the box (provision.sh): /srv/builds/<worktree> mirrors the Mac's igneum worktree ROOT (the directory that holds
|
||||
# igneum-pow/, app/, proving/ and vendor/), so the fork's relative path dependency `../../../../igneum-pow`
|
||||
# (vendor/igneum-node/consensus/pow/Cargo.toml) resolves on the box exactly as on the Mac:
|
||||
# Mac /Users/joshm/Projects/igneum-wt-ship0311/vendor/igneum-node-0311 -> box /srv/builds/igneum-wt-ship0311/vendor/igneum-node-0311
|
||||
# Mac /Users/joshm/Projects/igneum-wt-ship0311/igneum-pow -> box /srv/builds/igneum-wt-ship0311/igneum-pow
|
||||
# Mac /Users/joshm/Projects/igneum/app/igneum-app -> box /srv/builds/igneum/app/igneum-app
|
||||
# The fork's kaspa-build-info reads `git rev-parse HEAD` at build time and the release plans check the commit in the binary's
|
||||
# strings, so the fork tree on the box is a real clone of the bare mirror /srv/igneum-node.git checked out at the Mac's HEAD,
|
||||
# with the Mac's uncommitted changes rsynced on top. The igneum repo's crates get the same from /srv/igneum.git.
|
||||
|
||||
# shellcheck disable=SC2034 # shared with the scripts that source lib.sh
|
||||
BS_KEY="${IGNEUM_BUILD_KEY:-$HOME/.ssh/igneum_ed25519}"
|
||||
BS_HOST_FILE="${IGNEUM_BUILD_HOST_FILE:-$HOME/.config/igneum/build-server}" # one line: build@<ip>
|
||||
BS_ROOT_REMOTE=/srv/builds
|
||||
BS_MIRROR_REPO=/srv/igneum.git
|
||||
BS_MIRROR_NODE=/srv/igneum-node.git
|
||||
|
||||
bs_log() { printf '%s %s: %s\n' "$(date -u +%H:%M:%S)" "${BS_TOOL:-build-server}" "$*" >&2; }
|
||||
bs_die() { bs_log "ERROR: $*"; exit 1; }
|
||||
|
||||
bs_host() {
|
||||
BS_HOST="${BUILD_HOST:-}"
|
||||
if [ -z "$BS_HOST" ]; then
|
||||
[ -s "$BS_HOST_FILE" ] || bs_die "no build server: write build@<ip> to $BS_HOST_FILE (infra/build-server/run-from-mac.sh does) or set BUILD_HOST"
|
||||
BS_HOST="$(head -1 "$BS_HOST_FILE" | tr -d '[:space:]')"
|
||||
fi
|
||||
case "$BS_HOST" in *@*) ;; *) bs_die "BUILD_HOST must be user@host, got '$BS_HOST'" ;; esac
|
||||
[ -r "$BS_KEY" ] || bs_die "no ssh key at $BS_KEY"
|
||||
mkdir -p "$HOME/.ssh/cm"
|
||||
BS_SSH_OPTS=(-i "$BS_KEY" -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ServerAliveInterval=30 -o ServerAliveCountMax=6
|
||||
-o ControlMaster=auto -o ControlPath="$HOME/.ssh/cm/igneum-build-%r@%h:%p" -o ControlPersist=900)
|
||||
BS_SSH_CMD="ssh"; local o; for o in "${BS_SSH_OPTS[@]}"; do BS_SSH_CMD="$BS_SSH_CMD $(printf '%q' "$o")"; done
|
||||
}
|
||||
|
||||
bs_ssh() { ssh "${BS_SSH_OPTS[@]}" "$BS_HOST" "$@"; }
|
||||
bs_rsync() { rsync -e "$BS_SSH_CMD" "$@"; }
|
||||
|
||||
# the two sides' rustc must agree (the box is pinned by provision.sh RUST_TOOLCHAIN; the Mac runs rustup's stable):
|
||||
# a different compiler gives different bytes and, across a minor version, different lints and errors
|
||||
bs_toolchain_check() {
|
||||
local mac box
|
||||
mac=$("${CARGO_HOME:-$HOME/.cargo}/bin/rustc" --version 2>/dev/null | awk '{ print $2 }')
|
||||
box=$(bs_ssh '. /etc/profile.d/igneum-build.sh; rustc --version' 2>/dev/null | awk '{ print $2 }')
|
||||
[ -n "$box" ] || bs_die "cannot read rustc on $BS_HOST (is it provisioned? infra/build-server/run-from-mac.sh)"
|
||||
if [ "$mac" != "$box" ]; then
|
||||
if [ "${IGNEUM_TOOLCHAIN_MISMATCH:-}" = ok ]; then bs_log "WARNING: rustc $mac on the Mac, $box on the box (IGNEUM_TOOLCHAIN_MISMATCH=ok)"
|
||||
else bs_die "rustc $mac on the Mac, $box on the box; re-provision with RUST_TOOLCHAIN=$mac or set IGNEUM_TOOLCHAIN_MISMATCH=ok"; fi
|
||||
else bs_log "rustc $box on both sides"; fi
|
||||
}
|
||||
|
||||
# Where am I? Sets BS_KIND (node = a worktree of the fork under vendor/; repo = a crate of the igneum repo), BS_TOP (the git
|
||||
# top level of the crate's repo), BS_WT_ROOT (the igneum worktree root), BS_WT (its name = the directory on the box),
|
||||
# BS_CRATE (the crate dir, = $PWD), BS_CRATE_REL (relative to BS_WT_ROOT), BS_MIRROR, BS_BRANCH, BS_SHA, BS_REMOTE_WT,
|
||||
# BS_REMOTE_CRATE, and BS_LOCAL_DIRS (every directory of a path dependency, relative to BS_WT_ROOT, from cargo metadata).
|
||||
bs_context() {
|
||||
BS_CRATE="$PWD"
|
||||
[ -f "$BS_CRATE/Cargo.toml" ] || bs_die "no Cargo.toml in $BS_CRATE: run from the crate directory (the fork worktree, igneum-pow, app/igneum-app, proving/igneum-prove)"
|
||||
BS_TOP=$(git -C "$BS_CRATE" rev-parse --show-toplevel 2>/dev/null) || bs_die "$BS_CRATE is not inside a git worktree"
|
||||
case "$BS_TOP" in
|
||||
*/vendor/*)
|
||||
BS_KIND=node; BS_MIRROR=$BS_MIRROR_NODE
|
||||
BS_WT_ROOT=$(cd "$BS_TOP/../.." && pwd)
|
||||
[ -f "$BS_WT_ROOT/igneum-pow/Cargo.toml" ] || bs_die "$BS_TOP looks like a fork worktree but $BS_WT_ROOT/igneum-pow is missing"
|
||||
;;
|
||||
*)
|
||||
BS_KIND=repo; BS_MIRROR=$BS_MIRROR_REPO; BS_WT_ROOT="$BS_TOP"
|
||||
;;
|
||||
esac
|
||||
BS_WT=$(basename "$BS_WT_ROOT")
|
||||
BS_CRATE_REL=$(python3 -c 'import os, sys; print(os.path.relpath(sys.argv[1], sys.argv[2]))' "$BS_CRATE" "$BS_WT_ROOT")
|
||||
BS_TOP_REL=$(python3 -c 'import os, sys; print(os.path.relpath(sys.argv[1], sys.argv[2]))' "$BS_TOP" "$BS_WT_ROOT")
|
||||
case "$BS_CRATE_REL" in ..*) bs_die "$BS_CRATE is outside the worktree root $BS_WT_ROOT" ;; esac
|
||||
BS_BRANCH=$(git -C "$BS_TOP" branch --show-current 2>/dev/null || true)
|
||||
BS_SHA=$(git -C "$BS_TOP" rev-parse HEAD)
|
||||
[ -n "$BS_BRANCH" ] || BS_BRANCH="detached-$(git -C "$BS_TOP" rev-parse --short HEAD)"
|
||||
BS_REMOTE_WT="$BS_ROOT_REMOTE/$BS_WT"
|
||||
BS_REMOTE_CRATE="$BS_REMOTE_WT/$BS_CRATE_REL"
|
||||
# every local (path) package of the crate's dependency graph, as directories relative to the worktree root; the ones inside
|
||||
# BS_TOP are covered by the git checkout plus the overlay of BS_TOP itself (node kind) or synced one by one (repo kind)
|
||||
BS_LOCAL_DIRS=$(cd "$BS_CRATE" && "${CARGO_HOME:-$HOME/.cargo}/bin/cargo" metadata --format-version 1 2>/dev/null | python3 -c '
|
||||
import json, os, sys
|
||||
d = json.load(sys.stdin); root = sys.argv[1]; top = sys.argv[2]; kind = sys.argv[3]
|
||||
dirs = set()
|
||||
for p in d["packages"]:
|
||||
if p["source"] is not None: continue
|
||||
m = os.path.dirname(p["manifest_path"])
|
||||
if kind == "node" and (m == top or m.startswith(top + "/")): dirs.add(top); continue
|
||||
dirs.add(m)
|
||||
out = []
|
||||
for m in sorted(dirs):
|
||||
r = os.path.relpath(m, root)
|
||||
if r.startswith(".."): sys.exit("path dependency %s is outside the worktree root %s" % (m, root))
|
||||
out.append(r)
|
||||
print("\n".join(out))' "$BS_WT_ROOT" "$BS_TOP" "$BS_KIND") || bs_die "cargo metadata failed in $BS_CRATE"
|
||||
[ -n "$BS_LOCAL_DIRS" ] || bs_die "cargo metadata listed no local packages in $BS_CRATE"
|
||||
}
|
||||
|
||||
# push the crate repo's HEAD to its bare mirror on the box (fast after the first time), then check the remote tree out at that
|
||||
# commit ON A BRANCH of that name: kaspa-build-info (build-info/build.rs try_git_head) embeds the commit only when .git is a
|
||||
# directory AND HEAD is a symbolic ref to a loose branch file; a detached HEAD or a worktree's .git file gives an empty hash
|
||||
# (which is why the Mac's worktree builds print "igneumd 2.1.0" with no commit, 6 Oct 2026). The mirror doubles as the CI
|
||||
# runner's source later.
|
||||
bs_push_and_checkout() {
|
||||
local url="$BS_HOST:$BS_MIRROR" remote_top="$BS_REMOTE_WT/$BS_TOP_REL"
|
||||
[ "$BS_TOP_REL" = . ] && remote_top="$BS_REMOTE_WT"
|
||||
bs_log "push $BS_TOP HEAD $BS_SHA ($BS_BRANCH) -> $url"
|
||||
GIT_SSH_COMMAND="$BS_SSH_CMD" git -C "$BS_TOP" push -q --force "$url" "HEAD:refs/heads/$BS_BRANCH" || bs_die "push to the mirror failed"
|
||||
bs_ssh "set -e; mkdir -p '$BS_REMOTE_WT'
|
||||
if [ ! -d '$remote_top/.git' ]; then rm -rf '$remote_top'; git clone -q --no-checkout '$BS_MIRROR' '$remote_top'; fi
|
||||
cd '$remote_top'; git fetch -q origin '+refs/heads/*:refs/remotes/origin/*'; git checkout -q -B '$BS_BRANCH' '$BS_SHA'; git reset -q --hard '$BS_SHA'
|
||||
git status --porcelain | head -3" || bs_die "remote checkout at $remote_top failed"
|
||||
BS_REMOTE_TOP="$remote_top"
|
||||
}
|
||||
|
||||
# rsync one directory of the worktree to the same place on the box. By checksum and WITHOUT preserving times, so a file whose
|
||||
# content changed is written with the box's clock and nothing older than the last build slips past cargo's mtime check (the
|
||||
# stale-build class, 4 and 5 October 2026); the files rsync wrote are listed and re-stamped with touch as well, so the rule is
|
||||
# visible here and tools/ci/copied-sources-check.sh sees it. target dirs and .git never travel; --delete keeps the box equal to
|
||||
# the Mac inside the directory (excluded paths are protected).
|
||||
bs_overlay_dir() {
|
||||
local rel="$1" src="$BS_WT_ROOT/$1" dst="$BS_REMOTE_WT/$1" list nfiles ndirs
|
||||
[ -d "$src" ] || bs_die "no $src"
|
||||
list=$(mktemp)
|
||||
bs_ssh "mkdir -p '$dst'"
|
||||
bs_rsync -rlpgoD --checksum --delete --out-format='%n' \
|
||||
--exclude '/target' --exclude '/target-*' --exclude '/target/' --exclude 'target-*/' --exclude '.git' --exclude '.DS_Store' --exclude 'node_modules' \
|
||||
"$src/" "$BS_HOST:$dst/" > "$list" || { rm -f "$list"; bs_die "rsync of $rel failed"; }
|
||||
# files only: directories are listed whenever an attribute differs (a fresh clone's ownership), and a tree whose files
|
||||
# were all identical lists ONLY directories (first run of 6 October 2026: an empty file list failed the pipeline)
|
||||
nfiles=$(grep -vc '/$' "$list" || true); ndirs=$(grep -c '/$' "$list" || true)
|
||||
if [ "${nfiles:-0}" -gt 0 ]; then
|
||||
if ! grep -v '/$' "$list" | tr '\n' '\0' | bs_ssh "cd '$dst' && xargs -0 -r touch --no-create"; then rm -f "$list"; bs_die "re-stamp of $rel failed"; fi
|
||||
fi
|
||||
bs_log "overlay $rel -> $dst: ${nfiles:-0} file(s) written and re-stamped, ${ndirs:-0} dir(s)"
|
||||
rm -f "$list"
|
||||
}
|
||||
|
||||
bs_sync_sources() {
|
||||
local d
|
||||
bs_push_and_checkout
|
||||
for d in $BS_LOCAL_DIRS; do bs_overlay_dir "$d"; done
|
||||
}
|
||||
|
||||
bs_sha256() { shasum -a 256 "$1" | awk '{ print $1 }'; }
|
||||
bs_size() { stat -f %z "$1" 2>/dev/null || stat -c %s "$1"; }
|
||||
bs_fmt_secs() { local s=$1; printf '%d min %02d s' $((s / 60)) $((s % 60)); }
|
||||
|
||||
# kind of a run for the box's JSONL log (main's rule of 6 October 2026): <tool> <first cargo word>
|
||||
bs_kind() {
|
||||
local tool="$1" word="$2"
|
||||
case "$word" in test) echo suite; return ;; check|clippy) echo check; return ;; esac
|
||||
if [ "$tool" = cross-remote ]; then
|
||||
case "$BS_KIND:$BS_CRATE_REL" in node:*) echo node-windows ;; repo:app/igneum-app) echo app-windows ;; *) echo other ;; esac; return
|
||||
fi
|
||||
case "$BS_KIND:$BS_CRATE_REL" in node:*) echo node-linux ;; repo:app/igneum-app) echo app ;; repo:proving/igneum-prove) echo prove ;; *) echo other ;; esac
|
||||
}
|
||||
|
||||
# the remote runner (infra/build-server/remote-run.sh, piped to `bash -s` on the box behind the BR_* exports): takes one of the
|
||||
# box's build slots (never the Mac's), runs the command in the crate dir with sccache, prints the RESULT line and appends one
|
||||
# JSON line to /srv/builds/_log/builds.jsonl for the worker dashboard.
|
||||
# bs_remote_run <remote crate dir> <label> <shell command string>
|
||||
# with BR_KIND, BR_COMMAND, BR_TARGET and BR_ARTEFACTS set by the caller; the agent name is IGNEUM_AGENT (default the worktree)
|
||||
# and is appended to the label as "; agent=<name>".
|
||||
bs_remote_run() {
|
||||
local dir="$1" label="$2" cmd="$3" agent="${IGNEUM_AGENT:-$BS_WT}" v
|
||||
label="$label; agent=$agent"
|
||||
BR_DIR="$dir" BR_LABEL="$label" BR_CMD="$cmd" BR_TOOL="${BS_TOOL:-build-remote}" BR_WT="$BS_WT" BR_CRATE="$BS_CRATE_REL" \
|
||||
BR_BRANCH="$BS_BRANCH" BR_SHA="$BS_SHA" BR_AGENT="$agent" BR_KIND="${BR_KIND:-other}" BR_COMMAND="${BR_COMMAND:-}" \
|
||||
BR_TARGET="${BR_TARGET:-}" BR_ARTEFACTS="${BR_ARTEFACTS:-}" \
|
||||
bash -c '
|
||||
for v in BR_DIR BR_LABEL BR_CMD BR_TOOL BR_WT BR_CRATE BR_BRANCH BR_SHA BR_AGENT BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS; do
|
||||
printf "export %s=%q\n" "$v" "${!v}"
|
||||
done
|
||||
cat "$0"' "$(dirname "${BASH_SOURCE[0]}")/remote-run.sh" | bs_ssh 'bash -s'
|
||||
}
|
||||
398
infra/build-server/provision.sh
Executable file
398
infra/build-server/provision.sh
Executable file
|
|
@ -0,0 +1,398 @@
|
|||
#!/usr/bin/env bash
|
||||
# Provision igneum-build-1, the Hetzner dedicated build server (AX162-1-LTD: EPYC 9454P 48 cores / 96 threads, 128 GB,
|
||||
# 2x 3.84 TB NVMe, Falkenstein; ordered 6 October 2026). Idempotent: every step checks before it changes anything and
|
||||
# says "ok" (nothing to do) or "changed". Run it over ssh as root; nothing here reads a secret.
|
||||
#
|
||||
# infra/build-server/run-from-mac.sh <ip> the usual way (ships this file, fills WORKTREES, writes the host file)
|
||||
# ssh root@<ip> 'bash -s' < infra/build-server/provision.sh the bare way
|
||||
# ssh root@<ip> 'MODE=install bash -s' < infra/build-server/provision.sh force the rescue-system path
|
||||
#
|
||||
# Two modes, chosen by MODE (auto, install, provision; default auto):
|
||||
# install the box booted into Hetzner's rescue system (installimage present, hostname rescue*): run installimage in
|
||||
# batch mode for Ubuntu 24.04 with software RAID 1 over the two NVMe drives, no swap, the rescue system's
|
||||
# authorized_keys taken over, the image signature checked, then reboot. Run the script again after the reboot.
|
||||
# Ran on igneum-build-1 on 6 October 2026 at 17:16 to 17:20 UTC (16 steps, no prompt).
|
||||
# provision a running Ubuntu 24.04: user `build` with root's key, the compiler and cross toolchains, rustup pinned to
|
||||
# RUST_TOOLCHAIN with the x86_64-pc-windows-gnu target, sccache with a 100 GB disk cache, Node 22, git, tmux,
|
||||
# a swap-free tuned sysctl, the two bare mirrors (/srv/igneum.git, /srv/igneum-node.git), /srv/builds with one
|
||||
# directory per agent worktree, sshd key-only, ufw with 22 and the seed p2p ports.
|
||||
#
|
||||
# Settings (environment, all optional):
|
||||
# RUST_TOOLCHAIN 1.99.0 the Mac's `rustc --version` on 6 October 2026. Neither the repo nor the fork carries a
|
||||
# rust-toolchain file (checked 6 October 2026), so the pin lives here; the fork's Cargo.toml says
|
||||
# rust-version 1.91.0. tools/build-remote.sh compares the two sides and refuses a mismatch.
|
||||
# SCCACHE_GB 100 the local disk cache at /srv/sccache
|
||||
# SCCACHE_VERSION (unset) a `cargo install sccache --version` pin; unset = the newest on crates.io
|
||||
# NODE_MAJOR 22
|
||||
# WORKTREES "" space-separated agent worktree names, one /srv/builds/<name> each (run-from-mac.sh fills it from
|
||||
# `git worktree list` on the Mac; tools/build-remote.sh creates a missing one on first use)
|
||||
# SLOTS 1 remote build slots (tools/build-remote.sh takes one; with 1 slot every build gets the box)
|
||||
# P2P_PORTS "26611 26811" TCP ports ufw opens beside 22: the devnet seed's p2p (infra/seed-nodes/config.sh devnet
|
||||
# P2P_PORT=26611) and the testnet seed's (26811). A suffixed devnet (Devnet 2, the fleet's staging
|
||||
# chain) listens on the same 26611 (infra/cloud-devnet/config.sh P2P_PORT=26611). RPC ports
|
||||
# (26610, 28610, 26790 and the 268xx set) stay on loopback as on every seed, so they are not opened.
|
||||
# BOX_HOSTNAME igneum-build-1
|
||||
# SSH_PUBKEY (unset) a public key line for the build user when root has no authorized_keys (installimage installs it)
|
||||
#
|
||||
# Mirrors: the Mac pushes to them (never a clone from GitHub; the fork vendor/igneum-node exists only on the Mac):
|
||||
# git -C /Users/joshm/Projects/igneum remote add build build@<ip>:/srv/igneum.git
|
||||
# git -C /Users/joshm/Projects/igneum/vendor/igneum-node remote add build build@<ip>:/srv/igneum-node.git
|
||||
# git push build --all (tools/build-remote.sh pushes the branch it builds before every build)
|
||||
#
|
||||
# What this script does NOT do: install zig or cargo-zigbuild (the Mac's glibc 2.36 Linux cross-build, infra/cross/build-linux.sh,
|
||||
# stays on the Mac until the box is proven; a native build here links glibc 2.39, which Debian 13 seeds accept and HiveOS
|
||||
# does not), start any node, or copy a secret. The installimage flags and the image name were read from the live rescue system
|
||||
# on 6 October 2026 (`installimage -h`, /root/.oldroot/nfs/images); the script still reads the image list instead of hard-coding a name.
|
||||
set -euo pipefail
|
||||
|
||||
MODE="${MODE:-auto}"
|
||||
RUST_TOOLCHAIN="${RUST_TOOLCHAIN:-1.99.0}"
|
||||
SCCACHE_GB="${SCCACHE_GB:-100}"
|
||||
SCCACHE_VERSION="${SCCACHE_VERSION:-}"
|
||||
NODE_MAJOR="${NODE_MAJOR:-22}"
|
||||
WORKTREES="${WORKTREES:-}"
|
||||
SLOTS="${SLOTS:-1}"
|
||||
P2P_PORTS="${P2P_PORTS:-26611 26811}"
|
||||
BOX_HOSTNAME="${BOX_HOSTNAME:-igneum-build-1}"
|
||||
SSH_PUBKEY="${SSH_PUBKEY:-}"
|
||||
BUILD_USER=build
|
||||
BUILD_HOME=/home/$BUILD_USER
|
||||
|
||||
log() { printf '%s provision: %s\n' "$(date -u +%H:%M:%S)" "$*"; }
|
||||
die() { log "ERROR: $*" >&2; exit 1; }
|
||||
changed() { log "$1: changed${2:+ ($2)}"; }
|
||||
ok() { log "$1: ok${2:+ ($2)}"; }
|
||||
as_build() { su - "$BUILD_USER" -c "$*"; }
|
||||
|
||||
[ "$(id -u)" = 0 ] || die "run as root"
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# install mode: the rescue system
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
INSTALLIMAGE=/root/.oldroot/nfs/install/installimage # not on PATH in a non-interactive ssh shell (read 6 Oct 2026)
|
||||
|
||||
in_rescue() {
|
||||
[ -x "$INSTALLIMAGE" ] || return 1
|
||||
case "$(hostname)" in rescue*) return 0 ;; esac
|
||||
[ -d /root/.oldroot/nfs/images ]
|
||||
}
|
||||
|
||||
do_install() {
|
||||
local images drives image parts
|
||||
images=/root/.oldroot/nfs/images
|
||||
[ -d "$images" ] || die "no image directory at $images: not the Hetzner rescue system"
|
||||
# the Ubuntu 24.04 (noble) amd64 base image the rescue system offers (read on 6 October 2026: Ubuntu-2404-noble-amd64-base.tar.zst
|
||||
# with a detached .sig; read, not hard-coded, because Hetzner refreshes the names)
|
||||
image=$(find "$images" -maxdepth 1 -type f -iregex '.*/ubuntu-2404.*amd64.*\.tar\.\(zst\|gz\|xz\)' -printf '%f\n' | sort | tail -1 || true)
|
||||
[ -n "$image" ] || die "no Ubuntu 24.04 amd64 image under $images: $(find "$images" -maxdepth 1 -type f -printf '%f ' )"
|
||||
mapfile -t drives < <(lsblk -dn -o NAME,TYPE | awk '$2 == "disk" && $1 ~ /^nvme/ { print $1 }' | sort)
|
||||
[ "${#drives[@]}" = 2 ] || die "expected exactly two NVMe drives for RAID 1, found ${#drives[@]}: ${drives[*]:-none}"
|
||||
[ -s /root/.ssh/authorized_keys ] || die "/root/.ssh/authorized_keys is empty in the rescue system; -t yes would carry nothing into the image"
|
||||
[ -d /sys/firmware/efi ] || log "WARNING: no /sys/firmware/efi, the box booted in BIOS mode; the esp partition is harmless but grub goes to the MBR"
|
||||
# no swap partition: 128 GB of RAM and a swap-free sysctl (the provision mode checks no swap is active)
|
||||
parts="/boot/efi:esp:512M,/boot:ext4:1G,/:ext4:all"
|
||||
log "installimage: image $image, drives ${drives[*]} as software RAID 1, partitions $parts, hostname $BOX_HOSTNAME, rescue ssh keys taken over (-t yes), image signature checked (-g)"
|
||||
log "this WIPES ${drives[*]}"
|
||||
# flag form, read from `installimage -h` on 6 October 2026: -a batch, -n hostname, -r raid, -l level, -i image, -g verify
|
||||
# the detached signature, -p partitions mount:fs:size, -d drives, -t yes take over the rescue system's ssh keys (root's
|
||||
# authorized_keys), -G yes new host keys. The -c config form forbids every other flag, so the keys could not travel with it.
|
||||
TERM="${TERM:-xterm}" "$INSTALLIMAGE" -a -n "$BOX_HOSTNAME" -r yes -l 1 -i "$images/$image" -g -p "$parts" -d "$(IFS=,; echo "${drives[*]}")" -t yes -G yes
|
||||
log "installimage finished; rebooting into Ubuntu. Run this script again (MODE=provision or auto) once ssh answers (the host key is new: -G yes)."
|
||||
sync; reboot
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# provision mode: the installed Ubuntu
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
step_hostname() {
|
||||
if [ "$(hostnamectl --static 2>/dev/null || hostname)" = "$BOX_HOSTNAME" ]; then ok hostname "$BOX_HOSTNAME"; return; fi
|
||||
hostnamectl set-hostname "$BOX_HOSTNAME"; grep -q "$BOX_HOSTNAME" /etc/hosts || printf '127.0.1.1 %s\n' "$BOX_HOSTNAME" >> /etc/hosts
|
||||
changed hostname "$BOX_HOSTNAME"
|
||||
}
|
||||
|
||||
step_os_check() {
|
||||
. /etc/os-release
|
||||
[ "${ID:-}" = ubuntu ] && [ "${VERSION_ID:-}" = 24.04 ] || die "this is ${PRETTY_NAME:-unknown}, not Ubuntu 24.04"
|
||||
ok os "$PRETTY_NAME, $(nproc) threads, $(awk '/MemTotal/ { printf "%d GB", $2 / 1024 / 1024 }' /proc/meminfo)"
|
||||
}
|
||||
|
||||
# the proven Ubuntu 24.04 set: the PC build job's APT lists (app/igneum-app/src/jobbuild.rs: mingw-w64 posix threads so
|
||||
# libstdc++ has std::thread for rocksdb, clang for librocksdb-sys's bindgen, protoc for the node's proto crates) plus the
|
||||
# task's list (build-essential, clang, lld, pkg-config, libssl-dev, cmake, git, tmux) and what the scripts here call
|
||||
APT_PACKAGES=(
|
||||
build-essential clang lld llvm libclang-dev pkg-config libssl-dev cmake protobuf-compiler
|
||||
gcc-mingw-w64-x86-64 g++-mingw-w64-x86-64 binutils-mingw-w64-x86-64 mingw-w64-x86-64-dev mingw-w64-tools
|
||||
git tmux curl ca-certificates xz-utils zstd unzip rsync jq python3 ufw htop file
|
||||
)
|
||||
step_apt() {
|
||||
local need=() p
|
||||
for p in "${APT_PACKAGES[@]}"; do dpkg -s "$p" >/dev/null 2>&1 || need+=("$p"); done
|
||||
if [ "${#need[@]}" = 0 ]; then ok apt "${#APT_PACKAGES[@]} packages present"; return; fi
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq --no-install-recommends "${need[@]}"
|
||||
changed apt "installed ${need[*]}"
|
||||
}
|
||||
|
||||
step_mingw_alternatives() {
|
||||
# Ubuntu ships -posix and -win32 variants behind update-alternatives; the Windows exes want posix threads (jobbuild.rs)
|
||||
local tool want cur any=0
|
||||
for tool in gcc g++; do
|
||||
want="/usr/bin/x86_64-w64-mingw32-$tool-posix"
|
||||
cur=$(readlink -f "/etc/alternatives/x86_64-w64-mingw32-$tool" 2>/dev/null || true)
|
||||
[ -x "$want" ] || die "no $want after apt"
|
||||
if [ "$cur" != "$want" ]; then update-alternatives --set "x86_64-w64-mingw32-$tool" "$want" >/dev/null; any=1; fi
|
||||
done
|
||||
[ "$any" = 1 ] && changed mingw-alternatives "posix threads" || ok mingw-alternatives "posix threads"
|
||||
}
|
||||
|
||||
step_no_swap() {
|
||||
local any=0
|
||||
if [ -n "$(swapon --noheadings --show 2>/dev/null)" ]; then swapoff -a; any=1; fi
|
||||
if grep -qE '^[^#].*\sswap\s' /etc/fstab; then sed -i -E 's/^([^#].*\sswap\s.*)$/# \1 (disabled by infra\/build-server\/provision.sh)/' /etc/fstab; any=1; fi
|
||||
[ "$any" = 1 ] && changed swap "off, fstab entry commented" || ok swap "none"
|
||||
}
|
||||
|
||||
step_sysctl() {
|
||||
local f=/etc/sysctl.d/90-igneum-build.conf tmp
|
||||
tmp=$(mktemp)
|
||||
cat > "$tmp" <<'EOF'
|
||||
# igneum-build-1: a compile box with no swap (infra/build-server/provision.sh)
|
||||
vm.swappiness = 0
|
||||
vm.overcommit_memory = 0
|
||||
vm.dirty_ratio = 20
|
||||
vm.dirty_background_ratio = 5
|
||||
vm.max_map_count = 1048576
|
||||
fs.file-max = 4194304
|
||||
fs.inotify.max_user_watches = 1048576
|
||||
fs.inotify.max_user_instances = 8192
|
||||
kernel.pid_max = 4194304
|
||||
kernel.threads-max = 1048576
|
||||
net.core.somaxconn = 4096
|
||||
net.ipv4.tcp_fin_timeout = 15
|
||||
EOF
|
||||
if [ -f "$f" ] && cmp -s "$tmp" "$f"; then rm -f "$tmp"; ok sysctl "$f"; return; fi
|
||||
install -m 644 "$tmp" "$f"; rm -f "$tmp"; sysctl --system >/dev/null
|
||||
changed sysctl "$f applied"
|
||||
}
|
||||
|
||||
step_limits() {
|
||||
local f=/etc/security/limits.d/90-igneum-build.conf
|
||||
if [ -f "$f" ]; then ok limits; return; fi
|
||||
printf '%s soft nofile 1048576\n%s hard nofile 1048576\n%s soft nproc unlimited\n' "$BUILD_USER" "$BUILD_USER" "$BUILD_USER" > "$f"
|
||||
changed limits "$f"
|
||||
}
|
||||
|
||||
step_user() {
|
||||
local keys
|
||||
if ! id -u "$BUILD_USER" >/dev/null 2>&1; then useradd -m -s /bin/bash -G users "$BUILD_USER"; changed user "$BUILD_USER created"; else ok user "$BUILD_USER"; fi
|
||||
install -d -m 700 -o "$BUILD_USER" -g "$BUILD_USER" "$BUILD_HOME/.ssh"
|
||||
if [ -n "$SSH_PUBKEY" ]; then keys="$SSH_PUBKEY"; elif [ -s /root/.ssh/authorized_keys ]; then keys=$(cat /root/.ssh/authorized_keys); else die "no key for $BUILD_USER: root has no authorized_keys and SSH_PUBKEY is unset"; fi
|
||||
if [ -f "$BUILD_HOME/.ssh/authorized_keys" ] && [ "$(cat "$BUILD_HOME/.ssh/authorized_keys")" = "$keys" ]; then ok authorized_keys; else
|
||||
printf '%s\n' "$keys" > "$BUILD_HOME/.ssh/authorized_keys"; chmod 600 "$BUILD_HOME/.ssh/authorized_keys"; chown "$BUILD_USER:$BUILD_USER" "$BUILD_HOME/.ssh/authorized_keys"
|
||||
changed authorized_keys "$(printf '%s\n' "$keys" | grep -c .) key(s) from root"
|
||||
fi
|
||||
}
|
||||
|
||||
worktree_count() { find /srv/builds -mindepth 1 -maxdepth 1 -type d -not -name '_*' | wc -l | tr -d ' '; }
|
||||
|
||||
step_dirs() {
|
||||
local d any=0
|
||||
for d in /srv/builds /srv/builds/_locks /srv/sccache /srv/artefacts; do
|
||||
if [ ! -d "$d" ]; then install -d -m 755 -o "$BUILD_USER" -g "$BUILD_USER" "$d"; any=1; fi
|
||||
done
|
||||
if [ ! -f /srv/builds/_locks/slots ] || [ "$(cat /srv/builds/_locks/slots)" != "$SLOTS" ]; then printf '%s\n' "$SLOTS" > /srv/builds/_locks/slots; chown "$BUILD_USER:$BUILD_USER" /srv/builds/_locks/slots; any=1; fi
|
||||
for d in $WORKTREES; do
|
||||
case "$d" in */*|.*|_*) die "worktree name '$d' is not a plain directory name" ;; esac
|
||||
if [ ! -d "/srv/builds/$d" ]; then install -d -m 755 -o "$BUILD_USER" -g "$BUILD_USER" "/srv/builds/$d"; any=1; fi
|
||||
done
|
||||
[ "$any" = 1 ] && changed dirs "/srv/builds ($(worktree_count) worktree dirs), /srv/sccache, slots=$SLOTS" || ok dirs "$(worktree_count) worktree dirs, slots=$SLOTS"
|
||||
}
|
||||
|
||||
step_mirrors() {
|
||||
local r any=0
|
||||
for r in /srv/igneum.git /srv/igneum-node.git; do
|
||||
if [ ! -d "$r" ]; then install -d -m 755 -o "$BUILD_USER" -g "$BUILD_USER" "$r"; as_build "git init -q --bare -b master $r"; any=1; fi
|
||||
done
|
||||
as_build "git config --global --get safe.directory >/dev/null 2>&1 || git config --global --add safe.directory '*'"
|
||||
as_build "git config --global init.defaultBranch master; git config --global gc.auto 0"
|
||||
[ "$any" = 1 ] && changed mirrors "bare /srv/igneum.git and /srv/igneum-node.git (push from the Mac, see the header)" || ok mirrors
|
||||
}
|
||||
|
||||
step_rustup() {
|
||||
local cargo="$BUILD_HOME/.cargo/bin/cargo" rustup="$BUILD_HOME/.cargo/bin/rustup" any=0 t
|
||||
if [ ! -x "$rustup" ]; then
|
||||
as_build "curl -fsSL https://sh.rustup.rs | sh -s -- -y --profile minimal --no-modify-path --default-toolchain $RUST_TOOLCHAIN" >/dev/null
|
||||
any=1
|
||||
fi
|
||||
if ! as_build "$rustup toolchain list" | grep -q "^$RUST_TOOLCHAIN-"; then as_build "$rustup toolchain install $RUST_TOOLCHAIN --profile minimal" >/dev/null; any=1; fi
|
||||
if [ "$(as_build "$rustup default" | cut -d- -f1)" != "$RUST_TOOLCHAIN" ]; then as_build "$rustup default $RUST_TOOLCHAIN" >/dev/null; any=1; fi
|
||||
for t in x86_64-pc-windows-gnu x86_64-unknown-linux-gnu; do
|
||||
as_build "$rustup target list --installed --toolchain $RUST_TOOLCHAIN" | grep -qx "$t" || { as_build "$rustup target add $t --toolchain $RUST_TOOLCHAIN" >/dev/null; any=1; }
|
||||
done
|
||||
as_build "$rustup component list --installed --toolchain $RUST_TOOLCHAIN" | grep -q '^clippy' || { as_build "$rustup component add clippy rustfmt --toolchain $RUST_TOOLCHAIN" >/dev/null; any=1; }
|
||||
[ "$any" = 1 ] && changed rustup "$(as_build "$cargo --version"), targets: $(as_build "$rustup target list --installed" | tr '\n' ' ')" || ok rustup "$(as_build "$cargo --version"), targets: $(as_build "$rustup target list --installed" | tr '\n' ' ')"
|
||||
}
|
||||
|
||||
step_sccache() {
|
||||
local cargo="$BUILD_HOME/.cargo/bin/cargo" bin="$BUILD_HOME/.cargo/bin/sccache" cfgdir="$BUILD_HOME/.config/sccache" any=0 bytes tmp
|
||||
if [ ! -x "$bin" ] || { [ -n "$SCCACHE_VERSION" ] && ! "$bin" --version | grep -q " $SCCACHE_VERSION\$"; }; then
|
||||
as_build "$cargo install sccache --locked ${SCCACHE_VERSION:+--version $SCCACHE_VERSION}" >/dev/null 2>&1 || as_build "$cargo install sccache ${SCCACHE_VERSION:+--version $SCCACHE_VERSION}" >/dev/null
|
||||
any=1
|
||||
fi
|
||||
bytes=$(( SCCACHE_GB * 1024 * 1024 * 1024 ))
|
||||
install -d -m 755 -o "$BUILD_USER" -g "$BUILD_USER" "$cfgdir"
|
||||
tmp=$(mktemp)
|
||||
printf '[cache.disk]\ndir = "/srv/sccache"\nsize = %s\n' "$bytes" > "$tmp"
|
||||
if ! cmp -s "$tmp" "$cfgdir/config"; then install -m 644 -o "$BUILD_USER" -g "$BUILD_USER" "$tmp" "$cfgdir/config"; any=1; fi
|
||||
rm -f "$tmp"
|
||||
[ "$any" = 1 ] && changed sccache "$(as_build "$bin --version"), disk cache /srv/sccache, $SCCACHE_GB GB" || ok sccache "$(as_build "$bin --version"), /srv/sccache $SCCACHE_GB GB"
|
||||
}
|
||||
|
||||
step_cargo_config() {
|
||||
# the build user's cargo defaults: sccache in front of rustc, 90 jobs (96 threads, 6 left for ssh, rsync and the
|
||||
# system), lld for the native target through clang. The Windows target's compilers and flags are NOT here: they are
|
||||
# set per build by tools/cross-remote.sh, the same variables as the Mac's proto-cuda/windows-node/cross-build.sh and
|
||||
# the PC's jobbuild.rs, so a build's flags are visible in the script that runs it.
|
||||
local f="$BUILD_HOME/.cargo/config.toml" tmp
|
||||
tmp=$(mktemp)
|
||||
cat > "$tmp" <<'EOF'
|
||||
# igneum-build-1 (infra/build-server/provision.sh)
|
||||
[build]
|
||||
rustc-wrapper = "/home/build/.cargo/bin/sccache"
|
||||
jobs = 90
|
||||
|
||||
[target.x86_64-unknown-linux-gnu]
|
||||
linker = "clang"
|
||||
rustflags = ["-C", "link-arg=-fuse-ld=lld"]
|
||||
|
||||
[net]
|
||||
git-fetch-with-cli = true
|
||||
EOF
|
||||
if cmp -s "$tmp" "$f"; then rm -f "$tmp"; ok cargo-config "$f"; return; fi
|
||||
install -m 644 -o "$BUILD_USER" -g "$BUILD_USER" "$tmp" "$f"; rm -f "$tmp"
|
||||
changed cargo-config "$f"
|
||||
}
|
||||
|
||||
step_profile() {
|
||||
# sourced by tools/build-remote.sh's remote script (a non-login ssh shell reads no profile) and by login shells
|
||||
local f=/etc/profile.d/igneum-build.sh tmp
|
||||
tmp=$(mktemp)
|
||||
cat > "$tmp" <<EOF
|
||||
# igneum-build-1 (infra/build-server/provision.sh)
|
||||
export PATH="/home/build/.cargo/bin:/usr/local/bin:\$PATH"
|
||||
export SCCACHE_DIR=/srv/sccache
|
||||
export SCCACHE_CACHE_SIZE=${SCCACHE_GB}G
|
||||
export CARGO_INCREMENTAL=0
|
||||
export IGNEUM_BUILD_SLOTS_DIR=/srv/builds/_locks
|
||||
export IGNEUM_BUILD_ROOT=/srv/builds
|
||||
export IGNEUM_RUST_TOOLCHAIN=$RUST_TOOLCHAIN
|
||||
EOF
|
||||
if [ -f "$f" ] && cmp -s "$tmp" "$f"; then rm -f "$tmp"; ok profile "$f"; return; fi
|
||||
install -m 644 "$tmp" "$f"; rm -f "$tmp"; changed profile "$f"
|
||||
}
|
||||
|
||||
step_node() {
|
||||
local want have shasums tarball ver dir
|
||||
have=$(/usr/local/bin/node --version 2>/dev/null || true)
|
||||
case "$have" in v$NODE_MAJOR.*) ok node "$have"; return ;; esac
|
||||
# the newest $NODE_MAJOR release from nodejs.org, checked against its SHASUMS256.txt (https, the official host)
|
||||
shasums=$(curl -fsSL "https://nodejs.org/dist/latest-v$NODE_MAJOR.x/SHASUMS256.txt")
|
||||
tarball=$(printf '%s\n' "$shasums" | awk '$2 ~ /linux-x64\.tar\.xz$/ { print $2 }' | head -1)
|
||||
[ -n "$tarball" ] || die "no linux-x64 tarball in the Node $NODE_MAJOR SHASUMS"
|
||||
ver=${tarball#node-}; ver=${ver%-linux-x64.tar.xz}
|
||||
dir=/usr/local/lib/nodejs
|
||||
install -d "$dir"
|
||||
( cd "$dir" && curl -fsSLO "https://nodejs.org/dist/latest-v$NODE_MAJOR.x/$tarball" && printf '%s\n' "$shasums" | grep " $tarball\$" | sha256sum -c --quiet - && tar -xJf "$tarball" && rm -f "$tarball" )
|
||||
ln -sfn "$dir/node-$ver-linux-x64/bin/node" /usr/local/bin/node
|
||||
ln -sfn "$dir/node-$ver-linux-x64/bin/npm" /usr/local/bin/npm
|
||||
ln -sfn "$dir/node-$ver-linux-x64/bin/npx" /usr/local/bin/npx
|
||||
changed node "$(/usr/local/bin/node --version) from nodejs.org (sha256 checked)"
|
||||
}
|
||||
|
||||
step_sshd() {
|
||||
local f=/etc/ssh/sshd_config.d/10-igneum-build.conf tmp
|
||||
tmp=$(mktemp)
|
||||
cat > "$tmp" <<'EOF'
|
||||
# igneum-build-1 (infra/build-server/provision.sh): keys only
|
||||
PasswordAuthentication no
|
||||
KbdInteractiveAuthentication no
|
||||
ChallengeResponseAuthentication no
|
||||
PubkeyAuthentication yes
|
||||
PermitRootLogin prohibit-password
|
||||
PermitEmptyPasswords no
|
||||
X11Forwarding no
|
||||
MaxAuthTries 4
|
||||
ClientAliveInterval 60
|
||||
ClientAliveCountMax 10
|
||||
EOF
|
||||
if [ -f "$f" ] && cmp -s "$tmp" "$f"; then rm -f "$tmp"; ok sshd "$f"; return; fi
|
||||
install -m 644 "$tmp" "$f"; rm -f "$tmp"
|
||||
# Hetzner's installimage may leave a cloud-init drop-in that sets PasswordAuthentication yes; the lowest-numbered file wins
|
||||
if [ -f /etc/ssh/sshd_config.d/50-cloud-init.conf ] && grep -qi '^PasswordAuthentication yes' /etc/ssh/sshd_config.d/50-cloud-init.conf; then
|
||||
sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config.d/50-cloud-init.conf
|
||||
fi
|
||||
sshd -t || die "sshd -t rejected the configuration; the drop-in $f was NOT activated"
|
||||
systemctl reload ssh 2>/dev/null || systemctl reload sshd
|
||||
changed sshd "key-only, root prohibit-password"
|
||||
}
|
||||
|
||||
step_ufw() {
|
||||
local p want=() any=0 status
|
||||
status=$(ufw status verbose 2>/dev/null || true)
|
||||
grep -q 'Default: deny (incoming), allow (outgoing)' <<<"$status" || { ufw --force default deny incoming >/dev/null; ufw --force default allow outgoing >/dev/null; any=1; }
|
||||
want=(22)
|
||||
for p in $P2P_PORTS; do want+=("$p"); done
|
||||
for p in "${want[@]}"; do
|
||||
grep -qE "^$p/tcp +ALLOW IN +Anywhere *$" <<<"$status" || { ufw allow "$p/tcp" >/dev/null; any=1; }
|
||||
done
|
||||
grep -q '^Status: active' <<<"$status" || { ufw --force enable >/dev/null; any=1; }
|
||||
[ "$any" = 1 ] && changed ufw "22 and ${P2P_PORTS} open, everything else denied" || ok ufw "22 and ${P2P_PORTS}"
|
||||
}
|
||||
|
||||
step_summary() {
|
||||
log "summary:"
|
||||
{
|
||||
printf 'host %s, %s threads, %s RAM, root fs %s free\n' "$(hostname)" "$(nproc)" "$(awk '/MemTotal/ { printf "%d GB", $2 / 1024 / 1024 }' /proc/meminfo)" "$(df -h / | awk 'NR == 2 { print $4 }')"
|
||||
printf 'raid: %s\n' "$(grep -E '^md' /proc/mdstat 2>/dev/null | tr '\n' ';' || echo none)"
|
||||
printf 'rust: %s | %s | targets %s\n' "$(as_build "$BUILD_HOME/.cargo/bin/rustc --version")" "$(as_build "$BUILD_HOME/.cargo/bin/cargo --version")" "$(as_build "$BUILD_HOME/.cargo/bin/rustup target list --installed" | tr '\n' ' ')"
|
||||
printf 'sccache: %s, %s\n' "$(as_build "$BUILD_HOME/.cargo/bin/sccache --version")" "$(cat "$BUILD_HOME/.config/sccache/config" | tr '\n' ' ')"
|
||||
printf 'mingw: %s\n' "$(x86_64-w64-mingw32-gcc-posix --version | head -1)"
|
||||
printf 'clang: %s | lld: %s\n' "$(clang --version | head -1)" "$(ld.lld --version | head -1)"
|
||||
printf 'node: %s | git: %s | tmux: %s\n' "$(/usr/local/bin/node --version)" "$(git --version)" "$(tmux -V)"
|
||||
printf 'swap: %s\n' "$(swapon --noheadings --show 2>/dev/null | wc -l | awk '{ print ($1 == 0) ? "none" : $1 " device(s) ACTIVE" }')"
|
||||
printf 'mirrors: /srv/igneum.git (%s) /srv/igneum-node.git (%s)\n' "$(as_build 'git -C /srv/igneum.git branch --list | wc -l') branches" "$(as_build 'git -C /srv/igneum-node.git branch --list | wc -l') branches"
|
||||
printf 'builds: %s worktree dirs under /srv/builds, %s slot(s)\n' "$(worktree_count)" "$(cat /srv/builds/_locks/slots)"
|
||||
printf 'ufw: %s\n' "$(ufw status | grep -E 'ALLOW' | awk '{ print $1 }' | tr '\n' ' ')"
|
||||
printf 'ssh line: ssh -i ~/.ssh/igneum_ed25519 build@%s\n' "$(hostname -I 2>/dev/null | awk '{ print $1 }')"
|
||||
} | sed 's/^/ /'
|
||||
}
|
||||
|
||||
do_provision() {
|
||||
step_os_check
|
||||
step_hostname
|
||||
step_apt
|
||||
step_mingw_alternatives
|
||||
step_no_swap
|
||||
step_sysctl
|
||||
step_limits
|
||||
step_user
|
||||
step_dirs
|
||||
step_mirrors
|
||||
step_rustup
|
||||
step_sccache
|
||||
step_cargo_config
|
||||
step_profile
|
||||
step_node
|
||||
step_sshd
|
||||
step_ufw
|
||||
step_summary
|
||||
log "done"
|
||||
}
|
||||
|
||||
case "$MODE" in
|
||||
install) do_install ;;
|
||||
provision) do_provision ;;
|
||||
auto) if in_rescue; then log "rescue system detected: install mode"; do_install; else do_provision; fi ;;
|
||||
*) die "MODE must be auto, install or provision" ;;
|
||||
esac
|
||||
110
infra/build-server/remote-run.sh
Executable file
110
infra/build-server/remote-run.sh
Executable file
|
|
@ -0,0 +1,110 @@
|
|||
#!/usr/bin/env bash
|
||||
# The remote half of tools/build-remote.sh and tools/cross-remote.sh. It runs ON igneum-build-1, fed by lib.sh bs_remote_run
|
||||
# over `ssh build@<box> bash -s` with these exports prepended (never run it by hand on the Mac):
|
||||
# BR_DIR the crate directory on the box BR_CMD the shell string to run there (cargo ...)
|
||||
# BR_LABEL the slot-file label (ends with "; agent=<name>")
|
||||
# BR_TOOL build-remote | cross-remote BR_KIND node-linux | node-windows | app | app-windows | prove | suite | check | other
|
||||
# BR_WT the worktree name BR_CRATE the crate path relative to the worktree root
|
||||
# BR_COMMAND the cargo command as typed BR_TARGET the rust target triple
|
||||
# BR_BRANCH BR_SHA BR_AGENT the agent name (IGNEUM_AGENT on the Mac, else the worktree)
|
||||
# BR_ARTEFACTS space-separated paths (relative to BR_DIR) the Mac will fetch; empty for test, check, clippy
|
||||
#
|
||||
# 1. Takes a build slot: flock on $IGNEUM_BUILD_SLOTS_DIR/build-<k> for k below the count in .../slots (the box's own slot
|
||||
# files, never the Mac's); when every slot is busy it waits up to 2 h on build-0 and exits 75 if it gives up. The holder
|
||||
# line is `pid N since HH:MM:SSZ waited S s: <label>`, the format tools/lock/with-lock.sh writes on the Mac.
|
||||
# 2. Runs BR_CMD in BR_DIR with sccache, prints one `build-remote: RESULT rc= secs= compiles= sccache_hits_total= ...` line.
|
||||
# 3. Appends one JSON line to /srv/builds/_log/builds.jsonl (the worker dashboard reads it; asked for by main on 6 October
|
||||
# 2026): on success, on failure and on the slot give-up. UTC ISO 8601 Z times, numbers unquoted, unknown fields omitted,
|
||||
# artefacts with bytes and sha256 only when the run succeeded (a failed build would list the previous build's files),
|
||||
# the line kept under 4 KB.
|
||||
set -uo pipefail
|
||||
. /etc/profile.d/igneum-build.sh
|
||||
: "${BR_DIR:?}" "${BR_CMD:?}" "${BR_LABEL:?}" "${BR_TOOL:?}" "${BR_KIND:?}"
|
||||
BR_HOST=$(hostname); BR_PID=$$; BR_T0=$(date +%s)
|
||||
export BR_HOST BR_PID BR_T0
|
||||
LOG_DIR=/srv/builds/_log; mkdir -p "$LOG_DIR"
|
||||
|
||||
# jsonlog <exit> <slot> <wait_s> <start> <end> <secs> <compiles> <hits> <misses> <hits_total> <misses_total>
|
||||
jsonlog() {
|
||||
BR_EXIT="$1" BR_SLOT="$2" BR_WAIT="$3" BR_START="$4" BR_END="$5" BR_SECS="$6" BR_COMPILES="$7" \
|
||||
BR_HITS="$8" BR_MISSES="$9" BR_HITS_T="${10}" BR_MISSES_T="${11}" BR_LOG="$LOG_DIR/builds.jsonl" python3 - <<'PY' || echo "build-remote: WARNING the JSONL log line was not written" >&2
|
||||
import hashlib, json, os, time
|
||||
e = os.environ
|
||||
def iso(t):
|
||||
return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(int(t))) if t else None
|
||||
def num(v):
|
||||
try: return int(v)
|
||||
except (TypeError, ValueError): return None
|
||||
d = {
|
||||
"v": 1, "id": f"{e['BR_HOST']}-{e['BR_T0']}-{e['BR_PID']}", "host": e['BR_HOST'], "tool": e['BR_TOOL'],
|
||||
"worktree": e.get('BR_WT'), "crate": e.get('BR_CRATE'), "kind": e['BR_KIND'], "command": e.get('BR_COMMAND'),
|
||||
"target": e.get('BR_TARGET'), "branch": e.get('BR_BRANCH'), "sha": e.get('BR_SHA'), "label": e['BR_LABEL'],
|
||||
"agent": e.get('BR_AGENT'), "slot": num(e['BR_SLOT']), "wait_s": num(e['BR_WAIT']), "queued_at": iso(e['BR_T0']),
|
||||
"start": iso(e['BR_START']), "end": iso(e['BR_END']), "secs": num(e['BR_SECS']), "exit": num(e['BR_EXIT']),
|
||||
"compiles": num(e['BR_COMPILES']),
|
||||
}
|
||||
sc = {k: num(e[v]) for k, v in (("hits", "BR_HITS"), ("misses", "BR_MISSES"), ("hits_total", "BR_HITS_T"), ("misses_total", "BR_MISSES_T"))}
|
||||
sc = {k: v for k, v in sc.items() if v is not None}
|
||||
if sc: d["sccache"] = sc
|
||||
try:
|
||||
d["load_end"] = [float(x) for x in open('/proc/loadavg').read().split()[:3]]
|
||||
except OSError:
|
||||
pass
|
||||
arts = []
|
||||
if d["exit"] == 0:
|
||||
for p in e.get('BR_ARTEFACTS', '').split():
|
||||
fp = os.path.join(e['BR_DIR'], p)
|
||||
if os.path.isfile(fp):
|
||||
h = hashlib.sha256()
|
||||
with open(fp, 'rb') as f:
|
||||
for chunk in iter(lambda: f.read(1 << 20), b''):
|
||||
h.update(chunk)
|
||||
arts.append({"path": p, "bytes": os.path.getsize(fp), "sha256": h.hexdigest()})
|
||||
d["artefacts"] = arts
|
||||
d = {k: v for k, v in d.items() if v is not None and v != ""}
|
||||
line = json.dumps(d, separators=(',', ':'))
|
||||
if len(line) > 4000:
|
||||
for k in ("command", "label"):
|
||||
if k in d: d[k] = d[k][:200]
|
||||
line = json.dumps(d, separators=(',', ':'))
|
||||
with open(e['BR_LOG'], 'a') as f:
|
||||
f.write(line + "\n")
|
||||
PY
|
||||
}
|
||||
|
||||
slots=$(cat "$IGNEUM_BUILD_SLOTS_DIR/slots" 2>/dev/null || echo 1); [ "$slots" -ge 1 ] 2>/dev/null || slots=1
|
||||
got=""
|
||||
for k in $(seq 0 $((slots - 1))); do
|
||||
exec {fd}>"$IGNEUM_BUILD_SLOTS_DIR/build-$k"
|
||||
if flock -n "$fd"; then got=$k; break; fi
|
||||
exec {fd}>&-
|
||||
done
|
||||
if [ -z "$got" ]; then
|
||||
echo "build-remote: all $slots slot(s) busy, waiting (up to 2 h) for build-0: $(head -c 160 "$IGNEUM_BUILD_SLOTS_DIR/build-0" 2>/dev/null)" >&2
|
||||
exec {fd}>"$IGNEUM_BUILD_SLOTS_DIR/build-0"
|
||||
if ! flock -w 7200 "$fd"; then
|
||||
echo "build-remote: gave up waiting for a slot after 2 h" >&2
|
||||
jsonlog 75 0 $(( $(date +%s) - BR_T0 )) "" "$(date +%s)" "" "" "" "" "" ""
|
||||
exit 75
|
||||
fi
|
||||
got=0
|
||||
fi
|
||||
waited=$(( $(date +%s) - BR_T0 ))
|
||||
printf 'pid %s since %sZ waited %s s: %s\n' "$BR_PID" "$(date -u +%H:%M:%S)" "$waited" "$BR_LABEL" > "$IGNEUM_BUILD_SLOTS_DIR/build-$got"
|
||||
echo "build-remote: holding build-$got on $BR_HOST (waited $waited s)" >&2
|
||||
|
||||
cd "$BR_DIR" || { jsonlog 2 "$got" "$waited" "" "$(date +%s)" "" "" "" "" "" ""; exit 2; }
|
||||
sccache --start-server >/dev/null 2>&1 || true
|
||||
stat_field() { sccache --show-stats 2>/dev/null | awk -v key="$1" 'index($0, key) == 1 { print $NF; exit }'; }
|
||||
exec_before=$(stat_field "Compile requests executed"); hits_before=$(stat_field "Cache hits "); misses_before=$(stat_field "Cache misses ")
|
||||
t1=$(date +%s)
|
||||
eval "$BR_CMD"
|
||||
rc=$?
|
||||
t2=$(date +%s); secs=$(( t2 - t1 ))
|
||||
exec_after=$(stat_field "Compile requests executed"); hits_after=$(stat_field "Cache hits "); misses_after=$(stat_field "Cache misses ")
|
||||
compiles=$(( ${exec_after:-0} - ${exec_before:-0} )); hits=$(( ${hits_after:-0} - ${hits_before:-0} )); misses=$(( ${misses_after:-0} - ${misses_before:-0} ))
|
||||
printf 'build-remote: RESULT rc=%s secs=%s compiles=%s sccache_hits=%s sccache_misses=%s sccache_hits_total=%s sccache_misses_total=%s load=%s\n' \
|
||||
"$rc" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-?}" "${misses_after:-?}" "$(cut -d' ' -f1-3 /proc/loadavg)"
|
||||
jsonlog "$rc" "$got" "$waited" "$t1" "$t2" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-}" "${misses_after:-}"
|
||||
: > "$IGNEUM_BUILD_SLOTS_DIR/build-$got"
|
||||
exit "$rc"
|
||||
62
infra/build-server/run-from-mac.sh
Executable file
62
infra/build-server/run-from-mac.sh
Executable file
|
|
@ -0,0 +1,62 @@
|
|||
#!/usr/bin/env bash
|
||||
# Provision igneum-build-1 from this Mac and wire the Mac to it. Idempotent; run it again after any change to provision.sh.
|
||||
#
|
||||
# infra/build-server/run-from-mac.sh <ip> install (if in rescue) or provision, then wire the Mac
|
||||
# infra/build-server/run-from-mac.sh <ip> --wire-only skip provision.sh: only the host file, the remotes and the mirror push
|
||||
# RUST_TOOLCHAIN=1.99.0 SLOTS=2 infra/build-server/run-from-mac.sh <ip> settings pass through to provision.sh
|
||||
#
|
||||
# What it does: 1. ssh root@<ip> with provision.sh on stdin, WORKTREES filled from `git worktree list` of the igneum repo
|
||||
# (one /srv/builds/<name> per agent worktree); 2. writes build@<ip> to ~/.config/igneum/build-server (what tools/build-remote.sh
|
||||
# and tools/cross-remote.sh read); 3. adds the `build` remote to the igneum repo and to the fork vendor/igneum-node and pushes
|
||||
# every branch to the bare mirrors on the box (the fork exists only on this Mac; the mirror is its first copy elsewhere);
|
||||
# 4. prints the ssh line. If the box is still in the rescue system, provision.sh installs Ubuntu and reboots; run this again
|
||||
# when ssh answers (the host key changes: the old entry is removed here).
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO="$(cd "$HERE/../.." && pwd)"
|
||||
MAIN_REPO="${IGNEUM_MAIN_REPO:-/Users/joshm/Projects/igneum}" # the shared checkout: the fork lives under its vendor/
|
||||
# shellcheck disable=SC2034 # shared with the scripts that source lib.sh
|
||||
BS_TOOL=run-from-mac
|
||||
# shellcheck source=lib.sh
|
||||
. "$HERE/lib.sh"
|
||||
|
||||
IP="${1:-}"; shift || true
|
||||
[ -n "$IP" ] || bs_die "usage: run-from-mac.sh <ip> [--wire-only]"
|
||||
WIRE_ONLY=0; [ "${1:-}" = --wire-only ] && WIRE_ONLY=1
|
||||
PASS="" # a string, not an array: bash 3.2 (the Mac) treats an empty array as unbound under set -u
|
||||
for v in MODE RUST_TOOLCHAIN SCCACHE_GB SCCACHE_VERSION NODE_MAJOR SLOTS P2P_PORTS BOX_HOSTNAME SSH_PUBKEY; do
|
||||
[ -n "${!v:-}" ] && PASS="$PASS $v=$(printf '%q' "${!v}")"
|
||||
done
|
||||
|
||||
ROOT_SSH=(ssh -i "$BS_KEY" -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ServerAliveInterval=15 "root@$IP")
|
||||
|
||||
if [ "$WIRE_ONLY" = 0 ]; then
|
||||
WORKTREES=$(git -C "$MAIN_REPO" worktree list --porcelain | awk '/^worktree /{ print $2 }' | xargs -n1 basename | tr '\n' ' ')
|
||||
bs_log "provisioning root@$IP with $(printf '%s\n' "$WORKTREES" | wc -w | tr -d ' ') worktree names${PASS:+ and$PASS}"
|
||||
if ! "${ROOT_SSH[@]}" "WORKTREES='$WORKTREES'$PASS bash -s" < "$HERE/provision.sh"; then
|
||||
bs_log "provision.sh did not finish (an install-mode run ends with a reboot and a lost connection: wait for ssh, then run this again)"
|
||||
ssh-keygen -R "$IP" >/dev/null 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
if "${ROOT_SSH[@]}" 'hostname' 2>/dev/null | grep -q '^rescue'; then bs_die "still in the rescue system after provision.sh"; fi
|
||||
fi
|
||||
|
||||
mkdir -p "$(dirname "$BS_HOST_FILE")"
|
||||
printf 'build@%s\n' "$IP" > "$BS_HOST_FILE"
|
||||
bs_log "wrote $BS_HOST_FILE"
|
||||
bs_host
|
||||
bs_ssh 'hostname; nproc' >/dev/null || bs_die "build@$IP does not answer with $BS_KEY"
|
||||
|
||||
wire_remote() { # repo-dir mirror label
|
||||
local dir="$1" mirror="$2" label="$3" url="$BS_HOST:$2" cur
|
||||
cur=$(git -C "$dir" remote get-url build 2>/dev/null || true)
|
||||
if [ -z "$cur" ]; then git -C "$dir" remote add build "$url"; bs_log "$label: remote build = $url"
|
||||
elif [ "$cur" != "$url" ]; then git -C "$dir" remote set-url build "$url"; bs_log "$label: remote build -> $url"
|
||||
else bs_log "$label: remote build ok"; fi
|
||||
GIT_SSH_COMMAND="$BS_SSH_CMD" git -C "$dir" push -q --force build --all && bs_log "$label: every branch pushed to $mirror ($(git -C "$dir" branch --list | wc -l | tr -d ' ') branches)" || bs_die "$label: push to $mirror failed"
|
||||
}
|
||||
wire_remote "$MAIN_REPO" "$BS_MIRROR_REPO" "igneum"
|
||||
wire_remote "$MAIN_REPO/vendor/igneum-node" "$BS_MIRROR_NODE" "igneum-node (the fork)"
|
||||
|
||||
bs_log "ssh line: ssh -i $BS_KEY build@$IP"
|
||||
bs_log "next: cd <crate> && $REPO/tools/build-remote.sh (cross: tools/cross-remote.sh)"
|
||||
112
tools/build-remote.sh
Executable file
112
tools/build-remote.sh
Executable file
|
|
@ -0,0 +1,112 @@
|
|||
#!/usr/bin/env bash
|
||||
# Build on igneum-build-1 instead of this Mac. Run from any crate directory of any worktree on the Mac (a fork worktree under
|
||||
# vendor/, igneum-pow, app/igneum-app, proving/igneum-prove): the sources go to /srv/builds/<worktree>/<same relative path> on
|
||||
# the box (HEAD through the bare mirror, uncommitted changes by rsync, changed files re-stamped with touch in lib.sh's
|
||||
# bs_overlay_dir: the copied-sources rule), the cargo command runs there
|
||||
# with sccache and -j 90 under one of the box's build slots, and the artefacts come back into target-remote/ here.
|
||||
#
|
||||
# tools/build-remote.sh the default command for this crate (below)
|
||||
# tools/build-remote.sh -- build --release -p kaspad --features kaspad/igneum-pow
|
||||
# tools/build-remote.sh -- test --release -p kaspa-consensus-core --lib
|
||||
# tools/build-remote.sh --artefacts "target/release/igneumd" --out /tmp/x -- build --release -p kaspad --features kaspad/igneum-pow
|
||||
# tools/build-remote.sh --jobs 48 -- check
|
||||
# tools/build-remote.sh --target-dir target-exp -- build --release another persistent target dir on the box
|
||||
# tools/build-remote.sh --no-fetch -- clippy --all-targets nothing comes back (tests, check, clippy)
|
||||
#
|
||||
# Defaults by crate: a fork worktree builds `-p kaspad -p igneum-miner --features kaspad/igneum-pow` in release and fetches
|
||||
# target/release/{igneumd,igneum-miner} (what packaging/README-ship.md and infra/cross expect); app/igneum-app builds release
|
||||
# and fetches igneum-app, igneum-ota-sign, igneum-prove-verify; proving/igneum-prove fetches igneum-prove-host and
|
||||
# igneum-prove-export; any other crate builds release and fetches nothing unless --artefacts names files.
|
||||
#
|
||||
# Artefacts land in <crate>/target-remote/<path without the leading target/> (target-remote/release/igneumd), NEVER in
|
||||
# target/: the box builds x86_64 Linux ELF binaries (glibc 2.39, Ubuntu 24.04), which do not run on this Mac. Each one is
|
||||
# reported with size and sha256. For Windows exes use tools/cross-remote.sh.
|
||||
#
|
||||
# Slots: the box has its own slot files (/srv/builds/_locks/build-<k>, count in /srv/builds/_locks/slots, default 1); this
|
||||
# script takes one of THOSE, never the Mac's ~/.config/igneum/build-slots or tools/lock/with-lock.sh, so a remote build does
|
||||
# not hold a Mac slot. A build waits up to 2 h for a remote slot, as with-lock.sh does.
|
||||
#
|
||||
# Needs: ~/.config/igneum/build-server (build@<ip>, written by infra/build-server/run-from-mac.sh), ~/.ssh/igneum_ed25519,
|
||||
# the same rustc version on both sides (refused otherwise; IGNEUM_TOOLCHAIN_MISMATCH=ok overrides). IGNEUM_AGENT names the
|
||||
# agent in the slot-file label and the box's JSONL log (/srv/builds/_log/builds.jsonl); default the worktree name.
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC2034 # shared with the scripts that source lib.sh
|
||||
BS_TOOL=build-remote
|
||||
# shellcheck source=../infra/build-server/lib.sh
|
||||
. "$HERE/../infra/build-server/lib.sh"
|
||||
|
||||
JOBS="${JOBS:-90}"; OUT=""; ARTEFACTS=""; TARGET_DIR="target"; FETCH=1; CARGO_ARGS=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--jobs) JOBS="$2"; shift 2 ;;
|
||||
--out) OUT="$2"; shift 2 ;;
|
||||
--artefacts) ARTEFACTS="$2"; ARTEFACTS_SET=1; shift 2 ;;
|
||||
--target-dir) TARGET_DIR="$2"; shift 2 ;;
|
||||
--no-fetch) FETCH=0; shift ;;
|
||||
--) shift; CARGO_ARGS=("$@"); break ;;
|
||||
-h|--help) sed -n '2,32p' "$0"; exit 0 ;;
|
||||
*) CARGO_ARGS=("$@"); break ;;
|
||||
esac
|
||||
done
|
||||
[ "${CARGO_ARGS[0]:-}" = cargo ] && CARGO_ARGS=("${CARGO_ARGS[@]:1}")
|
||||
|
||||
bs_host
|
||||
bs_context
|
||||
|
||||
# defaults per crate
|
||||
case "$BS_KIND:$BS_CRATE_REL" in
|
||||
node:*)
|
||||
[ "${#CARGO_ARGS[@]}" -gt 0 ] || CARGO_ARGS=(build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow)
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/release/igneumd $TARGET_DIR/release/igneum-miner" ;;
|
||||
repo:app/igneum-app)
|
||||
[ "${#CARGO_ARGS[@]}" -gt 0 ] || CARGO_ARGS=(build --release)
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/release/igneum-app $TARGET_DIR/release/igneum-ota-sign $TARGET_DIR/release/igneum-prove-verify" ;;
|
||||
repo:proving/igneum-prove)
|
||||
[ "${#CARGO_ARGS[@]}" -gt 0 ] || CARGO_ARGS=(build --release)
|
||||
[ -n "$ARTEFACTS" ] || ARTEFACTS="$TARGET_DIR/release/igneum-prove-host $TARGET_DIR/release/igneum-prove-export" ;;
|
||||
*)
|
||||
[ "${#CARGO_ARGS[@]}" -gt 0 ] || CARGO_ARGS=(build --release) ;;
|
||||
esac
|
||||
case "${CARGO_ARGS[0]}" in build) ;; *) [ -n "${ARTEFACTS_SET:-}" ] || { FETCH=0; ARTEFACTS=""; } ;; esac # test, check, clippy: nothing to fetch
|
||||
[ -n "$OUT" ] || OUT="$BS_CRATE/target-remote"
|
||||
|
||||
bs_log "$BS_KIND crate $BS_WT/$BS_CRATE_REL at $BS_SHA ($BS_BRANCH) -> $BS_HOST:$BS_REMOTE_CRATE; cargo ${CARGO_ARGS[*]} -j $JOBS; target dir $TARGET_DIR"
|
||||
bs_toolchain_check
|
||||
t_sync0=$(date +%s)
|
||||
bs_sync_sources
|
||||
bs_log "sources in place after $(( $(date +%s) - t_sync0 )) s"
|
||||
|
||||
# the fork's kaspa-build-info embeds the commit through a build script that, having once found no branch, emits no
|
||||
# rerun-if-changed and is never run again by cargo (release-0.3.11 plan: `cargo clean -p kaspa-build-info` first); so when
|
||||
# the commit the box builds differs from the last one built in this target dir, that one crate is cleaned (a relink, seconds)
|
||||
pre=""
|
||||
if [ "$BS_KIND" = node ] && [ "${CARGO_ARGS[0]}" = build ]; then
|
||||
pre="[ \"\$(cat '.build-remote-sha-$TARGET_DIR' 2>/dev/null)\" = '$BS_SHA' ] || CARGO_TARGET_DIR='$TARGET_DIR' cargo clean -q --release -p kaspa-build-info 2>/dev/null; "
|
||||
fi
|
||||
cmd="${pre}CARGO_TARGET_DIR='$TARGET_DIR' cargo $(printf '%q ' "${CARGO_ARGS[@]}")-j $JOBS 2>&1 | tee -a '$BS_REMOTE_WT/.build-remote.log'; rc=\${PIPESTATUS[0]}; [ \$rc = 0 ] && echo '$BS_SHA' > '.build-remote-sha-$TARGET_DIR'; ( exit \$rc )" # a subshell exit: the runner reads \$? and still prints its RESULT line
|
||||
label="$BS_WT/$BS_CRATE_REL cargo ${CARGO_ARGS[*]}"
|
||||
BR_KIND=$(bs_kind build-remote "${CARGO_ARGS[0]}"); BR_COMMAND="cargo ${CARGO_ARGS[*]}"; BR_TARGET=x86_64-unknown-linux-gnu
|
||||
for ((i = 0; i < ${#CARGO_ARGS[@]}; i++)); do [ "${CARGO_ARGS[$i]}" = --target ] && BR_TARGET="${CARGO_ARGS[$((i + 1))]:-}"; done
|
||||
BR_ARTEFACTS=""; [ "$FETCH" = 1 ] && BR_ARTEFACTS="$ARTEFACTS"
|
||||
export BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS
|
||||
t0=$(date +%s)
|
||||
set +e
|
||||
bs_remote_run "$BS_REMOTE_CRATE" "$label" "$cmd" 2>&1 | tee "/tmp/build-remote-$$.log"
|
||||
rc=${PIPESTATUS[0]}
|
||||
set -e
|
||||
secs=$(( $(date +%s) - t0 ))
|
||||
result=$(grep -m1 '^build-remote: RESULT' "/tmp/build-remote-$$.log" || true); rm -f "/tmp/build-remote-$$.log"
|
||||
if [ "$rc" != 0 ]; then bs_die "remote cargo failed (rc $rc) after $(bs_fmt_secs "$secs"); $result"; fi
|
||||
bs_log "remote cargo ${CARGO_ARGS[0]} done in $(bs_fmt_secs "$secs") wall from the Mac; ${result#build-remote: RESULT }"
|
||||
|
||||
if [ "$FETCH" = 1 ] && [ -n "$ARTEFACTS" ]; then
|
||||
mkdir -p "$OUT"
|
||||
for a in $ARTEFACTS; do
|
||||
rel="${a#"$TARGET_DIR"/}"; dest="$OUT/$rel"; mkdir -p "$(dirname "$dest")"
|
||||
bs_rsync -p "$BS_HOST:$BS_REMOTE_CRATE/$a" "$dest" || bs_die "no $a on the box after the build"
|
||||
bs_log "artefact $dest: $(bs_size "$dest") bytes, sha256 $(bs_sha256 "$dest"), $(file -b "$dest" | cut -c1-60)"
|
||||
# the commit-string gate (rule of 6 October 2026): a node binary without its commit in its strings fails the run
|
||||
case "$BS_KIND:$(basename "$dest")" in node:igneumd) "$HERE/ci/commit-string-check.sh" "$dest" "$BS_SHA" || bs_die "commit-string gate failed for $a" ;; esac # only kaspad depends on kaspa-build-info
|
||||
done
|
||||
fi
|
||||
124
tools/cross-remote.sh
Executable file
124
tools/cross-remote.sh
Executable file
|
|
@ -0,0 +1,124 @@
|
|||
#!/usr/bin/env bash
|
||||
# The Windows cross-build on igneum-build-1: what proto-cuda/windows-node/cross-build.sh does on the Mac (Homebrew mingw-w64)
|
||||
# and what the PC build job does in WSL2 (app/igneum-app/src/jobbuild.rs: Ubuntu 24.04 mingw-w64 posix threads, static
|
||||
# libgcc), run on the box with sccache and -j 90 (sources synced and re-stamped with touch by lib.sh's bs_overlay_dir, the
|
||||
# copied-sources rule). Run from a fork worktree (igneumd.exe, igneum-miner.exe) or from
|
||||
# app/igneum-app (igneum-app.exe, igneum-ota-sign.exe, igneum-prove-verify.exe).
|
||||
#
|
||||
# tools/cross-remote.sh the default command for this crate
|
||||
# tools/cross-remote.sh --compare target-integration/x86_64-pc-windows-gnu/release sha256 against the Mac's exes
|
||||
# tools/cross-remote.sh -- build --release -p kaspad --features igneum-pow --target x86_64-pc-windows-gnu
|
||||
# tools/cross-remote.sh --jobs 48 --target-dir target-win
|
||||
#
|
||||
# Output: <crate>/target-remote/x86_64-pc-windows-gnu/release/<name>.exe, one line each with size, sha256 and the DLL import
|
||||
# list (x86_64-w64-mingw32-objdump -p on the box, as the Mac script prints it). With --compare <dir>, the Mac's exe of the same
|
||||
# name is hashed too and the line says identical or differs.
|
||||
#
|
||||
# Reproducible (main's decision, 6 October 2026): -Wl,--no-insert-timestamp zeroes the PE header timestamp the mingw linker
|
||||
# writes, so two builds of one tree give one hash (verified 6 Oct: two runs, both exes identical); the Mac's cross-build.sh and
|
||||
# the PC job (jobbuild.rs) carry the same flag.
|
||||
# Byte identity (6 October 2026): the same rustc (1.99.0 both sides, refused otherwise) and the same flags give the same Rust
|
||||
# code, but two things still differ between the Mac's exe and the box's: the C and C++ objects (rocksdb, snappy, zstd, lz4,
|
||||
# secp256k1, mimalloc) come from Homebrew's mingw gcc on the Mac and Ubuntu's gcc 13 here, and source paths embedded by
|
||||
# rustc (panic locations, /Users/joshm/... against /srv/builds/...) differ unless both sides pass --remap-path-prefix, which
|
||||
# the Mac script does not. So: identical bytes build to build ON THE BOX (sccache does not change output), a different hash
|
||||
# from the Mac's exe is expected, and the number that matters is the DLL list (must be none since the fork's database/build.rs
|
||||
# links libstdc++ statically) and that the exe runs on the PC. The same holds for the PC-built exes today.
|
||||
#
|
||||
# Environment of the build (the PC recipe; the Mac's -static-libstdc++ added, harmless since housekeeping made the C++ runtime
|
||||
# static in the fork): CC/CXX/AR_x86_64_pc_windows_gnu = the posix mingw compilers, the linker the same gcc, RUSTFLAGS
|
||||
# -static -static-libgcc -static-libstdc++, LIBCLANG_PATH = Ubuntu's llvm lib dir (bindgen for librocksdb-sys),
|
||||
# BINDGEN_EXTRA_CLANG_ARGS pointed at /usr/x86_64-w64-mingw32, IGNEUM_WINDRES for the app's .rc.
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC2034 # shared with lib.sh
|
||||
BS_TOOL=cross-remote
|
||||
# shellcheck source=../infra/build-server/lib.sh
|
||||
. "$HERE/../infra/build-server/lib.sh"
|
||||
|
||||
TARGET=x86_64-pc-windows-gnu
|
||||
JOBS="${JOBS:-90}"; OUT=""; COMPARE=""; TARGET_DIR="target"; CARGO_ARGS=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--jobs) JOBS="$2"; shift 2 ;;
|
||||
--out) OUT="$2"; shift 2 ;;
|
||||
--compare) COMPARE="$2"; shift 2 ;;
|
||||
--target-dir) TARGET_DIR="$2"; shift 2 ;;
|
||||
--) shift; CARGO_ARGS=("$@"); break ;;
|
||||
-h|--help) sed -n '2,30p' "$0"; exit 0 ;;
|
||||
*) CARGO_ARGS=("$@"); break ;;
|
||||
esac
|
||||
done
|
||||
[ "${CARGO_ARGS[0]:-}" = cargo ] && CARGO_ARGS=("${CARGO_ARGS[@]:1}")
|
||||
|
||||
bs_host
|
||||
bs_context
|
||||
case "$BS_KIND:$BS_CRATE_REL" in
|
||||
node:*)
|
||||
[ "${#CARGO_ARGS[@]}" -gt 0 ] || CARGO_ARGS=(build --release -p kaspad -p igneum-miner --features igneum-pow --target "$TARGET")
|
||||
EXES="igneumd.exe igneum-miner.exe" ;;
|
||||
repo:app/igneum-app)
|
||||
[ "${#CARGO_ARGS[@]}" -gt 0 ] || CARGO_ARGS=(build --release --target "$TARGET")
|
||||
EXES="igneum-app.exe igneum-ota-sign.exe igneum-prove-verify.exe" ;;
|
||||
*) bs_die "cross-remote builds the fork (run from a vendor/igneum-node* worktree) or app/igneum-app, not $BS_CRATE_REL" ;;
|
||||
esac
|
||||
[ -n "$OUT" ] || OUT="$BS_CRATE/target-remote"
|
||||
[ -z "$COMPARE" ] && [ -d "$BS_CRATE/target-integration/$TARGET/release" ] && COMPARE="$BS_CRATE/target-integration/$TARGET/release"
|
||||
|
||||
bs_log "$BS_KIND crate $BS_WT/$BS_CRATE_REL at $BS_SHA ($BS_BRANCH) -> $BS_HOST:$BS_REMOTE_CRATE; cargo ${CARGO_ARGS[*]} -j $JOBS; target dir $TARGET_DIR"
|
||||
bs_toolchain_check
|
||||
t_sync0=$(date +%s)
|
||||
bs_sync_sources
|
||||
bs_log "sources in place after $(( $(date +%s) - t_sync0 )) s"
|
||||
|
||||
# the build environment, as one shell string for the remote runner (every value is a literal; nothing from the Mac's env)
|
||||
env_block='LLVM_LIB=$(ls -d /usr/lib/llvm-*/lib 2>/dev/null | sort -V | tail -1); [ -n "$LLVM_LIB" ] || { echo "no /usr/lib/llvm-*/lib on the box (apt clang libclang-dev)"; exit 2; }
|
||||
export CC_x86_64_pc_windows_gnu=x86_64-w64-mingw32-gcc-posix CXX_x86_64_pc_windows_gnu=x86_64-w64-mingw32-g++-posix AR_x86_64_pc_windows_gnu=x86_64-w64-mingw32-ar
|
||||
export CARGO_TARGET_X86_64_PC_WINDOWS_GNU_LINKER=x86_64-w64-mingw32-gcc-posix
|
||||
export CARGO_TARGET_X86_64_PC_WINDOWS_GNU_RUSTFLAGS="-C link-arg=-static -C link-arg=-static-libgcc -C link-arg=-static-libstdc++ -C link-arg=-Wl,--no-insert-timestamp"
|
||||
export IGNEUM_WINDRES=x86_64-w64-mingw32-windres LIBCLANG_PATH="$LLVM_LIB"
|
||||
export BINDGEN_EXTRA_CLANG_ARGS_x86_64_pc_windows_gnu="--target=x86_64-w64-mingw32 --sysroot=/usr/x86_64-w64-mingw32 -I/usr/x86_64-w64-mingw32/include"
|
||||
echo "cross-remote: $(x86_64-w64-mingw32-gcc-posix --version | head -1); libclang $LLVM_LIB"'
|
||||
pre="" # the same kaspa-build-info clean on a new commit as build-remote.sh (the Windows target has its own fingerprint)
|
||||
[ "$BS_KIND" = node ] && pre="[ \"\$(cat '.cross-remote-sha-$TARGET_DIR' 2>/dev/null)\" = '$BS_SHA' ] || CARGO_TARGET_DIR='$TARGET_DIR' cargo clean -q --release -p kaspa-build-info --target $TARGET 2>/dev/null; "
|
||||
cmd="$env_block
|
||||
${pre}CARGO_TARGET_DIR='$TARGET_DIR' cargo $(printf '%q ' "${CARGO_ARGS[@]}")-j $JOBS 2>&1 | tee -a '$BS_REMOTE_WT/.cross-remote.log'; rc=\${PIPESTATUS[0]}; [ \$rc = 0 ] && echo '$BS_SHA' > '.cross-remote-sha-$TARGET_DIR'
|
||||
for exe in $EXES; do f='$TARGET_DIR/$TARGET/release/'\$exe; [ -f \"\$f\" ] && echo \"cross-remote: DLLS \$exe: \$(x86_64-w64-mingw32-objdump -p \"\$f\" | awk '/DLL Name/ { print \$3 }' | sort -u | tr '\n' ' ')\"; done
|
||||
( exit \$rc )" # a subshell exit: the runner reads \$? and still prints its RESULT line
|
||||
label="$BS_WT/$BS_CRATE_REL cross $TARGET"
|
||||
BR_KIND=$(bs_kind cross-remote "${CARGO_ARGS[0]}"); BR_COMMAND="cargo ${CARGO_ARGS[*]}"; BR_TARGET="$TARGET"
|
||||
BR_ARTEFACTS=""; for exe in $EXES; do BR_ARTEFACTS="$BR_ARTEFACTS $TARGET_DIR/$TARGET/release/$exe"; done
|
||||
export BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS
|
||||
t0=$(date +%s)
|
||||
set +e
|
||||
bs_remote_run "$BS_REMOTE_CRATE" "$label" "$cmd" 2>&1 | tee "/tmp/cross-remote-$$.log"
|
||||
rc=${PIPESTATUS[0]}
|
||||
set -e
|
||||
secs=$(( $(date +%s) - t0 ))
|
||||
result=$(grep -m1 '^build-remote: RESULT' "/tmp/cross-remote-$$.log" || true)
|
||||
dlls=$(grep '^cross-remote: DLLS' "/tmp/cross-remote-$$.log" || true); rm -f "/tmp/cross-remote-$$.log"
|
||||
if [ "$rc" != 0 ]; then bs_die "remote cross-build failed (rc $rc) after $(bs_fmt_secs "$secs"); $result"; fi
|
||||
bs_log "remote cross-build done in $(bs_fmt_secs "$secs") wall from the Mac; ${result#build-remote: RESULT }"
|
||||
|
||||
mkdir -p "$OUT/$TARGET/release"
|
||||
for exe in $EXES; do
|
||||
dest="$OUT/$TARGET/release/$exe"
|
||||
bs_rsync -p "$BS_HOST:$BS_REMOTE_CRATE/$TARGET_DIR/$TARGET/release/$exe" "$dest" || bs_die "no $exe on the box after the build"
|
||||
sum=$(bs_sha256 "$dest"); line="$exe: $(bs_size "$dest") bytes, sha256 $sum"
|
||||
d=$(printf '%s\n' "$dlls" | grep "DLLS $exe:" | sed 's/.*DLLS [^:]*: *//'); line="$line, DLLs: ${d:-none}"
|
||||
if [ -n "$COMPARE" ] && [ -f "$COMPARE/$exe" ]; then
|
||||
msum=$(bs_sha256 "$COMPARE/$exe")
|
||||
if [ "$msum" = "$sum" ]; then line="$line; IDENTICAL to $COMPARE/$exe"; else line="$line; differs from $COMPARE/$exe ($(bs_size "$COMPARE/$exe") bytes, sha256 ${msum:0:16}...; expected, see the header)"; fi
|
||||
fi
|
||||
bs_log "$line"
|
||||
# the commit-string gate (rule of 6 October 2026): a node exe without its commit in its strings fails the run
|
||||
case "$BS_KIND:$exe" in node:igneumd.exe) "$HERE/ci/commit-string-check.sh" "$dest" "$BS_SHA" || bs_die "commit-string gate failed for $exe" ;; esac # only kaspad depends on kaspa-build-info
|
||||
done
|
||||
# an exe that imports libstdc++-6.dll (a fork before the housekeeping commit that made the C++ runtime static) needs the
|
||||
# three runtime DLLs OF THIS TOOLCHAIN beside it (the PC job does the same; push-inputs.sh takes them from next to the exes)
|
||||
if printf '%s\n' "$dlls" | grep -q 'libstdc++-6.dll'; then
|
||||
bs_ssh 'd=$(dirname "$(x86_64-w64-mingw32-gcc-posix -print-file-name=libstdc++-6.dll)"); mkdir -p /tmp/igneum-mingw-dlls; cp "$d/libstdc++-6.dll" "$d/libgcc_s_seh-1.dll" /usr/x86_64-w64-mingw32/lib/libwinpthread-1.dll /tmp/igneum-mingw-dlls/ && ls /tmp/igneum-mingw-dlls' >/dev/null \
|
||||
&& bs_rsync -p "$BS_HOST:/tmp/igneum-mingw-dlls/*.dll" "$OUT/$TARGET/release/" || bs_die "could not fetch the mingw runtime DLLs"
|
||||
for dll in libstdc++-6.dll libgcc_s_seh-1.dll libwinpthread-1.dll; do bs_log "runtime $dll: $(bs_size "$OUT/$TARGET/release/$dll") bytes, sha256 $(bs_sha256 "$OUT/$TARGET/release/$dll") (GCC 13 posix, beside the exes)"; done
|
||||
fi
|
||||
bs_log "exes in $OUT/$TARGET/release"
|
||||
51
tools/fleet/bps-collect.py
Normal file
51
tools/fleet/bps-collect.py
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
#!/usr/bin/env python3
|
||||
"""The block-rate run's collector (docs/analysis/block-rate-devnet2.md). Every 60 s, from the Devnet 2 seed and every
|
||||
Devnet 2 miner box (the library's chain-side readers): height, daa, blue score, tips, peers, exec tip; the seed's
|
||||
counts of accepted blocks and red blocks, reorg depths, finality lock lines (delay from determined to LOCKED, voters
|
||||
and weight), p2p bytes (the node's /proc/<pid>/net or ss), CPU and RSS of the node. One JSON line per minute to
|
||||
~/Desktop/fleet/bps/<run>.jsonl; `--report <run>` prints the run's table and the payout intervals per tier.
|
||||
"""
|
||||
import sys, os, json, time, datetime, re
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))); from lib import Box, Registry
|
||||
ROOT = os.path.expanduser("~/Desktop/fleet/bps"); os.makedirs(ROOT, exist_ok=True)
|
||||
def now(): return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
SEED_CMD = r"""L=/root/fleet/dn2-node.log; P=$(pgrep -f '^/root/fleet/in/igneumd' | head -1)
|
||||
echo accepted=$(grep -c 'PoW accepted' $L) red=$(grep -ciE 'red block|colored red|is red' $L) reorg_max=$(grep -oE 'selected-chain reorg: [0-9]+ chain blocks' $L | grep -oE '[0-9]+ chain' | awk '{if ($1>m) m=$1} END {print m+0}') locks=$(grep -c 'LOCKED' $L)
|
||||
grep -E 'Finality: (checkpoint [0-9]+ determined|checkpoint [0-9]+ LOCKED|certificate at index [0-9]+ received)' $L | tail -40 | sed -E 's/^([0-9-]+ [0-9:.]+)[^]]*\] //' | cut -c1-170
|
||||
echo cpu_rss=$(ps -o %cpu=,rss= -p $P 2>/dev/null | tr -s ' ') rx_tx=$(cat /proc/$P/net/dev 2>/dev/null | awk 'NR>2 && $1!="lo:" {rx+=$2; tx+=$10} END {print rx, tx}')
|
||||
/opt/igneum/pkg/bin/igneum-miner watch 1 grpc://127.0.0.1:26610 2>/dev/null | grep -o 'blocks=[0-9]*.*synced=[a-z]*' | sed -E 's/difficulty=[0-9.]* sink=[0-9a-f]* //' | tail -1
|
||||
curl -s -m 6 -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","id":1,"method":"igneum_getExecStatus","params":[]}' http://127.0.0.1:26790/ | python3 -c 'import sys,json; r=sys.stdin.read(); d=json.loads(r).get("result",{}) if r.strip() else {}; print("exec", int(d.get("executedTip","0x0"),16))' 2>/dev/null"""
|
||||
def sample(run):
|
||||
reg = Registry.load()
|
||||
seed = next((Box(b["ssh_host"], b["ssh_port"], b["label"], i, b.get("wallet"), b.get("provider")) for i, b in reg.items() if b.get("dn2_seed") and b.get("state") != "destroyed"), None)
|
||||
row = {"t": now(), "run": run}
|
||||
if seed:
|
||||
rc, out, err = seed.run(SEED_CMD, 60); row["seed_raw"] = out[-4000:]
|
||||
for l in out.split("\n"):
|
||||
if l.startswith("accepted="): row.update(dict(kv.split("=") for kv in l.split()))
|
||||
elif l.startswith("blocks="): row.update({"w_" + k: v for k, v in (kv.split("=") for kv in l.split() if "=" in kv)})
|
||||
elif l.startswith("exec "): row["exec_tip"] = int(l.split()[1])
|
||||
elif l.startswith("cpu_rss="): row["cpu_rss"] = l
|
||||
fin = [l for l in out.split("\n") if "Finality:" in l]; row["finality_tail"] = fin[-12:]
|
||||
miners = [b for b in reg.values() if b.get("state") != "destroyed" and b.get("ssh_host") and (b.get("devnet2") or b.get("stage") == "bps")]
|
||||
row["miner_boxes"] = len(miners)
|
||||
with open(os.path.join(ROOT, f"{run}.jsonl"), "a") as f: f.write(json.dumps(row) + "\n")
|
||||
print(row["t"], run, {k: row.get(k) for k in ("accepted", "red", "reorg_max", "locks", "w_blocks", "w_daa", "w_blue", "w_tips", "w_peers", "exec_tip", "miner_boxes")}, flush=True)
|
||||
def report(run):
|
||||
rows = [json.loads(l) for l in open(os.path.join(ROOT, f"{run}.jsonl"))]
|
||||
if len(rows) < 2: print("too few rows"); return
|
||||
a, z = rows[0], rows[-1]
|
||||
secs = (datetime.datetime.strptime(z["t"], "%Y-%m-%dT%H:%M:%SZ") - datetime.datetime.strptime(a["t"], "%Y-%m-%dT%H:%M:%SZ")).total_seconds()
|
||||
blocks = int(z.get("w_blocks", 0)) - int(a.get("w_blocks", 0)); blue = int(z.get("w_blue", 0)) - int(a.get("w_blue", 0))
|
||||
print(f"run {run}: {secs/60:.1f} min, blocks {blocks} ({blocks/max(secs,1):.2f}/s), blue score +{blue} ({blue/max(secs,1):.2f}/s), red share {100*(1-blue/max(blocks,1)):.1f}%, max reorg {max(int(r.get('reorg_max',0)) for r in rows)}, exec lag at end {int(z.get('w_blocks',0)) - int(z.get('exec_tip',0))} blocks, locks {int(z.get('locks',0)) - int(a.get('locks',0))}")
|
||||
# payout interval per tier at the measured block rate: blocks/s x (miner hash / network hash)
|
||||
bps = blocks / max(secs, 1)
|
||||
for net_name, net in (("tonight (2.0 GH/s)", 2.0e9), ("1 TH/s", 1e12), ("10 TH/s", 1e13), ("100 TH/s", 1e14)):
|
||||
for card, mhs in (("4070 (28 MH/s)", 28e6), ("5090 (128 MH/s)", 128e6), ("8x 4090 rig (459 MH/s)", 459e6)):
|
||||
per_s = bps * mhs / net; iv = 1 / per_s if per_s else float("inf")
|
||||
print(f" {net_name:<20} {card:<24} one block every {iv/60:,.1f} min ({iv/3600:,.2f} h)")
|
||||
if __name__ == "__main__":
|
||||
if sys.argv[1] == "--report": report(sys.argv[2])
|
||||
else:
|
||||
run = sys.argv[1]
|
||||
while True: sample(run); time.sleep(60)
|
||||
Loading…
Reference in a new issue