Build server: remote-run.sh with the JSONL build log, benchmark plan docs/plans/build-server.md, commit hash in box builds
remote-run.sh runs on the box behind BR_* exports: slot, sccache stats, the RESULT line and one JSON line per run in /srv/builds/_log/builds.jsonl (v 1, id, host, tool, worktree, crate, kind, command, target, branch, sha, label, agent, slot, wait_s, queued_at, start, end, secs, exit, compiles, sccache, load_end, artefacts; written on success, failure and the 2 h slot give-up), the label ending in '; agent=<name>' (IGNEUM_AGENT, default the worktree). The remote checkout is a branch and build-remote.sh cleans kaspa-build-info (release profile) on a new commit, so the box's igneumd carries its commit hash (no Mac worktree build does: .git is a file there). cross-remote.sh fetches the GCC 13 runtime DLLs beside an exe that imports libstdc++-6.dll. The plan holds the three benchmarks: clean node build 1 min 27 s (Mac 12 to 18 min), incremental 7 s (Mac 2 to 15 min), Windows cross 1 min 44 s (Mac 4 min 49 s to 12 min 28 s), their consequences and the proposed rules. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs
parent
a1ecb37bef
commit
0b3ca31d87
5 changed files with 264 additions and 41 deletions
94
docs/plans/build-server.md
Normal file
94
docs/plans/build-server.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# Build server: igneum-build-1 (plan, benchmarks, rules)
|
||||
|
||||
6 October 2026. the project lead ordered a Hetzner dedicated server at 18:2x UK: AX162-1-LTD, EPYC 9454P (48 cores, 96 threads),
|
||||
128 GB, 2x 3.84 TB NVMe, Falkenstein (FSN1-DC24, Hetzner #3088308), 188.40.146.49, IPv6 2a01:4f8:2240:205a::/64.
|
||||
Purpose: this Mac is the compile queue for ten agents (8-minute rebuilds under contention) and the Windows cross-builds;
|
||||
the box takes over Rust builds, cross-builds and a shared compile cache, and later a CI runner and the Devnet 2 seed.
|
||||
|
||||
## 1. What is in place (all on branch `build-server`)
|
||||
|
||||
| Piece | File | State 6 Oct 2026 |
|
||||
|---|---|---|
|
||||
| Install + provision | `infra/build-server/provision.sh` | ran: installimage 17:16 to 17:20 UTC (Ubuntu 24.04, RAID 1, no swap), provision 17:24 to 17:27 UTC, second run 2 s with zero changes (idempotent) |
|
||||
| Mac side | `infra/build-server/run-from-mac.sh <ip>` | ran: `~/.config/igneum/build-server` = `build@188.40.146.49`, `build` remotes added, 124 igneum branches and 48 fork branches pushed to the bare mirrors |
|
||||
| Shared library | `infra/build-server/lib.sh` | host line, ssh options, mirror push, remote checkout, overlay, remote slot runner |
|
||||
| Remote runner | `infra/build-server/remote-run.sh` | runs on the box: slot, sccache, RESULT line, one JSON line per run in `/srv/builds/_log/builds.jsonl` (the worker dashboard's feed, fields as main asked on 6 Oct) |
|
||||
| Remote cargo | `tools/build-remote.sh` | any crate dir, any worktree: `tools/build-remote.sh [-- cargo args]`; `IGNEUM_AGENT=<name>` tags the slot label and the log |
|
||||
| Remote Windows cross | `tools/cross-remote.sh` | fork worktree or app/igneum-app: `tools/cross-remote.sh [--compare <dir of the Mac's exes>]` |
|
||||
|
||||
ssh line: `ssh -i ~/.ssh/igneum_ed25519 build@188.40.146.49` (root works with the same key; passwords are off).
|
||||
|
||||
Box facts (provision summary): rustc 1.99.0 (the Mac's version; NO rust-toolchain file exists in the repo or the fork, the
|
||||
pin is `RUST_TOOLCHAIN` in provision.sh), targets x86_64-unknown-linux-gnu and x86_64-pc-windows-gnu, sccache 0.18.0 with a
|
||||
100 GB disk cache at /srv/sccache, mingw-w64 GCC 13 posix threads (the PC job's recipe), clang and lld 18.1.3, Node v22.23.3,
|
||||
git 2.43.0, tmux 3.4, ufw 22 + 26611 + 26811 (the devnet and testnet seed p2p ports; RPC stays on loopback as on every seed),
|
||||
md0 /boot and md1 / as RAID 1, 3.3 TB free, swap none, 1 build slot in `/srv/builds/_locks/slots`.
|
||||
|
||||
## 2. How a build travels
|
||||
|
||||
1. `bs_context` (lib.sh) reads the crate: a fork worktree under `vendor/` (kind node, mirror `/srv/igneum-node.git`) or a crate
|
||||
of the igneum repo (kind repo, mirror `/srv/igneum.git`). `cargo metadata` lists every path dependency.
|
||||
2. HEAD is pushed to the mirror; the box clones or fetches it at `/srv/builds/<worktree>/<same relative path>` and checks out
|
||||
that commit. A real `.git` is needed: the fork's `kaspa-build-info` runs `git rev-parse HEAD` at build time and every release
|
||||
plan checks the commit in the binary's strings.
|
||||
3. Uncommitted changes go by `rsync --checksum` without `-t` (files that changed are written with the box's clock) and the
|
||||
written files are re-stamped with `touch` (the copied-sources rule, `tools/ci/copied-sources-check.sh` passes).
|
||||
4. The cargo command runs in the crate dir under one of the box's slot files (`flock` on `/srv/builds/_locks/build-<k>`,
|
||||
never the Mac's `~/.config/igneum/build-slots`), with sccache and `-j 90`, logging wall time, compile count and cache hits.
|
||||
5. Artefacts come back into `<crate>/target-remote/...` with size and sha256. NEVER into `target/`: the box's native
|
||||
binaries are x86_64 Linux (glibc 2.39) and do not run on the Mac.
|
||||
|
||||
`/srv/builds/<worktree>` mirrors the Mac's igneum worktree ROOT because the fork's path dependency is
|
||||
`../../../../igneum-pow` (vendor/igneum-node/consensus/pow/Cargo.toml).
|
||||
|
||||
## 3. Benchmarks
|
||||
|
||||
The Mac numbers are from the logs (docs/bench-log.md, docs/plans/release-0.3.10.md, release-0.3.11.md); no fresh Mac run
|
||||
was made, because a Mac build would take a slot from the agents and the logs already hold several readings per case.
|
||||
Mac = Apple M5 Max (18 cores), builds at `nice -n 19` with 4 to 6 jobs under the build lock, usually loaded by other agents.
|
||||
Box = igneum-build-1, `-j 90`, nothing else running.
|
||||
|
||||
| Case | Mac (logged) | Box | Box run |
|
||||
|---|---|---|---|
|
||||
| Clean node build (`cargo build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow`, every crate) | 12 min 36 s (0.3.10, new target dir, -j 4); 17 min 53 s at load 140 and 8 min 58 s second time (fud ledger, -j 4); rusty-kaspa kaspad alone 2 min 36 s on an idle Mac | **1 min 27 s** cargo wall (1 min 34 s end to end from the Mac: push and sync 1 s, build, fetch of both binaries) | cold: 518 crates downloaded inside that time, sccache 0 hits / 993 misses, 563 crates compiled; igneumd 48,220,896 B, igneum-miner 9,107,232 B, ELF x86-64 PIE; `igneumd --version` runs on the box; 17:30:58 to 17:32:32 UTC |
|
||||
| Incremental rebuild (one file changed, same target dir) | 2 min 08 s (0.3.10, 17:49Z); 3 min 19 s (0.3.11); 5 min 06 s (txgossip); 15 min 18 s at load 110 to 134 (M31); 35 s to 4 min (execution layer); "8 min under contention" (main, 6 Oct) | **7 s** cargo wall (10 s end to end: sync 1 s, build 6.97 s, fetch) | one line appended to kaspad/src/main.rs in the fork worktree, uncommitted, carried by the overlay (1 file written and re-stamped); 1 crate compiled, igneumd relinked; box idle (load 12 from the clean build a minute earlier); 17:33:11 to 17:33:21 UTC |
|
||||
| Windows cross-build (`--target x86_64-pc-windows-gnu`, igneumd.exe + igneum-miner.exe) | 8 min 25 s clean (-j 6, 3 Oct); 4 min 49 s and 4 min 53 s with warm dependencies (4 Oct); 12 min 28 s (finality-fixes, 4 Oct) | **1 min 44 s** cargo wall (1 min 48 s end to end) | cold: 995 compiles, sccache 0 hits; igneumd.exe 49,434,624 B, igneum-miner.exe 10,201,600 B; mingw GCC 13 posix, libclang 18; igneumd.exe imports libstdc++-6.dll (this fork head predates the housekeeping commit that made the C++ runtime static), so cross-remote.sh now fetches the three GCC 13 runtime DLLs beside the exes as the PC job does; 17:33:58 to 17:35:46 UTC. Second run on the same target dir, no source change: 8 s |
|
||||
|
||||
Also logged for context: the Mac's Linux cross-build with zig (`infra/cross/build-linux.sh`) took 30 min 56 s cold and
|
||||
3 min 20 s incremental; PC 1 built Linux + Windows node and app and ran both test suites in 7 min 38 s cold, 5 min 09 s warm
|
||||
(CLAUDE.md, 5 Oct).
|
||||
|
||||
### What the numbers mean and what follows
|
||||
|
||||
| Number | Means | Done or proposed |
|
||||
|---|---|---|
|
||||
| Clean build 1 min 27 s against 12 to 18 min on the loaded Mac (8 to 12x) | a new worktree costs an agent a minute and a half, not a slot for a quarter of an hour; the first build of every one of the 123 worktree dirs on the box is this cold case, later ones are the 7 s case | R1 proposed; sccache was cold (0 hits of 1,982 requests) because every run so far was the first of its kind; the cache fills as agents build the same crate versions from different worktrees, so the second worktree's clean build will be mostly hits (measure when it happens, write the number here) |
|
||||
| Incremental 7 s against 2 to 15 min on the Mac (the "8 min under contention") | an edit-build loop of seconds for Linux targets; end to end 10 s because sync is 1 s and the two binaries (57 MB) come back in 2 s | R1; the slot count stays 1 until two agents collide, then 2 with `-j 48` each (`SLOTS=2 run-from-mac.sh` and `--jobs 48`) |
|
||||
| Windows cross 1 min 44 s against 4 min 49 s to 12 min 28 s on the Mac (3 to 7x), 8 s incremental | a Windows exe per commit is cheap enough to build on every push; the PC `build` job (7 min 38 s cold, 5 min 09 s warm for Linux + Windows + tests) stays the second source | R2 proposed |
|
||||
| Mac arm64 binaries: not built here | agents who run nodes on the Mac (local devnets, the DMG) still take Mac slots; the box cannot remove that contention | R3; the real relief is to move test networks to the fleet or to a Devnet 2 seed on this box (its unit, ports and ufw are ready) |
|
||||
| The commit hash is EMPTY in every Mac worktree build and was empty in the first box builds | `kaspa-build-info` (build-info/build.rs) needs `.git` to be a directory AND HEAD to be a symbolic ref to a loose branch file; a worktree's `.git` is a file and a detached HEAD is not a ref; and once it has found nothing it emits no `rerun-if-changed`, so cargo never runs it again in that target dir (release-0.3.11: `cargo clean -p kaspa-build-info`) | fixed on the box: the remote checkout is `git checkout -B <branch> <sha>` and build-remote.sh runs `cargo clean --release -p kaspa-build-info` whenever the commit differs from the last one built in that target dir (`.build-remote-sha-<target dir>`); verified 17:40 UTC: 2 string hits for 3bfe346f, binary +1,024 B. The release plans' "commit in its strings" checks were passing against builds from the fork's MAIN checkout (a real .git directory on a branch), not from worktrees. Proposed: the PC build job and the Mac's release recipe adopt the same two steps, or the fork's build.rs learns to read a worktree's gitdir file |
|
||||
| igneumd.exe hash changes build to build with no source change (c424aae0 then bfbff015) while the Linux igneumd stays byte-identical across three builds | the mingw linker writes a timestamp into the PE header; the Linux ELF has none | not changed (the Mac and PC exes have the same property); `-C link-arg=-Wl,--no-insert-timestamp` would make the exe reproducible and is a one-line change to cross-remote.sh, the Mac script and jobbuild.rs together if main wants reproducible Windows builds |
|
||||
| sccache misses 1,982 of 1,982 | expected for first builds; the cache is 494 MB after three builds, capped at 100 GB | nothing; the hit rate is in every RESULT line and every JSONL line, read it after the first repeat build |
|
||||
| Disk 3.3 TB free, RAM 125 GB, load peaked at 24 during the Windows build with 96 threads | room for 2 slots and the Devnet 2 seed without contention | nothing now |
|
||||
|
||||
## 4. Rules (proposed for CLAUDE.md once the box passes; main decides)
|
||||
|
||||
| Rule | Text |
|
||||
|---|---|
|
||||
| R1 | Every agent's `cargo build`, `cargo test`, `cargo check` and `cargo clippy` for Linux goes through `tools/build-remote.sh` from the crate directory. The box takes one remote slot per build; nobody runs cargo over ssh by hand. |
|
||||
| R2 | Windows exes come from `tools/cross-remote.sh` (fork worktree: igneumd.exe, igneum-miner.exe; app/igneum-app: igneum-app.exe and the two tools). The PC `build` job stays as the second source until two releases have shipped from the box. |
|
||||
| R3 | The Mac keeps what only it can do: aarch64-apple-darwin binaries (the DMG, nodes that agents run locally), tests that need Metal (proto-metal, the Metal worker), and measurements. Those still use `tools/lock/with-lock.sh` and the Mac's build slots. |
|
||||
| R4 | A worktree builds once on the box per commit plus overlay; the next build is incremental in `/srv/builds/<worktree>/.../target`. Nobody deletes another worktree's target dir on the box. |
|
||||
| R5 | `RUST_TOOLCHAIN` in provision.sh is bumped in the same commit as the Mac's `rustup update`; build-remote.sh refuses a mismatch. Add a `rust-toolchain.toml` to the fork and the repo (none exists today) so both sides pin from one file. |
|
||||
| R6 | The box is never a node host for the live devnet and never holds a secret (no `~/.config/igneum` there). The Devnet 2 seed on it runs under its own unit with `--devnet --devnet-suffix=<n>` on 26611 (ufw already open) when that work starts. |
|
||||
| R7 | CLAUDE.md line "nothing is built on a server" and "Windows builds go to the GitHub runner, Linux binaries come from infra/cross/build-linux.sh" are rewritten when R1 and R2 are adopted; until then the box is the measured option, not the rule. |
|
||||
|
||||
## 5. What the box does not do yet
|
||||
|
||||
| Gap | Why it matters | Next step |
|
||||
|---|---|---|
|
||||
| No zig / cargo-zigbuild | the devnet seed (Debian 12, glibc 2.36) takes the Mac's zig build; a native box build links glibc 2.39, which Debian 13 seeds accept and HiveOS (Ubuntu 18/20 base) does not | install zig 0.17 + cargo-zigbuild in provision.sh, add `--target x86_64-unknown-linux-gnu.2.36` mode to build-remote.sh |
|
||||
| No macOS target | agents who run nodes on the Mac still build there | out of scope (needs the macOS SDK on Linux); the fleet or the box's own Devnet 2 seed takes the test-network runs instead |
|
||||
| No CI runner | GitHub `ci.yml` and `windows.yml` run on GitHub's machines | install a self-hosted runner as user build once R1 is in |
|
||||
| Byte identity with the Mac's exes | different C/C++ toolchain (Homebrew mingw vs Ubuntu GCC 13) and embedded source paths | not a goal; the box is identical with itself build to build, cross-remote.sh reports sha256 and the DLL list per exe |
|
||||
| Robot API | `~/.config/igneum/hetzner-token` is the Cloud token (hcloud); the dedicated box lives in Robot, a separate credential | main sets the Robot server name in the UI; a webservice user goes to `~/.config/igneum/robot-credentials` when needed |
|
||||
|
|
@ -102,7 +102,10 @@ print("\n".join(out))' "$BS_WT_ROOT" "$BS_TOP" "$BS_KIND") || bs_die "cargo meta
|
|||
}
|
||||
|
||||
# push the crate repo's HEAD to its bare mirror on the box (fast after the first time), then check the remote tree out at that
|
||||
# commit: a real .git for kaspa-build-info, and the mirror doubles as the CI runner's source later
|
||||
# commit ON A BRANCH of that name: kaspa-build-info (build-info/build.rs try_git_head) embeds the commit only when .git is a
|
||||
# directory AND HEAD is a symbolic ref to a loose branch file; a detached HEAD or a worktree's .git file gives an empty hash
|
||||
# (which is why the Mac's worktree builds print "igneumd 2.1.0" with no commit, 6 Oct 2026). The mirror doubles as the CI
|
||||
# runner's source later.
|
||||
bs_push_and_checkout() {
|
||||
local url="$BS_HOST:$BS_MIRROR" remote_top="$BS_REMOTE_WT/$BS_TOP_REL"
|
||||
[ "$BS_TOP_REL" = . ] && remote_top="$BS_REMOTE_WT"
|
||||
|
|
@ -110,7 +113,7 @@ bs_push_and_checkout() {
|
|||
GIT_SSH_COMMAND="$BS_SSH_CMD" git -C "$BS_TOP" push -q --force "$url" "HEAD:refs/heads/$BS_BRANCH" || bs_die "push to the mirror failed"
|
||||
bs_ssh "set -e; mkdir -p '$BS_REMOTE_WT'
|
||||
if [ ! -d '$remote_top/.git' ]; then rm -rf '$remote_top'; git clone -q --no-checkout '$BS_MIRROR' '$remote_top'; fi
|
||||
cd '$remote_top'; git fetch -q origin '+refs/heads/*:refs/remotes/origin/*'; git checkout -q --detach '$BS_SHA'; git reset -q --hard '$BS_SHA'
|
||||
cd '$remote_top'; git fetch -q origin '+refs/heads/*:refs/remotes/origin/*'; git checkout -q -B '$BS_BRANCH' '$BS_SHA'; git reset -q --hard '$BS_SHA'
|
||||
git status --porcelain | head -3" || bs_die "remote checkout at $remote_top failed"
|
||||
BS_REMOTE_TOP="$remote_top"
|
||||
}
|
||||
|
|
@ -148,39 +151,31 @@ bs_sha256() { shasum -a 256 "$1" | awk '{ print $1 }'; }
|
|||
bs_size() { stat -f %z "$1" 2>/dev/null || stat -c %s "$1"; }
|
||||
bs_fmt_secs() { local s=$1; printf '%d min %02d s' $((s / 60)) $((s % 60)); }
|
||||
|
||||
# the remote runner: takes a build slot (flock on /srv/builds/_locks/build-<k>, the box's own slot files, never the Mac's
|
||||
# ~/.config/igneum/build-slots), runs the command in the crate dir with the box's profile, reports time and sccache stats.
|
||||
# $1 remote crate dir, $2 holder label, $3... the command (already a shell string: `cargo build ...`)
|
||||
bs_remote_run() {
|
||||
local dir="$1" label="$2" cmd="$3"
|
||||
bs_ssh "bash -s" <<EOF
|
||||
set -uo pipefail
|
||||
. /etc/profile.d/igneum-build.sh
|
||||
slots=\$(cat "\$IGNEUM_BUILD_SLOTS_DIR/slots" 2>/dev/null || echo 1); [ "\$slots" -ge 1 ] 2>/dev/null || slots=1
|
||||
got=""; t0=\$(date +%s)
|
||||
for k in \$(seq 0 \$((slots - 1))); do
|
||||
exec {fd}>"\$IGNEUM_BUILD_SLOTS_DIR/build-\$k"
|
||||
if flock -n "\$fd"; then got=\$k; break; fi
|
||||
exec {fd}>&-
|
||||
done
|
||||
if [ -z "\$got" ]; then
|
||||
echo "build-remote: all \$slots slot(s) busy, waiting (up to 2 h) for build-0: \$(head -c 160 "\$IGNEUM_BUILD_SLOTS_DIR/build-0" 2>/dev/null)" >&2
|
||||
exec {fd}>"\$IGNEUM_BUILD_SLOTS_DIR/build-0"; flock -w 7200 "\$fd" || { echo "build-remote: gave up waiting for a slot after 2 h" >&2; exit 75; }; got=0
|
||||
fi
|
||||
printf 'pid %s since %sZ waited %s s: %s\n' "\$\$" "\$(date -u +%H:%M:%S)" "\$(( \$(date +%s) - t0 ))" "$label" > "\$IGNEUM_BUILD_SLOTS_DIR/build-\$got"
|
||||
echo "build-remote: holding build-\$got on \$(hostname) (waited \$(( \$(date +%s) - t0 )) s)" >&2
|
||||
cd '$dir' || exit 2
|
||||
sccache --start-server >/dev/null 2>&1 || true
|
||||
before=\$(sccache --show-stats 2>/dev/null | awk '/^Compile requests executed/ { print \$4 }')
|
||||
t1=\$(date +%s)
|
||||
$cmd
|
||||
rc=\$?
|
||||
secs=\$(( \$(date +%s) - t1 ))
|
||||
after=\$(sccache --show-stats 2>/dev/null | awk '/^Compile requests executed/ { print \$4 }')
|
||||
hits=\$(sccache --show-stats 2>/dev/null | awk '/^Cache hits / { print \$3 }')
|
||||
misses=\$(sccache --show-stats 2>/dev/null | awk '/^Cache misses / { print \$3 }')
|
||||
printf 'build-remote: RESULT rc=%s secs=%s compiles=%s sccache_hits_total=%s sccache_misses_total=%s load=%s\n' "\$rc" "\$secs" "\$(( \${after:-0} - \${before:-0} ))" "\${hits:-?}" "\${misses:-?}" "\$(cut -d' ' -f1-3 /proc/loadavg)"
|
||||
: > "\$IGNEUM_BUILD_SLOTS_DIR/build-\$got"
|
||||
exit \$rc
|
||||
EOF
|
||||
# kind of a run for the box's JSONL log (main's rule of 6 October 2026): <tool> <first cargo word>
|
||||
bs_kind() {
|
||||
local tool="$1" word="$2"
|
||||
case "$word" in test) echo suite; return ;; check|clippy) echo check; return ;; esac
|
||||
if [ "$tool" = cross-remote ]; then
|
||||
case "$BS_KIND:$BS_CRATE_REL" in node:*) echo node-windows ;; repo:app/igneum-app) echo app-windows ;; *) echo other ;; esac; return
|
||||
fi
|
||||
case "$BS_KIND:$BS_CRATE_REL" in node:*) echo node-linux ;; repo:app/igneum-app) echo app ;; repo:proving/igneum-prove) echo prove ;; *) echo other ;; esac
|
||||
}
|
||||
|
||||
# the remote runner (infra/build-server/remote-run.sh, piped to `bash -s` on the box behind the BR_* exports): takes one of the
|
||||
# box's build slots (never the Mac's), runs the command in the crate dir with sccache, prints the RESULT line and appends one
|
||||
# JSON line to /srv/builds/_log/builds.jsonl for the worker dashboard.
|
||||
# bs_remote_run <remote crate dir> <label> <shell command string>
|
||||
# with BR_KIND, BR_COMMAND, BR_TARGET and BR_ARTEFACTS set by the caller; the agent name is IGNEUM_AGENT (default the worktree)
|
||||
# and is appended to the label as "; agent=<name>".
|
||||
bs_remote_run() {
|
||||
local dir="$1" label="$2" cmd="$3" agent="${IGNEUM_AGENT:-$BS_WT}" v
|
||||
label="$label; agent=$agent"
|
||||
BR_DIR="$dir" BR_LABEL="$label" BR_CMD="$cmd" BR_TOOL="${BS_TOOL:-build-remote}" BR_WT="$BS_WT" BR_CRATE="$BS_CRATE_REL" \
|
||||
BR_BRANCH="$BS_BRANCH" BR_SHA="$BS_SHA" BR_AGENT="$agent" BR_KIND="${BR_KIND:-other}" BR_COMMAND="${BR_COMMAND:-}" \
|
||||
BR_TARGET="${BR_TARGET:-}" BR_ARTEFACTS="${BR_ARTEFACTS:-}" \
|
||||
bash -c '
|
||||
for v in BR_DIR BR_LABEL BR_CMD BR_TOOL BR_WT BR_CRATE BR_BRANCH BR_SHA BR_AGENT BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS; do
|
||||
printf "export %s=%q\n" "$v" "${!v}"
|
||||
done
|
||||
cat "$0"' "$(dirname "${BASH_SOURCE[0]}")/remote-run.sh" | bs_ssh 'bash -s'
|
||||
}
|
||||
|
|
|
|||
110
infra/build-server/remote-run.sh
Executable file
110
infra/build-server/remote-run.sh
Executable file
|
|
@ -0,0 +1,110 @@
|
|||
#!/usr/bin/env bash
|
||||
# The remote half of tools/build-remote.sh and tools/cross-remote.sh. It runs ON igneum-build-1, fed by lib.sh bs_remote_run
|
||||
# over `ssh build@<box> bash -s` with these exports prepended (never run it by hand on the Mac):
|
||||
# BR_DIR the crate directory on the box BR_CMD the shell string to run there (cargo ...)
|
||||
# BR_LABEL the slot-file label (ends with "; agent=<name>")
|
||||
# BR_TOOL build-remote | cross-remote BR_KIND node-linux | node-windows | app | app-windows | prove | suite | check | other
|
||||
# BR_WT the worktree name BR_CRATE the crate path relative to the worktree root
|
||||
# BR_COMMAND the cargo command as typed BR_TARGET the rust target triple
|
||||
# BR_BRANCH BR_SHA BR_AGENT the agent name (IGNEUM_AGENT on the Mac, else the worktree)
|
||||
# BR_ARTEFACTS space-separated paths (relative to BR_DIR) the Mac will fetch; empty for test, check, clippy
|
||||
#
|
||||
# 1. Takes a build slot: flock on $IGNEUM_BUILD_SLOTS_DIR/build-<k> for k below the count in .../slots (the box's own slot
|
||||
# files, never the Mac's); when every slot is busy it waits up to 2 h on build-0 and exits 75 if it gives up. The holder
|
||||
# line is `pid N since HH:MM:SSZ waited S s: <label>`, the format tools/lock/with-lock.sh writes on the Mac.
|
||||
# 2. Runs BR_CMD in BR_DIR with sccache, prints one `build-remote: RESULT rc= secs= compiles= sccache_hits_total= ...` line.
|
||||
# 3. Appends one JSON line to /srv/builds/_log/builds.jsonl (the worker dashboard reads it; asked for by main on 6 October
|
||||
# 2026): on success, on failure and on the slot give-up. UTC ISO 8601 Z times, numbers unquoted, unknown fields omitted,
|
||||
# artefacts with bytes and sha256 only when the run succeeded (a failed build would list the previous build's files),
|
||||
# the line kept under 4 KB.
|
||||
set -uo pipefail
|
||||
. /etc/profile.d/igneum-build.sh
|
||||
: "${BR_DIR:?}" "${BR_CMD:?}" "${BR_LABEL:?}" "${BR_TOOL:?}" "${BR_KIND:?}"
|
||||
BR_HOST=$(hostname); BR_PID=$$; BR_T0=$(date +%s)
|
||||
export BR_HOST BR_PID BR_T0
|
||||
LOG_DIR=/srv/builds/_log; mkdir -p "$LOG_DIR"
|
||||
|
||||
# jsonlog <exit> <slot> <wait_s> <start> <end> <secs> <compiles> <hits> <misses> <hits_total> <misses_total>
|
||||
jsonlog() {
|
||||
BR_EXIT="$1" BR_SLOT="$2" BR_WAIT="$3" BR_START="$4" BR_END="$5" BR_SECS="$6" BR_COMPILES="$7" \
|
||||
BR_HITS="$8" BR_MISSES="$9" BR_HITS_T="${10}" BR_MISSES_T="${11}" BR_LOG="$LOG_DIR/builds.jsonl" python3 - <<'PY' || echo "build-remote: WARNING the JSONL log line was not written" >&2
|
||||
import hashlib, json, os, time
|
||||
e = os.environ
|
||||
def iso(t):
|
||||
return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(int(t))) if t else None
|
||||
def num(v):
|
||||
try: return int(v)
|
||||
except (TypeError, ValueError): return None
|
||||
d = {
|
||||
"v": 1, "id": f"{e['BR_HOST']}-{e['BR_T0']}-{e['BR_PID']}", "host": e['BR_HOST'], "tool": e['BR_TOOL'],
|
||||
"worktree": e.get('BR_WT'), "crate": e.get('BR_CRATE'), "kind": e['BR_KIND'], "command": e.get('BR_COMMAND'),
|
||||
"target": e.get('BR_TARGET'), "branch": e.get('BR_BRANCH'), "sha": e.get('BR_SHA'), "label": e['BR_LABEL'],
|
||||
"agent": e.get('BR_AGENT'), "slot": num(e['BR_SLOT']), "wait_s": num(e['BR_WAIT']), "queued_at": iso(e['BR_T0']),
|
||||
"start": iso(e['BR_START']), "end": iso(e['BR_END']), "secs": num(e['BR_SECS']), "exit": num(e['BR_EXIT']),
|
||||
"compiles": num(e['BR_COMPILES']),
|
||||
}
|
||||
sc = {k: num(e[v]) for k, v in (("hits", "BR_HITS"), ("misses", "BR_MISSES"), ("hits_total", "BR_HITS_T"), ("misses_total", "BR_MISSES_T"))}
|
||||
sc = {k: v for k, v in sc.items() if v is not None}
|
||||
if sc: d["sccache"] = sc
|
||||
try:
|
||||
d["load_end"] = [float(x) for x in open('/proc/loadavg').read().split()[:3]]
|
||||
except OSError:
|
||||
pass
|
||||
arts = []
|
||||
if d["exit"] == 0:
|
||||
for p in e.get('BR_ARTEFACTS', '').split():
|
||||
fp = os.path.join(e['BR_DIR'], p)
|
||||
if os.path.isfile(fp):
|
||||
h = hashlib.sha256()
|
||||
with open(fp, 'rb') as f:
|
||||
for chunk in iter(lambda: f.read(1 << 20), b''):
|
||||
h.update(chunk)
|
||||
arts.append({"path": p, "bytes": os.path.getsize(fp), "sha256": h.hexdigest()})
|
||||
d["artefacts"] = arts
|
||||
d = {k: v for k, v in d.items() if v is not None and v != ""}
|
||||
line = json.dumps(d, separators=(',', ':'))
|
||||
if len(line) > 4000:
|
||||
for k in ("command", "label"):
|
||||
if k in d: d[k] = d[k][:200]
|
||||
line = json.dumps(d, separators=(',', ':'))
|
||||
with open(e['BR_LOG'], 'a') as f:
|
||||
f.write(line + "\n")
|
||||
PY
|
||||
}
|
||||
|
||||
slots=$(cat "$IGNEUM_BUILD_SLOTS_DIR/slots" 2>/dev/null || echo 1); [ "$slots" -ge 1 ] 2>/dev/null || slots=1
|
||||
got=""
|
||||
for k in $(seq 0 $((slots - 1))); do
|
||||
exec {fd}>"$IGNEUM_BUILD_SLOTS_DIR/build-$k"
|
||||
if flock -n "$fd"; then got=$k; break; fi
|
||||
exec {fd}>&-
|
||||
done
|
||||
if [ -z "$got" ]; then
|
||||
echo "build-remote: all $slots slot(s) busy, waiting (up to 2 h) for build-0: $(head -c 160 "$IGNEUM_BUILD_SLOTS_DIR/build-0" 2>/dev/null)" >&2
|
||||
exec {fd}>"$IGNEUM_BUILD_SLOTS_DIR/build-0"
|
||||
if ! flock -w 7200 "$fd"; then
|
||||
echo "build-remote: gave up waiting for a slot after 2 h" >&2
|
||||
jsonlog 75 0 $(( $(date +%s) - BR_T0 )) "" "$(date +%s)" "" "" "" "" "" ""
|
||||
exit 75
|
||||
fi
|
||||
got=0
|
||||
fi
|
||||
waited=$(( $(date +%s) - BR_T0 ))
|
||||
printf 'pid %s since %sZ waited %s s: %s\n' "$BR_PID" "$(date -u +%H:%M:%S)" "$waited" "$BR_LABEL" > "$IGNEUM_BUILD_SLOTS_DIR/build-$got"
|
||||
echo "build-remote: holding build-$got on $BR_HOST (waited $waited s)" >&2
|
||||
|
||||
cd "$BR_DIR" || { jsonlog 2 "$got" "$waited" "" "$(date +%s)" "" "" "" "" "" ""; exit 2; }
|
||||
sccache --start-server >/dev/null 2>&1 || true
|
||||
stat_field() { sccache --show-stats 2>/dev/null | awk -v key="$1" 'index($0, key) == 1 { print $NF; exit }'; }
|
||||
exec_before=$(stat_field "Compile requests executed"); hits_before=$(stat_field "Cache hits "); misses_before=$(stat_field "Cache misses ")
|
||||
t1=$(date +%s)
|
||||
eval "$BR_CMD"
|
||||
rc=$?
|
||||
t2=$(date +%s); secs=$(( t2 - t1 ))
|
||||
exec_after=$(stat_field "Compile requests executed"); hits_after=$(stat_field "Cache hits "); misses_after=$(stat_field "Cache misses ")
|
||||
compiles=$(( ${exec_after:-0} - ${exec_before:-0} )); hits=$(( ${hits_after:-0} - ${hits_before:-0} )); misses=$(( ${misses_after:-0} - ${misses_before:-0} ))
|
||||
printf 'build-remote: RESULT rc=%s secs=%s compiles=%s sccache_hits=%s sccache_misses=%s sccache_hits_total=%s sccache_misses_total=%s load=%s\n' \
|
||||
"$rc" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-?}" "${misses_after:-?}" "$(cut -d' ' -f1-3 /proc/loadavg)"
|
||||
jsonlog "$rc" "$got" "$waited" "$t1" "$t2" "$secs" "$compiles" "$hits" "$misses" "${hits_after:-}" "${misses_after:-}"
|
||||
: > "$IGNEUM_BUILD_SLOTS_DIR/build-$got"
|
||||
exit "$rc"
|
||||
|
|
@ -27,7 +27,8 @@
|
|||
# not hold a Mac slot. A build waits up to 2 h for a remote slot, as with-lock.sh does.
|
||||
#
|
||||
# Needs: ~/.config/igneum/build-server (build@<ip>, written by infra/build-server/run-from-mac.sh), ~/.ssh/igneum_ed25519,
|
||||
# the same rustc version on both sides (refused otherwise; IGNEUM_TOOLCHAIN_MISMATCH=ok overrides).
|
||||
# the same rustc version on both sides (refused otherwise; IGNEUM_TOOLCHAIN_MISMATCH=ok overrides). IGNEUM_AGENT names the
|
||||
# agent in the slot-file label and the box's JSONL log (/srv/builds/_log/builds.jsonl); default the worktree name.
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC2034 # shared with the scripts that source lib.sh
|
||||
|
|
@ -76,8 +77,19 @@ t_sync0=$(date +%s)
|
|||
bs_sync_sources
|
||||
bs_log "sources in place after $(( $(date +%s) - t_sync0 )) s"
|
||||
|
||||
cmd="CARGO_TARGET_DIR='$TARGET_DIR' cargo $(printf '%q ' "${CARGO_ARGS[@]}")-j $JOBS 2>&1 | tee -a '$BS_REMOTE_WT/.build-remote.log'; exit \${PIPESTATUS[0]}"
|
||||
# the fork's kaspa-build-info embeds the commit through a build script that, having once found no branch, emits no
|
||||
# rerun-if-changed and is never run again by cargo (release-0.3.11 plan: `cargo clean -p kaspa-build-info` first); so when
|
||||
# the commit the box builds differs from the last one built in this target dir, that one crate is cleaned (a relink, seconds)
|
||||
pre=""
|
||||
if [ "$BS_KIND" = node ] && [ "${CARGO_ARGS[0]}" = build ]; then
|
||||
pre="[ \"\$(cat '.build-remote-sha-$TARGET_DIR' 2>/dev/null)\" = '$BS_SHA' ] || CARGO_TARGET_DIR='$TARGET_DIR' cargo clean -q --release -p kaspa-build-info 2>/dev/null; "
|
||||
fi
|
||||
cmd="${pre}CARGO_TARGET_DIR='$TARGET_DIR' cargo $(printf '%q ' "${CARGO_ARGS[@]}")-j $JOBS 2>&1 | tee -a '$BS_REMOTE_WT/.build-remote.log'; rc=\${PIPESTATUS[0]}; [ \$rc = 0 ] && echo '$BS_SHA' > '.build-remote-sha-$TARGET_DIR'; ( exit \$rc )" # a subshell exit: the runner reads \$? and still prints its RESULT line
|
||||
label="$BS_WT/$BS_CRATE_REL cargo ${CARGO_ARGS[*]}"
|
||||
BR_KIND=$(bs_kind build-remote "${CARGO_ARGS[0]}"); BR_COMMAND="cargo ${CARGO_ARGS[*]}"; BR_TARGET=x86_64-unknown-linux-gnu
|
||||
for ((i = 0; i < ${#CARGO_ARGS[@]}; i++)); do [ "${CARGO_ARGS[$i]}" = --target ] && BR_TARGET="${CARGO_ARGS[$((i + 1))]:-}"; done
|
||||
BR_ARTEFACTS=""; [ "$FETCH" = 1 ] && BR_ARTEFACTS="$ARTEFACTS"
|
||||
export BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS
|
||||
t0=$(date +%s)
|
||||
set +e
|
||||
bs_remote_run "$BS_REMOTE_CRATE" "$label" "$cmd" 2>&1 | tee "/tmp/build-remote-$$.log"
|
||||
|
|
|
|||
|
|
@ -76,11 +76,16 @@ export CARGO_TARGET_X86_64_PC_WINDOWS_GNU_RUSTFLAGS="-C link-arg=-static -C link
|
|||
export IGNEUM_WINDRES=x86_64-w64-mingw32-windres LIBCLANG_PATH="$LLVM_LIB"
|
||||
export BINDGEN_EXTRA_CLANG_ARGS_x86_64_pc_windows_gnu="--target=x86_64-w64-mingw32 --sysroot=/usr/x86_64-w64-mingw32 -I/usr/x86_64-w64-mingw32/include"
|
||||
echo "cross-remote: $(x86_64-w64-mingw32-gcc-posix --version | head -1); libclang $LLVM_LIB"'
|
||||
pre="" # the same kaspa-build-info clean on a new commit as build-remote.sh (the Windows target has its own fingerprint)
|
||||
[ "$BS_KIND" = node ] && pre="[ \"\$(cat '.cross-remote-sha-$TARGET_DIR' 2>/dev/null)\" = '$BS_SHA' ] || CARGO_TARGET_DIR='$TARGET_DIR' cargo clean -q --release -p kaspa-build-info --target $TARGET 2>/dev/null; "
|
||||
cmd="$env_block
|
||||
CARGO_TARGET_DIR='$TARGET_DIR' cargo $(printf '%q ' "${CARGO_ARGS[@]}")-j $JOBS 2>&1 | tee -a '$BS_REMOTE_WT/.cross-remote.log'; rc=\${PIPESTATUS[0]}
|
||||
${pre}CARGO_TARGET_DIR='$TARGET_DIR' cargo $(printf '%q ' "${CARGO_ARGS[@]}")-j $JOBS 2>&1 | tee -a '$BS_REMOTE_WT/.cross-remote.log'; rc=\${PIPESTATUS[0]}; [ \$rc = 0 ] && echo '$BS_SHA' > '.cross-remote-sha-$TARGET_DIR'
|
||||
for exe in $EXES; do f='$TARGET_DIR/$TARGET/release/'\$exe; [ -f \"\$f\" ] && echo \"cross-remote: DLLS \$exe: \$(x86_64-w64-mingw32-objdump -p \"\$f\" | awk '/DLL Name/ { print \$3 }' | sort -u | tr '\n' ' ')\"; done
|
||||
exit \$rc"
|
||||
( exit \$rc )" # a subshell exit: the runner reads \$? and still prints its RESULT line
|
||||
label="$BS_WT/$BS_CRATE_REL cross $TARGET"
|
||||
BR_KIND=$(bs_kind cross-remote "${CARGO_ARGS[0]}"); BR_COMMAND="cargo ${CARGO_ARGS[*]}"; BR_TARGET="$TARGET"
|
||||
BR_ARTEFACTS=""; for exe in $EXES; do BR_ARTEFACTS="$BR_ARTEFACTS $TARGET_DIR/$TARGET/release/$exe"; done
|
||||
export BR_KIND BR_COMMAND BR_TARGET BR_ARTEFACTS
|
||||
t0=$(date +%s)
|
||||
set +e
|
||||
bs_remote_run "$BS_REMOTE_CRATE" "$label" "$cmd" 2>&1 | tee "/tmp/cross-remote-$$.log"
|
||||
|
|
@ -104,4 +109,11 @@ for exe in $EXES; do
|
|||
fi
|
||||
bs_log "$line"
|
||||
done
|
||||
# an exe that imports libstdc++-6.dll (a fork before the housekeeping commit that made the C++ runtime static) needs the
|
||||
# three runtime DLLs OF THIS TOOLCHAIN beside it (the PC job does the same; push-inputs.sh takes them from next to the exes)
|
||||
if printf '%s\n' "$dlls" | grep -q 'libstdc++-6.dll'; then
|
||||
bs_ssh 'd=$(dirname "$(x86_64-w64-mingw32-gcc-posix -print-file-name=libstdc++-6.dll)"); mkdir -p /tmp/igneum-mingw-dlls; cp "$d/libstdc++-6.dll" "$d/libgcc_s_seh-1.dll" /usr/x86_64-w64-mingw32/lib/libwinpthread-1.dll /tmp/igneum-mingw-dlls/ && ls /tmp/igneum-mingw-dlls' >/dev/null \
|
||||
&& bs_rsync -p "$BS_HOST:/tmp/igneum-mingw-dlls/*.dll" "$OUT/$TARGET/release/" || bs_die "could not fetch the mingw runtime DLLs"
|
||||
for dll in libstdc++-6.dll libgcc_s_seh-1.dll libwinpthread-1.dll; do bs_log "runtime $dll: $(bs_size "$OUT/$TARGET/release/$dll") bytes, sha256 $(bs_sha256 "$OUT/$TARGET/release/$dll") (GCC 13 posix, beside the exes)"; done
|
||||
fi
|
||||
bs_log "exes in $OUT/$TARGET/release"
|
||||
|
|
|
|||
Loading…
Reference in a new issue