diff --git a/docs/evidence/reproduced/0.3.14.md b/docs/evidence/reproduced/0.3.14.md new file mode 100644 index 000000000..558e1adfa --- /dev/null +++ b/docs/evidence/reproduced/0.3.14.md @@ -0,0 +1,14 @@ +# Reproduced: Igneum Miner 0.3.14 (node 4c6b129d75c3d77a3689d22f1e1dc721b556aebb, app a90f6a5371ed19c62511255a4713eb36191848b9) + +06 October 2026, 19:53 UTC on igneum-build-1 by `infra/build-server/repro/rebuild-on-box.sh` (driven by `tools/repro/rebuild-release.sh`): a clean clone of the fork at the node commit on branch `release-0.3.14-node` under a clean clone of the repo at the app commit, 2 independent clean passes per target in one target path each (no sccache, SOURCE_DATE_EPOCH 1791305478, TZ UTC), rustc 1.99.0, x86_64-w64-mingw32-gcc-posix (GCC) 13-posix, clang 18.1.3, glibc 2.39. Shipped hashes read from the public downloads (no token) where marked. Whole run 1 s; log `/srv/builds/_repro/0.3.14/rebuild.log`. + +| Artefact | Shipped sha256 (source) | Box pass A | Box pass B | A vs shipped | A vs B | Reason for a DIFFER | +|---|---|---|---|---|---|---| +| igneumd | 934f393cacc31a06d0c45a9fe2e2f504941a32533110b51851968e70cf90fa3a (given) | 03f35e056922fa1e... (49600096 B) | 03f35e056922fa1e... (49600096 B) | **DIFFER** | **MATCH** | toolchain: the shipped binary came from the Mac's infra/cross/build-linux.sh (zig, glibc 2.36 target, docs/plans/release-0.3.14.md), the box's from the native clang/lld against glibc 2.39 (the box binary needs GLIBC_2.39); the shipped binary was not on hand this run (the public download failed), so its symbol versions were not read; commit string in the box's binary: 1 hit(s); the shipped exe was built from a worktree before the two-step clean, so by the empty-commit class it carries none (not read: no file) | +| igneum-miner | 7e296541 (given) | 900c1f0bf8a3b504... (9842168 B) | 900c1f0bf8a3b504... (9842168 B) | **DIFFER** | **MATCH** | toolchain: the shipped binary came from the Mac's infra/cross/build-linux.sh (zig, glibc 2.36 target, docs/plans/release-0.3.14.md), the box's from the native clang/lld against glibc 2.39 (the box binary needs GLIBC_2.39); the shipped binary was not on hand this run (the public download failed), so its symbol versions were not read | +| igneumd.exe | 44fa74c02415ff258b5956ef55909dc97890e3f29fca3d2db26f7152ef4ce541 (given) | 166e604e01c668e6... (51758592 B) | 166e604e01c668e6... (51758592 B) | **DIFFER** | **MATCH** | toolchain: the shipped exe came from the Mac's proto-cuda/windows-node/cross-build.sh (Homebrew mingw) at 16:52Z on 6 Oct 2026, the box's from Ubuntu GCC 13 posix with -Wl,--no-insert-timestamp (the fix landed 17:48Z, after this cut's exes, so the shipped PE header carries a build time); the shipped exe itself is not on hand (innoextract 1.9 cannot read the Inno Setup 6 installer: setup loader revision 2), so its hash is the release plan's and its PE header was not read; commit string in the box's binary: 1 hit(s); the shipped exe was built from a worktree before the two-step clean, so by the empty-commit class it carries none (not read: no file) | +| igneum-miner.exe | 819ea9ce (given) | fefd266c3bd6470f... (10994688 B) | fefd266c3bd6470f... (10994688 B) | **DIFFER** | **MATCH** | toolchain: the shipped exe came from the Mac's proto-cuda/windows-node/cross-build.sh (Homebrew mingw) at 16:52Z on 6 Oct 2026, the box's from Ubuntu GCC 13 posix with -Wl,--no-insert-timestamp (the fix landed 17:48Z, after this cut's exes, so the shipped PE header carries a build time); the shipped exe itself is not on hand (innoextract 1.9 cannot read the Inno Setup 6 installer: setup loader revision 2), so its hash is the release plan's and its PE header was not read | + +Full box hashes: igneumd A 03f35e056922fa1e406e14d35963e8d3ca57f98f0d58a04c3f7cc450a26d1fdc; igneum-miner A 900c1f0bf8a3b504dfb2a7fa7abb91f3286eb0d020ed1e0aa5f3ab808c0489eb; igneumd.exe A 166e604e01c668e69b88556cad959429c67b040ec1e024cf7e514e1b5fc8edae; igneum-miner.exe A fefd266c3bd6470f61476a96453d4ea0cb54c42b8b23038cf5ef5a3397399514; + +Reading: MATCH against the shipped bytes is the goal; A vs B MATCH with a DIFFER against the shipped bytes means the box is deterministic and the shipped build came from another toolchain (the reason column says which facts differ); A vs B DIFFER is a non-determinism on the box itself and is the row to fix first. diff --git a/docs/plans/build-server.md b/docs/plans/build-server.md index ba7a841fb..eb610dbbd 100644 --- a/docs/plans/build-server.md +++ b/docs/plans/build-server.md @@ -152,6 +152,82 @@ Consequences: ci.yml's `pow` and `sims` jobs would run on a pinned 1.99.0 (GitHu Open: a worktree whose remote-run.sh predates this keeps the old behaviour until it has master with it (the script is piped from each Mac worktree per build), so until every agent rebases, a build from an old worktree still asks `-j 90` beside a new one at 45. The build-server agent was told at 19:28Z. +### 7.2 The night battery (DONE 19:42Z installed, dry run 19:45 to 19:49Z) + +`infra/build-server/night/night-battery.sh`, run by `igneum-night-battery.timer` at 02:00 Europe/London (the box's clock is +Europe/Berlin, so the unit names the zone: next run Wed 2026-10-07 03:00 CEST = 02:00 BST; not Persistent, a missed night is +not run by day) through `igneum-night-battery.service` (User build, Nice 19, idle IO, 8 h limit), whose ExecStart is +`remote-run.sh` with the battery as BR_CMD, so ONE build slot spans the whole invocation and one JSONL line records it. +Installed by provision.sh `step_night` from the mirror at `NIGHT_REF` (box-work tonight, master once merged); the battery +re-execs itself from master's checkout at run time. Every row: `cargo test --release --no-fail-fast` per crate (the repo's +five crates and the proving workspace's host, core and export; every fork workspace member, kaspad with `igneum-pow`), +igneum-pow's two fuzz tests at 2,000 programs (10x), the three simulators in full, the fast-time harnesses +(`tools/finality-attacks/run.mjs --fast-time`, `tools/harness/run.mjs s3 s4 --fast-time --no-bench-log`, +`tools/exec-sync/reorg.mjs`) on igneumd, igneum-miner, igneum-harness-sim and igneum-p2p-probe built into the night +checkout's `target-integration`, clippy per crate dir, `cargo audit` per Cargo.lock; then `docs/benchmarks/night/.md` +(pass/fail table, "new since last night" against the newest earlier report, commits moved), committed as igneum-labs on +branch `night-battery` (rebuilt on master each night, earlier unmerged reports carried over) and force-pushed to +`/srv/igneum.git` only. Main merges: `git fetch build night-battery` from the main checkout. The fork branch defaults to the +newest `release-*-node` on the mirror (tonight release-0.3.15-node 713ef876). + +Dry run (`NIGHT_SUBSET=1`, the second slot while the repro held the first, so CARGO_BUILD_JOBS 45): **3 min 54 s** wall, +10 pass, 1 FAIL, 1 skip; report `docs/benchmarks/night/2026-10-06-dryrun.md` on the mirror's night-battery branch (fbb72e5). + +| Row | Result | Time | Detail | +|---|---|---|---| +| suite igneum-pow | pass | 51 s | 99 passed | +| suite fork/kaspa-pow | pass | 1 min 04 s | 7 passed | +| suite fork/igneum-miner | pass | 49 s | 18 passed | +| fuzz igneum-pow x200 | pass | 11 s | 200 mx8 programs and 200 scratch programs, 800 units each | +| sim finality_sim.py, finality_v2.py --quick, difficulty/sim.py --quick | pass | 3 s, 41 s, 5 s | 249, 117, 8 table lines | +| clippy igneum-pow | pass | 3 s | 28 warnings | +| audit igneum-pow | pass | 3 s | 0 vulnerabilities | +| audit vendor/igneum-node | **FAIL** | 2 s | 22 advisories in the fork's lock file: h2 (RUSTSEC-2026-0258, unbounded empty DATA frames), quinn-proto (2026-0185, remote memory exhaustion), rustls (2026-0285, TLS 1.3 handshake across encryption levels), ruint (2026-0220), crossbeam-epoch, anyhow (2026-0190), event-listener, faster-hex (2026-0306, AVX2 read past src), lru (2026-0253), tracing-subscriber (2025-0055), chacha20, spin; 17 unmaintained-crate warnings (async-std discontinued, atty, bincode, derivative, instant, mach, paste, proc-macro-error, rustls-pemfile) | +| harness | skip | | not in the subset; its first run is the 02:00 battery, so the first full report will show whether the Node harnesses run on Linux unchanged (c4, fud and v3.mjs default IGNEUM_NODE_ROOT to /Users/joshm/Projects/igneum/; the battery sets it) | + +What the FAIL means and what follows: every node binary shipped so far (and the 0.3.15 one tonight) links h2, quinn-proto and +rustls at versions with published advisories; h2 and quinn-proto are in the gRPC and QUIC paths a peer can reach, so these are +the remote ones. The fix is a dependency bump in the fork (`cargo update -p h2 -p quinn-proto -p rustls -p ruint -p +crossbeam-epoch -p anyhow -p event-listener -p faster-hex -p lru -p tracing-subscriber`, then the suites), a consensus +engineer's hour on a quiet branch, and the row goes green by itself the next night. Until then the row stays FAIL every night +and "new since last night" stays quiet about it. The unmaintained-crate warnings are upstream rusty-kaspa's and do not fail +the row. + +Expected full-run time (not measured yet): the three suites above compile the fork's test targets once (about 1 min each for +the first crates, seconds after), so 75 fork members plus the repo crates are estimated at 40 to 70 min; the full sims about +10 min (finality_v2.py is 5 min on the Mac); the harnesses 15 to 30 min; clippy and audit under 10 min. Under 2 h, inside +the 8 h limit; the first report at 02:00 BST writes the real number. + +### 7.3 Reproducible builds (DONE 19:52Z; A vs B MATCH on all four, DIFFER against the shipped bytes, both explained) + +`tools/repro/rebuild-release.sh ` (the Mac) reads the pins from `docs/plans/release-.md` (the heading +"(node , app )" and the bold hashes of the Linux and Windows rows), makes sure both commits are on the mirrors, and +runs `infra/build-server/repro/rebuild-on-box.sh` on the box: a clean clone of the fork at the node commit on a branch under +a clean clone of the repo at the app commit, two clean passes per target in ONE target path each, no sccache, under build +slots through remote-run.sh, `SOURCE_DATE_EPOCH` = the node commit's time, `TZ=UTC`; the shipped hashes come token-free from +the public downloads (the HiveOS tarball for the Linux pair; the installer for the exes, when innoextract can open it) with +the plan's hashes as the fallback; the evidence goes to `docs/evidence/reproduced/.md`. The 0.3.14 run (node +4c6b129d, app a90f6a5): four passes of 70 to 78 s, whole run 5 min 06 s. + +| Artefact | A vs shipped | A vs B | Why the shipped bytes differ (read off the binaries) | +|---|---|---|---| +| igneumd (box 03f35e05..., 49,600,096 B) | DIFFER | **MATCH** | shipped 934f393c... was the Mac's zig build for glibc 2.36; the box's needs GLIBC_2.39 (native clang and lld). Commit string 4c6b129d in the box's: 1 hit | +| igneum-miner (box 900c1f0b..., 9,842,168 B) | DIFFER | **MATCH** | the same toolchain difference | +| igneumd.exe (box 166e604e..., 51,758,592 B) | DIFFER | **MATCH** | shipped 44fa74c0... came from the Mac's Homebrew mingw at 16:52Z, before the --no-insert-timestamp fix (17:48Z) and from a worktree (the empty-commit class); the box's is Ubuntu GCC 13 posix with a zero PE timestamp and the commit string (1 hit). innoextract 1.9 cannot open the Inno Setup 6 installer (setup loader revision 2), so the shipped exe's own header was not read | +| igneum-miner.exe (box fefd266c..., 10,994,688 B) | DIFFER | **MATCH** | the same | + +Two non-determinisms found on the way, both in the SHIPPED builds too (first run 19:43Z, passes A and B differed on all four): + +| Class | Fact | Fix | +|---|---|---| +| OUT_DIR path in the binary | prost's generated `protowire.rs` (kaspa-grpc-core, kaspa-p2p-lib) embeds its OUT_DIR path; a pass in a target dir of another NAME differs (igneum-miner matched byte for byte once the path was the same) | one target path per target in the repro; for cross-machine identity a `--remap-path-prefix` of the target dir and the home (not done: the Mac and the box differ in every path anyway) | +| Build clock in the binary | libmimalloc-sys compiles mimalloc's C with `__DATE__` and `__TIME__` ("Oct 6 2026", "21:38:15" sat in libmimalloc.a, next to the mimalloc option names); two builds a minute apart differ | `SOURCE_DATE_EPOCH` exported for every pass (GCC and clang take the date and time from it); PROPOSED for build-remote.sh, cross-remote.sh, cross-build.sh and the PC job: export it from the commit time so two builds of one commit give one hash. The earlier "byte-identical across three builds" on the box was under sccache, which returns the first build's object and hides this class | + +What it means: the box is deterministic for a given commit and path, so a release built on it can be checked by anyone with the +same toolchain by rebuilding and comparing; the shipped 0.3.14 bytes cannot be reproduced anywhere because they came from +two Mac toolchains with a build clock inside, and that is the reason to ship from the box from 0.3.16 (R2) with +SOURCE_DATE_EPOCH set. Open: `rebuild-release.sh` for 0.3.15 the moment it ships (one command, 5 min). + ### 7.4 The CPU prover trial (DONE 19:30Z; verdict: the box is NOT a prover) `infra/build-server/prover/cpu-trial.sh` on the box under the measure hold (builds excluded), `igneum-prove-host` from master diff --git a/infra/build-server/night/igneum-night-battery.service b/infra/build-server/night/igneum-night-battery.service new file mode 100644 index 000000000..0ff06cb56 --- /dev/null +++ b/infra/build-server/night/igneum-night-battery.service @@ -0,0 +1,34 @@ +# igneum-build-1: the night battery (infra/build-server/night/night-battery.sh) under one build slot through remote-run.sh. +# Installed by infra/build-server/provision.sh step_night; started by igneum-night-battery.timer at 02:00 Europe/London. +[Unit] +Description=Igneum night battery (every test suite, fuzz, simulators, harnesses, clippy, audit; report on branch night-battery) +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=build +Group=build +Nice=19 +IOSchedulingClass=idle +WorkingDirectory=/srv/builds/_night +Environment=BR_DIR=/srv/builds/_night +Environment=BR_CMD=/srv/builds/_bin/night-battery.sh +Environment="BR_LABEL=night battery; agent=night" +Environment=BR_TOOL=night-battery +Environment=BR_KIND=other +Environment=BR_WT=_night +Environment=BR_CRATE=. +Environment=BR_BRANCH=master +Environment=BR_SHA= +Environment=BR_AGENT=night +Environment="BR_COMMAND=night-battery.sh (suites, fuzz, sims, harness, clippy, audit)" +Environment=BR_TARGET=x86_64-unknown-linux-gnu +Environment=IGNEUM_AGENT=night +ExecStart=/bin/bash /srv/builds/_bin/remote-run.sh +TimeoutStartSec=8h +StandardOutput=append:/srv/builds/_log/night-battery.log +StandardError=append:/srv/builds/_log/night-battery.log + +[Install] +WantedBy=multi-user.target diff --git a/infra/build-server/night/igneum-night-battery.timer b/infra/build-server/night/igneum-night-battery.timer new file mode 100644 index 000000000..fdc5b114b --- /dev/null +++ b/infra/build-server/night/igneum-night-battery.timer @@ -0,0 +1,12 @@ +# igneum-build-1: 02:00 London every night (the box's clock is Europe/Berlin; the time zone is named here, so BST and GMT both +# land at 02:00 UK). Not Persistent: a missed night is not run during the day. +[Unit] +Description=Igneum night battery at 02:00 Europe/London + +[Timer] +OnCalendar=*-*-* 02:00:00 Europe/London +Persistent=false +AccuracySec=1min + +[Install] +WantedBy=timers.target diff --git a/infra/build-server/night/night-battery.sh b/infra/build-server/night/night-battery.sh new file mode 100755 index 000000000..4a93d3ef1 --- /dev/null +++ b/infra/build-server/night/night-battery.sh @@ -0,0 +1,207 @@ +#!/usr/bin/env bash +# The night battery on igneum-build-1 (6 October 2026): one invocation, one build slot, every test the repo and the fork have. +# Runs ON the box as user build at 02:00 Europe/London (igneum-night-battery.timer) through remote-run.sh, which holds the +# slot for the whole run, sets CARGO_BUILD_JOBS (90 alone, 45 beside another build) and writes the JSONL line. By hand: +# /srv/builds/_bin/night-battery.sh the full battery (hours) +# NIGHT_SUBSET=1 /srv/builds/_bin/night-battery.sh the dry-run subset (igneum-pow suite and fuzz, two fork crates, quick sims, clippy and audit on igneum-pow) +# NIGHT_NODE_BRANCH=release-0.3.15-node ... the fork branch (default: the newest release-*-node on the mirror) +# NIGHT_SKIP="harness sims" ... skip stages by name +# Stages, each a row (pass, FAIL, skip) with seconds and a detail: +# checkout /srv/builds/_night/igneum from /srv/igneum.git master (the battery re-execs itself from that checkout, so the +# script that runs is master's), vendor/igneum-node from /srv/igneum-node.git at the fork branch +# suites cargo test --release --no-fail-fast per crate: the repo's crates (igneum-pow, igneum-census, pool, proto-vdf, +# app/igneum-app, every member of proving/igneum-prove) and every workspace member of the fork (kaspad with +# --features igneum-pow); the pass and fail counts are read from the `test result:` lines +# fuzz igneum-pow's two fuzz tests at 10x their default (IGNEUM_MIXER_FUZZ and IGNEUM_SCRATCH_FUZZ 2000) +# sims sim/finality_sim.py, sim/finality_v2.py and sim/difficulty/sim.py in full (the subset runs --quick) +# harness the fork's igneumd, igneum-miner, igneum-harness-sim and igneum-p2p-probe built into target-integration, then +# tools/finality-attacks/run.mjs --fast-time, tools/harness/run.mjs s3 s4 --fast-time --no-bench-log and +# tools/exec-sync/reorg.mjs (loopback ports 27200+, 27800+, 29870+; nothing of the live devnet) +# clippy cargo clippy --release --all-targets per crate dir (warnings counted; a row fails on an error) +# audit cargo audit in every directory with a Cargo.lock +# report docs/benchmarks/night/.md (a pass/fail table and "new since last night" against the newest earlier +# report), committed as igneum-labs on branch night-battery (based on master, earlier reports carried over) and +# pushed to /srv/igneum.git night-battery; main merges (`git fetch build night-battery` on the Mac). Never master. +set -uo pipefail +_slots_env="${IGNEUM_BUILD_SLOTS_DIR:-}"; [ -f /etc/profile.d/igneum-build.sh ] && . /etc/profile.d/igneum-build.sh; [ -n "$_slots_env" ] && IGNEUM_BUILD_SLOTS_DIR="$_slots_env" +NIGHT_ROOT="${NIGHT_ROOT:-/srv/builds/_night}"; REPO_MIRROR=/srv/igneum.git; NODE_MIRROR=/srv/igneum-node.git +SUBSET="${NIGHT_SUBSET:-0}"; SKIP=" ${NIGHT_SKIP:-} " +DATE=$(date -u +%Y-%m-%d); STAMP=$(date -u +%Y%m%dT%H%M%SZ); T_ALL=$(date +%s) +REPORT_NAME="$DATE$( [ "$SUBSET" = 1 ] && echo -dryrun ).md" +LOGDIR="$NIGHT_ROOT/logs/$STAMP"; mkdir -p "$LOGDIR" +ROWS="$LOGDIR/rows.tsv"; : > "$ROWS" +say() { printf '%s night: %s\n' "$(date -u +%H:%M:%S)" "$*" >&2; } +row() { printf '%s\t%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "$4" "$5" >> "$ROWS"; say "$1 | $2 | $3 | ${4}s | $5"; } # stage name status secs detail +skip() { case "$SKIP" in *" $1 "*) return 0 ;; esac; return 1; } +cargo_jobs="${CARGO_BUILD_JOBS:-90}" + +# checkout +IG="$NIGHT_ROOT/igneum"; FORK="$IG/vendor/igneum-node" +t0=$(date +%s) +if [ ! -d "$IG/.git" ]; then git clone -q "$REPO_MIRROR" "$IG" || { row checkout repo FAIL 0 "clone failed"; exit 1; }; fi +git -C "$IG" fetch -q origin '+refs/heads/*:refs/remotes/origin/*' || { row checkout repo FAIL 0 "fetch failed"; exit 1; } +git -C "$IG" checkout -q -- . 2>/dev/null; git -C "$IG" clean -qfd -e target -e 'target-*' -e vendor +git -C "$IG" checkout -q -B night-battery origin/master +# earlier reports not yet merged into master ride along +git -C "$IG" checkout -q origin/night-battery -- docs/benchmarks/night 2>/dev/null || true +REPO_SHA=$(git -C "$IG" rev-parse --short HEAD) +if [ "${NIGHT_REEXEC:-0}" != 1 ] && [ -f "$IG/infra/build-server/night/night-battery.sh" ] && ! cmp -s "$0" "$IG/infra/build-server/night/night-battery.sh"; then + say "re-exec from master's copy ($REPO_SHA)"; NIGHT_REEXEC=1 exec bash "$IG/infra/build-server/night/night-battery.sh" +fi +NODE_BRANCH="${NIGHT_NODE_BRANCH:-$(git -C "$NODE_MIRROR" branch --list 'release-*-node' | tr -d ' *' | sort -V | tail -1)}" +[ -n "$NODE_BRANCH" ] || NODE_BRANCH=master +mkdir -p "$IG/vendor" +if [ ! -d "$FORK/.git" ]; then git clone -q --no-checkout "$NODE_MIRROR" "$FORK" || { row checkout fork FAIL 0 "clone failed"; exit 1; }; fi +git -C "$FORK" fetch -q origin '+refs/heads/*:refs/remotes/origin/*' +git -C "$FORK" checkout -q -- . 2>/dev/null; git -C "$FORK" clean -qfd -e target -e 'target-*' +git -C "$FORK" checkout -q -B "$NODE_BRANCH" "origin/$NODE_BRANCH" || { row checkout fork FAIL 0 "no branch $NODE_BRANCH"; exit 1; } +NODE_SHA=$(git -C "$FORK" rev-parse --short HEAD) +row checkout "repo master $REPO_SHA, fork $NODE_BRANCH $NODE_SHA" pass $(( $(date +%s) - t0 )) "$IG; jobs $cargo_jobs; subset $SUBSET" + +# helpers +run_to() { # : runs in the current dir, returns the exit code, 124 on timeout + local log="$1" to="$2"; shift 2 + timeout --signal=TERM --kill-after=60 "$to" "$@" > "$log" 2>&1; echo $? +} +counts() { # pass/fail totals from cargo test output + awk '/^test result:/ { for (i = 1; i <= NF; i++) { if ($(i+1) == "passed;") p += $i; if ($(i+1) == "failed;") f += $i } } END { printf "%d passed, %d failed", p, f }' "$1" +} +suite() { #