diff --git a/docs/evidence/reproduced/0.3.14.md b/docs/evidence/reproduced/0.3.14.md index 558e1adf..32a4d037 100644 --- a/docs/evidence/reproduced/0.3.14.md +++ b/docs/evidence/reproduced/0.3.14.md @@ -1,13 +1,13 @@ # Reproduced: Igneum Miner 0.3.14 (node 4c6b129d75c3d77a3689d22f1e1dc721b556aebb, app a90f6a5371ed19c62511255a4713eb36191848b9) -06 October 2026, 19:53 UTC on igneum-build-1 by `infra/build-server/repro/rebuild-on-box.sh` (driven by `tools/repro/rebuild-release.sh`): a clean clone of the fork at the node commit on branch `release-0.3.14-node` under a clean clone of the repo at the app commit, 2 independent clean passes per target in one target path each (no sccache, SOURCE_DATE_EPOCH 1791305478, TZ UTC), rustc 1.99.0, x86_64-w64-mingw32-gcc-posix (GCC) 13-posix, clang 18.1.3, glibc 2.39. Shipped hashes read from the public downloads (no token) where marked. Whole run 1 s; log `/srv/builds/_repro/0.3.14/rebuild.log`. +06 October 2026, 20:03 UTC on igneum-build-1 by `infra/build-server/repro/rebuild-on-box.sh` (driven by `tools/repro/rebuild-release.sh`): a clean clone of the fork at the node commit on branch `release-0.3.14-node` under a clean clone of the repo at the app commit, 2 independent clean passes per target in one target path each (no sccache, SOURCE_DATE_EPOCH 1791305478, TZ UTC), rustc 1.99.0, x86_64-w64-mingw32-gcc-posix (GCC) 13-posix, clang 18.1.3, glibc 2.39. Shipped hashes read from the public downloads (no token) where marked. Whole run 3 s; log `/srv/builds/_repro/0.3.14/rebuild.log`. | Artefact | Shipped sha256 (source) | Box pass A | Box pass B | A vs shipped | A vs B | Reason for a DIFFER | |---|---|---|---|---|---|---| -| igneumd | 934f393cacc31a06d0c45a9fe2e2f504941a32533110b51851968e70cf90fa3a (given) | 03f35e056922fa1e... (49600096 B) | 03f35e056922fa1e... (49600096 B) | **DIFFER** | **MATCH** | toolchain: the shipped binary came from the Mac's infra/cross/build-linux.sh (zig, glibc 2.36 target, docs/plans/release-0.3.14.md), the box's from the native clang/lld against glibc 2.39 (the box binary needs GLIBC_2.39); the shipped binary was not on hand this run (the public download failed), so its symbol versions were not read; commit string in the box's binary: 1 hit(s); the shipped exe was built from a worktree before the two-step clean, so by the empty-commit class it carries none (not read: no file) | -| igneum-miner | 7e296541 (given) | 900c1f0bf8a3b504... (9842168 B) | 900c1f0bf8a3b504... (9842168 B) | **DIFFER** | **MATCH** | toolchain: the shipped binary came from the Mac's infra/cross/build-linux.sh (zig, glibc 2.36 target, docs/plans/release-0.3.14.md), the box's from the native clang/lld against glibc 2.39 (the box binary needs GLIBC_2.39); the shipped binary was not on hand this run (the public download failed), so its symbol versions were not read | -| igneumd.exe | 44fa74c02415ff258b5956ef55909dc97890e3f29fca3d2db26f7152ef4ce541 (given) | 166e604e01c668e6... (51758592 B) | 166e604e01c668e6... (51758592 B) | **DIFFER** | **MATCH** | toolchain: the shipped exe came from the Mac's proto-cuda/windows-node/cross-build.sh (Homebrew mingw) at 16:52Z on 6 Oct 2026, the box's from Ubuntu GCC 13 posix with -Wl,--no-insert-timestamp (the fix landed 17:48Z, after this cut's exes, so the shipped PE header carries a build time); the shipped exe itself is not on hand (innoextract 1.9 cannot read the Inno Setup 6 installer: setup loader revision 2), so its hash is the release plan's and its PE header was not read; commit string in the box's binary: 1 hit(s); the shipped exe was built from a worktree before the two-step clean, so by the empty-commit class it carries none (not read: no file) | -| igneum-miner.exe | 819ea9ce (given) | fefd266c3bd6470f... (10994688 B) | fefd266c3bd6470f... (10994688 B) | **DIFFER** | **MATCH** | toolchain: the shipped exe came from the Mac's proto-cuda/windows-node/cross-build.sh (Homebrew mingw) at 16:52Z on 6 Oct 2026, the box's from Ubuntu GCC 13 posix with -Wl,--no-insert-timestamp (the fix landed 17:48Z, after this cut's exes, so the shipped PE header carries a build time); the shipped exe itself is not on hand (innoextract 1.9 cannot read the Inno Setup 6 installer: setup loader revision 2), so its hash is the release plan's and its PE header was not read | +| igneumd | 934f393cacc31a06d0c45a9fe2e2f504941a32533110b51851968e70cf90fa3a (given) | 03f35e056922fa1e... (49600096 B) | 03f35e056922fa1e... (49600096 B) | **DIFFER** | **MATCH** | the shipped binary was not on hand this run (no public artefact), so only the hash is compared; the box's needs GLIBC_2.39, embeds /srv/builds/_repro/0.3.14, clock string 16:51:18; commit string 4c6b129d75c3d77a3689d22f1e1dc721b556aebb: 1 hit(s) in the box's binary | +| igneum-miner | 7e296541 (given) | 900c1f0bf8a3b504... (9842168 B) | 900c1f0bf8a3b504... (9842168 B) | **DIFFER** | **MATCH** | the shipped binary was not on hand this run (no public artefact), so only the hash is compared; the box's needs GLIBC_2.39, embeds /srv/builds/_repro/0.3.14, clock string none | +| igneumd.exe | 44fa74c02415ff258b5956ef55909dc97890e3f29fca3d2db26f7152ef4ce541 (given) | 166e604e01c668e6... (51758592 B) | 166e604e01c668e6... (51758592 B) | **DIFFER** | **MATCH** | box exe: Ubuntu GCC 13 posix, -Wl,--no-insert-timestamp (PE timestamp 'Jan 1 01:00:00 1970'), build path /srv/builds/_repro/0.3.14, clock string 16:51:18; the shipped exe itself is not on hand (innoextract 1.9 cannot read the Inno Setup 6 installer: setup loader revision 2; or no public installer for this version), so its hash is the given one and its header was not read; commit string 4c6b129d75c3d77a3689d22f1e1dc721b556aebb: 1 hit(s) in the box's binary | +| igneum-miner.exe | 819ea9ce (given) | fefd266c3bd6470f... (10994688 B) | fefd266c3bd6470f... (10994688 B) | **DIFFER** | **MATCH** | box exe: Ubuntu GCC 13 posix, -Wl,--no-insert-timestamp (PE timestamp 'Jan 1 01:00:00 1970'), build path /srv/builds/_repro/0.3.14, clock string none; the shipped exe itself is not on hand (innoextract 1.9 cannot read the Inno Setup 6 installer: setup loader revision 2; or no public installer for this version), so its hash is the given one and its header was not read | Full box hashes: igneumd A 03f35e056922fa1e406e14d35963e8d3ca57f98f0d58a04c3f7cc450a26d1fdc; igneum-miner A 900c1f0bf8a3b504dfb2a7fa7abb91f3286eb0d020ed1e0aa5f3ab808c0489eb; igneumd.exe A 166e604e01c668e69b88556cad959429c67b040ec1e024cf7e514e1b5fc8edae; igneum-miner.exe A fefd266c3bd6470f61476a96453d4ea0cb54c42b8b23038cf5ef5a3397399514; diff --git a/docs/evidence/reproduced/0.3.15.md b/docs/evidence/reproduced/0.3.15.md new file mode 100644 index 00000000..6f28b653 --- /dev/null +++ b/docs/evidence/reproduced/0.3.15.md @@ -0,0 +1,14 @@ +# Reproduced: Igneum Miner 0.3.15 (node 713ef876073d3661e9b48d2ead9a515afd1b2156, app 563485b769868ee34a530f1c40f7419109cc493f) + +06 October 2026, 20:03 UTC on igneum-build-1 by `infra/build-server/repro/rebuild-on-box.sh` (driven by `tools/repro/rebuild-release.sh`): a clean clone of the fork at the node commit on branch `release-0.3.15-node` under a clean clone of the repo at the app commit, 2 independent clean passes per target in one target path each (no sccache, SOURCE_DATE_EPOCH 1791312828, TZ UTC), rustc 1.99.0, x86_64-w64-mingw32-gcc-posix (GCC) 13-posix, clang 18.1.3, glibc 2.39. Shipped hashes read from the public downloads (no token) where marked. Whole run 2 s; log `/srv/builds/_repro/0.3.15/rebuild.log`. + +| Artefact | Shipped sha256 (source) | Box pass A | Box pass B | A vs shipped | A vs B | Reason for a DIFFER | +|---|---|---|---|---|---|---| +| igneumd | 1e51bfb6401e2d86022eeaddf03750a72d6eb200fa879b7c58fb5c3afbf38a50 (igneum-hive-0.3.15.tar.gz (1715e58ea1d4a1ed...)) | 1f1b6eee4aaf4cdf... (49720224 B) | 1f1b6eee4aaf4cdf... (49720224 B) | **DIFFER** | **MATCH** | toolchain: the shipped binary needs GLIBC_2.34, the box's GLIBC_2.39 (a glibc GLIBC_2.34 build is the Mac's infra/cross/build-linux.sh with zig; the box links the native clang/lld); build path in the shipped binary: /Users/joshm/.cargo/registry, in the box's: /srv/builds/_repro/0.3.15 (prost's protowire.rs embeds OUT_DIR, so a different path is a different binary); build clock string (mimalloc's __TIME__) in the shipped binary: 11:05:57, in the box's: 18:53:48 (with SOURCE_DATE_EPOCH the string is the commit's time of day, the same in every build; none = no mimalloc in the binary); commit string 713ef876073d3661e9b48d2ead9a515afd1b2156: 1 hit(s) in the box's binary, 1 in the shipped one (0 = the empty-commit class: a worktree build before the two-step clean) | +| igneum-miner | c5b489105d933b01e4dceff8dc31e2db8e9aa53de06dddafc6eb12ee85b8f7a6 (igneum-hive-0.3.15.tar.gz (1715e58ea1d4a1ed...)) | a34e0a56c859a667... (9978968 B) | a34e0a56c859a667... (9978968 B) | **DIFFER** | **MATCH** | toolchain: the shipped binary needs GLIBC_2.34, the box's GLIBC_2.39 (a glibc GLIBC_2.34 build is the Mac's infra/cross/build-linux.sh with zig; the box links the native clang/lld); build path in the shipped binary: /Users/joshm/.cargo/registry, in the box's: /srv/builds/_repro/0.3.15 (prost's protowire.rs embeds OUT_DIR, so a different path is a different binary); build clock string (mimalloc's __TIME__) in the shipped binary: none, in the box's: none (with SOURCE_DATE_EPOCH the string is the commit's time of day, the same in every build; none = no mimalloc in the binary) | +| igneumd.exe | none | 9b377455ac9cc3b9... (51847168 B) | 9b377455ac9cc3b9... (51847168 B) | NO SHIPPED HASH | **MATCH** | | +| igneum-miner.exe | none | 65b30edd266d137f... (11129856 B) | 65b30edd266d137f... (11129856 B) | NO SHIPPED HASH | **MATCH** | | + +Full box hashes: igneumd A 1f1b6eee4aaf4cdfec07a165e8242b6d69a9fe20eb1b73f3caf672ec3ff53f91; igneum-miner A a34e0a56c859a667200600d3bca32f9ae22e33deb98cd226e8f54e40a6e354a4; igneumd.exe A 9b377455ac9cc3b90f081bd12d954e400ad1549c158ac179ccf07d16b6bf2c8e; igneum-miner.exe A 65b30edd266d137f3320d4d477e83bf73e0795ac3c697cbcf46b05e7dba0f8a7; + +Reading: MATCH against the shipped bytes is the goal; A vs B MATCH with a DIFFER against the shipped bytes means the box is deterministic and the shipped build came from another toolchain (the reason column says which facts differ); A vs B DIFFER is a non-determinism on the box itself and is the row to fix first. diff --git a/docs/plans/build-server.md b/docs/plans/build-server.md index eb610dbb..80384006 100644 --- a/docs/plans/build-server.md +++ b/docs/plans/build-server.md @@ -223,10 +223,21 @@ Two non-determinisms found on the way, both in the SHIPPED builds too (first run | OUT_DIR path in the binary | prost's generated `protowire.rs` (kaspa-grpc-core, kaspa-p2p-lib) embeds its OUT_DIR path; a pass in a target dir of another NAME differs (igneum-miner matched byte for byte once the path was the same) | one target path per target in the repro; for cross-machine identity a `--remap-path-prefix` of the target dir and the home (not done: the Mac and the box differ in every path anyway) | | Build clock in the binary | libmimalloc-sys compiles mimalloc's C with `__DATE__` and `__TIME__` ("Oct 6 2026", "21:38:15" sat in libmimalloc.a, next to the mimalloc option names); two builds a minute apart differ | `SOURCE_DATE_EPOCH` exported for every pass (GCC and clang take the date and time from it); PROPOSED for build-remote.sh, cross-remote.sh, cross-build.sh and the PC job: export it from the commit time so two builds of one commit give one hash. The earlier "byte-identical across three builds" on the box was under sccache, which returns the first build's object and hides this class | +0.3.15 as well (run 19:56 to 20:00Z, the moment it reached dl/public; node 713ef876, app 563485b; four clean passes of 63 to +76 s): A vs B **MATCH on all four** again (igneumd 1f1b6eee..., igneum-miner a34e0a56..., igneumd.exe 9b377455..., +igneum-miner.exe 65b30edd...). Against the shipped Linux pair in `igneum-hive-0.3.15.tar.gz` (igneumd 1e51bfb6..., +igneum-miner c5b48910...): DIFFER, and the binaries say why: the shipped pair needs GLIBC_2.34 and embeds +`/Users/joshm/.cargo/registry` and the clock string 11:05:57, so the HiveOS package carries the Mac's zig build, not the +box's 06211d55... of 19:00Z (the box build needs GLIBC_2.39, which HiveOS cannot run; the zig lane is right for that +package). The Windows exes have no shipped hash yet (the public installer is still 0.3.14). `docs/evidence/reproduced/0.3.15.md`. + What it means: the box is deterministic for a given commit and path, so a release built on it can be checked by anyone with the -same toolchain by rebuilding and comparing; the shipped 0.3.14 bytes cannot be reproduced anywhere because they came from -two Mac toolchains with a build clock inside, and that is the reason to ship from the box from 0.3.16 (R2) with -SOURCE_DATE_EPOCH set. Open: `rebuild-release.sh` for 0.3.15 the moment it ships (one command, 5 min). +same toolchain by rebuilding and comparing; the shipped 0.3.14 and 0.3.15 Linux bytes came from the Mac's zig lane with a +build clock inside and cannot be reproduced anywhere, and the Windows 0.3.14 exes carried a PE timestamp. The reason to ship +every target from the box from 0.3.16 (R2) is this table, with SOURCE_DATE_EPOCH exported in every build script; for HiveOS +(glibc 2.36 and under) the box needs zig + cargo-zigbuild first (section 6, row 1), or the package keeps the Mac's build and +stays unreproducible until then. Open: a `--reuse` re-report of 0.3.15 once its Windows installer is public, and +`rebuild-release.sh 0.3.16` the moment it ships. ### 7.4 The CPU prover trial (DONE 19:30Z; verdict: the box is NOT a prover) diff --git a/infra/build-server/repro/rebuild-on-box.sh b/infra/build-server/repro/rebuild-on-box.sh index 6d09b584..ff23c9ee 100755 --- a/infra/build-server/repro/rebuild-on-box.sh +++ b/infra/build-server/repro/rebuild-on-box.sh @@ -111,21 +111,34 @@ fi # 4. the evidence glibc_need() { objdump -T "$1" 2>/dev/null | grep -o 'GLIBC_[0-9.]*' | sort -uV | tail -1; } -pe_stamp() { x86_64-w64-mingw32-objdump -p "$1" 2>/dev/null | awk '/Time\/Date/ { $1=""; $2=""; print; exit }' | sed 's/^ *//'; } +# the producers run inside `{ ...; || true; }`: a consumer that stops early (grep -m1, awk exit) sends SIGPIPE to the producer, and +# under pipefail that failed the pipeline and ran the `|| echo` fallback AS WELL, which put a second line into a table cell +# (6 October 2026, 20:01Z, the 0.3.15 evidence; lib.sh met the same class with grep -q this morning) +pe_stamp() { { x86_64-w64-mingw32-objdump -p "$1" 2>/dev/null || true; } | awk '/Time\/Date/ { $1=""; $2=""; print; exit }' | sed 's/^ *//'; } commit_hits() { strings -n 7 "$1" 2>/dev/null | grep -c "$NODE_FULL" || true; } -reason() { # - local a="$1" sp="$2" r="" +build_path() { { strings -n 12 "$1" 2>/dev/null || true; } | grep -o -m1 -E '(/srv/builds/[^/ ]+/[^/ ]+|/Users/[^/ ]+/[^/ ]+/[^/ ]+)' || echo "no build path string"; } +clock_str() { { strings -n 8 "$1" 2>/dev/null || true; } | grep -m1 -E '^[0-9]{2}:[0-9]{2}:[0-9]{2}$' || echo "none"; } +reason() { # : facts read off both binaries, then what they mean + local a="$1" sp="$2" bp="${PASS_PATH[$a:A]}" r="" case "$a" in igneumd|igneum-miner) - r="toolchain: the shipped binary came from the Mac's infra/cross/build-linux.sh (zig, glibc 2.36 target, docs/plans/release-$VERSION.md), the box's from the native clang/lld against glibc 2.39 (the box binary needs $(glibc_need "${PASS_PATH[$a:A]}"))" - [ -n "$sp" ] && r="$r; the shipped binary needs $(glibc_need "$sp" 2>/dev/null || echo 'no GLIBC_ version read')" || r="$r; the shipped binary was not on hand this run (the public download failed), so its symbol versions were not read" ;; + if [ -n "$sp" ]; then + local gs gb; gs=$(glibc_need "$sp" 2>/dev/null || echo none); gb=$(glibc_need "$bp") + if [ "$gs" != "$gb" ]; then r="toolchain: the shipped binary needs $gs, the box's $gb (a glibc $gs build is the Mac's infra/cross/build-linux.sh with zig; the box links the native clang/lld)" + else r="same glibc need ($gs) on both: not a toolchain difference"; fi + r="$r; build path in the shipped binary: $(build_path "$sp"), in the box's: $(build_path "$bp") (prost's protowire.rs embeds OUT_DIR, so a different path is a different binary)" + r="$r; build clock string (mimalloc's __TIME__) in the shipped binary: $(clock_str "$sp"), in the box's: $(clock_str "$bp") (with SOURCE_DATE_EPOCH the string is the commit's time of day, the same in every build; none = no mimalloc in the binary)" + else + r="the shipped binary was not on hand this run (no public artefact), so only the hash is compared; the box's needs $(glibc_need "$bp"), embeds $(build_path "$bp"), clock string $(clock_str "$bp")" + fi ;; igneumd.exe|igneum-miner.exe) - r="toolchain: the shipped exe came from the Mac's proto-cuda/windows-node/cross-build.sh (Homebrew mingw) at 16:52Z on 6 Oct 2026, the box's from Ubuntu GCC 13 posix with -Wl,--no-insert-timestamp (the fix landed 17:48Z, after this cut's exes, so the shipped PE header carries a build time)" - [ -n "$sp" ] && r="$r; shipped PE timestamp '$(pe_stamp "$sp")' against the box's '$(pe_stamp "${PASS_PATH[$a:A]}")'" || r="$r; the shipped exe itself is not on hand (innoextract 1.9 cannot read the Inno Setup 6 installer: setup loader revision 2), so its hash is the release plan's and its PE header was not read" ;; + r="box exe: Ubuntu GCC 13 posix, -Wl,--no-insert-timestamp (PE timestamp '$(pe_stamp "$bp")'), build path $(build_path "$bp"), clock string $(clock_str "$bp")" + if [ -n "$sp" ]; then r="$r; shipped exe: PE timestamp '$(pe_stamp "$sp")', build path $(build_path "$sp"), clock string $(clock_str "$sp") (a non-zero PE timestamp = built before the --no-insert-timestamp fix of 6 Oct 2026 17:48Z; a /Users path = the Mac's Homebrew mingw cross-build.sh)" + else r="$r; the shipped exe itself is not on hand (innoextract 1.9 cannot read the Inno Setup 6 installer: setup loader revision 2; or no public installer for this version), so its hash is the given one and its header was not read"; fi ;; esac if [ "$a" = igneumd ] || [ "$a" = igneumd.exe ]; then - if [ -n "$sp" ]; then r="$r; commit string $NODE_FULL in the shipped binary: $(commit_hits "$sp") hit(s), in the box's: $(commit_hits "${PASS_PATH[$a:A]}") (0 in the shipped one = the empty-commit class)" - else r="$r; commit string in the box's binary: $(commit_hits "${PASS_PATH[$a:A]}") hit(s); the shipped exe was built from a worktree before the two-step clean, so by the empty-commit class it carries none (not read: no file)"; fi + r="$r; commit string $NODE_FULL: $(commit_hits "$bp") hit(s) in the box's binary" + [ -n "$sp" ] && r="$r, $(commit_hits "$sp") in the shipped one (0 = the empty-commit class: a worktree build before the two-step clean)" fi echo "$r" }