From 6bed846b5f134865c60352f20e9a14c505f83e80 Mon Sep 17 00:00:00 2001 From: igneum-josh <337424239+igneum-josh@users.noreply.github.com> Date: Wed, 7 Oct 2026 15:51:30 +0100 Subject: [PATCH 01/15] Counter ASIC 3.0 status: the suite at 017e7037 (103 of 103; the lib tests' 2^20 cost on CI time) Co-Authored-By: Claude Fable 5.1 --- docs/plans/counter-asic-3-status.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plans/counter-asic-3-status.md b/docs/plans/counter-asic-3-status.md index 76c3ccf73..1da4012c2 100644 --- a/docs/plans/counter-asic-3-status.md +++ b/docs/plans/counter-asic-3-status.md @@ -421,7 +421,7 @@ Every failing seed is ONE low-entropy load site; the mechanism is lineage-blind | Margin | 0.015 to each side | | Cost per chosen candidate | one 2^20 pass, 2.1 to 2.2 s on one box-2 core, once an hour on a node | -Known-failed test class_v4_distinct_ratio_rejects_the_low_entropy_band green on box 2 (12.95 s): p15 attempt 3 (52638ea2e8b0fd68) site 2 at 0.943; p18 attempt 2 (9a37e9489d8ba698) site 6 at 0.927; p19 attempt 0 (79d7441de0689223) site 15 at 0.933; p56 attempt 2 (486a8ad2701ec3b5) site 2 at 0.965; p23 attempt 1 (d65122675f16a1c7) draws site 7 from r5 and passes, and with r6 put back is refused by (a') UnfreshLoadSource and, run anyway, by the ratio at 0.836. Stream unchanged from ddacfbd3 (re-export diff 0 on all eight packs): id a785001687d8688a, the seven fingerprints and the packs zip sha256 4f2445c5... as recorded. In flight on box 2: the crate suite and the 4,096 + F8 census at 017e7037; then the attack-pass lane's 64-seed gate at 2^24 and 10^6 exhaustion count. Owed as before: G2, G3, the ladder re-measure, AMD (PC 1), the 2019-class core, the fleet and PC 2 G1 on the sub-version 3 packs (a re-run of a known result). 0.3.21's STAGING (the node lane): the order dry-merges onto 55768f88 with nothing moving to 0.3.22; the late-join fix is 52e96c94 (70e4601e rebased onto 55768f88, exec suite 33 green with both new tests); f067f7c1, b0444f51 and 437f0438 merge clean in order; 2e32d5f6's one conflict (DST_ADDRESS beside pool-finish's DST_BINDING in consensus/core/src/finality.rs) kept both; the live-file digest eada4bda after each (every switch at never); the staging waits on the shipper's sweep-end word; the re-pin held. PC 2 DOWN AGAIN (main, 16:5x UK): Josh takes PC 2 down for cable work (PC 1 back but his desk); both PCs out of the sweep's waves, each updates on its poller on return; no PC job to PC 1; the Windows G1 completed before the outage, nothing reruns. 0.3.21's SECOND GATE LINE on 55768f88 (sha256 279b1b690e854fc9): the ten-minute mixed-version gate beside the 5899f603 pair, 13:37:40Z to 13:47:52Z, SUMMARY PASS (one digest b0afb2ee on five nodes; 223 new and 381 old blocks accepted by the old hub, 0 rejected; counts equal at 319, 486 and 604 through both clean joins and the restart step at 13:45:22Z; no panic); the node lane's two lines on 0.3.21's first candidate complete, in plan 6.9 on ca3-v4-node; the fleet's set on it (the bare-child 12 GB line, the wipe, the kept read, the cases) is the fleet's. 0.3.21's FIRST GATE LINE on 55768f88 (sha256 279b1b690e854fc9, the string read back; pairing igneum-pow 8c728ca3 at byte 5): the digest gate 13:35:41Z to 13:37:19Z SUMMARY PASS (a89be8a7 on both binaries with the peers; db9a85f9 refused, no peer; the live file's eada4bda unmoved); the ten-minute mixed-version gate from 13:37:40Z, line about 13:50Z. The 0.3.21 order as the shipper sent it: 55768f88; f067f7c1 and 70e4601e; b0444f51; 6eb21fc9; db28d331; then the re-pin from 8bdcbdd8 on the coordinator's word; suites between, the digest read after every one; the mirror's release-0.3.20-node back at the pin c4459193, release-0.3.21-node open at 55768f88. THE LATE-JOIN COMMIT (N9's second half, the node lane): 70e4601e on the box mirror as branch proof-hold-fix, from c4459193, two files (igneum/exec/src/proving.rs, protocol/flows/src/v10/proving.rs); the gap was the fetch side on the joiner (the served record ran the native check against the joiner's trailing exec state before anything was stored, the check refused it, the proof was never held, the body rule read "not held" for 20 s and failed the IBD); the fix holds the proof by hash before the checks (the pool entry still needs them) and the serve side says when it holds fewer than asked; the exec suite 32 passed at 13:26Z with the known-failed shape first, the flows check green 13:28Z, igneumd on build-1 at the 0321 worktree path built 13:32Z, sha256 17649eeb2f7d1290, string read back; with the testnet lane (the resume form, B alone); it joins the 0.3.21 staging as its own commit. THE WIPE CANARY ON c19-1, c4459193 (sha 45be9b02d1b002f5, string read back): FORM END rc 0 at 13:50:53Z. Wipe synced 13:35:50Z (57 minutes, inside the 98-minute class); mining 13:36:00Z to 13:47:07Z, 66 mined, 66 accepted, 0 rejected, isSynced true at the tip throughout; the hub holds 41 of its blocks in its last 700 with 0 rejects (13:47:09Z); the restart on its kept datadir at 13:47:15Z: the old process stopped at once (the new process's first lock line seven seconds after the marker; the watchdog held nothing, the b7cc37e7 fault closed), synced again at 13:48:39Z after 84 s, 109 templates read with max 3,432 ms and 0 timeouts; the kept read on pool-1's 0.3.17 copy on the same pod passed at 13:38Z (the rewrite line once, a clean second start). The pin's set on c4459193: the digest gate PASS, the mixed-version gate PASS, the wipe canary PASS, the kept read PASS, the restart PASS, the 12 GB line proves and verifies (paid is a race, not a gate); CASES END from c20-1 (about 14:50Z) is the last pin line. THE INTEROP FACT stands from the void run: the 5899f603 hub accepted 235 object-byte-5 blocks from the 8097d600 node with 0 rejected, one digest on all five nodes on the live sixteen-field file. The gates: the digest test and the kaspa-pow vector test (the amended devnet epoch-0 id 1a4230699a6b9c60 must equal, c120d7963abdcd96 must differ, the v3 control unchanged) on the box; the mixed-version Devnet 2 gate (the amended 0.3.20 node beside a 5899f603 node for ten minutes on the live file without the v4 fields) after the Mac build; the fresh-join canary the 0.3.20 cut's | +Known-failed test class_v4_distinct_ratio_rejects_the_low_entropy_band green on box 2 (12.95 s): p15 attempt 3 (52638ea2e8b0fd68) site 2 at 0.943; p18 attempt 2 (9a37e9489d8ba698) site 6 at 0.927; p19 attempt 0 (79d7441de0689223) site 15 at 0.933; p56 attempt 2 (486a8ad2701ec3b5) site 2 at 0.965; p23 attempt 1 (d65122675f16a1c7) draws site 7 from r5 and passes, and with r6 put back is refused by (a') UnfreshLoadSource and, run anyway, by the ratio at 0.836. Stream unchanged from ddacfbd3 (re-export diff 0 on all eight packs): id a785001687d8688a, the seven fingerprints and the packs zip sha256 4f2445c5... as recorded. THE SUITE AT 017e7037 on box 2: 103 of 103 (64 lib + 7 derive + 4 mixer + 19 packs + 2 recheck + 7 scratch, 3 diag ignored), rc 0; the lib tests 140.8 s against the 41 s whole-suite line at ddacfbd3, because every class v4 draw in the tests pays the 2^20 ratio pass on its chosen candidate (2.2 s each): a CI-time cost, not a node cost (one pass per epoch draw). In flight on box 2: the 4,096 + F8 census at 017e7037; then the attack-pass lane's 64-seed gate at 2^24 and 10^6 exhaustion count. Owed as before: G2, G3, the ladder re-measure, AMD (PC 1), the 2019-class core, the fleet and PC 2 G1 on the sub-version 3 packs (a re-run of a known result). 0.3.21's STAGING (the node lane): the order dry-merges onto 55768f88 with nothing moving to 0.3.22; the late-join fix is 52e96c94 (70e4601e rebased onto 55768f88, exec suite 33 green with both new tests); f067f7c1, b0444f51 and 437f0438 merge clean in order; 2e32d5f6's one conflict (DST_ADDRESS beside pool-finish's DST_BINDING in consensus/core/src/finality.rs) kept both; the live-file digest eada4bda after each (every switch at never); the staging waits on the shipper's sweep-end word; the re-pin held. PC 2 DOWN AGAIN (main, 16:5x UK): Josh takes PC 2 down for cable work (PC 1 back but his desk); both PCs out of the sweep's waves, each updates on its poller on return; no PC job to PC 1; the Windows G1 completed before the outage, nothing reruns. 0.3.21's SECOND GATE LINE on 55768f88 (sha256 279b1b690e854fc9): the ten-minute mixed-version gate beside the 5899f603 pair, 13:37:40Z to 13:47:52Z, SUMMARY PASS (one digest b0afb2ee on five nodes; 223 new and 381 old blocks accepted by the old hub, 0 rejected; counts equal at 319, 486 and 604 through both clean joins and the restart step at 13:45:22Z; no panic); the node lane's two lines on 0.3.21's first candidate complete, in plan 6.9 on ca3-v4-node; the fleet's set on it (the bare-child 12 GB line, the wipe, the kept read, the cases) is the fleet's. 0.3.21's FIRST GATE LINE on 55768f88 (sha256 279b1b690e854fc9, the string read back; pairing igneum-pow 8c728ca3 at byte 5): the digest gate 13:35:41Z to 13:37:19Z SUMMARY PASS (a89be8a7 on both binaries with the peers; db9a85f9 refused, no peer; the live file's eada4bda unmoved); the ten-minute mixed-version gate from 13:37:40Z, line about 13:50Z. The 0.3.21 order as the shipper sent it: 55768f88; f067f7c1 and 70e4601e; b0444f51; 6eb21fc9; db28d331; then the re-pin from 8bdcbdd8 on the coordinator's word; suites between, the digest read after every one; the mirror's release-0.3.20-node back at the pin c4459193, release-0.3.21-node open at 55768f88. THE LATE-JOIN COMMIT (N9's second half, the node lane): 70e4601e on the box mirror as branch proof-hold-fix, from c4459193, two files (igneum/exec/src/proving.rs, protocol/flows/src/v10/proving.rs); the gap was the fetch side on the joiner (the served record ran the native check against the joiner's trailing exec state before anything was stored, the check refused it, the proof was never held, the body rule read "not held" for 20 s and failed the IBD); the fix holds the proof by hash before the checks (the pool entry still needs them) and the serve side says when it holds fewer than asked; the exec suite 32 passed at 13:26Z with the known-failed shape first, the flows check green 13:28Z, igneumd on build-1 at the 0321 worktree path built 13:32Z, sha256 17649eeb2f7d1290, string read back; with the testnet lane (the resume form, B alone); it joins the 0.3.21 staging as its own commit. THE WIPE CANARY ON c19-1, c4459193 (sha 45be9b02d1b002f5, string read back): FORM END rc 0 at 13:50:53Z. Wipe synced 13:35:50Z (57 minutes, inside the 98-minute class); mining 13:36:00Z to 13:47:07Z, 66 mined, 66 accepted, 0 rejected, isSynced true at the tip throughout; the hub holds 41 of its blocks in its last 700 with 0 rejects (13:47:09Z); the restart on its kept datadir at 13:47:15Z: the old process stopped at once (the new process's first lock line seven seconds after the marker; the watchdog held nothing, the b7cc37e7 fault closed), synced again at 13:48:39Z after 84 s, 109 templates read with max 3,432 ms and 0 timeouts; the kept read on pool-1's 0.3.17 copy on the same pod passed at 13:38Z (the rewrite line once, a clean second start). The pin's set on c4459193: the digest gate PASS, the mixed-version gate PASS, the wipe canary PASS, the kept read PASS, the restart PASS, the 12 GB line proves and verifies (paid is a race, not a gate); CASES END from c20-1 (about 14:50Z) is the last pin line. THE INTEROP FACT stands from the void run: the 5899f603 hub accepted 235 object-byte-5 blocks from the 8097d600 node with 0 rejected, one digest on all five nodes on the live sixteen-field file. The gates: the digest test and the kaspa-pow vector test (the amended devnet epoch-0 id 1a4230699a6b9c60 must equal, c120d7963abdcd96 must differ, the v3 control unchanged) on the box; the mixed-version Devnet 2 gate (the amended 0.3.20 node beside a 5899f603 node for ten minutes on the live file without the v4 fields) after the Mac build; the fresh-join canary the 0.3.20 cut's | | Main's rulings (7 October, morning) | no generator change to v4 on the live devnet; the record's null is the window model with numbers, sent by the hash lane to the attack-pass lane so AP-F8-1 re-gates against it; a fault beyond the model (a low-entropy source at site 15) stops at the coordinator with the two options priced (a 0.3.19 class amendment before the flip, or the flip held at the floor), nothing shipping without Josh's word; the tighter tail, an acceptance bound on the hot-set share, is a CLASS V5 item (sent to the v5 lane a6410f3b8abefb762 with the 64-seed census as its gate; the bound's number follows from the model) | ### AP-F4-1, the weak-day MUL draw (the attack-pass lane, 7 October, morning): PASS against v4, a class v5 rule From c9d161e2edb2bc7ec90873d0c90bae937a138df4 Mon Sep 17 00:00:00 2001 From: igneum-josh <337424239+igneum-josh@users.noreply.github.com> Date: Wed, 7 Oct 2026 16:05:47 +0100 Subject: [PATCH 02/15] RTX 5060 Ti (16 GB, Razer Core X V2 Thunderbolt) on PC 2: detection, G1 bit-exact on class v4 sub-version 1 and the v3 control, the 10-minute stock window, the bench-log entry, the card-picker entry and the public table row The first 16 GB card and the first Thunderbolt card. The app's first poll listed it (vendor nvidia, 16,311 MiB, driver 610.47 shared with the 5090, kind discrete: the app does not know it is an eGPU) and started a miner on it by itself. Job run-5060ti-bench-20261007-b (tools/bench-5060ti/pc2-5060ti-bench.ps1, the card alone under the runner's --cards-off, the 5090 mining, no power limit written: 180 W default read back unchanged, PC 2 lost power twice that day): G1 PASS with 867dbc45cfb36b4d (sub-version 1) and 90f794dd556f7a3b equal to the Mac's; the window 1,105 dispatches of 2^24 in 602 s at 30.882 MH/s, 114.8 W mean at utilisation 90 percent and over, 0.269 MH/W, PCIe 4.0 x4 through the enclosure (the hash is latency-bound, the link costs nothing measurable). Owed: the efficient point (the app's Ember Tune; PC 2's Power Helper refuses every request since the restart) and the prove-beside read (the shipped WSL2 host has no IGNEUM_CUDA_DEVICE selector; the floor server is on PC 2; the prover-floor host is the cut). The kit (tools/bench-5060ti/make-kit.sh) takes the sub-version 1 packs from ca3-v4-amend a0aaca92, since master's packs-ca3-v4 is the 6 October stream. Co-Authored-By: Claude Fable 5.1 --- docs/bench-log.md | 29 +++ site/miner-bench.json | 11 + site/yourcard.js | 2 +- tools/bench-5060ti/README.md | 32 +++ tools/bench-5060ti/make-kit.sh | 34 +++ tools/bench-5060ti/pc2-5060ti-bench.ps1 | 289 ++++++++++++++++++++++++ 6 files changed, 396 insertions(+), 1 deletion(-) create mode 100644 tools/bench-5060ti/README.md create mode 100755 tools/bench-5060ti/make-kit.sh create mode 100644 tools/bench-5060ti/pc2-5060ti-bench.ps1 diff --git a/docs/bench-log.md b/docs/bench-log.md index 267e2fff3..11a9f5cc8 100644 --- a/docs/bench-log.md +++ b/docs/bench-log.md @@ -2652,3 +2652,32 @@ in the same shape and reports a box behind its wanted binary. | Rig | the same, and a rig that leaves is itself a weight removal: at 459 MH/s on tonight's devnet it is about 20 percent of the weight, over the hour's budget by itself | | Pool | a pool node is one voter carrying its members' whole weight; a pool restart is the largest single removal on the network and must be sliced like the fleet's | | The network | finality by miner weight is only as steady as the miners' uptime; until public hash dwarfs the fleet, the fleet's supervisor is a consensus component | + +## 7 October 2026, the first 16 GB card: an RTX 5060 Ti in a Thunderbolt enclosure on PC 2 (branch bench-5060ti) + +Machine: PC 2 (`1ccfe586`, Windows 11), an ASUS Dual GeForce RTX 5060 Ti (16 GB GDDR7, Blackwell sm_120, PnP `PCI\VEN_10DE&DEV_2D04&SUBSYS_8A111043`) in a Razer Core X V2 Thunderbolt enclosure ("USB4 Router (2.0), Razer - Core X V2", bus `0B:00.0`), beside the RTX 5090 on its own supply; NVIDIA driver 610.47 (WDDM 32.0.16.1047, the 5090's driver, nothing installed for the new card); the installed app 0.3.19 and its own `igneum-worker-cuda.exe` (NVRTC 12.8). Jobs `fetch-5060ti-packs-20261007` (the kit: `tools/bench-5060ti/make-kit.sh`, the class v4 pack at sub-version 1 and the v3 control, sha256 `fd8393ed...`, 105,892 bytes) and `run-5060ti-bench-20261007-b` (`tools/bench-5060ti/pc2-5060ti-bench.ps1`, 14:44:19Z, ran 14:45:05 to 14:58:11Z, exit 0; the 5060 Ti alone through the runner's `--cards-off`, the 5090 mining throughout; run `-a` died in 1 s on an argument-binding fault in the nvidia-smi query and is void). PC 2 lost power twice that day, so the job WRITES NO POWER LIMIT: it reads `power.limit` against `power.default_limit` (180 W = 180 W, range 150 to 198 W) and the row says `limit_is_stock=yes`. Read back with `node tools/jobs.mjs run-5060ti-bench-20261007-b`. + +**Detection** (the app's first poll after the restart, run `win-1ccfe586-20261007-143611`, 14:36:16Z): `GPUs: NVIDIA GeForce RTX 5090 (CUDA); NVIDIA GeForce RTX 5060 Ti (CUDA); AMD Radeon(TM) Graphics (OpenCL, gfx1036)` and `cards: NVIDIA GeForce RTX 5090 [discrete, off] | NVIDIA GeForce RTX 5060 Ti [discrete, off] | ...`; the app started a miner on it by itself (`nvidia-1ccfe586-2`, `--device 1`, 8 identities, the default). The kind reads `discrete`, not `external`: the app does not know it is an eGPU. nvidia-smi in the job: index 1, 16,311 MiB, PCIe link gen 4 x4 current against gen 4 x16 maximum (the Thunderbolt link: a quarter of the slot's lanes), 43 C idle. The freeze lane's cause class for the 15:10 UK hang on the first boot with the card: not the card (no TDR, no Thunderbolt or PCIe link event; Kernel-Power 41 + 6008, no bugcheck, the power shape again). + +**G1 and the window** (the installed CUDA worker, `--bench --batch-log2 24 --block-warps 1`, the card alone, `CUDA_VISIBLE_DEVICES` on its UUID so every row names the device; nvidia-smi every 2 s on the card, the loaded samples at utilisation 90 percent and over): + +| Pack | Dispatches of 2^24 | Self-test | Fingerprint 2^24 at base 0 | MH/s | +|---|---|---|---|---| +| mx8-devnet-epoch0 (the class v3 control) | 5 | PASS | 90f794dd556f7a3b (= the control everywhere) | 30.895 | +| v4-devnet-epoch0 (class v4, sub-version 1, program id 1a4230699a6b9c60) | 5 | PASS | 867dbc45cfb36b4d (= Metal, Apple OpenCL, the RTX 5090) | 30.879 | +| v4-devnet-epoch0, the 10-minute window at the stock limit | 1,105 (602 s) | PASS | 867dbc45cfb36b4d | 30.882 | + +| Row | Value | +|---|---| +| NVIDIA RTX 5060 Ti 16 GB, class v4, CUDA (NVRTC), driver 610.47, PCIe 4.0 x4 through the enclosure | 30.9 MH/s over 10 minutes on the card alone | +| Watts at the stock limit (180 W default, unchanged) | 114.8 W mean, 115 W p50 over the window; 0.269 MH/W; SM 2,753 MHz, memory 13,801 MHz, 60 C maximum | +| The class v4 shadow against the control | 0.1 percent (the 5090 paid 0.2, the 9070 XT 3, the B580 0.1) | +| The efficient point | OWED to the app's Ember Tune: nothing set by the job; PC 2's Power Helper refused every request since the restart ("the helper did not run sequence 0 within 15 s", 14:39Z), so no ladder ran on either card | +| Prove beside the miner (16 GB tier) | BLOCKED, not measured: the shipped WSL2 host (sha `71bc2438...`) carries no `IGNEUM_CUDA_DEVICE` selector, so aimed at anything it proves on CUDA device 0 (the 5090) through the app's own socket `/tmp/sp1-cuda-0.sock`; the selector lives in the prover-floor host (`proof_system.rs`, branch prover-floor) and is the owed cut. The job's inventory: the floor server IS on PC 2 (`/opt/igneum-floor/bin/sp1-gpu-server`, 6.8.1 build `e911facb...`, 166,665,880 bytes) beside the stock one (`~/.sp1/bin`, `c2642ad1...`), WSL sees the card as CUDA device 1 | +| Card-picker entry (`site/yourcard.js`) | `['NVIDIA RTX 5060 Ti', 30.9]`, added; the public table row in `site/miner-bench.json` | + +Against the 5090 on the same PC (122 MH/s at 308 W, 0.396 MH/W): 25.3 percent of its hash at 37 percent of its draw, 68 percent of its hash per watt. The dependent-read ceiling was not probed (the memprobe step is not in this job); at 128 loads a hash 30.9 MH/s is 3.95 G dependent reads a second, between the 9070 XT (2.4 to 2.7 G) and the 5090 (16 to 18 G). + +Consequences per tier (the rule of 5 October 2026): a 5060 Ti owner (16 GB, Windows) mines at 30.9 MH/s and 115 W from the box with nothing to set: about 5,100 blocks a day at the 522 MH/s the devnet showed at 14:44Z (one every 17 s, approximate: the network rate moves), about a quarter of a 5090 owner's 20,200, for 2.76 kWh a day (£0.79 at 28.5 p against the 5090's £2.11); through a Thunderbolt enclosure the x4 link costs nothing measurable (the hash is bound by the card's own memory latency, not the link; the 5090's PCIe-slot rows are the comparison), so a laptop with a Thunderbolt 4 port and this enclosure is a 31 MH/s miner. The 8 GB 5060 Ti: the same hash is the expectation (the 1 GiB dataset fits), a line owed. Proving on the 16 GB tier: the fleet's 4060 Ti 16 GB row (9.0 GB peak beside the miner on the patched server) says this card would mine and prove with about 7 GB spare, approximate until the host with the device selector ships; today the app's prover default leaves it off ("a full shard needs a 24 GB card") and the measured read is owed to the prover-floor host cut. Linux and HiveOS take the same CUDA worker (owed a line). What the lane does next: the prover-floor host's selector into the shipped WSL2 bundle, then the prove-beside read on this card; the Power Helper fault on PC 2 to the Ember lane (no efficient point on any PC 2 card until it answers). + +Found on the way: inside a PowerShell `@( ... )` the comma binds before `+`, so `'--query-gpu=' + $f, '--format=csv'` is one argument (run a, void in 1 s; the query string is built first now); a bare string inside a function that also returns a value is swallowed into the caller's variable (the sampler line; `[Console]::Out.WriteLine` now); the app's `kind` for a Thunderbolt card reads `discrete` (a word for the Cards page to earn: `external`, which the state already names). diff --git a/site/miner-bench.json b/site/miner-bench.json index 4256e0100..1a1cfa40d 100644 --- a/site/miner-bench.json +++ b/site/miner-bench.json @@ -66,6 +66,17 @@ "source": "bench log: 4 October 2026, first outside machine on the devnet: an Apple silicon laptop through the Igneum Miner app", "by": "reported by the fleet", "note": "21.0 MH/s average over 7 minutes, 24.3 MH/s at the moment of the report, 33 accepted blocks" + }, + { + "card": "NVIDIA RTX 5060 Ti (16 GB)", + "generator": "v2", + "mh_s": 30.9, + "mh_per_w": 0.269, + "miner": "Igneum Miner 0.3.19 package, prebuilt NVRTC worker (bench mode, class v4 program)", + "date": "2026-10-07", + "source": "bench log: 7 October 2026, the first 16 GB card: an RTX 5060 Ti in a Thunderbolt enclosure (run b)", + "by": "measured by the team", + "note": "class v4 program, 128 loads per hash, 1 GiB dataset, 10-minute window on the card alone at the stock 180 W limit: 114.8 W mean; PCIe 4.0 x4 through the enclosure" } ] } diff --git a/site/yourcard.js b/site/yourcard.js index fb2f80b7b..4ad14adb7 100644 --- a/site/yourcard.js +++ b/site/yourcard.js @@ -2,7 +2,7 @@ second is the target, so the wait is network hashes per second over the card's rate. The rates are the bench table's measured numbers (site/miners.html); nothing here is a promise. Reads /api/live every 15 s. */ (function () { - var CARDS = [['NVIDIA RTX 5090', 127.7], ['NVIDIA RTX 4070', 28.8], ['Apple M5 Max', 26.7], ['AMD RX 9070 XT', 17.8]]; + var CARDS = [['NVIDIA RTX 5090', 127.7], ['NVIDIA RTX 5060 Ti', 30.9], ['NVIDIA RTX 4070', 28.8], ['Apple M5 Max', 26.7], ['AMD RX 9070 XT', 17.8]]; var els = document.querySelectorAll('[data-yourcard]'); if (!els.length) return; var net = null, chosen = (function () { try { return localStorage.getItem('igneum-card'); } catch (e) { return null; } })(); function dur(sec) { if (!(sec > 0)) return 'pending'; if (sec < 90) return Math.round(sec) + ' s'; if (sec < 5400) return Math.round(sec / 60) + ' min'; if (sec < 172800) { var h = Math.floor(sec / 3600), m = Math.round((sec % 3600) / 60); return h + ' h' + (m ? ' ' + m + ' min' : ''); } return Math.round(sec / 86400) + ' days'; } diff --git a/tools/bench-5060ti/README.md b/tools/bench-5060ti/README.md new file mode 100644 index 000000000..ec25c2cd6 --- /dev/null +++ b/tools/bench-5060ti/README.md @@ -0,0 +1,32 @@ +# The RTX 5060 Ti in a Razer Core X V2 on PC 2: the jobs (7 October 2026, branch bench-5060ti) + +The card: an ASUS Dual GeForce RTX 5060 Ti (16 GB, PnP `PCI\VEN_10DE&DEV_2D04&SUBSYS_8A111043`) in a Razer Core X V2 Thunderbolt enclosure ("USB4 Router (2.0), Razer - Core X V2") on PC 2 (`1ccfe586`, DESKTOP-KMCV30N), beside the RTX 5090 on its own supply. PC 2 lost power twice on 7 October, so NOTHING here writes a power limit: the playbook reads `power.limit` against `power.default_limit` and says whether they were equal; it carries no `nvidia-smi -pl`, `-lgc` or `-lmc`. Every job is a signed `run` job through `packaging/ota/publish-jobs.sh` from the main checkout (its signer binary lives there), published under the PC 2 mkdir lock (`/tmp/igneum-devnet/pc2-ca3.lock`, a holder note inside, removed when the bench ends), with the RUNNER's `--cards-off` for the 5060 Ti alone (the 5090 keeps mining), never `--stop-miners`, never pausing, resuming or quitting the installed app (`tools/ci/playbook-quit-check.sh`). + +## 1. Detection (the intake, no job needed) + +The app's first poll after the restart that followed the 15:10 UK freeze (run `win-1ccfe586-20261007-143611`, 14:36:16Z): `GPUs: NVIDIA GeForce RTX 5090 (CUDA); NVIDIA GeForce RTX 5060 Ti (CUDA); AMD Radeon(TM) Graphics (OpenCL, gfx1036)` and `cards: NVIDIA GeForce RTX 5090 [discrete, off] | NVIDIA GeForce RTX 5060 Ti [discrete, off] | AMD Radeon(TM) Graphics [integrated, off]`; the app then started a miner on it (`nvidia-1ccfe586-2`, `--device 1`, 8 identities, the default). The kind reads `discrete`, not `external`: the app does not know it is an eGPU. The freeze lane's `pc2-freeze-check-20261007` (14:38:22Z): nvidia-smi index 1, driver 610.47 (WDDM 32.0.16.1047, the 5090's driver, nothing to install), 16,311 MiB, bus `0B:00.0`, status OK; the freeze was NOT the card (no TDR, no Thunderbolt or PCIe link event; Kernel-Power 41 + 6008, no bugcheck: the power shape again). + +## 2. The kit and the bench + +The kit (packs only; the INSTALLED `igneum-worker-cuda.exe` is the worker under test): the class v4 pack `v4-devnet-epoch0` at SUB-VERSION 1 (the stream 0.3.20 ships at object byte 5, program id 1a4230699a6b9c60, fingerprint 867dbc45cfb36b4d) and the v3 control `mx8-devnet-epoch0` (90f794dd556f7a3b), both as commit a0aaca92 on ca3-v4-amend exported them (master's `proto-cuda/packs-ca3-v4` is the older 6 October stream, f410c731b6bc2d31, not the one to measure against). + + tools/bench-5060ti/make-kit.sh "$TMPDIR/igneum-5060ti-packs.zip" # prints sha256 and bytes; checks the vectors prefix 756301bf + packaging/ota/publish-jobs.sh add --kind fetch --target 1ccfe586 --id fetch-5060ti-packs-20261007 \ + --file "$TMPDIR/igneum-5060ti-packs.zip" --dir jobs --extract --title "RTX 5060 Ti bench packs (class v4 sub-version 1 and the v3 control)" --deploy + +Published 14:40:24Z: sha256 `fd8393ed37c4f311b4e34b131440f9e606e0557b7006e12d1696686753d3ba71`, 105,892 bytes; landed 14:41:11Z, sha256 ok (an app update wipes the jobs folder: republish under a new id after one). + +Then the bench, the live network rate and the kit id substituted: + + NET=$(curl -s "https://igneum.network/api/live?window=30" | python3 -c "import json,sys;print(round(json.load(sys.stdin)['state']['hashes_per_second_estimate']/1e6,1))") + sed -e "s/__NETWORK_MHS__/$NET/" -e "s/__KIT_ID__/fetch-5060ti-packs-20261007/" tools/bench-5060ti/pc2-5060ti-bench.ps1 > "$TMPDIR/pc2-5060ti-bench.ps1" + packaging/ota/publish-jobs.sh add --kind run --target 1ccfe586 --id run-5060ti-bench- \ + --cards-off "nvidia:NVIDIA GeForce RTX 5060 Ti,nvidia:1:NVIDIA GeForce RTX 5060 Ti" \ + --script "$TMPDIR/pc2-5060ti-bench.ps1" --timeout-minutes 45 --title "RTX 5060 Ti (Razer Core X V2) on PC 2: ..." --deploy + +What it prints (`node tools/jobs.mjs `): `RESULT gpu ...` (every card's nvidia-smi row: driver, VRAM, the PCIe gen and width = the Thunderbolt link, power.limit against power.default_limit), `RESULT DETECT ...`, `RESULT app_card ...` (read only), `RESULT card ... quiet` (no compute process on the card's UUID: the runner's `--cards-off` took), `RESULT G1 pack=... fingerprint= match=yes|no` for both packs (the worker sees the 5060 Ti alone through `CUDA_VISIBLE_DEVICES` on its UUID and every row names the device it ran on; a row on another card is void), `RESULT sampler stock ...` (nvidia-smi every 2 s on the card over the 10-minute window: mean, p50 and max watts at utilisation 90 percent and over, clocks, temperature), `RESULT bench v4-stock-window ...` (the worker's `--bench` with enough 2^24 dispatches for 600 s: the MH/s is the mean over them), `RESULT tune ...` (the app's own TUNE lines for the card, read only: the efficient point is Ember Tune's to find, its ladder only steps DOWN), `RESULT PROVE ...` (16 GB only: the app's WSL2 host aimed at the 5060 Ti with `IGNEUM_CUDA_DEVICE` on one live shard while a bench loop hashes on the card, VRAM peak from the sampler; BLOCKED with the reason when the shipped host carries no selector, since aiming the stock host proves on CUDA device 0, the 5090, through the app's own socket), then `RESULT ROW ...`, `RESULT PICKER ['NVIDIA RTX 5060 Ti', ]`, `RESULT CONSEQUENCES ...` and a `SUMMARY {json}` line. + +| Run | Published | Outcome | +|---|---|---| +| `run-5060ti-bench-20261007` | 14:42:27Z | VOID in 1 s: the nvidia-smi query and its format flag were one argument (inside `@( ... )` the comma binds before `+`), "temperature.gpu --format=csv is not a valid field"; the query string is built first now. Run b's sampler line was swallowed the same way the Intel lane's Bench lines were (a bare string in a function that returns a value goes into the caller's variable); `[Console]::Out.WriteLine` now | +| `run-5060ti-bench-20261007-b` | 14:44:19Z | ran 14:45:05 to 14:58:11Z, exit 0: the card alone from 0 s, G1 PASS with 867dbc45cfb36b4d and 90f794dd556f7a3b equal to the Mac's (30.879 and 30.895 MH/s at five dispatches), the window 1,105 dispatches in 602 s at 30.882 MH/s and 114.8 W mean (p50 115) at the stock 180 W limit, 0.269 MH/W, SM 2,753 MHz, 60 C; no TUNE line for the card; the prove read BLOCKED (the shipped host has no `IGNEUM_CUDA_DEVICE` selector; the floor server is on PC 2 at `/opt/igneum-floor/bin`, WSL device 1); the bench-log entry, the picker entry and the table row | diff --git a/tools/bench-5060ti/make-kit.sh b/tools/bench-5060ti/make-kit.sh new file mode 100755 index 000000000..41e489fb6 --- /dev/null +++ b/tools/bench-5060ti/make-kit.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# The packs kit for tools/bench-5060ti/pc2-5060ti-bench.ps1 (7 October 2026, branch bench-5060ti): the class v4 pack +# v4-devnet-epoch0 at SUB-VERSION 1 (generator 4, program id 1a4230699a6b9c60, the stream 0.3.20 ships at object byte 5; +# 2^24 fingerprint 867dbc45cfb36b4d, Metal = Apple OpenCL = the RTX 5090 on PC 2, job run-ca3-v4-amend-g1-pc2-20261007) +# and the class v3 control mx8-devnet-epoch0 (73bcbfe8ccf988f1, fingerprint 90f794dd556f7a3b), both as commit a0aaca92 +# on ca3-v4-amend exported them (the branch moved on to sub-versions 2 and 3; master's proto-cuda/packs-ca3-v4 is the +# 6 October stream c120d7963abdcd96, fingerprint f410c731b6bc2d31, which is NOT the one the bench measures against). +# Packs only: the INSTALLED igneum-worker-cuda.exe on PC 2 is the worker under test, so no exe travels. +# tools/bench-5060ti/make-kit.sh [out.zip] default $TMPDIR/igneum-5060ti-packs.zip +# Prints the zip's sha256 and size: the README's fetch line takes them. Checks the vectors.json sha256 prefixes the +# ledger recorded for sub-version 1 (756301bf for the devnet epoch 0 pack) before packing. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +ROOT="$(cd "$HERE/../.." && pwd)" +OUT="${1:-${TMPDIR:-/tmp}/igneum-5060ti-packs.zip}" +COMMIT=a0aaca92 +STAGE="$(mktemp -d)" +git -C "$ROOT" cat-file -e "$COMMIT^{commit}" 2>/dev/null || { echo "commit $COMMIT (ca3-v4-amend, the sub-version 1 packs) is not in this repository" >&2; exit 1; } +git -C "$ROOT" archive "$COMMIT" proto-cuda/packs-ca3-v4/mx8-devnet-epoch0 proto-cuda/packs-ca3-v4/v4-devnet-epoch0 | tar -x -C "$STAGE" +mkdir -p "$STAGE/packs" +mv "$STAGE/proto-cuda/packs-ca3-v4/mx8-devnet-epoch0" "$STAGE/proto-cuda/packs-ca3-v4/v4-devnet-epoch0" "$STAGE/packs/" +rm -rf "$STAGE/proto-cuda" +v="$(shasum -a 256 "$STAGE/packs/v4-devnet-epoch0/vectors.json" | cut -c1-8)" +[ "$v" = 756301bf ] || { echo "v4-devnet-epoch0/vectors.json sha256 prefix $v, expected 756301bf (sub-version 1)" >&2; exit 1; } +grep -q '"sub_version": 1' "$STAGE/packs/v4-devnet-epoch0/program.json" || { echo "v4-devnet-epoch0 is not sub-version 1" >&2; exit 1; } +grep -q '"program_id": "0x1a4230699a6b9c60"' "$STAGE/packs/v4-devnet-epoch0/program.json" || { echo "v4-devnet-epoch0 program id is not 1a4230699a6b9c60" >&2; exit 1; } +# a copied tree is re-stamped before anything uses it (CLAUDE.md, the stale-build class); the packs are read, not built +find "$STAGE" -type f -exec touch {} + +( cd "$STAGE" && find . -type f ! -name SHA256SUMS | sort | xargs shasum -a 256 > SHA256SUMS ) +rm -f "$OUT" +( cd "$STAGE" && zip -X -q -r "$OUT" . ) +rm -rf "$STAGE" +echo "kit $OUT (packs from $COMMIT: v4-devnet-epoch0 sub-version 1, mx8-devnet-epoch0 the control)" +echo "sha256 $(shasum -a 256 "$OUT" | cut -c1-64) bytes $(wc -c < "$OUT" | tr -d ' ')" diff --git a/tools/bench-5060ti/pc2-5060ti-bench.ps1 b/tools/bench-5060ti/pc2-5060ti-bench.ps1 new file mode 100644 index 000000000..d1b6495a5 --- /dev/null +++ b/tools/bench-5060ti/pc2-5060ti-bench.ps1 @@ -0,0 +1,289 @@ +# Igneum run job: the ASUS Dual GeForce RTX 5060 Ti in a Razer Core X V2 (Thunderbolt) on PC 2 (machine 1ccfe586), +# 7 October 2026 (branch bench-5060ti). Published as a signed `run` job with the RUNNER's --cards-off +# (NOT --stop-miners): the runner switches the 5060 Ti off through the app's own card path before this script and puts +# it back on ANY exit; the RTX 5090 keeps mining. This script never switches a card, never stops, holds or ends the +# installed app (the playbook-quit rule, tools/ci/playbook-quit-check.sh); it reads api/state once, read only. +# POWER RULE (Josh, 7 October 2026: PC 2 lost power twice today): nothing here raises any card's power limit. There is +# no nvidia-smi -pl, -lgc or -lmc anywhere in this file; the limits are READ (power.limit, power.default_limit) and the +# row says whether they were equal. The efficient point is the app's own Ember Tune's to find (its ladder only steps +# DOWN from the stock limit); this job sets nothing and reads the app's TUNE lines for the card, if any. +# After the fetch job that places the two packs (tools/bench-5060ti/make-kit.sh: v4-devnet-epoch0 at sub-version 1, +# the stream 0.3.20 ships, fingerprint 867dbc45cfb36b4d; mx8-devnet-epoch0 the class v3 control, 90f794dd556f7a3b). +# Steps: detect (nvidia-smi: name, VRAM, driver, the PCIe link gen and width = the Thunderbolt link; the Windows +# adapter rows; the app's own card row, read only), the card quiet (no compute process on its UUID), G1 (self-test and +# the 2^24 fingerprint on both packs, bit-exact against the Mac), the 10-minute window at the stock limit (the worker's +# --bench with enough dispatches for 600 s, nvidia-smi sampled every 2 s on the card alone: MH/s and mean watts at +# utilisation 90 percent and over), the app's tune lines (read only), then, on a 16 GB card, the prove-beside-miner +# read: the app's own WSL2 prover host aimed at the 5060 Ti (IGNEUM_CUDA_DEVICE, the prover-floor host's selector) +# on one live shard while a bench loop hashes on the card, VRAM peak from the sampler; when the shipped host carries +# no selector the read is BLOCKED and the line says so (aiming the stock host would prove on the 5090 instead). +# Lines start with RESULT; a SUMMARY {json} line ends the job. Read back with `node tools/jobs.mjs `. +$ErrorActionPreference = 'Continue' +$jobName = 'pc2-5060ti-bench' +$kitId = '__KIT_ID__' # the publisher substitutes the fetch job id that placed the packs +$started = Get-Date +function Stamp { (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ') } +function Summary([string] $status, [hashtable] $extra) { + $o = [ordered]@{ job = $jobName; status = $status; duration_s = [int]((Get-Date) - $started).TotalSeconds; finished_at = (Stamp) } + foreach ($k in $extra.Keys) { $o[$k] = $extra[$k] } + 'SUMMARY ' + ($o | ConvertTo-Json -Compress -Depth 4) +} +$expected = @{ 'v4-devnet-epoch0' = '867dbc45cfb36b4d'; 'mx8-devnet-epoch0' = '90f794dd556f7a3b' } # sub-version 1: Metal = Apple OpenCL = the RTX 5090 (job run-ca3-v4-amend-g1-pc2-20261007) +$networkMhs = [double] '__NETWORK_MHS__' # the publisher substitutes live_state.hashes_per_second_estimate / 1e6; 0 = unknown, the line says so +if ($networkMhs -isnot [double] -or [double]::IsNaN($networkMhs)) { $networkMhs = 0 } +$subsidyIgn = 31.69 # IGN a block, spec 02 section 2.5 (1 block a second) +$ref5090Mhs = 122.0; $ref5090W = 308.0 # the 5090's row on PC 2 (main's figures for the per-tier sentence) +$windowS = 600 +"RESULT start $(Stamp) job=$jobName machine=$env:COMPUTERNAME app_version=$env:IGNEUM_APP_VERSION network_mhs=$networkMhs" +$jobs = Split-Path $env:IGNEUM_JOB_DIR +$kit = Join-Path $jobs $kitId +if (-not (Test-Path (Join-Path $kit 'packs\v4-devnet-epoch0\program.h'))) { "RESULT error kit missing at $kit (the fetch job $kitId runs first; an app update wipes the jobs folder: republish it under a new id after one)"; Summary 'failed' @{ error = 'kit missing' }; exit 2 } +$job = $env:IGNEUM_JOB_DIR; if (-not $job) { $job = Join-Path $env:TEMP 'igneum-5060ti' }; New-Item -ItemType Directory -Force -Path $job | Out-Null +$inst = @("$env:LOCALAPPDATA\Programs\Igneum Miner", "$env:ProgramFiles\Igneum Miner") | Where-Object { Test-Path (Join-Path $_ 'igneum-worker-cuda.exe') } | Select-Object -First 1 +$appDir = if ($env:IGNEUM_APP_DIR) { $env:IGNEUM_APP_DIR } else { Join-Path $env:LOCALAPPDATA 'igneum\app' } +if (-not $inst) { "RESULT error no installed igneum-worker-cuda.exe"; Summary 'failed' @{ error = 'no worker' }; exit 2 } +$exe = Join-Path $inst 'igneum-worker-cuda.exe' +"RESULT app install=$inst app_dir=$appDir worker=$exe sha256=$((Get-FileHash -Algorithm SHA256 $exe).Hash.ToLower())" +foreach ($f in (Get-ChildItem (Join-Path $kit 'packs') -Recurse -File | Sort-Object FullName)) { "RESULT kitfile $($f.FullName.Substring($kit.Length + 1).Replace('\', '/')) sha256 $((Get-FileHash -Algorithm SHA256 $f.FullName).Hash.ToLower()) bytes $($f.Length)" } + +# ---- detect: nvidia-smi's rows (the Thunderbolt link is the PCIe gen and width the card reports), Windows' adapter rows, the app's row ---- +function Smi([string[]] $argv) { try { @(& nvidia-smi @argv 2>&1 | ForEach-Object { "$_" }) } catch { @("nvidia-smi failed: $($_.Exception.Message)") } } +$gpuFields = 'index,uuid,name,driver_version,memory.total,memory.used,pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,pcie.link.width.max,power.draw,power.limit,power.default_limit,power.min_limit,power.max_limit,clocks.max.sm,clocks.max.mem,temperature.gpu' +# the query is ONE string built first: inside @( ... ) the comma binds before +, so '--query-gpu=' + $f, '--format=...' +# became a single argument carrying the format words (run a, 14:43:29Z: "temperature.gpu --format=csv is not a valid field") +$gpuQuery = "--query-gpu=$gpuFields" +$rows = @(Smi @($gpuQuery, '--format=csv,noheader,nounits')) +$rows | ForEach-Object { "RESULT gpu $_" } +$card = $null +foreach ($r in $rows) { + $c = @($r -split ',\s*') + if ($c.Count -ge 18 -and $c[2] -match '5060 Ti') { $card = [pscustomobject]@{ index = [int]$c[0]; uuid = $c[1]; name = $c[2]; driver = $c[3]; mem_total = [int]$c[4]; gen = $c[6]; gen_max = $c[7]; width = $c[8]; width_max = $c[9]; plimit = [double]$c[11]; pdefault = [double]$c[12]; pmin = $c[13]; pmax = $c[14]; clk_sm_max = $c[15]; clk_mem_max = $c[16] }; break } +} +try { + foreach ($a in @(Get-CimInstance Win32_VideoController -ErrorAction Stop)) { "RESULT adapter name=`"$($a.Name)`" driver=$($a.DriverVersion) status=$($a.Status) code=$($a.ConfigManagerErrorCode) pnp=$($a.PNPDeviceID)" } +} catch { "RESULT adapter error=$($_.Exception.Message)" } +if ($null -eq $card) { "RESULT error no RTX 5060 Ti in nvidia-smi's list (the rows above are what the driver sees; is the enclosure's Thunderbolt link up and the card seated?)"; Summary 'failed' @{ error = 'no 5060 Ti in nvidia-smi' }; exit 2 } +$tier = if ($card.mem_total -ge 15000) { 16 } elseif ($card.mem_total -ge 7000) { 8 } else { [math]::Round($card.mem_total / 1024) } +"RESULT DETECT name=`"$($card.name)`" vendor=nvidia vram_mib=$($card.mem_total) tier_gb=$tier driver=$($card.driver) pcie_link=gen$($card.gen)x$($card.width) pcie_max=gen$($card.gen_max)x$($card.width_max) power_limit_w=$($card.plimit) power_default_w=$($card.pdefault) power_range_w=$($card.pmin)-$($card.pmax) limit_is_stock=$(if ($card.plimit -eq $card.pdefault) { 'yes' } else { 'NO' }) smi_index=$($card.index) uuid=$($card.uuid)" +# READ ONLY: the app's view of the card (its key is what the publisher's --cards-off named; the runner, not this script, switched it) +$cardKey = ''; $appDevice = '' +try { + $urlFile = Join-Path $appDir 'app.url' + if (Test-Path $urlFile) { + $url = (Get-Content -LiteralPath $urlFile -Raw).Trim() + $st = Invoke-RestMethod -Uri ($url + 'api/state') -Method GET -TimeoutSec 10 + $all = $st.mining.cards; if (-not $all) { $all = $st.cards } + foreach ($c in @($all | Where-Object { $_.name -match '5060' })) { $cardKey = $c.key; $appDevice = "$($c.device)"; "RESULT app_card key=$($c.key) vendor=$($c.vendor) kind=$($c.kind) enabled=$($c.enabled) identities=$($c.identities) state=$($c.state) device=$($c.device) hash_now=$($c.hash_now) vram_mb=$($c.vram_mb) reason=`"$($c.reason)`" message=`"$($c.message)`"" } + if (-not $cardKey) { "RESULT app_card none: the app's state lists no card named 5060 (cards: $(@($all | ForEach-Object { $_.name }) -join ' | '))" } + } else { "RESULT app_card no app.url under $appDir" } +} catch { "RESULT app_card error=$($_.Exception.Message)" } + +# the card quiet? read only, by the driver's compute list on the card's UUID: the runner's --cards-off took, or not +function CardApps { @(Smi @('--query-compute-apps=pid,process_name,used_memory,gpu_uuid', '--format=csv,noheader,nounits') | Where-Object { $_ -match [regex]::Escape($card.uuid) }) } +$cardAlone = $false; $t = 0 +while ($t -lt 150) { if ((@(CardApps)).Count -eq 0) { $cardAlone = $true; break }; Start-Sleep -Seconds 5; $t += 5 } +if ($cardAlone) { "RESULT card $(Stamp) quiet after $t s: no compute process on $($card.uuid) (the card to itself)" } else { "RESULT card $(Stamp) UNCONFIRMED after $t s: a compute process still runs on the card ($((CardApps) -join '; ')); was the job published with --cards-off $cardKey ? the rows below are LOADED-card figures" } + +# The worker sees ONLY the 5060 Ti (CUDA_VISIBLE_DEVICES on its UUID) and takes --device 0; every RESULT line names +# the device it ran on, and a line naming another card voids the row. The 5090's process is never touched. +$env:CUDA_VISIBLE_DEVICES = $card.uuid +# Runs the worker as a child under a wall-clock cap; a child past its cap is ended by ITS OWN pid (taskkill /PID, never +# a name). Output comes back through a file; a trailing "timeout s" line says when the cap fired. +function Capped([string] $tag, [string[]] $argv, [int] $capS) { + $out = Join-Path $job ("$tag.out"); $errf = Join-Path $job "$tag.err" + Remove-Item -LiteralPath $out, $errf -Force -ErrorAction SilentlyContinue + $p = Start-Process -FilePath $exe -ArgumentList $argv -NoNewWindow -PassThru -RedirectStandardOutput $out -RedirectStandardError $errf + $done = $p.WaitForExit($capS * 1000) + if (-not $done) { try { & taskkill /T /F /PID $p.Id 2>&1 | Out-Null } catch { }; Start-Sleep -Seconds 2 } + $lines = @(); if (Test-Path -LiteralPath $out) { $lines = @(Get-Content -LiteralPath $out -ErrorAction SilentlyContinue) } + if (Test-Path -LiteralPath $errf) { $lines += @(Get-Content -LiteralPath $errf -ErrorAction SilentlyContinue) } + if (-not $done) { $lines += "timeout ${capS}s: the worker was ended by its pid $($p.Id)" } + if ($done) { try { $p.Refresh() } catch { } } + $script:lastExit = if ($done) { $p.ExitCode } else { 124 } + return $lines +} +$script:lastExit = 0 +# the nvidia-smi sampler on the card's index, 2 s, to a file; ended by its pid +function SamplerStart([string] $tag) { + $f = Join-Path $job "$tag.csv"; Remove-Item -LiteralPath $f -Force -ErrorAction SilentlyContinue + $p = Start-Process -FilePath 'nvidia-smi' -ArgumentList @('-i', "$($card.index)", '--query-gpu=timestamp,power.draw,power.limit,clocks.sm,clocks.mem,temperature.gpu,utilization.gpu,memory.used', '--format=csv,noheader,nounits', '-lms', '2000') -NoNewWindow -PassThru -RedirectStandardOutput $f -RedirectStandardError (Join-Path $job "$tag.smi.err") + return [pscustomobject]@{ pid = $p.Id; file = $f } +} +function SamplerStop($s) { try { Stop-Process -Id $s.pid -Force -ErrorAction SilentlyContinue } catch { }; Start-Sleep -Seconds 1 } +# the window's numbers from a sampler file: samples at utilisation 90 and over are the loaded ones +function SamplerRead($s, [string] $tag) { + $all = @(); if (Test-Path -LiteralPath $s.file) { $all = @(Get-Content -LiteralPath $s.file -ErrorAction SilentlyContinue | Where-Object { $_ -match ',' }) } + $pw = @(); $pl = @(); $sm = @(); $mem = @(); $tmp = @(); $used = @(); $n = 0 + foreach ($l in $all) { $c = @($l -split ',\s*'); if ($c.Count -lt 8) { continue }; $n++; $u = [double]$c[6]; $used += [double]$c[7]; if ($u -ge 90) { $pw += [double]$c[1]; $pl += [double]$c[2]; $sm += [double]$c[3]; $mem += [double]$c[4]; $tmp += [double]$c[5] } } + $o = [ordered]@{ samples = $n; loaded = $pw.Count; w_mean = 0; w_p50 = 0; w_max = 0; limit_max = 0; sm_mhz = 0; mem_mhz = 0; temp_max = 0; mem_used_peak_mib = 0 } + if ($pw.Count -gt 0) { $sorted = $pw | Sort-Object; $o.w_mean = [math]::Round(($pw | Measure-Object -Average).Average, 1); $o.w_p50 = [math]::Round($sorted[[int][math]::Floor($sorted.Count / 2)], 1); $o.w_max = [math]::Round(($pw | Measure-Object -Maximum).Maximum, 1); $o.limit_max = ($pl | Measure-Object -Maximum).Maximum; $o.sm_mhz = [math]::Round(($sm | Measure-Object -Average).Average); $o.mem_mhz = [math]::Round(($mem | Measure-Object -Average).Average); $o.temp_max = ($tmp | Measure-Object -Maximum).Maximum } + if ($used.Count -gt 0) { $o.mem_used_peak_mib = ($used | Measure-Object -Maximum).Maximum } + # the line goes to the host, not the pipeline: a bare string here was swallowed into the caller's variable (run b, 14:55Z: the ROW carried the watts, the sampler line never reached the report) + [Console]::Out.WriteLine("RESULT sampler $tag samples=$($o.samples) loaded=$($o.loaded) watts_mean=$($o.w_mean) watts_p50=$($o.w_p50) watts_max=$($o.w_max) limit_seen_w=$($o.limit_max) sm_mhz=$($o.sm_mhz) mem_mhz=$($o.mem_mhz) temp_max=$($o.temp_max) mem_used_peak_mib=$($o.mem_used_peak_mib) file=$($s.file)") + return $o +} + +# ---- the benches ---- +# A function's uncaptured output is its return value: Bench writes its lines into $script:benchOut; Flush prints them. +$script:benchOut = @() +function Say([string] $l) { $script:benchOut += $l } +function Flush { $script:benchOut | ForEach-Object { $_ }; $script:benchOut = @() } +function Bench([string] $label, [string] $packDir, [int] $batches, [int] $capS) { + if (-not (Test-Path $packDir)) { Say "RESULT bench $label error=pack missing at $packDir"; return $null } + $t0 = (Get-Date).ToUniversalTime() + Say "RESULT bench $label start $(Stamp) pack=$packDir batches=$batches" + $lines = @(Capped "bench-$label" @('--bench', '--pack', $packDir, '--batches', "$batches", '--batch-log2', '24', '--block-warps', '1', '--device', '0') $capS) + $rc = $script:lastExit + $wall = [int]((Get-Date).ToUniversalTime() - $t0).TotalSeconds + $lines | Where-Object { $_ -match '^RESULT|^warm-up|^info|^pack |^nvrtc|^self-test|FAIL|error|MISMATCH|^timeout' } | Select-Object -First 30 | ForEach-Object { Say "RESULT bench $label $_" } + Say "RESULT bench $label end $(Stamp) exit=$rc wall_s=$wall" + $res = $lines | Where-Object { $_ -match '^RESULT pack=' } | Select-Object -First 1 + if ($null -eq $rc -or "$rc" -eq '') { $rc = 0 } # a null exit after WaitForExit on a -PassThru process: the RESULT line is the verdict + if (-not $res -or $rc -ne 0) { + $err = ($lines | Where-Object { $_ -match 'FAIL|error|MISMATCH' } | Select-Object -First 1); if (-not $err) { $err = "exit $rc, no RESULT line" } + Say "RESULT G1 pack=$label error=$($err -replace '\s+', ' ')" + return $null + } + $fp = ''; $mhs = ''; $check = ''; $dev = '' + if ($res -match 'fingerprint=([0-9a-f]{16})') { $fp = $Matches[1] } + if ($res -match 'mhs=([\d.]+)') { $mhs = $Matches[1] } + if ($res -match 'check=(\S+)') { $check = $Matches[1] } + if ($res -match 'device=(\S+)') { $dev = $Matches[1] } + if ($dev -notmatch '5060') { Say "RESULT G1 pack=$label error=ran on device `"$dev`", not the 5060 Ti: the row is void"; return $null } + $want = $expected[$label] + $match = if ($want) { if ($fp -eq $want) { 'yes' } else { 'no' } } else { 'n/a' } + Say "RESULT G1 pack=$label fingerprint=$fp match=$match expected=$want check=$check device=$dev batches=$batches wall_s=$wall harness=cuda-installed" + return [pscustomobject]@{ label = $label; mhs = [double]$mhs; fp = $fp; match = $match; check = $check; wall_s = $wall; device = $dev } +} +# G1: five dispatches each, the fingerprints +$v4 = Bench 'v4-devnet-epoch0' (Join-Path $kit 'packs\v4-devnet-epoch0') 5 300; Flush +$v3 = Bench 'mx8-devnet-epoch0' (Join-Path $kit 'packs\mx8-devnet-epoch0') 5 300; Flush +# the 10-minute window at the stock limit: as many 2^24 dispatches as the five-batch rate says fit in 600 s +$stock = $null; $stockSmi = $null +if ($v4 -and $v4.mhs -gt 0) { + $n = [int][math]::Ceiling($windowS * $v4.mhs * 1e6 / 16777216); if ($n -lt 50) { $n = 50 }; if ($n -gt 6000) { $n = 6000 } + $s = SamplerStart 'stock' + Start-Sleep -Seconds 4 + $stock = Bench 'v4-stock-window' (Join-Path $kit 'packs\v4-devnet-epoch0') $n ($windowS + 300); Flush + SamplerStop $s + $stockSmi = SamplerRead $s 'stock' +} else { "RESULT window skipped: no class v4 rate from G1" } + +# ---- the efficient point: the app's Ember Tune's, read only (this job sets nothing; the ladder only ever steps DOWN) ---- +try { + $logDir = $env:IGNEUM_LOG_DIR; if (-not $logDir) { $logDir = Join-Path $appDir 'logs' } + $newest = Get-ChildItem -Path $logDir -Filter 'app-*.log' -ErrorAction SilentlyContinue | Sort-Object LastWriteTime -Descending | Select-Object -First 1 + $tuneLines = @() + if ($newest) { $tuneLines = @(Get-Content -LiteralPath $newest.FullName -ErrorAction SilentlyContinue | Where-Object { $_ -match '^\d+(\.\d+)? (TUNE|tune) ' -and $_ -match '5060' } | Select-Object -Last 8) } + if ($tuneLines.Count -gt 0) { $tuneLines | ForEach-Object { "RESULT tune $_" } } else { "RESULT tune none: the app has no TUNE line for the 5060 Ti yet (the card was off for this job; the app's tune runs when the card mines and the Power Helper answers)" } +} catch { "RESULT tune error=$($_.Exception.Message)" } + +# ---- prove beside the miner, 16 GB only: the app's own WSL2 host aimed at the 5060 Ti while a bench loop hashes on it ---- +$prove = [ordered]@{ ran = $false; verdict = 'not attempted'; peak_mib = 0; base_mib = 0; prove_s = 0; why = '' } +if ($tier -ge 16) { + $hostWin = Join-Path $inst 'wsl2\bin\igneum-prove-host' + if (-not (Test-Path -LiteralPath $hostWin)) { $prove.why = "no WSL2 host at $hostWin"; "RESULT prove blocked why=$($prove.why)" } + else { + function WslPath($p) { $w = (& wsl.exe -d Ubuntu-24.04 -- wslpath -a ($p -replace '\\', '/') 2>$null); if ($w) { ($w -replace "`0", '').Trim() } else { '/mnt/c' + ($p.Substring(2) -replace '\\', '/') } } + $hostW = WslPath $hostWin; $jobW = WslPath $job + # the fixture: one live block from PC 2's own node (read only, the exec RPC on loopback), cut with the app's exporter + $evmPort = $null + foreach ($p in 26790, 26800, 26810) { try { $r = Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:$p" -ContentType 'application/json' -Body '{"jsonrpc":"2.0","id":1,"method":"eth_blockNumber","params":[]}' -TimeoutSec 20; if ($r.result) { $evmPort = $p; $tipHex = $r.result; break } } catch { } } + if (-not $evmPort) { $prove.why = 'no exec RPC on 26790/26800/26810'; "RESULT prove blocked why=$($prove.why)" } + else { + $tip = [Convert]::ToInt64($tipHex, 16); $blk = $tip - 30 + $body = (@{jsonrpc='2.0'; id=1; method='igneum_exportSegments'; params=@('0x0', ('0x{0:x}' -f $blk))} | ConvertTo-Json -Compress) + $bodyFile = Join-Path $job 'export-request.json'; $seqFile = Join-Path $job 'seq.json' + [IO.File]::WriteAllText($bodyFile, $body, (New-Object System.Text.UTF8Encoding $false)) + & curl.exe -s -S -m 600 -X POST "http://127.0.0.1:$evmPort" -H 'Content-Type: application/json' --data-binary "@$bodyFile" -o $seqFile 2>&1 | ForEach-Object { "RESULT prove curl $_" } + $seqLen = if (Test-Path $seqFile) { (Get-Item $seqFile).Length } else { 0 } + "RESULT prove export port=$evmPort tip=$tip block=$blk bytes=$seqLen" + # the bash side (LF, no BOM, run as the WSL default user = the app's prover user, never as the root user: the + # socket rule of tools/ci/prover-socket-check.sh; the server this run starts is ended by the pid on ITS socket). + # Inventory first; the prove only when the host carries the selector and the card is not WSL device 0 (device 0's + # socket /tmp/sp1-cuda-0.sock is the app's own server on the 5090 and is never touched). + $bashBody = @' +set -uo pipefail +H="$1"; JOB="$2"; UUID="$3"; BLK="$4" +X="$(dirname "$H")/igneum-prove-export" +CUDA_DIR="$(ls -d /usr/local/cuda-12.* 2>/dev/null | sort -V | tail -1 || true)"; [ -n "$CUDA_DIR" ] && export PATH="$CUDA_DIR/bin:$PATH" && export LD_LIBRARY_PATH="$CUDA_DIR/lib64:/usr/lib/wsl/lib:${LD_LIBRARY_PATH:-}" +export PATH="$HOME/.sp1/bin:$PATH" +stamp() { date -u +%Y-%m-%dT%H:%M:%SZ; } +echo "RESULT prove host=$H exists=$([ -x "$H" ] && echo yes || echo no) sha=$(sha256sum "$H" 2>/dev/null | cut -c1-16) export_exists=$([ -x "$X" ] && echo yes || echo no) user=$(id -un) home=$HOME" +SEL=$(grep -c IGNEUM_CUDA_DEVICE "$H" 2>/dev/null || true); SEL=${SEL:-0} +echo "RESULT prove selector IGNEUM_CUDA_DEVICE strings_in_host=$SEL" +for s in "$HOME/.sp1/bin/sp1-gpu-server" /opt/igneum-floor/bin/sp1-gpu-server; do if [ -x "$s" ]; then echo "RESULT prove server $s bytes=$(stat -c %s "$s") sha=$(sha256sum "$s" | cut -c1-16) version=$("$s" --version 2>&1 | head -1)"; else echo "RESULT prove server $s absent"; fi; done +echo "RESULT prove sockets_before $(ls /tmp/sp1-cuda-*.sock 2>/dev/null | tr '\n' ' ')" +nvidia-smi --query-gpu=index,uuid,name,memory.total --format=csv,noheader 2>/dev/null | sed 's/^/RESULT prove wslgpu /' +IDX="$(nvidia-smi --query-gpu=index,uuid --format=csv,noheader 2>/dev/null | awk -F', ' -v u="$UUID" '$2==u {print $1}' | head -1)" +echo "RESULT prove wsl_index_of_5060ti=${IDX:-none}" +if [ "$SEL" = 0 ]; then echo "RESULT prove blocked why=the shipped host has no IGNEUM_CUDA_DEVICE selector: aimed at nothing it proves on CUDA device 0 (the 5090) through the app's own socket; the prover-floor host carries the selector (proof_system.rs) and is the owed cut"; exit 0; fi +if [ -z "$IDX" ] || [ "$IDX" = 0 ]; then echo "RESULT prove blocked why=the 5060 Ti is WSL CUDA device '${IDX:-none}': device 0 is the app's own prover socket and is never touched by a job"; exit 0; fi +SOCK="/tmp/sp1-cuda-$IDX.sock" +rm -f "$SOCK" +python3 -c "import json; d=json.load(open('$JOB/seq.json')); json.dump(d['result'], open('$JOB/export.json','w'))" || { echo "RESULT prove blocked why=the export reply did not unwrap"; exit 0; } +FIX="$JOB/block-$BLK.json" +if ! "$X" "$JOB/export.json" "$BLK" "$FIX" --source "PC 2 live devnet export, the 5060 Ti prove-beside read" 2>&1 | tail -1 | sed 's/^/RESULT prove cut /'; then echo "RESULT prove blocked why=the cut failed"; exit 0; fi +echo "RESULT prove start $(stamp) fixture=$(basename "$FIX") device=$IDX socket=$SOCK" +t0=$(date +%s) +IGNEUM_CUDA_DEVICE="$IDX" SP1_PROVER=cuda RUST_LOG=off timeout 900 "$H" "$FIX" --mode compressed --shard 0 --prover 0xCAfc6e74000000000000000000000000000000c2 --out "$JOB/prove-5060ti.json" > "$JOB/prove-5060ti.log" 2>&1; rc=$? +echo "RESULT prove end $(stamp) rc=$rc wall_s=$(( $(date +%s) - t0 ))" +grep -E '^(RESULT|STAGE|segment proof|FLOOR|thread|Error|error)' "$JOB/prove-5060ti.log" | head -20 | sed 's/^/RESULT prove host-out /' +# the server this run started answers on ITS socket: ended by that pid, never by a name; the socket unlinked +PID="$(fuser "$SOCK" 2>/dev/null | tr -d ' ' || true)" +if [ -n "$PID" ]; then kill "$PID" 2>/dev/null; sleep 2; kill -9 "$PID" 2>/dev/null; echo "RESULT prove server_ended pid=$PID"; fi +rm -f "$SOCK" +echo "RESULT prove sockets_after $(ls /tmp/sp1-cuda-*.sock 2>/dev/null | tr '\n' ' ')" +'@ + $bashFile = Join-Path $job 'prove-5060ti.sh' + [IO.File]::WriteAllText($bashFile, ("#!/bin/bash`n" + ($bashBody -replace "`r`n", "`n").TrimEnd() + "`n"), (New-Object System.Text.UTF8Encoding $false)) + $bashW = WslPath $bashFile + # the miner-shaped load on the 5060 Ti for the read: a bench loop of about six minutes, by pid, file output + $loopN = if ($v4 -and $v4.mhs -gt 0) { [int][math]::Ceiling(360 * $v4.mhs * 1e6 / 16777216) } else { 300 } + $loopOut = Join-Path $job 'beside-loop.out' + $loop = Start-Process -FilePath $exe -ArgumentList @('--bench', '--pack', (Join-Path $kit 'packs\v4-devnet-epoch0'), '--batches', "$loopN", '--batch-log2', '24', '--block-warps', '1', '--device', '0') -NoNewWindow -PassThru -RedirectStandardOutput $loopOut -RedirectStandardError (Join-Path $job 'beside-loop.err') + $s2 = SamplerStart 'beside' + Start-Sleep -Seconds 45 # the loop's build, self-test and warm-up: the base reading is the miner-shaped resident set + $baseRow = @(Get-Content -LiteralPath $s2.file -ErrorAction SilentlyContinue | Where-Object { $_ -match ',' } | Select-Object -Last 1) + if ($baseRow.Count -gt 0) { $c = @($baseRow[0] -split ',\s*'); if ($c.Count -ge 8) { $prove.base_mib = [double]$c[7] } } + "RESULT prove base_mib=$($prove.base_mib) (the bench loop resident on the card before the host starts)" + $wslOut = @(& wsl.exe -d Ubuntu-24.04 -- bash $bashW $hostW $jobW $card.uuid "$blk" 2>&1 | ForEach-Object { ("$_" -replace "`0", '') }) + $wslOut | ForEach-Object { $_ } + $prove.ran = ($wslOut | Where-Object { $_ -match '^RESULT prove end ' }).Count -gt 0 + $endLine = $wslOut | Where-Object { $_ -match '^RESULT prove end ' } | Select-Object -First 1 + if ($endLine -and $endLine -match 'rc=(\d+) wall_s=(\d+)') { $prove.prove_s = [int]$Matches[2]; $prove.verdict = if ($Matches[1] -eq '0') { 'proved' } else { "failed rc=$($Matches[1])" } } + $blocked = $wslOut | Where-Object { $_ -match '^RESULT prove blocked ' } | Select-Object -First 1 + if ($blocked) { $prove.verdict = 'blocked'; $prove.why = ($blocked -replace '^RESULT prove blocked why=', '') } + if (-not $loop.HasExited) { $loop.WaitForExit(120000) | Out-Null } + if (-not $loop.HasExited) { try { & taskkill /T /F /PID $loop.Id 2>&1 | Out-Null } catch { } } + SamplerStop $s2 + $besideSmi = SamplerRead $s2 'beside' + $prove.peak_mib = $besideSmi.mem_used_peak_mib + $loopRes = @(Get-Content -LiteralPath $loopOut -ErrorAction SilentlyContinue | Where-Object { $_ -match '^RESULT pack=' } | Select-Object -First 1) + $loopMhs = if ($loopRes.Count -gt 0 -and $loopRes[0] -match 'mhs=([\d.]+)') { $Matches[1] } else { 'none' } + "RESULT prove loop_mhs=$loopMhs (the bench loop's mean over its dispatches, the prover beside it when it ran)" + $own = if ($prove.ran -and $prove.verdict -eq 'proved') { [math]::Round(($prove.peak_mib - $prove.base_mib) / 1024, 1) } else { 0 } + "RESULT PROVE tier_gb=$tier verdict=$($prove.verdict) prove_s=$($prove.prove_s) peak_mib=$($prove.peak_mib) base_mib=$($prove.base_mib) own_gb=$own vram_mib=$($card.mem_total) spare_gb=$(if ($prove.peak_mib -gt 0) { [math]::Round(($card.mem_total - $prove.peak_mib) / 1024, 1) } else { 'n/a' }) why=`"$($prove.why)`"" + } + } +} else { "RESULT PROVE tier_gb=$tier verdict=not-a-16gb-card: the prove-beside read is for the 16 GB tier; an 8 GB card mines only (the 12 GB floor is 13.9 GB for an empty shard)" } + +# ---- the row, the picker entry and the consequences ---- +$rate = if ($stock) { $stock.mhs } elseif ($v4) { $v4.mhs } else { 0 } +$watts = if ($stockSmi -and $stockSmi.loaded -gt 0) { $stockSmi.w_mean } else { 0 } +$mhw = if ($watts -gt 0 -and $rate -gt 0) { [math]::Round($rate / $watts, 3) } else { 0 } +$blocksDay = if ($networkMhs -gt 0 -and $rate -gt 0) { [math]::Round($rate / $networkMhs * 86400, 2) } else { 0 } +$hoursBlock = if ($blocksDay -gt 0) { [math]::Round(24 / $blocksDay, 1) } else { 0 } +$ignDay = [math]::Round($blocksDay * $subsidyIgn, 1) +$shadowCost = if ($v4 -and $v3 -and $v3.mhs -gt 0) { [math]::Round((1 - $v4.mhs / $v3.mhs) * 100, 1) } else { 'owed' } +$of5090 = if ($rate -gt 0) { [math]::Round($rate / $ref5090Mhs * 100, 1) } else { 0 } +$ref5090Mhw = [math]::Round($ref5090Mhs / $ref5090W, 3) +$mhwVs5090 = if ($mhw -gt 0) { [math]::Round($mhw / $ref5090Mhw * 100) } else { 0 } +"RESULT ROW card=`"$($card.name)`" vram_gb=$tier driver=$($card.driver) pcie=gen$($card.gen)x$($card.width) g1_v4_mhs=$(if ($v4) { $v4.mhs } else { 'failed' }) g1_v3_mhs=$(if ($v3) { $v3.mhs } else { 'failed' }) stock_window_mhs=$(if ($stock) { $stock.mhs } else { 'failed' }) stock_window_s=$(if ($stock) { $stock.wall_s } else { 0 }) watts_mean=$watts watts_p50=$(if ($stockSmi) { $stockSmi.w_p50 } else { 0 }) limit_w=$($card.plimit) default_w=$($card.pdefault) mhw=$mhw sm_mhz=$(if ($stockSmi) { $stockSmi.sm_mhz } else { 0 }) mem_mhz=$(if ($stockSmi) { $stockSmi.mem_mhz } else { 0 }) temp_max=$(if ($stockSmi) { $stockSmi.temp_max } else { 0 }) shadow_cost_pct=$shadowCost efficient_point=app-tune-owed network_mhs=$networkMhs blocks_per_day=$blocksDay hours_per_block=$hoursBlock ign_per_day=$ignDay card_alone=$cardAlone" +"RESULT PICKER ['NVIDIA RTX 5060 Ti', $rate] (site/yourcard.js CARDS entry; measured on the card alone, so it may go on the site)" +"RESULT CONSEQUENCES a 5060 Ti owner ($tier GB, Windows, in a Thunderbolt enclosure at gen$($card.gen)x$($card.width)): $rate MH/s at $watts W ($mhw MH/W), $of5090 percent of the 5090's $ref5090Mhs MH/s at $ref5090W W ($ref5090Mhw MH/W: $mhwVs5090 percent of its hash per watt); $blocksDay blocks a day at today's $networkMhs MH/s network (one every $hoursBlock h, about $ignDay IGN a day at $subsidyIgn IGN a block, approximate: the network rate moves); prove-beside verdict $($prove.verdict); the efficient point is the app's tune to find and is owed (nothing set by this job)" +$status = if ($v4 -and $v4.match -eq 'yes' -and $v3 -and $v3.match -eq 'yes' -and $stock) { 'done' } elseif ($v4 -or $stock) { 'partial' } else { 'failed' } +"RESULT end $(Stamp)" +Summary $status @{ name = $card.name; tier_gb = $tier; driver = $card.driver; pcie = "gen$($card.gen)x$($card.width)"; rate_mhs = $rate; watts = $watts; mhw = $mhw; limit_w = $card.plimit; default_w = $card.pdefault; v4_fp = $(if ($v4) { $v4.fp } else { '' }); v4_match = $(if ($v4) { $v4.match } else { 'failed' }); v3_match = $(if ($v3) { $v3.match } else { 'failed' }); prove = $prove.verdict; prove_peak_mib = $prove.peak_mib; blocks_per_day = $blocksDay; card_alone = $cardAlone } +if ($status -eq 'failed') { exit 1 } +exit 0 From 32fd127f96a4a137121cbf297e2ee3c1278f9708 Mon Sep 17 00:00:00 2001 From: igneum-josh <337424239+igneum-josh@users.noreply.github.com> Date: Wed, 7 Oct 2026 16:05:56 +0100 Subject: [PATCH 03/15] Gate: the box lock self-tests read the same on an idle box and on one at load 120 (private lock directories, no pinning in the fixture, file-driven waits, a 3 s settle, a quiet that outlives the slot settle); the check runs them one at a time with one retry each (the horizon lane, 7 Oct 2026: one red on a full gate, green a minute later) Co-Authored-By: Claude Fable 5.1 --- infra/build-server/lease.sh | 25 +++++++++++++++---------- infra/build-server/remote-run.sh | 14 ++++++++------ tools/ci/box-locks-check.sh | 17 +++++++++++++---- 3 files changed, 36 insertions(+), 20 deletions(-) diff --git a/infra/build-server/lease.sh b/infra/build-server/lease.sh index 09baf3551..5138474bb 100755 --- a/infra/build-server/lease.sh +++ b/infra/build-server/lease.sh @@ -82,7 +82,7 @@ run_cores() { say "holding cores $set (waited $waited s): $label" # the keeper closes the lock descriptors first: an inherited flock would outlive the release in its orphaned sleep ( for fd in "${fds[@]}"; do exec {fd}>&-; done; while kill -0 $$ 2>/dev/null; do touch "$LOCKS/lease-$$" 2>/dev/null; [ -s "$LOCKS/lease-$$" ] || printf '%s\n' "$line" > "$LOCKS/lease-$$"; sleep 20; done ) & local keeper=$! - nice -n "$nice" taskset -c "$set" "$@"; local rc=$? + if [ "${LEASE_NO_PIN:-0}" = 1 ]; then "$@"; else nice -n "$nice" taskset -c "$set" "$@"; fi; local rc=$? # LEASE_NO_PIN: the self-test (lock semantics only; a pinned fixture on a loaded box ran seconds late) pkill -P "$keeper" 2>/dev/null; kill "$keeper" 2>/dev/null; wait "$keeper" 2>/dev/null rm -f "$LOCKS/lease-$$"; say "released cores $set after $(( $(date +%s) - t0 )) s, exit $rc" exit $rc @@ -110,29 +110,34 @@ run_quiet() { } self_test() { t=$(mktemp -d); trap 'rm -rf "$t"' EXIT - export IGNEUM_BUILD_SLOTS_DIR="$t/locks" IGNEUM_BUILD_LOG_DIR="$t/log" LEASE_REAP_S=2; mkdir -p "$t/locks"; echo 2 > "$t/locks/slots" + # a PRIVATE lock directory (never the live _locks) and no pinning: the test is about the locks, and it must read the same on an + # idle box and on one at load 120 (the horizon lane, 7 Oct 2026: one red on a full gate, green a minute later) + export IGNEUM_BUILD_SLOTS_DIR="$t/locks" IGNEUM_BUILD_LOG_DIR="$t/log" LEASE_REAP_S=2 LEASE_NO_PIN=1; mkdir -p "$t/locks"; echo 2 > "$t/locks/slots" local me="$0" fail=0 f() { echo "lease self-test: FAIL: $*"; fail=1; } # 1. two leases on disjoint cores run together; the same core waits - bash "$me" cores 0-1 --label A -- bash -c "date +%s.%N > '$t/a.start'; sleep 2; date +%s.%N > '$t/a.end'" 2>/dev/null & - sleep 0.5; bash "$me" cores 2-3 --label B -- bash -c "date +%s.%N > '$t/b.start'" 2>/dev/null; sleep 0.2 - python3 -c "import sys; sys.exit(0 if float(open('$t/b.start').read()) < float(open('$t/a.start').read()) + 1.5 else 1)" || f "a lease on other cores waited for an unrelated lease" + bash "$me" cores 0-1 --label A -- bash -c "date +%s.%N > '$t/a.start'; sleep 6; date +%s.%N > '$t/a.end'" 2>/dev/null & + for i in $(seq 1 50); do [ -f "$t/a.start" ] && break; sleep 0.1; done + bash "$me" cores 2-3 --label B -- bash -c "date +%s.%N > '$t/b.start'" 2>/dev/null; sleep 0.2 + # B ran while A still held its cores (A sleeps 6 s): B's start is before A's end + python3 -c "import sys; sys.exit(0 if float(open('$t/b.start').read()) < float(open('$t/a.start').read()) + 5.5 else 1)" || f "a lease on other cores waited for an unrelated lease" bash "$me" cores 1-2 --label C -- bash -c "date +%s.%N > '$t/c.start'" 2>/dev/null python3 -c "import sys; sys.exit(0 if float(open('$t/c.start').read()) >= float(open('$t/a.end').read()) else 1)" || f "a lease sharing a core started before the holder released it" wait [ -z "$(ls "$t/locks" | grep -E '^(lease|wait)-')" ] || f "lease or wait files left behind: $(ls "$t/locks")" # 2. quiet is refused while a slot is held, and runs on an idle box with a holder line naming the owner - ( exec 9>>"$t/locks/build-0"; flock 9; echo "pid $BASHPID since x waited 0 s: fake build" > "$t/locks/build-0"; sleep 2 ) & - sleep 0.3; bash "$me" quiet --label Q --owner tester -- true 2>/dev/null; [ $? = 73 ] || f "quiet was not refused while a slot was held" + ( exec 9>>"$t/locks/build-0"; flock 9; echo "pid $BASHPID since x waited 0 s: fake build" > "$t/locks/build-0"; sleep 6 ) & + for i in $(seq 1 50); do [ -s "$t/locks/build-0" ] && break; sleep 0.1; done + bash "$me" quiet --label Q --owner tester -- true 2>/dev/null; [ $? = 73 ] || f "quiet was not refused while a slot was held" wait; : > "$t/locks/build-0" bash "$me" quiet --label Q --owner tester -- bash -c "grep -q 'owner=tester' '$t/locks/quiet'" || f "quiet did not run on an idle box with the owner in its holder line" bash "$me" quiet --label Q --owner tester --cap-s 1 -- sleep 5; [ $? = 124 ] || f "the quiet cap did not end the command" # 3. a quiet is refused while a core is leased - bash "$me" cores 0 --label L -- sleep 2 2>/dev/null & sleep 0.5 + bash "$me" cores 0 --label L -- sleep 6 2>/dev/null & for i in $(seq 1 50); do ls "$t"/locks/lease-* >/dev/null 2>&1 && break; sleep 0.1; done bash "$me" quiet --label Q --owner tester -- true 2>/dev/null; [ $? = 73 ] || f "quiet was not refused while a core was leased" wait # 4. reap: a stopped holder older than the window is killed and its file cleared, with a line - bash "$me" cores 5 --label S -- sleep 60 2>/dev/null & local lp=$!; sleep 0.6 + bash "$me" cores 5 --label S -- sleep 60 2>/dev/null & local lp=$!; for i in $(seq 1 50); do ls "$t"/locks/lease-* >/dev/null 2>&1 && break; sleep 0.1; done; sleep 0.3 local hp; hp=$(sed -n 's/^pid \([0-9]*\) .*/\1/p' "$t"/locks/lease-* | head -1); kill -STOP "$hp"; sleep 2.5 touch -d '-10 seconds' "$t"/locks/lease-* 2>/dev/null [ "$(bash "$me" reap 2>/dev/null)" = 1 ] || f "the stopped lease holder was not reaped" @@ -140,7 +145,7 @@ self_test() { [ -z "$(ls "$t/locks" | grep '^lease-')" ] || f "the reaped lease file remains" kill -KILL "$lp" 2>/dev/null; wait 2>/dev/null # 5. a running (not stopped) holder is left alone - bash "$me" cores 6 --label R -- sleep 3 2>/dev/null & sleep 0.6; touch -d '-10 seconds' "$t"/locks/lease-* + bash "$me" cores 6 --label R -- sleep 6 2>/dev/null & for i in $(seq 1 50); do ls "$t"/locks/lease-* >/dev/null 2>&1 && break; sleep 0.1; done; sleep 0.3; touch -d '-10 seconds' "$t"/locks/lease-* [ "$(bash "$me" reap 2>/dev/null)" = 0 ] || f "a running holder was reaped"; wait [ "$fail" = 0 ] && echo "lease self-test: disjoint leases run together, a shared core waits, quiet is refused beside a slot or a lease and capped, a stopped holder is reaped after the window, a running one is kept" return $fail diff --git a/infra/build-server/remote-run.sh b/infra/build-server/remote-run.sh index eddaf6620..3468a28fb 100755 --- a/infra/build-server/remote-run.sh +++ b/infra/build-server/remote-run.sh @@ -178,7 +178,8 @@ fi if [ "${1:-}" = --self-test-slots ]; then me="${IGNEUM_REMOTE_RUN_UNDER_TEST:-$0}" t=$(mktemp -d); trap 'rm -rf "$t"' EXIT - mkdir -p "$t/locks" "$t/log" "$t/dir"; echo 2 > "$t/locks/slots" + # a PRIVATE lock directory, never the live _locks; a 3 s settle so two fakes that start seconds apart on a loaded box still see each other + mkdir -p "$t/locks" "$t/log" "$t/dir"; echo 2 > "$t/locks/slots"; export BR_SETTLE_S=3 fake() { # [BR_MEASURE=1]: a fake run that records its start and end epoch and the job count it was given local name="$1" secs="$2" measure="${3:-0}" IGNEUM_BUILD_SLOTS_DIR="$t/locks" IGNEUM_BUILD_LOG_DIR="$t/log" BR_MEASURE="$measure" BR_CORES="${BR_CORES:-0}" BR_NICE="${BR_NICE:-0}" BR_DIR="$t/dir" BR_CMD="date +%s.%N > '$t/$name.start'; echo JOBS=\${CARGO_BUILD_JOBS:-none} > '$t/$name.jobs'; sleep $secs; date +%s.%N > '$t/$name.end'" \ @@ -188,19 +189,19 @@ if [ "${1:-}" = --self-test-slots ]; then fail() { echo "self-test-slots: FAIL: $*"; exit 1; } after() { python3 -c "import sys; sys.exit(0 if float(open(sys.argv[1]).read()) >= float(open(sys.argv[2]).read()) else 1)" "$1" "$2"; } # 1. two concurrent builds: 45 jobs each - fake a 3 & fake b 3 & wait + fake a 6 & fake b 6 & wait [ "$(cat "$t/a.jobs")" = JOBS=45 ] && [ "$(cat "$t/b.jobs")" = JOBS=45 ] || fail "two concurrent builds got $(cat "$t/a.jobs" "$t/b.jobs" | tr '\n' ' ') (want JOBS=45 JOBS=45)" # 2. one build alone: 90 fake c 1 [ "$(cat "$t/c.jobs")" = JOBS=90 ] || fail "a lone build got $(cat "$t/c.jobs") (want JOBS=90)" # 3. a quiet measurement blocks an unbounded build (it starts only after the quiet ended) and lets a bounded suite run beside it - fake m 3 1 & sleep 0.5; fake d 1 & BR_CORES=1 BR_NICE=10 fake s 1 & wait + fake m 9 1 & sleep 0.5; fake d 1 & BR_CORES=1 BR_NICE=10 fake s 1 & wait # the quiet holds 9 s: longer than a slot take plus the 3 s settle after "$t/d.start" "$t/m.end" || fail "an unbounded build started while a quiet measurement held the box (build start $(cat "$t/d.start"), quiet end $(cat "$t/m.end"))" python3 -c "import sys; sys.exit(0 if float(open('$t/s.start').read()) < float(open('$t/m.end').read()) else 1)" || fail "a bounded suite waited for the quiet measurement" [ "$(cat "$t/m.jobs")" = JOBS=none ] || fail "a measurement was given a job count" grep -q 'owner=self-test' "$t/m.out" "$t/log/builds.jsonl" 2>/dev/null || fail "the quiet holder line lacks its owner" # 4. a quiet measurement is REFUSED (exit 73) while a build holds a slot or a core is leased - fake e 3 & sleep 0.5; fake n 1 1; rc=$?; wait + fake e 6 & sleep 0.5; fake n 1 1; rc=$?; wait [ "$rc" = 73 ] && [ ! -f "$t/n.start" ] || fail "a quiet measurement was not refused while a build ran (rc $rc)" ( exec 9>>"$t/locks/core-7"; flock 9; sleep 6 ) & sleep 0.5; fake o 1 1; rc=$?; wait [ "$rc" = 73 ] || fail "a quiet measurement was not refused while a core was leased (rc $rc)" @@ -382,8 +383,9 @@ else fi waited=$(( $(date +%s) - BR_T0 )) holder_line "$waited" > "$SLOTS_DIR/build-$got" - # the job count: let a build that started in the same second take its slot, then count the slots held (this one included) - sleep 1 + # the job count: let a build that started in the same second take its slot, then count the slots held (this one included); + # BR_SETTLE_S widens it for the self-test on a loaded box + sleep "${BR_SETTLE_S:-1}" held=0 for k in $(seq 0 $((slots - 1))); do if [ "$k" = "$got" ]; then held=$((held + 1)); continue; fi diff --git a/tools/ci/box-locks-check.sh b/tools/ci/box-locks-check.sh index 227695c9e..a9ea41c38 100755 --- a/tools/ci/box-locks-check.sh +++ b/tools/ci/box-locks-check.sh @@ -19,9 +19,18 @@ stamp="ci-$$-$(date +%s)" scp -q "${BS_SSH_OPTS[@]}" infra/build-server/lease.sh "$BS_HOST:/tmp/lease-$stamp.sh" scp -q "${BS_SSH_OPTS[@]}" infra/build-server/remote-run.sh "$BS_HOST:/tmp/rr-$stamp.sh" rc=0 -# the two self-tests run in parallel in one ssh session (each against its own scratch lock directory); their last lines are printed -bs_ssh "set -o pipefail; (bash /tmp/lease-$stamp.sh --self-test 2>&1 | grep -E '^lease self-test'; echo lease=\${PIPESTATUS[0]} >> /tmp/locks-$stamp.rc) & (IGNEUM_REMOTE_RUN_UNDER_TEST=/tmp/rr-$stamp.sh bash /tmp/rr-$stamp.sh --self-test-slots 2>&1 | grep -E '^self-test-slots'; echo slots=\${PIPESTATUS[0]} >> /tmp/locks-$stamp.rc) & wait; cat /tmp/locks-$stamp.rc; rm -f /tmp/lease-$stamp.sh /tmp/rr-$stamp.sh /tmp/locks-$stamp.rc" | tee /tmp/box-locks-$$.out || rc=1 -grep -q '^lease=0$' /tmp/box-locks-$$.out && grep -q '^slots=0$' /tmp/box-locks-$$.out || rc=1 -rm -f /tmp/box-locks-$$.out +# one at a time (two fixtures at once on a box at load 120 ran seconds late), each retried once on a failure (a transient ssh or +# a momentary stall; the fixtures use private lock directories, never the live _locks, so the box's state is not in the result) +try_twice() { #