diff --git a/tools/ca3-pc1-amd/README.md b/tools/ca3-pc1-amd/README.md index 318eaf4fa..289d8b328 100644 --- a/tools/ca3-pc1-amd/README.md +++ b/tools/ca3-pc1-amd/README.md @@ -37,12 +37,22 @@ The fetch lands at `\fetch-ca3-pc1-amd-20261006\{bin,packs,src,SHA256SUMS} | # | Script | Publish (after the fetch; `--id` fixed so the read-back is one command) | Length | Card | Rows it fills | |---|---|---|---|---|---| | 1 | `pc1-amd-g1-shadow.ps1` | `packaging/ota/publish-jobs.sh add --kind run --target ae432dc7 --id run-ca3-pc1-amd-g1-shadow-20261006 --script tools/ca3-pc1-amd/pc1-amd-g1-shadow.ps1 --timeout-minutes 40 --title "CA3 PC 1: G1 + AMD ladder on the 9070 XT" --deploy` | about 15 min (7 benches of 60 to 90 dispatches of 2^24 at 0.9 to 2 s each, the card switch up to 150 s, the installed-worker cross-check 40 dispatches) | 9070 XT ALONE (its worker stopped; 5090 and 4070 keep mining) | G1 for `mx8+sh256x27` on the AMD vendor (section 5: `RESULT G1 pack=sh256x27 fingerprint=3d2e8245cc084d07 match=yes` turns "AMD vendor NOT RUN" green; the other five packs give the same for the ladder); the 9070 XT column of the item 8 tables (section 3: `RESULT LADDER pack=

ops= mhs= watts= uj= gclk_mhz=`), where the card leaves the latency bound by the 5 percent rule; item 1's AMD per-joule row from the control (`RESULT LADDER pack=mx8-devnet-epoch0 ... uj=`: section 3 "Consequences per tier (item 1)", 16 GB AMD row); section 7 rows "item 8: the 9070 XT rows", "item 1: the 9070 XT rows" | +| 1b (the watts re-run) | `pc1-amd-watts.ps1` | `packaging/ota/publish-jobs.sh add --kind run --target ae432dc7 --id run-ca3-pc1-amd-watts-20261006 --script tools/ca3-pc1-amd/pc1-amd-watts.ps1 --timeout-minutes 20 --title "CA3 PC 1: watts on the 9070 XT (control + sh256x27)" --deploy` | about 5 min (2 x 100 dispatches at the 18.9 to 19.3 MH/s job 1 measured = 87 to 89 s each, the switch, 12 s of proof) | 9070 XT ALONE | the two `watts= uj=` cells job 1 left owed: item 1's AMD per-joule row (the control) and the candidate's AMD row in the item 8 table (sh256x27); the sampler is proven on the 12 idle seconds first (`RESULT sampler_raw` x3, `RESULT sampler_idle lines= lines_9070_with_watts=`) and the benches do not run without a 9070 XT watts line (`RESULT watts error=`, status failed, card restored) | | 2 | `pc1-amd-family.ps1` | `packaging/ota/publish-jobs.sh add --kind run --target ae432dc7 --id run-ca3-pc1-amd-family-20261006 --script tools/ca3-pc1-amd/pc1-amd-family.ps1 --timeout-minutes 15 --title "CA3 PC 1: family step costs on the 9070 XT" --deploy` | about 3 min (3 runs x up to 3 AMD devices, 26 variants each, builds dominate) | beside the miners (ratios; every row says `card_state=`) | the "RX 9070 XT" column of the item 6 table (section 3: `RESULT FAMILYBEST name= ratio= exact= path=` per family, `RESULT FAMILY ... variant=` for the form each took: `shfla_bperm` = `ds_bpermute_b32`, the one number that could move R3; `shflx_swz` / `shflx_bperm`; `perm_amd` = `v_perm_b32` through `amd_perm` or `perm_c` emulated; `bfe_amd`; `dot4_amd` = the 5 October sudot4 row re-measured; `mm8_gfx12` / `mm8_gfx11` = the WMMA builtins, exact=unverified); section 7 row "item 6: the 9070 XT step costs" | | 3 | `pc1-amd-derive.ps1` | `packaging/ota/publish-jobs.sh add --kind run --target ae432dc7 --id run-ca3-pc1-amd-derive-20261006 --script tools/ca3-pc1-amd/pc1-amd-derive.ps1 --timeout-minutes 15 --title "CA3 PC 1: dr736 on the 9070 XT" --deploy` | about 4 min (3 packs x a 60-dispatch pass and a 3-dispatch pass) | beside the miners (build, compile and bit-exact rows; the rate as a ratio to mx8 in the same job) | the item 2 table's 9070 XT cells (section 3: `RESULT DERIVE pack=dr736-genesis build_ms= build_ms_cached= build_ms_over_mx8= dataset_ms= fingerprint= match= mhs= ratio_to_mx8=`): the OpenCL compile cost of the derivation program against mx8's (the NVRTC +1.1 s finding on AMD), the 1 GiB daily build against mx8's 72 to 77 ms (5 October), bit-exactness on the AMD vendor; section 7 row "item 2: the 9070 XT rows" | | 4 (optional, confirmed by main) | `pc1-4070-shadow.ps1` | `packaging/ota/publish-jobs.sh add --kind run --target ae432dc7 --id run-ca3-pc1-4070-shadow-20261006 --script tools/ca3-pc1-amd/pc1-4070-shadow.ps1 --timeout-minutes 30 --title "CA3 PC 1: item 8 ladder on the RTX 4070" --deploy` | about 11 min (7 benches of 100 to 200 dispatches at an expected 0.4 to 1 s each, approximate: no 4070 row exists yet; NVRTC per pack; the card switch) | 4070 ALONE (its CUDA worker stopped; 5090 and 9070 XT keep mining) | the "small NVIDIA card binds near 100,000" row of item 8 (section 3, consequences per tier at N = 100,000; section 7 "the 4060-class rows"): `RESULT LADDER pack=

ops= mhs= watts= uj= sm_mhz=` with nvidia-smi at 1 Hz on that index; G1 fingerprints on a second NVIDIA card | Read back: `node tools/jobs.mjs run-ca3-pc1-amd-g1-shadow-20261006` (and `--all` for the 5-minute progress uploads). The run_id on the intake is `job--ae432dc7`. +## Job 1's first run (run-ca3-pc1-amd-g1-shadow-20261006, 15:46:27 to 15:56:54Z, exit 0) and what it found + +G1 on the AMD vendor 7 of 7 (every fingerprint equal to the Mac's and the 5090's, self-test PASS, 96 of 96 lanes), the card to itself (worker pid 7060 gone after 5 s, back as pid 4900 after the restore), the ladder's rates on the kit worker (the installed worker's cross-check on the control 18.922 against 18.923 MH/s): 18.92 MH/s at 930 ops, 19.31 at 49,700, 19.29 at 102,100, 19.11 at 199,600, 19.60 at 330,700, 19.07 for the 64-instruction block at 49,700; OpenCL build 42 to 51 ms for mx8, 418 to 429 ms for the 256-instruction shadow packs, 342 for the 64-instruction one; the daily build 70 to 74 ms. Two faults, both fixed on this branch (commit after e054ed7): + +| Fault | Cause | Fix | +|---|---|---| +| every LADDER row `watts=owed samples=0` though the one-shot helper call printed the 9070 XT at 140.0 W | `Start-Process powershell.exe -ArgumentList @(..., $tele, $samples)`: Windows PowerShell 5.1 joins an ArgumentList with spaces and does not quote, and the helper's path holds "Igneum Miner", so the wrapper's `$Exe` was `...\Programs\Igneum`, its `& $Exe -l 1` failed and nothing was written | the wrapper reads its paths from `IGNEUM_SAMPLER_EXE` and `IGNEUM_SAMPLER_OUT` in the environment (inherited by the child, never on a command line), the `-File` path is quoted, the wrapper logs its own start line with `exists=`; the sampler must prove itself in the 12 idle seconds (first three raw lines printed, a 9070 XT line with watts required) or the benches are not run and the job fails loudly with `RESULT watts error=` | +| the restore posted one entry's identities (2) to every key form while settings.json holds three gfx1201 entries (`amd:3:gfx1201` off / 2, `amd:gfx1201` on / 8, `amd:1:gfx1201` on / 2); the worker came back (pid 4900), but if the app's live key is `amd:gfx1201` its identities moved from 8 to 2 | one `$cardIdent` for every form | every entry is restored with ITS OWN enabled flag and identities (and cap, job 4), the derived forms first and the key as written last so an entry's own settings win on the live key; job 4 carries the same shape. OWED: a read of PC 1's settings.json (or the app's card tile) to confirm which gfx1201 key the app runs and whether its identities read 8 or 2 after the 15:56Z restore; if 2 and the app runs `amd:gfx1201`, one `POST api/cards {key: amd:gfx1201, enabled: true, identities: 8}` puts it back (the coordinator's call, not done by any job here) | + ## The AMD watts readback on PC 1 (what exists) `igneum-gpu-telemetry.exe` (proto-opencl/gpu-telemetry.c on master, shipped by `packaging/windows/make-payload.sh` since 0.3.10; the Ember Tune playbook found it in the install folder or under a `jobs\amd-kit-*\kit\` folder) prints one line per AMD card per sample: `amd bus kind discrete name "AMD Radeon RX 9070 XT" watts temp_c fan_rpm fan_pct

mclk_mhz gclk_mhz util_pct source adlx`, then `end N card(s)`, at `-l 1` once a second with `fflush` per sample. The watts field is ADLX `GPUPower` with `GPUTotalBoardPower` as the fallback (gpu-telemetry.c lines 120 to 121 on master); on 5 October it read the 9070 XT at 198.9 W at 17.73 MH/s (release-0.3.10.md, job `tele-measure-1`), the figure the app's MH per watt line uses. So a board-watts readback EXISTS and job 1 uses it: a second copy of the helper at 1 Hz through a wrapper that stamps each line (UTC, ms) so the samples window onto each bench (after its first 12 s, before its last 2 s, the PC 2 shape), ended by the wrapper's pid in `finally`. The job prints `RESULT tele helper= sha256= watts_source=adlx` after one sample; if the helper is missing or prints `watts -` for the 9070 XT the rows carry `watts=owed uj=owed` and the SUMMARY says `watts: owed`, never a guess. Whether ADLX `GPUPower` on RDNA 4 is total board power or ASIC power is not verified here: the row says `adlx` and the number is what the app itself reports. @@ -54,10 +64,10 @@ Read back: `node tools/jobs.mjs run-ca3-pc1-amd-g1-shadow-20261006` (and `--all` | `proto-opencl/family-probe.c` on Apple OpenCL (`cc -framework OpenCL`, M5 Max) | every C-form and emulated row bit-exact on both 32-lane groups in 3 repetitions; ratios to the alu chain beside the Metal probe's (rotr 1.14 against Metal 1.13, shl 0.89 / 0.85, shr 0.89 / 0.86, bfe 0.82 / 0.77, andn 0.84 / 0.75, popc 1.00 / 0.87, clz 1.11 / 1.01, sel 0.82 / 0.76, perm emulated 1.52 / 1.13, shflx and shfla through `__local` 1.96 and 2.04 (Metal native 0.86 and 1.91), dot4 emulated signed 4.66 / 4.73); the Apple `dot(char4,char4)` row exact=no as the 5 October finding said; every AMD builtin skipped (`build=skipped`), the khr, intel and amd_media_ops2 forms `build=failed` with the log line, the run went on; `FAMILYBEST` picked the exact rows | | `proto-opencl/host.c` with the build line, `--bench-pack` on all eight kit packs on Apple OpenCL | self-test PASS (96 of 96 lanes) and the 2^24 fingerprint at base 0 equal to the recorded one on every pack: mx8-devnet-epoch0 `90f794dd556f7a3b`, sh256x13 `59ac286fe2a5a9ef`, sh256x27 `3d2e8245cc084d07`, sh256x53 `4f824b15cf2b124a`, sh256x88 `0572522e39a94d8a`, sh64x52 `9dd010f79d8ca9f4`, dr736-genesis `50e3eaa779da4f1e`, dr736-devnet-epoch0 `9553f6d5c667205a`; `build` line: mx8 70 ms, shadow packs 70 to 88 ms, dr736 337 to 356 ms (+270 ms on Apple OpenCL: item 2's compile cost shows on this harness too) | | mingw cross-builds of both exes | clean (`-Wall -Wextra`), 116 KB and 35 KB | -| `tools/ci/bash-body-check.sh` on the four scripts | 0 bash bodies, all parse | -| `tools/ci/kit-path-check.sh` on the four scripts | 4 of 4 kit paths checked before use | +| `tools/ci/bash-body-check.sh` on the five scripts | 0 bash bodies, all parse | +| `tools/ci/kit-path-check.sh` on the five scripts | 5 of 5 kit paths checked before use | | `tools/ci/prover-socket-check.sh`, `copied-sources-check.sh`, `no-conflict-markers.sh` | pass | -| The Windows PowerShell 5.1 parse (`windows.yml` parse job) | the real parser runs only on the Windows runner and only over `relay/playbooks` and the other listed folders (not `tools/`); on the Mac the same rule's approximation (`tools/ci/check-workflow-shell.mjs`'s drive-reference regex, applied to these four files by hand) fired once on `"$dev: --stop-miners"` and the line was fixed to `${dev}:`; 0 findings after; brace, paren and bracket depth 0 by the bash-body-check tokenizer | +| The Windows PowerShell 5.1 parse (`windows.yml` parse job) | the real parser runs only on the Windows runner and only over `relay/playbooks` and the other listed folders (not `tools/`); on the Mac the same rule's approximation (`tools/ci/check-workflow-shell.mjs`'s drive-reference regex, applied to these five files by hand) fired once on `"$dev: --stop-miners"` and the line was fixed to `${dev}:`; 0 findings after; brace, paren and bracket depth 0 by the bash-body-check tokenizer | ## Not tested on the Mac (the PC side) diff --git a/tools/ca3-pc1-amd/pc1-4070-shadow.ps1 b/tools/ca3-pc1-amd/pc1-4070-shadow.ps1 index dfe9b43e4..e49a92aa2 100644 --- a/tools/ca3-pc1-amd/pc1-4070-shadow.ps1 +++ b/tools/ca3-pc1-amd/pc1-4070-shadow.ps1 @@ -56,29 +56,37 @@ if ($null -eq $idx) { "RESULT error no RTX 4070 in nvidia-smi's list"; Summary ' "RESULT device nvidia-smi index $idx name '$gpuName' limits $(Smi 'power.limit,power.min_limit,power.max_limit,clocks.max.sm' $idx)" # ---- the card's keys from settings.json (READ ONLY), every form ---- -$cardKeys = @(); $cardEnabled = $false; $cardIdent = 1; $cardPower = 0 +$entries = @() try { $sj = Get-Content -LiteralPath (Join-Path $appDir 'settings.json') -Raw | ConvertFrom-Json foreach ($prop in $sj.cards.PSObject.Properties) { if ($prop.Name -like 'nvidia:*' -and $prop.Name -match '4070') { $parts = $prop.Name -split ':' - $cardKeys += $prop.Name - if ($parts.Count -ge 3) { $cardKeys += ($parts[0] + ':' + $parts[1]); $cardKeys += ($parts[0] + ':' + (($parts | Select-Object -Skip 2) -join ':')) } - $cardEnabled = [bool] $prop.Value.enabled - if ($prop.Value.identities) { $cardIdent = [int] $prop.Value.identities } - if ($prop.Value.power_pct) { $cardPower = [int] $prop.Value.power_pct } - "RESULT card settings_key=$($prop.Name) enabled=$cardEnabled identities=$cardIdent power_pct=$cardPower forms=[$($cardKeys -join ' | ')]" + $forms = @() + if ($parts.Count -ge 3) { $forms += ($parts[0] + ':' + $parts[1]); $forms += ($parts[0] + ':' + (($parts | Select-Object -Skip 2) -join ':')) } + $ident = 1; if ($prop.Value.identities) { $ident = [int] $prop.Value.identities } + $pp = 0; if ($prop.Value.power_pct) { $pp = [int] $prop.Value.power_pct } + $entries += [pscustomobject]@{ key = $prop.Name; forms = $forms; enabled = [bool] $prop.Value.enabled; identities = $ident; power_pct = $pp } + "RESULT card settings_key=$($prop.Name) enabled=$([bool] $prop.Value.enabled) identities=$ident power_pct=$pp derived_forms=[$($forms -join ' | ')]" } } } catch { "RESULT settings_read_failed $($_.Exception.Message)" } -if ($cardKeys.Count -eq 0) { "RESULT card none in settings.json matching nvidia:*4070*: no switch; the rows below carry whatever load the card has" } +if ($entries.Count -eq 0) { "RESULT card none in settings.json matching nvidia:*4070*: no switch; the rows below carry whatever load the card has" } +# off: every form of every entry; restore: the derived forms first, then each key as written with ITS OWN settings, so an +# entry's own identities and cap win on the live key (the class found in job 1's first run: one entry's identities posted to every form) function PostCards([bool] $enabled) { if (-not $base) { "RESULT cards no app.url: nothing posted"; return } - foreach ($k in $cardKeys) { - $c = @{ key = $k; enabled = $enabled; identities = $cardIdent } - if ($cardPower -gt 0) { $c.power_pct = $cardPower } - $body = @{ cards = @($c) } | ConvertTo-Json -Depth 5 - try { Invoke-RestMethod -Uri "$base/api/cards" -Method POST -Body $body -ContentType 'application/json' -TimeoutSec 10 | Out-Null; "RESULT cards key=$k enabled=$enabled posted" } catch { "RESULT cards key=$k enabled=$enabled error=$($_.Exception.Message)" } + foreach ($pass in @('derived', 'written')) { + foreach ($e in $entries) { + $keys = if ($pass -eq 'derived') { $e.forms } else { @($e.key) } + $on = if ($enabled) { $e.enabled } else { $false } + foreach ($k in $keys) { + $c = @{ key = $k; enabled = $on; identities = $e.identities } + if ($e.power_pct -gt 0) { $c.power_pct = $e.power_pct } + $body = @{ cards = @($c) } | ConvertTo-Json -Depth 5 + try { Invoke-RestMethod -Uri "$base/api/cards" -Method POST -Body $body -ContentType 'application/json' -TimeoutSec 10 | Out-Null; "RESULT cards key=$k enabled=$on identities=$($e.identities) posted (from entry $($e.key))" } catch { "RESULT cards key=$k enabled=$on error=$($_.Exception.Message)" } + } + } } } function Workers { @(Get-CimInstance Win32_Process -Filter "Name = 'igneum-worker-opencl.exe' OR Name = 'igneum-worker-cuda.exe'" -ErrorAction SilentlyContinue | ForEach-Object { "$($_.Name):$($_.ProcessId):[$($_.CommandLine -replace '\s+', ' ')]" }) } @@ -88,11 +96,12 @@ function CudaApps { @(& nvidia-smi -i $idx --query-compute-apps=pid,process_name $samples = Join-Path $job 'smi-samples.csv' $sampler = $null +$samplerOk = $false $results = @() $cardAlone = $false $switched = $false try { - if ($cardKeys.Count -gt 0 -and $base) { + if ($entries.Count -gt 0 -and $base) { $before = @(CardWorkers) if ($before.Count -eq 0 -and (@(CudaApps)).Count -eq 0) { "RESULT card already quiet before the switch (no cuda worker with --device $idx, no compute app on index $idx)"; $cardAlone = $true } else { @@ -104,7 +113,7 @@ try { else { "RESULT card $(Stamp) UNCONFIRMED after $t s: worker pids [$($left -join ' ')] compute apps [$($apps -join '; ')]; the rows below are LOADED-card figures" } Start-Sleep -Seconds 5 } - } else { "RESULT card not switched (no key or no app.url): the rows below carry whatever load the card has" } + } else { "RESULT card not switched (no entry or no app.url): the rows below carry whatever load the card has" } "RESULT workers_during $(Stamp) $((Workers) -join ' ')" "RESULT gpu_before $(Stamp) $(Smi 'power.draw,power.limit,clocks.sm,clocks.mem,temperature.gpu,memory.used' $idx)" # the sampler: nvidia-smi on this index every second into a file (its own timestamps), ended in finally @@ -113,9 +122,20 @@ try { Remove-Item -LiteralPath $samples -Force -ErrorAction SilentlyContinue $sampler = Start-Process -FilePath $smiExe -ArgumentList @('-i', "$idx", '--query-gpu=timestamp,power.draw,utilization.gpu,clocks.sm,clocks.mem,temperature.gpu,memory.used', '--format=csv,noheader,nounits', '-l', '1') -RedirectStandardOutput $samples -NoNewWindow -PassThru Start-Sleep -Seconds 12 - "RESULT idle $(Stamp) 12 s of idle samples before the first bench: $(Smi 'power.draw,utilization.gpu,clocks.sm' $idx)" + # the sampler PROVES itself in the idle window: the first three raw nvidia-smi lines, and a line with a power number, or no benches + $raw = @(); if (Test-Path -LiteralPath $samples) { $raw = @(Get-Content -LiteralPath $samples) } + $raw | Select-Object -First 3 | ForEach-Object { "RESULT sampler_raw $_" } + $nPower = @($raw | Where-Object { $_ -match '^\d{4}/\d\d/\d\d \d\d:\d\d:\d\d\.\d+,\s*[\d.]+,' }).Count + "RESULT sampler_idle lines=$($raw.Count) lines_with_power=$nPower sampler_alive=$(-not $sampler.HasExited) now: $(Smi 'power.draw,utilization.gpu,clocks.sm' $idx)" + $samplerOk = $nPower -gt 0 + if (-not $samplerOk) { + $why = if ($raw.Count -eq 0) { 'nvidia-smi -l 1 wrote nothing in 12 s' } else { 'nvidia-smi wrote ' + $raw.Count + ' line(s) but none carries a timestamp and a power number (first lines above)' } + "RESULT watts error=$why; the ladder is NOT run blind" + foreach ($pk in $order) { "RESULT LADDER pack=$pk error=$why" } + } else { "RESULT idle $(Stamp) sampler proven on the idle window" } $i = 0 foreach ($pk in $order) { + if (-not $samplerOk) { break } $i++ $d = Join-Path $packs $pk if (-not (Test-Path $d)) { "RESULT G1 pack=$pk error=pack missing at $d"; "RESULT LADDER pack=$pk error=pack missing"; continue } @@ -152,10 +172,12 @@ try { } finally { if ($sampler) { try { Stop-Process -Id $sampler.Id -Force -ErrorAction SilentlyContinue; "RESULT sampler ended pid=$($sampler.Id)" } catch { } } if ($switched) { - if ($cardEnabled) { PostCards $true; "RESULT card restore posted enabled=true (settings.json had it enabled)" } else { "RESULT card restore skipped: settings.json had the card disabled before this job" } + PostCards $true + "RESULT card restore posted: each settings.json entry with its own enabled flag, identities and cap" $t = 0; $back = @() - while ($t -lt 90 -and $cardEnabled) { Start-Sleep -Seconds 5; $t += 5; $back = @(CardWorkers); if ($back.Count -gt 0) { break } } - "RESULT card_workers_after $(Stamp) pids=[$($back -join ' ')] after $t s $(if ($back.Count -gt 0) { '(the card mines again)' } elseif ($cardEnabled) { '(NOT back yet: check the app)' } else { '' })" + $anyEnabled = ($entries | Where-Object { $_.enabled }).Count -gt 0 + while ($t -lt 90 -and $anyEnabled) { Start-Sleep -Seconds 5; $t += 5; $back = @(CardWorkers); if ($back.Count -gt 0) { break } } + "RESULT card_workers_after $(Stamp) pids=[$($back -join ' ')] after $t s $(if ($back.Count -gt 0) { '(the card mines again)' } elseif ($anyEnabled) { '(NOT back yet: check the app)' } else { '(no entry was enabled before this job)' })" } "RESULT workers_after $(Stamp) $((Workers) -join ' ')" "RESULT gpu_after $(Stamp) $(Smi 'power.draw,power.limit,clocks.sm,clocks.mem,temperature.gpu,memory.used' $idx)" @@ -186,8 +208,8 @@ foreach ($r in $results) { "RESULT LADDER pack=$($r.pack) ops=$($r.ops) shadow_instrs=$($r.instrs) mhs=$($r.mhs) watts=$watts uj=$uj watts_min=$wmin watts_max=$wmax sm_mhz=$sm mem_mhz=$mem util=$util temp_max=$tmax samples=$($in.Count) window_s=$([int]($r.t1 - $r.t0).TotalSeconds) nvrtc_ms=$($r.nvrtc_ms) dataset_ms=$($r.dataset_ms) fingerprint=$($r.fp) match=$($r.match) card_alone=$cardAlone device=$($r.dev) tag=$($r.tag)" } foreach ($r in $rows) { "RESULT sample $($r.ts.ToString('HH:mm:ss')) $($r.w) $($r.util) $($r.sm) $($r.mem) $($r.temp) $($r.used)" } -$status = if ($results.Count -eq $order.Count) { 'done' } elseif ($results.Count -gt 0) { 'partial' } else { 'failed' } +$status = if (-not $samplerOk) { 'failed' } elseif ($results.Count -eq $order.Count) { 'done' } elseif ($results.Count -gt 0) { 'partial' } else { 'failed' } "RESULT end $(Stamp)" -Summary $status @{ packs_run = $results.Count; packs_wanted = $order.Count; g1_match = $g1Match; card_alone = $cardAlone; smi_index = $idx; device = $gpuName } +Summary $status @{ packs_run = $results.Count; packs_wanted = $order.Count; g1_match = $g1Match; sampler_proven = $samplerOk; card_alone = $cardAlone; smi_index = $idx; device = $gpuName } if ($status -eq 'failed') { exit 1 } exit 0 diff --git a/tools/ca3-pc1-amd/pc1-amd-g1-shadow.ps1 b/tools/ca3-pc1-amd/pc1-amd-g1-shadow.ps1 index 6182e0151..452e98cd0 100644 --- a/tools/ca3-pc1-amd/pc1-amd-g1-shadow.ps1 +++ b/tools/ca3-pc1-amd/pc1-amd-g1-shadow.ps1 @@ -78,26 +78,35 @@ if ($null -eq $dev) { "RESULT error no gfx1201 device in --list (the eGPU is off "RESULT device $dev gfx1201 (list from $listExe)" # ---- the card's keys from settings.json (READ ONLY), every form ---- -$cardKeys = @(); $cardEnabled = $false; $cardIdent = 1 +$entries = @() try { $sj = Get-Content -LiteralPath (Join-Path $appDir 'settings.json') -Raw | ConvertFrom-Json foreach ($prop in $sj.cards.PSObject.Properties) { if ($prop.Name -like 'amd:*' -and $prop.Name -match 'gfx1201') { $parts = $prop.Name -split ':' - $cardKeys += $prop.Name - if ($parts.Count -ge 3) { $cardKeys += ($parts[0] + ':' + $parts[1]); $cardKeys += ($parts[0] + ':' + $parts[2]) } - $cardEnabled = [bool] $prop.Value.enabled - if ($prop.Value.identities) { $cardIdent = [int] $prop.Value.identities } - "RESULT card settings_key=$($prop.Name) enabled=$cardEnabled identities=$cardIdent forms=[$($cardKeys -join ' | ')]" + $forms = @() + if ($parts.Count -ge 3) { $forms += ($parts[0] + ':' + $parts[1]); $forms += ($parts[0] + ':' + $parts[2]) } + $ident = 1; if ($prop.Value.identities) { $ident = [int] $prop.Value.identities } + $entries += [pscustomobject]@{ key = $prop.Name; forms = $forms; enabled = [bool] $prop.Value.enabled; identities = $ident } + "RESULT card settings_key=$($prop.Name) enabled=$([bool] $prop.Value.enabled) identities=$ident derived_forms=[$($forms -join ' | ')]" } } } catch { "RESULT settings_read_failed $($_.Exception.Message)" } -if ($cardKeys.Count -eq 0) { "RESULT card none in settings.json matching amd:*gfx1201*: no switch; the rows below carry whatever load the card has" } +if ($entries.Count -eq 0) { "RESULT card none in settings.json matching amd:*gfx1201*: no switch; the rows below carry whatever load the card has" } +# off: every form of every entry; restore: the derived forms first, then each key as written with ITS OWN enabled flag and +# identities, so an entry's own settings win on the live key (the first run of this job, 15:46Z, posted one entry's +# identities to every form while settings.json held three gfx1201 entries: the class fixed here) function PostCards([bool] $enabled) { if (-not $base) { "RESULT cards no app.url: nothing posted"; return } - foreach ($k in $cardKeys) { - $body = @{ cards = @(@{ key = $k; enabled = $enabled; identities = $cardIdent }) } | ConvertTo-Json -Depth 5 - try { Invoke-RestMethod -Uri "$base/api/cards" -Method POST -Body $body -ContentType 'application/json' -TimeoutSec 10 | Out-Null; "RESULT cards key=$k enabled=$enabled posted" } catch { "RESULT cards key=$k enabled=$enabled error=$($_.Exception.Message)" } + foreach ($pass in @('derived', 'written')) { + foreach ($e in $entries) { + $keys = if ($pass -eq 'derived') { $e.forms } else { @($e.key) } + $on = if ($enabled) { $e.enabled } else { $false } + foreach ($k in $keys) { + $body = @{ cards = @(@{ key = $k; enabled = $on; identities = $e.identities }) } | ConvertTo-Json -Depth 5 + try { Invoke-RestMethod -Uri "$base/api/cards" -Method POST -Body $body -ContentType 'application/json' -TimeoutSec 10 | Out-Null; "RESULT cards key=$k enabled=$on identities=$($e.identities) posted (from entry $($e.key))" } catch { "RESULT cards key=$k enabled=$on error=$($_.Exception.Message)" } + } + } } } function Workers { @@ -127,7 +136,7 @@ $cardAlone = $false $switched = $false try { # ---- the card off, confirmed by the process list (never api/state) ---- - if ($cardKeys.Count -gt 0 -and $base) { + if ($entries.Count -gt 0 -and $base) { $before = @(CardWorkers) "RESULT card_workers_before pids=[$($before -join ' ')]" if ($before.Count -eq 0) { "RESULT card already quiet before the switch (no opencl worker with --device ${dev}: --stop-miners or the card is off in the app)"; $cardAlone = $true } @@ -140,27 +149,46 @@ try { else { "RESULT card $(Stamp) UNCONFIRMED after $t s: opencl worker pids [$($left -join ' ')] still carry --device $dev; the rows below are LOADED-card figures" } Start-Sleep -Seconds 5 } - } else { "RESULT card not switched (no key or no app.url): the rows below carry whatever load the card has" } + } else { "RESULT card not switched (no entry or no app.url): the rows below carry whatever load the card has" } "RESULT workers_during $(Stamp) $((Workers) -join ' ')" - # ---- the sampler: the helper at 1 Hz through a wrapper (timestamps added per line), ended by its own pid in finally ---- + # ---- the sampler: the helper at 1 Hz through a wrapper; its paths through the ENVIRONMENT (inherited by the child, + # never on a command line: Start-Process -ArgumentList in Windows PowerShell 5.1 joins arguments WITHOUT quoting, + # and "Igneum Miner" holds a space, which is why the 15:46Z run wrote no samples), the -File path quoted; ended + # by its own pid in finally. The sampler PROVES itself in the 12 idle seconds: the first three raw lines are + # printed, and without a 9070 XT line with watts the ladder is NOT run blind (the rates are on record from 15:46Z). ---- + $samplerOk = $false if ($tele -and $wattsSource -ne 'owed') { $wrapper = Join-Path $job 'sampler.ps1' $wrapperText = @' -param([string] $Exe, [string] $Out) -& $Exe -l 1 2>&1 | ForEach-Object { Add-Content -LiteralPath $Out -Value ((Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fff') + ' ' + $_) } +$exe = $env:IGNEUM_SAMPLER_EXE +$out = $env:IGNEUM_SAMPLER_OUT +Add-Content -LiteralPath $out -Value ('wrapper start ' + (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fff') + ' exe=' + $exe + ' exists=' + (Test-Path -LiteralPath $exe)) +& $exe -l 1 2>&1 | ForEach-Object { Add-Content -LiteralPath $out -Value ((Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fff') + ' ' + $_) } +Add-Content -LiteralPath $out -Value ('wrapper end ' + (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fff') + ' exit=' + $LASTEXITCODE) '@ [IO.File]::WriteAllText($wrapper, $wrapperText, (New-Object System.Text.UTF8Encoding $false)) Remove-Item -LiteralPath $samples -Force -ErrorAction SilentlyContinue - $sampler = Start-Process -FilePath 'powershell.exe' -ArgumentList @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', $wrapper, $tele, $samples) -WindowStyle Hidden -PassThru - "RESULT sampler pid=$($sampler.Id) file=$samples" + $env:IGNEUM_SAMPLER_EXE = $tele + $env:IGNEUM_SAMPLER_OUT = $samples + $sampler = Start-Process -FilePath 'powershell.exe' -ArgumentList @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', ('"' + $wrapper + '"')) -WindowStyle Hidden -PassThru + "RESULT sampler pid=$($sampler.Id) wrapper=$wrapper file=$samples exe_via_env=$tele" Start-Sleep -Seconds 12 - "RESULT idle $(Stamp) 12 s of idle samples taken before the first bench" - } + $raw = @(); if (Test-Path -LiteralPath $samples) { $raw = @(Get-Content -LiteralPath $samples) } + $raw | Select-Object -First 3 | ForEach-Object { "RESULT sampler_raw $_" } + $n9070 = @($raw | Where-Object { $_ -match '^\S+ amd \d+ .*name "[^"]*9070[^"]*" watts [\d.]+ ' }).Count + "RESULT sampler_idle lines=$($raw.Count) lines_9070_with_watts=$n9070 sampler_alive=$(-not $sampler.HasExited)" + if ($n9070 -eq 0) { + $why = if ($raw.Count -eq 0) { 'the sampler wrote nothing in 12 s (file absent or empty)' } else { 'the sampler wrote ' + $raw.Count + ' line(s) but none is a 9070 XT line with a watts number (first lines above)' } + "RESULT watts error=$why; the ladder is NOT run blind" + foreach ($pk in $order) { "RESULT LADDER pack=$pk error=$why" } + } else { $samplerOk = $true; "RESULT idle $(Stamp) sampler proven on the idle window" } + } else { "RESULT watts error=no helper or no watts in its one-shot sample: the ladder runs with watts owed"; $samplerOk = $true } # ---- the ladder ---- $i = 0 foreach ($pk in $order) { + if (-not $samplerOk) { break } $i++ $d = Join-Path $packs $pk if (-not (Test-Path $d)) { "RESULT G1 pack=$pk error=pack missing at $d"; "RESULT LADDER pack=$pk error=pack missing"; continue } @@ -219,10 +247,12 @@ param([string] $Exe, [string] $Out) # ---- whatever happened: the sampler ended by its own pid (never by name: the app runs its own copy of the helper), the card back ---- if ($sampler) { try { & taskkill /T /F /PID $sampler.Id 2>&1 | Out-Null; "RESULT sampler ended pid=$($sampler.Id)" } catch { "RESULT sampler end error=$($_.Exception.Message)" } } if ($switched) { - if ($cardEnabled) { PostCards $true; "RESULT card restore posted enabled=true (settings.json had it enabled)" } else { "RESULT card restore skipped: settings.json had the card disabled before this job" } + PostCards $true + "RESULT card restore posted: each settings.json entry with its own enabled flag and identities" $t = 0; $back = @() - while ($t -lt 90 -and $cardEnabled) { Start-Sleep -Seconds 5; $t += 5; $back = @(CardWorkers); if ($back.Count -gt 0) { break } } - "RESULT card_workers_after $(Stamp) pids=[$($back -join ' ')] after $t s $(if ($back.Count -gt 0) { '(the card mines again)' } elseif ($cardEnabled) { '(NOT back yet: check the app)' } else { '' })" + $anyEnabled = ($entries | Where-Object { $_.enabled }).Count -gt 0 + while ($t -lt 90 -and $anyEnabled) { Start-Sleep -Seconds 5; $t += 5; $back = @(CardWorkers); if ($back.Count -gt 0) { break } } + "RESULT card_workers_after $(Stamp) pids=[$($back -join ' ')] after $t s $(if ($back.Count -gt 0) { '(the card mines again)' } elseif ($anyEnabled) { '(NOT back yet: check the app)' } else { '(no entry was enabled before this job)' })" } "RESULT workers_after $(Stamp) $((Workers) -join ' ')" } @@ -263,8 +293,8 @@ if ($rows.Count -gt 0) { if ($idle.Count -gt 0) { "RESULT idle_power samples=$($idle.Count) watts_mean=$([math]::Round(($idle | Measure-Object -Property w -Average).Average, 1)) (the card switched off in the app, before the first bench)" } foreach ($r in $rows) { "RESULT sample $($r.ts.ToString('HH:mm:ss')) $($r.w) $($r.temp) $($r.fan) $($r.mclk) $($r.gclk) $($r.util) $($r.src)" } } -$status = if ($results.Count -eq $order.Count) { 'done' } elseif ($results.Count -gt 0) { 'partial' } else { 'failed' } +$status = if (-not $samplerOk) { 'failed' } elseif ($results.Count -eq $order.Count) { 'done' } elseif ($results.Count -gt 0) { 'partial' } else { 'failed' } "RESULT end $(Stamp)" -Summary $status @{ packs_run = $results.Count; packs_wanted = $order.Count; g1_match = $g1Match; ladder_rows = $ladderRows; watts = $wattsSource; card_alone = $cardAlone; device = $dev } +Summary $status @{ packs_run = $results.Count; packs_wanted = $order.Count; g1_match = $g1Match; ladder_rows = $ladderRows; watts = $wattsSource; sampler_proven = $samplerOk; card_alone = $cardAlone; device = $dev } if ($status -eq 'failed') { exit 1 } exit 0 diff --git a/tools/ca3-pc1-amd/pc1-amd-watts.ps1 b/tools/ca3-pc1-amd/pc1-amd-watts.ps1 new file mode 100644 index 000000000..687eebcd3 --- /dev/null +++ b/tools/ca3-pc1-amd/pc1-amd-watts.ps1 @@ -0,0 +1,240 @@ +# Counter ASIC 3.0, PC 1 AMD job 1b (6 October 2026, 16:xx UTC): the WATTS re-run for the two rows job 1 left owed. +# Job 1 (run-ca3-pc1-amd-g1-shadow-20261006, 15:46 to 15:56Z) gave G1 7 of 7 and the ladder's rates but every +# LADDER row read watts=owed with samples=0: the helper's one-shot call worked (140.0 W on the 9070 XT) and the 1 Hz +# sampler wrote nothing, because Start-Process -ArgumentList in Windows PowerShell 5.1 joins its arguments WITHOUT +# quoting, and the helper's path holds a space ("Igneum Miner"), so the wrapper received a split path and its +# `& $Exe -l 1` failed silently. Here the wrapper takes its paths from environment variables (inherited by the child, +# no quoting) and the -File path is quoted; and the sampler PROVES ITSELF in the 12 idle seconds before any bench: +# the first three raw helper lines are printed, and if no 9070 XT line with watts arrived the job prints +# `RESULT watts error=` and ends WITHOUT the benches (the card restored in finally), never running blind. +# Packs: the class v3 control mx8-devnet-epoch0 and the class v4 candidate sh256x27, 90 s each, the 9070 XT ALONE +# (switched off in the installed app through POST api/cards with every key form of every settings.json entry for +# the card, confirmed quiet by the process list, each entry restored with ITS OWN settings in finally). Plain signed +# `run` job (NOT --stop-miners): the 5090 and the 4070 keep mining. Never quits, pauses, resumes or updates the app, +# never writes settings.json. Watts = ADLX board watts from the installed app's own igneum-gpu-telemetry.exe +# (a second copy at -l 1), the mean over each bench window after its first 12 s and before its last 2 s; +# microjoules per hash = watts / MH/s. Lines: `RESULT LADDER pack= ops= mhs= watts= uj= ...`, +# `RESULT G1 ...` again for the two packs, a `SUMMARY {json}` line. Read back with `node tools/jobs.mjs `. +$ErrorActionPreference = 'Continue' +$jobName = 'pc1-amd-watts' +$kitId = 'fetch-ca3-pc1-amd-20261006' +$started = Get-Date +function Stamp { (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ') } +function Summary([string] $status, [hashtable] $extra) { + $o = [ordered]@{ job = $jobName; status = $status; duration_s = [int]((Get-Date) - $started).TotalSeconds; finished_at = (Stamp) } + foreach ($k in $extra.Keys) { $o[$k] = $extra[$k] } + 'SUMMARY ' + ($o | ConvertTo-Json -Compress -Depth 4) +} +$expected = @{ 'mx8-devnet-epoch0' = '90f794dd556f7a3b'; 'sh256x27' = '3d2e8245cc084d07' } +$opsPerHash = @{ 'mx8-devnet-epoch0' = 930; 'sh256x27' = 102100 } +# 100 dispatches of 2^24 at the 18.9 to 19.3 MH/s job 1 measured = 87 to 89 s per pack +$batches = @{ 'mx8-devnet-epoch0' = 100; 'sh256x27' = 100 } +$order = @('mx8-devnet-epoch0', 'sh256x27') +"RESULT start $(Stamp) job=$jobName machine=$env:COMPUTERNAME app_version=$env:IGNEUM_APP_VERSION" +$jobs = Split-Path $env:IGNEUM_JOB_DIR +$kit = Join-Path $jobs $kitId +if (-not (Test-Path $kit)) { "RESULT error kit missing at $kit (the fetch job $kitId runs first; republish it after any app update)"; Summary 'failed' @{ error = 'kit missing' }; exit 2 } +$exe = Join-Path $kit 'bin\igneum-worker-opencl.exe' +$packs = Join-Path $kit 'packs' +if (-not (Test-Path $exe)) { "RESULT error worker missing at $exe"; Summary 'failed' @{ error = 'worker missing' }; exit 2 } +if (-not (Test-Path $packs)) { "RESULT error packs missing at $packs"; Summary 'failed' @{ error = 'packs missing' }; exit 2 } +"RESULT worker kit $exe sha256 $((Get-FileHash -Algorithm SHA256 $exe).Hash.ToLower()) bytes $((Get-Item $exe).Length)" +$job = $env:IGNEUM_JOB_DIR; if (-not $job) { $job = Join-Path $env:TEMP 'igneum-ca3-pc1-amd' }; New-Item -ItemType Directory -Force -Path $job | Out-Null +$inst = @("$env:LOCALAPPDATA\Programs\Igneum Miner", "$env:ProgramFiles\Igneum Miner") | Where-Object { Test-Path (Join-Path $_ 'igneum-app.exe') } | Select-Object -First 1 +$appDir = if ($env:IGNEUM_APP_DIR) { $env:IGNEUM_APP_DIR } else { Join-Path $env:LOCALAPPDATA 'igneum\app' } +$urlFile = Join-Path $appDir 'app.url' +$base = $null +if (Test-Path $urlFile) { $base = (Get-Content -LiteralPath $urlFile -Raw).Trim().TrimEnd('/') } +"RESULT app install=$inst app_dir=$appDir url_present=$([bool]$base)" +$listExe = $exe +if ($inst -and (Test-Path (Join-Path $inst 'igneum-worker-opencl.exe'))) { $listExe = Join-Path $inst 'igneum-worker-opencl.exe' } +$list = @(& $listExe --list 2>&1 | ForEach-Object { "$_" }) +$list | ForEach-Object { "RESULT list $_" } +$dev = $null +foreach ($l in $list) { if ($l -match '^\s*\[(\d+)\].*gfx1201' -and $l -notmatch 'dup') { $dev = [int]$Matches[1]; break } } +if ($null -eq $dev) { "RESULT error no gfx1201 device in --list (the eGPU is off the bus: every row OWED)"; Summary 'failed' @{ error = 'no gfx1201' }; exit 2 } +"RESULT device $dev gfx1201 (list from $listExe)" + +# ---- every settings.json entry for the card (READ ONLY), each with its own settings and its key forms ---- +$entries = @() +try { + $sj = Get-Content -LiteralPath (Join-Path $appDir 'settings.json') -Raw | ConvertFrom-Json + foreach ($prop in $sj.cards.PSObject.Properties) { + if ($prop.Name -like 'amd:*' -and $prop.Name -match 'gfx1201') { + $parts = $prop.Name -split ':' + $forms = @() + if ($parts.Count -ge 3) { $forms += ($parts[0] + ':' + $parts[1]); $forms += ($parts[0] + ':' + $parts[2]) } + $ident = 1; if ($prop.Value.identities) { $ident = [int] $prop.Value.identities } + $entries += [pscustomobject]@{ key = $prop.Name; forms = $forms; enabled = [bool] $prop.Value.enabled; identities = $ident } + "RESULT card settings_key=$($prop.Name) enabled=$([bool] $prop.Value.enabled) identities=$ident derived_forms=[$($forms -join ' | ')]" + } + } +} catch { "RESULT settings_read_failed $($_.Exception.Message)" } +if ($entries.Count -eq 0) { "RESULT card none in settings.json matching amd:*gfx1201*: no switch; the rows below carry whatever load the card has" } +# off: every form of every entry; restore: the derived forms first, then each key as written with ITS OWN settings, so an +# entry's own identities win on the live key (job 1 posted one entry's identities to every form: the class fixed here) +function PostCards([bool] $enabled) { + if (-not $base) { "RESULT cards no app.url: nothing posted"; return } + foreach ($pass in @('derived', 'written')) { + foreach ($e in $entries) { + $keys = if ($pass -eq 'derived') { $e.forms } else { @($e.key) } + $on = if ($enabled) { $e.enabled } else { $false } + foreach ($k in $keys) { + $body = @{ cards = @(@{ key = $k; enabled = $on; identities = $e.identities }) } | ConvertTo-Json -Depth 5 + try { Invoke-RestMethod -Uri "$base/api/cards" -Method POST -Body $body -ContentType 'application/json' -TimeoutSec 10 | Out-Null; "RESULT cards key=$k enabled=$on identities=$($e.identities) posted (from entry $($e.key))" } catch { "RESULT cards key=$k enabled=$on error=$($_.Exception.Message)" } + } + } + } +} +function Workers { @(Get-CimInstance Win32_Process -Filter "Name = 'igneum-worker-opencl.exe' OR Name = 'igneum-worker-cuda.exe'" -ErrorAction SilentlyContinue | ForEach-Object { "$($_.Name):$($_.ProcessId):[$($_.CommandLine -replace '\s+', ' ')]" }) } +function CardWorkers { @(Get-CimInstance Win32_Process -Filter "Name = 'igneum-worker-opencl.exe'" -ErrorAction SilentlyContinue | Where-Object { $_.CommandLine -match "--device\s+$dev(\s|$)" } | ForEach-Object { $_.ProcessId }) } +"RESULT workers_before $(Stamp) $((Workers) -join ' ')" + +# ---- the AMD helper: one in-process sample first (the path that worked in job 1) ---- +$tele = $null +if ($inst -and (Test-Path (Join-Path $inst 'igneum-gpu-telemetry.exe'))) { $tele = Join-Path $inst 'igneum-gpu-telemetry.exe' } +if (-not $tele) { $found = Get-ChildItem -Path (Join-Path $appDir 'jobs') -Recurse -Filter 'igneum-gpu-telemetry.exe' -ErrorAction SilentlyContinue | Where-Object { $_.FullName -match 'amd-kit' } | Sort-Object LastWriteTime -Descending | Select-Object -First 1; if ($found) { $tele = $found.FullName } } +if (-not $tele) { "RESULT watts error=no igneum-gpu-telemetry.exe in the install folder or under an amd-kit job folder"; Summary 'failed' @{ error = 'no helper' }; exit 2 } +$one = @(& $tele 2>&1 | ForEach-Object { "$_" }) +$one | ForEach-Object { "RESULT tele_once $_" } +$line9070 = $one | Where-Object { $_ -match '^amd \d+ .*name "[^"]*9070[^"]*" watts (\S+) ' } | Select-Object -First 1 +if (-not $line9070 -or $line9070 -notmatch ' watts ([\d.]+) ') { "RESULT watts error=the helper's one-shot sample has no 9070 XT line with a watts number"; Summary 'failed' @{ error = 'no 9070 watts in the one-shot sample' }; exit 2 } +"RESULT tele helper=$tele sha256=$((Get-FileHash -Algorithm SHA256 $tele).Hash.ToLower()) one_shot_watts=$($Matches[1]) (ADLX GPUPower, GPUTotalBoardPower as the fallback)" + +$samples = Join-Path $job 'amd-samples.log' +$sampler = $null +$results = @() +$cardAlone = $false +$switched = $false +$samplerOk = $false +try { + # ---- the card off, confirmed by the process list ---- + if ($entries.Count -gt 0 -and $base) { + $before = @(CardWorkers) + "RESULT card_workers_before pids=[$($before -join ' ')]" + if ($before.Count -eq 0) { "RESULT card already quiet before the switch (no opencl worker with --device ${dev}: --stop-miners or the card is off in the app)"; $cardAlone = $true } + else { + PostCards $false; $switched = $true + $t = 0 + while ($t -lt 150) { Start-Sleep -Seconds 5; $t += 5; if ((@(CardWorkers)).Count -eq 0) { break } } + $left = @(CardWorkers) + if ($left.Count -eq 0) { "RESULT card $(Stamp) quiet after $t s: no igneum-worker-opencl process with --device $dev (the card to itself)"; $cardAlone = $true } + else { "RESULT card $(Stamp) UNCONFIRMED after $t s: opencl worker pids [$($left -join ' ')] still carry --device $dev; the rows below are LOADED-card figures" } + Start-Sleep -Seconds 5 + } + } else { "RESULT card not switched (no entry or no app.url): the rows below carry whatever load the card has" } + "RESULT workers_during $(Stamp) $((Workers) -join ' ')" + + # ---- the sampler: the helper at 1 Hz through a wrapper; paths through the ENVIRONMENT (inherited, never quoted on a + # command line), the -File path quoted; ended by its own pid in finally ---- + $wrapper = Join-Path $job 'sampler.ps1' + $wrapperText = @' +$exe = $env:IGNEUM_SAMPLER_EXE +$out = $env:IGNEUM_SAMPLER_OUT +Add-Content -LiteralPath $out -Value ('wrapper start ' + (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fff') + ' exe=' + $exe + ' exists=' + (Test-Path -LiteralPath $exe)) +& $exe -l 1 2>&1 | ForEach-Object { Add-Content -LiteralPath $out -Value ((Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fff') + ' ' + $_) } +Add-Content -LiteralPath $out -Value ('wrapper end ' + (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fff') + ' exit=' + $LASTEXITCODE) +'@ + [IO.File]::WriteAllText($wrapper, $wrapperText, (New-Object System.Text.UTF8Encoding $false)) + Remove-Item -LiteralPath $samples -Force -ErrorAction SilentlyContinue + $env:IGNEUM_SAMPLER_EXE = $tele + $env:IGNEUM_SAMPLER_OUT = $samples + $sampler = Start-Process -FilePath 'powershell.exe' -ArgumentList @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', ('"' + $wrapper + '"')) -WindowStyle Hidden -PassThru + "RESULT sampler pid=$($sampler.Id) wrapper=$wrapper file=$samples exe_via_env=$tele" + Start-Sleep -Seconds 12 + # ---- the proof: the first three raw lines, and a 9070 XT line with watts inside the idle window, or no benches ---- + $raw = @(); if (Test-Path -LiteralPath $samples) { $raw = @(Get-Content -LiteralPath $samples) } + $raw | Select-Object -First 3 | ForEach-Object { "RESULT sampler_raw $_" } + $n9070 = @($raw | Where-Object { $_ -match '^\S+ amd \d+ .*name "[^"]*9070[^"]*" watts [\d.]+ ' }).Count + "RESULT sampler_idle lines=$($raw.Count) lines_9070_with_watts=$n9070 sampler_alive=$(-not $sampler.HasExited)" + if ($n9070 -eq 0) { + $why = if ($raw.Count -eq 0) { 'the sampler wrote nothing in 12 s (file absent or empty)' } else { 'the sampler wrote ' + $raw.Count + ' line(s) but none is a 9070 XT line with a watts number (first lines above)' } + "RESULT watts error=$why; the benches are NOT run blind" + "RESULT LADDER pack=mx8-devnet-epoch0 error=$why" + "RESULT LADDER pack=sh256x27 error=$why" + } else { + $samplerOk = $true + "RESULT idle $(Stamp) sampler proven on the idle window" + $i = 0 + foreach ($pk in $order) { + $i++ + $d = Join-Path $packs $pk + if (-not (Test-Path $d)) { "RESULT G1 pack=$pk error=pack missing at $d"; "RESULT LADDER pack=$pk error=pack missing"; continue } + $instrs = 0 + try { $pj = Get-Content -LiteralPath (Join-Path $d 'program.json') -Raw | ConvertFrom-Json; if ($pj.shadow) { $instrs = [int] $pj.shadow.instrs_per_hash } } catch { } + $n = $batches[$pk] + $tag = "$pk#$i" + $t0 = (Get-Date).ToUniversalTime() + "RESULT bench $tag start $(Stamp) batches=$n shadow_instrs_per_hash=$instrs" + $lines = @(& $exe --bench-pack --pack $d --batches $n --batch-log2 24 --device $dev 2>&1 | ForEach-Object { "$_" }) + $rc = $LASTEXITCODE + $t1 = (Get-Date).ToUniversalTime() + $lines | ForEach-Object { "RESULT bench $tag $_" } + "RESULT bench $tag end $(Stamp) exit=$rc wall_s=$([int]($t1 - $t0).TotalSeconds)" + $res = $lines | Where-Object { $_ -match '^RESULT pack=' } | Select-Object -First 1 + $fp = ''; $mhs = ''; $check = '' + if ($res -and $res -match 'fingerprint=([0-9a-f]{16})') { $fp = $Matches[1] } + if ($res -and $res -match 'mhs=([\d.]+)') { $mhs = $Matches[1] } + if ($res -and $res -match 'check=(\S+)') { $check = $Matches[1] } + if (-not $res -or $rc -ne 0) { + $err = ($lines | Where-Object { $_ -match 'FAIL|error|MISMATCH' } | Select-Object -First 1); if (-not $err) { $err = "exit $rc, no RESULT line" } + "RESULT G1 pack=$pk error=$($err -replace '\s+', ' ')"; "RESULT LADDER pack=$pk error=$($err -replace '\s+', ' ')"; continue + } + $match = if ($fp -eq $expected[$pk]) { 'yes' } else { 'no' } + "RESULT G1 pack=$pk fingerprint=$fp match=$match expected=$($expected[$pk]) check=$check card=gfx1201 harness=opencl-kit" + $results += [pscustomobject]@{ tag = $tag; pack = $pk; instrs = $instrs; ops = $opsPerHash[$pk]; mhs = $mhs; fp = $fp; match = $match; t0 = $t0; t1 = $t1 } + Start-Sleep -Seconds 3 + } + } +} finally { + if ($sampler) { try { & taskkill /T /F /PID $sampler.Id 2>&1 | Out-Null; "RESULT sampler ended pid=$($sampler.Id)" } catch { "RESULT sampler end error=$($_.Exception.Message)" } } + if ($switched) { + PostCards $true + "RESULT card restore posted: each settings.json entry with its own enabled flag and identities" + $t = 0; $back = @() + $anyEnabled = ($entries | Where-Object { $_.enabled }).Count -gt 0 + while ($t -lt 90 -and $anyEnabled) { Start-Sleep -Seconds 5; $t += 5; $back = @(CardWorkers); if ($back.Count -gt 0) { break } } + "RESULT card_workers_after $(Stamp) pids=[$($back -join ' ')] after $t s $(if ($back.Count -gt 0) { '(the card mines again)' } elseif ($anyEnabled) { '(NOT back yet: check the app)' } else { '(no entry was enabled before this job)' })" + } + "RESULT workers_after $(Stamp) $((Workers) -join ' ')" +} + +# ---- the samples: board watts and clocks inside each bench window (after its first 12 s, before its last 2 s) ---- +$rows = @() +if (Test-Path -LiteralPath $samples) { + foreach ($l in (Get-Content -LiteralPath $samples)) { + if ($l -match '^(\S+) amd (\d+) bus (\S+) kind (\S+) name "([^"]*)" watts (\S+) temp_c (\S+) fan_rpm (\S+) fan_pct (\S+) mclk_mhz (\S+) gclk_mhz (\S+) util_pct (\S+) source (\S+)') { + if ($Matches[5] -notmatch '9070') { continue } + try { $ts = [datetime]::ParseExact($Matches[1], 'yyyy-MM-ddTHH:mm:ss.fff', [Globalization.CultureInfo]::InvariantCulture) } catch { continue } + $w = if ($Matches[6] -eq '-') { $null } else { [double] $Matches[6] } + $rows += [pscustomobject]@{ ts = $ts; w = $w; temp = $Matches[7]; fan = $Matches[8]; mclk = $Matches[10]; gclk = $Matches[11]; util = $Matches[12]; src = $Matches[13] } + } + } +} +"RESULT samples total $($rows.Count) file $samples (9070 XT lines)" +function Window([datetime] $t0, [datetime] $t1) { @($rows | Where-Object { $_.ts -gt $t0.AddSeconds(12) -and $_.ts -lt $t1.AddSeconds(-2) -and $null -ne $_.w }) } +$withWatts = 0 +foreach ($r in $results) { + $in = @(Window $r.t0 $r.t1) + $watts = 'owed'; $uj = 'owed'; $gclk = 'owed'; $mclk = 'owed'; $tmax = 'owed'; $wmin = ''; $wmax = '' + if ($in.Count -gt 0) { + $m = $in | Measure-Object -Property w -Average -Minimum -Maximum + $watts = [math]::Round($m.Average, 1); $wmin = $m.Minimum; $wmax = $m.Maximum + $gclk = [math]::Round((($in | ForEach-Object { [double] $_.gclk }) | Measure-Object -Average).Average) + $mclk = [math]::Round((($in | ForEach-Object { [double] $_.mclk }) | Measure-Object -Average).Average) + $tmax = (($in | ForEach-Object { [double] $_.temp }) | Measure-Object -Maximum).Maximum + if ($r.mhs -and [double] $r.mhs -gt 0) { $uj = [math]::Round($m.Average / [double] $r.mhs, 3) } + $withWatts++ + } + "RESULT LADDER pack=$($r.pack) ops=$($r.ops) shadow_instrs=$($r.instrs) mhs=$($r.mhs) watts=$watts uj=$uj watts_min=$wmin watts_max=$wmax gclk_mhz=$gclk mclk_mhz=$mclk temp_max=$tmax samples=$($in.Count) window_s=$([int]($r.t1 - $r.t0).TotalSeconds) fingerprint=$($r.fp) match=$($r.match) card_alone=$cardAlone tag=$($r.tag)" +} +if ($rows.Count -gt 0) { + $idle = @($rows | Where-Object { ($results.Count -eq 0 -or $_.ts -lt $results[0].t0) -and $null -ne $_.w }) + if ($idle.Count -gt 0) { "RESULT idle_power samples=$($idle.Count) watts_mean=$([math]::Round(($idle | Measure-Object -Property w -Average).Average, 1)) (the card switched off in the app, before the first bench)" } + foreach ($r in $rows) { "RESULT sample $($r.ts.ToString('HH:mm:ss')) $($r.w) $($r.temp) $($r.fan) $($r.mclk) $($r.gclk) $($r.util) $($r.src)" } +} +$status = if (-not $samplerOk) { 'failed' } elseif ($withWatts -eq $order.Count) { 'done' } elseif ($results.Count -gt 0) { 'partial' } else { 'failed' } +"RESULT end $(Stamp)" +Summary $status @{ packs_run = $results.Count; packs_wanted = $order.Count; rows_with_watts = $withWatts; sampler_proven = $samplerOk; card_alone = $cardAlone; device = $dev } +if ($status -eq 'failed') { exit 1 } +exit 0