The page lives at an unlisted path on dl.igneum.network (name in ~/.config/igneum/fleet-path) and reads fleet.json
every 30 s; this script copies a fleet.json in and deploys, then checks the edge serves it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The console-window class (the project lead, 5 October 2026: "Windows Command Processor" windows on PC 1 whenever a remote job runs).
Measured on PC 1 (ae432dc7, Windows 11 Pro 26200, default terminal "Let Windows decide" = Windows Terminal 1.24) with
tools/windows/console-watch.ps1 (job run-20261005-182528): no child a job script starts from the app's headless
console opens a window (powershell, cmd, query, curl, nvidia-smi, wsl --status, a distro, interop cmd and powershell,
powershell -WindowStyle Hidden: 0 windows each); Start-Process in a new console opens a Terminal window (the known-failed
case: 2 windows), the same with -WindowStyle Hidden opens none (the known-finished case). The elevated path
(Start-Process -Verb RunAs -WindowStyle Hidden through the AppInfo service) is the one road left; its watcher
(console-watch-elevated.ps1, job run-20261005-184610) was cancelled at the UAC prompt.
- platform.rs: elevated_ps_line + elevated_command build the one PowerShell line every elevated launch uses (the NVIDIA
power cap, the sweep helper, the clock sync, an elevated remote job), -WindowStyle Hidden by construction; unit
tests on the line, the quoting and the Command.
- jobrun.rs: the elevated job path uses it; the relaunch helper's Start-Process carries the reason it has no
-WindowStyle Hidden (igneum-app.exe is a windows-subsystem program).
- tools/ci/windows-spawn-check.mjs (+ ci.yml): fails when a Command::new in app/igneum-app/src is not quieted,
a creation_flags is not CREATE_NO_WINDOW alone, a Start-Process the Rust code writes lacks -WindowStyle Hidden or
-NoNewWindow, or host.cpp spawns without CREATE_NO_WINDOW / SW_HIDE; self-test on known-good and known-bad samples.
- tools/windows/console-watch.ps1, console-watch-bg.ps1, console-watch-elevated.ps1: the watchers (run jobs).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The statement and the pinned guests are unchanged; every fixture proof verifies as before. The defaults stay (batch-log2 22, SP1 defaults): the one knob that moves a mining card's prover costs a fifth of the hash rate; the plan carries the trade for the project lead and the batch fold for the next pin. Measured: docs/bench-log.md "aggregation cost on the RTX 5090"; the plan line: docs/plans/proving-v1.md "Aggregation cost (5 October, night)". Also: make-package's gate skips the exporter's .node-plan.json side files and takes the run lock for its execute step; the state-reply class (/api/state answering {} once paid_wei passes u64::MAX) found on the way and fixed on the app branch at 42f36b3.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit ea38ece9eaa5949dd657cbfea5308c4948177ff8)
Source: the scratch engine's own log in collect ember-c35-collect-1 (06:59Z): 22:31:02Z '0.3.10 is available: downloading',
22:31:05Z 'update: starting the installer first ... ota-apply.ps1', and the installed app's 'quit:' at 22:31:06Z.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PC 1, 22:31 UTC: the installed engine's quit hung 24 minutes in the jobs runner's abort, waiting for EOF on the script's
stdout pipe whose write end the second engine and its miners had inherited (Process.Start with redirection inherits
every inheritable handle), while the orphaned miners mined on against the relaunched app.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The app's install clears the jobs folder on a PC, so a run job whose kit was fetched by an earlier fetch job finds
nothing after an update and fails in seconds (5 October 2026, 21:49Z, the AMD kit; bench-log 4df339f). Rule: a run
playbook that reaches a path under the jobs folder other than its own tests the kit is there before its first use,
and the fetch is republished under a new id after any app update.
tools/ci/kit-path-check.sh reads every *.ps1 under relay/playbooks/ and tools/. A kit root is a path derived from
the jobs folder (`$jobs = Split-Path $env:IGNEUM_JOB_DIR` then `Join-Path $jobs '<fetch id>'`, the race-5090.ps1
shape) or one carrying a literal `jobs\` (the amd-card-test.ps1 shape); every path built from it belongs to that kit.
A presence check (Test-Path, [IO.File]::Exists, [IO.Directory]::Exists, Get-Item or Get-ChildItem with -ErrorAction)
on the root or anything under it covers the whole kit. A use before that line fails with "kit path used before a
presence check: republish the fetch after any app update", as does a literal jobs\ path in a command with no check.
The job's own folder ($env:IGNEUM_JOB_DIR) is not a kit path.
Fixtures: kit-path-ok.ps1 (both shapes, checked; a sibling pack file covered by the worker's check) and
kit-path-unchecked.ps1 (the worker run before its check, a literal never checked); --self-test asserts the lines.
Wired into ci.yml after the bash-body step, and into publish-jobs.sh add --kind run beside the other two checks;
test-publish-jobs.sh gains the refusal (34 passed, 0 failed). The current tree: race-5090.ps1 is the one playbook
with a kit, checked before use. README-ship.md: the rule.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A PC job is published from a worktree by packaging/ota/publish-jobs.sh and never passes CI before it runs; tonight
the root-socket fault came back from a job on a branch without the check. `add --kind run` now runs, on the script
being published and before anything is signed: tools/ci/bash-body-check.sh for a PowerShell script (every inline
bash body parses; a body it cannot read fails, never skips), `bash -n` for a .sh script, and
tools/ci/prover-socket-check.sh for both (a root prover run kills sp1-gpu-server and unlinks its socket). A failure
refuses the publish with the check's output; a missing check file refuses too. Kinds without a script (fetch,
collect, restart, update-now, shard-benchmark, build) are untouched.
tools/ci/prover-socket-check.sh is copied from proving-v1 (344cba8; master lacks it) with two additions: file
arguments check those files only (the publisher's call), and an allow list for packaging/ota/test-publish-jobs.sh,
which carries a known-bad root prover script on purpose. Its ci.yml step is left to proving-v1 to avoid a duplicate.
packaging/ota/test-publish-jobs.sh: four refusals (a lost quote in a PowerShell bash body, an unreadable body, a
.sh with a lost quote, a root prover script without the cleanup) and the envelope unchanged after a refusal.
32 passed, 0 failed on this Mac with the main checkout's signer. packaging/README-ship.md: the publish-time gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The known-failed case, from PC 2's 0.3.9 log (run win-1ccfe586-20261005-200114): 1791234223 pause -> 'stopping the
miners (paused)' (every slot's restart_at cleared, the 5090 'off'); 1791235511 '[ok] mining resumed'; then
'0.00 MH/s, waiting' at every 30-s status line until the 0.3.10 restart at 21:49:41Z. Cause: Cmd::Resume re-armed
only slots whose watchdog said faulted; the 5090's slot was healthy and stopped, so nothing restarted it. The test
the_pc2_resume_of_21_25_11z_restarts_under_the_new_rule_and_not_the_old encodes that slot (faulted false, live
false): the old rule returns [] (the defect), the new rule [0]. cargo test -p igneum-app resume: 3 passed;
provedefault: 6 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Twice on 5 October 2026 a PowerShell job script carried a bash body inside a string, a quote was lost on the way
through PowerShell, and bash refused the body: pc1-cpu-prove.ps1 (first version) reported exit 0 having done
nothing, the 0.3.10 installer job failed in 4 s. tools/amd-prove/check-job-bash.sh covered only its own here-string.
tools/ci/bash-body-check.sh reads every *.ps1 under relay/playbooks/ and tools/, finds each bash body however it is
handed over (bash -c "...", bash -lc '...', bash -c $var, a + concatenation in parentheses, the Start-Process argument
list, a here-string written to a file that is later run with bash), unescapes it the way PowerShell would (backtick
escapes and "" in double-quoted strings, '' in single-quoted strings, here-strings verbatim; $var left as-is, a $(...)
subexpression replaced by ${PS_SUBEXPR}), and runs bash -n on it. One line per body with the file line of the error.
A body it sees but cannot read is "unextractable body" and fails too: a skip would be a hole in the class check.
bash 3.2 compatible; python3 for the extractor.
--self-test runs three fixtures under tools/ci/fixtures/: the correct shapes (8 bodies, must pass), the lost quotes
(the awk apostrophe, a dropped closing quote in a literal and in a variable; must fail with the line), and three
unreadable bodies (must fail). Wired into ci.yml next to the copied-sources check, self-test first. The current tree:
7 inline bodies in 3 playbooks, all parse. packaging/README-ship.md: the job-script rule (body to a file, bash <file>).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
the project lead, 5 October 2026, 22:45 BST: "make sure we have ember tuning every single card for efficiency out of the box, the
more data = the better the tune, make an awesome system." Built on lever 3 (docs/plans/miner-eff.md), lever 2's signed
tuning section (docs/design/miner-tuning.md), the AMD telemetry helper (423936b, its --tune/--set-gmax/--set-plimit/
--reset contract) and the Power control switch (057f0ec). Design, data flow, tiers and the privacy line:
docs/plans/ember-tune.md.
- src/ember.rs (new): two knobs per card (power limit %, core clock cap MHz; memory clock never touched), the full plan
(power ladder 100..50%, then the clock ladder 90..60% at the chosen power), the confirm plan (the fleet prior and one
neighbour), the baseline plan (measure only), the marks (faulted, hot, memory_clock_dropped, unapplied, no_readings),
the choice (best MH/W within 1% of the top rate, then rate, then draw), the fleet record (a hash of the install id,
no address), the prior lookup and the kill switch (tuning.ember), the state machine on a fake clock. 9 unit tests.
- engine.rs: tick_sweep schedules every NVIDIA, AMD and Apple card (120 s steady, 600 s to the boundary, no job hold,
no pause, weekly, again after a driver major or program-class change, never under the manifest kill switch); the
probe (nvidia-smi clocks.max.gr + driver_version and the direct/helper mode; igneum-gpu-telemetry --tune for AMD);
tune_apply (nvidia-smi -pl / -lgc 0,<MHz> / -rgc directly or through the helper; the AMD helper per request);
Cmd::TuneProbe, Cmd::TuneSet; faults from rejected and mismatched hashes mark the step; the TUNE lines and the TUNE
{json} record, uploaded with the log; the Tuned line on the card state. The NVIDIA helper starts only with Power
control on: the --sweep job never counts as permission (no prompt on a PC with nobody there).
- sweep.rs: the helper protocol gains lgc/rgc (clock cap and reset) and resets the clocks after 20 idle minutes.
- state.rs, config.rs: the tune fields (clock cap, driver, class, source, the Tuned line); the nvidia-smi telemetry
query carries clocks.gr and clocks.mem; the AMD sample line's plimit_pct and gmax_mhz are parsed.
- ui: "Tuned: X MH/s at Y W (Z MH/W)" with the point, the source and when; measure-only cards say why; the Ember Tune
switch; tune-line.test.mjs.
- relay/lib/ember.mjs + relay/test/ember.test.mjs: the aggregation per (card model | driver major | program class):
median point, MH/W, spread, samples, machines; five samples converge, an outlier does not move the median, baselines
make no prior, de-duplication, the manifest merge keeps lever 2's cards. api/console.mjs fn=tuning and
tools/console.mjs tuning; tools/tuning.mjs --priors [--write tuning.json] [--site] [--tuning-off].
- site: the fleet priors table on /miners (site/miner-priors.json), the lever text.
- relay/playbooks/ember-tune-pc1.ps1: the PC 1 run (second engine with --sweep from a scratch copy of the install).
Measured tonight: see the bench log entry that follows the PC 1 run. The 9070 XT left PC 1's bus at 20:40 UTC and the
5090 needs the administrator prompt the project lead cannot answer asleep, so tonight's PC 1 run is the baseline plan on the 5090
through the whole pipeline; the two-knob tune on both cards is owed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
main.swift: servePackProgram reads program.h with the packfile.h checks (generator 2 or 3, the class line against
the generator, the seed bytes, IGNEUM_SEEDW_INIT against attempt_words, class and era against the line) and
compiles program_bound.metal; the program store keys on (seed, class, era); a v3 job with no resident v3 pack
program answers need + error; v2 lines unchanged (Swift generation, the variant race); a pack program never races.
verify.rs: Epoch::chain_dataset_day(day, class, days_since_genesis, genesis_dataset_log2) and days_since_genesis,
the entry the node builds every day cache through (the ca2-mixer growth rule fills the body).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
