MF-11 update-return (0.3.21), the app half on release-0.3.21: the Windows update helper owns the return (the exe set kept beside the app, the app launched by the helper, api/state polled 120 s, the kept set restored when nothing answers, one intake line either way; the installer stays silent under /IGNOTA=2), the window host restarts a dead engine and answers the Restart Manager, the first act after an update is the read-back line and a FAULT pc-restart line when the PC came up from a power loss; the tuner never asks above a card's measured efficient point (the 5090 at 308 W, floored at the vendor's 400 W) unless Power control is on and the user raised the cap, and a refused cap is asked again at 2 and 10 min then reported as FAULT power-cap; the job-channel ping on every wake request (7 October 2026, PC 2 lost power at 10:46Z; the relay half stays on update-return)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-labs 2026-10-07 14:29:41 +00:00
parent db4212ca52
commit e68ba08ccc
9 changed files with 752 additions and 57 deletions

View file

@ -0,0 +1,73 @@
//! Did this PC come up from a power loss or a hard reset? (MF-11, 7 October 2026: PC 2 dropped twice in one day with
//! Kernel-Power 41 and EventLog 6008 at the next boot, no bugcheck, no dump, and nothing said so until a person read
//! the event log.) Windows: the System log's event 41 (Kernel-Power, critical) or 6008 (EventLog, "the previous
//! shutdown was unexpected") inside the last 15 minutes, read once at the engine's start through wevtutil; the engine
//! logs one `FAULT pc-restart:` line to the intake. Other platforms: nothing (a Mac's power log is not this class).
use std::process::Command;
use std::time::Duration;
/// How far back the start-up check looks: an engine starts at login, inside a minute or two of the boot.
pub const WINDOW_MS: u64 = 15 * 60 * 1000;
/// One line naming the event, or None when the boot was clean, the query failed, or this is not Windows.
pub fn unexpected_restart() -> Option<String> {
if !cfg!(windows) {
return None;
}
let query = format!("*[System[(EventID=41 or EventID=6008) and TimeCreated[timediff(@SystemTime) <= {WINDOW_MS}]]]");
let mut c = Command::new(crate::platform::tool("wevtutil"));
c.args(["qe", "System", &format!("/q:{query}"), "/f:text", "/c:2", "/rd:true"]);
let out = crate::detect::run_timeout(&mut c, None, Duration::from_secs(20))?;
parse_events(&out)
}
/// The reading of wevtutil's text output: the newest 41 or 6008 as "event 41 (Kernel-Power) at <time>" or
/// "event 6008 (the previous shutdown was unexpected) at <time>"; None when neither is in the text.
pub fn parse_events(text: &str) -> Option<String> {
let mut date = String::new();
let mut id = String::new();
for line in text.lines() {
let t = line.trim();
if let Some(d) = t.strip_prefix("Date:") {
date = d.trim().to_string();
} else if let Some(i) = t.strip_prefix("Event ID:") {
id = i.trim().to_string();
if id == "41" || id == "6008" {
break;
}
}
}
match id.as_str() {
"41" => Some(format!("event 41 (Kernel-Power: the system rebooted without shutting down cleanly) at {date}")),
"6008" => Some(format!("event 6008 (the previous shutdown was unexpected) at {date}")),
_ => None,
}
}
#[cfg(test)]
mod tests {
use super::parse_events;
const PC2: &str = "Event[0]\n Log Name: System\n Source: Microsoft-Windows-Kernel-Power\n Date: 2026-10-07T14:37:19.4360000Z\n Event ID: 41\n Task: N/A\n Level: Critical\n Opcode: Info\n Keyword: N/A\n User: S-1-5-18\n Computer: X\n Description: \n\nEvent[1]\n Log Name: System\n Source: EventLog\n Date: 2026-10-07T14:37:29.0380000Z\n Event ID: 6008\n Task: None\n Level: Error\n";
/// PC 2's boot of 13:37Z on 7 October 2026 (the collection job's wevtutil text): the known-failed case reads as one.
#[test]
fn pc2_boot_reads_as_a_power_loss() {
assert_eq!(parse_events(PC2), Some("event 41 (Kernel-Power: the system rebooted without shutting down cleanly) at 2026-10-07T14:37:19.4360000Z".into()));
}
#[test]
fn a_6008_alone_is_the_unexpected_shutdown() {
let t = "Event[0]\n Log Name: System\n Source: EventLog\n Date: 2026-10-07T08:22:34.0000000Z\n Event ID: 6008\n Level: Error\n";
assert_eq!(parse_events(t), Some("event 6008 (the previous shutdown was unexpected) at 2026-10-07T08:22:34.0000000Z".into()));
}
/// A clean boot (the known-good case): nothing, including other ids inside the window and an empty answer.
#[test]
fn a_clean_boot_reads_as_nothing() {
assert_eq!(parse_events(""), None);
assert_eq!(parse_events("Event[0]\n Log Name: System\n Source: Microsoft-Windows-Kernel-General\n Date: 2026-10-07T14:37:17.7400000Z\n Event ID: 12\n Level: Information\n"), None);
assert_eq!(parse_events("wevtutil: access denied"), None);
}
}

View file

@ -25,6 +25,31 @@ pub const POWER_STEPS_PCT: [u32; 6] = [100, 90, 80, 70, 60, 50];
pub const CLOCK_STEPS_PCT: [u32; 7] = [100, 90, 80, 70, 60, 50, 45];
/// A card's clock floor when the vendor reports none: this share of its maximum core clock.
pub const CLOCK_FLOOR_PCT: u32 = 45;
/// The measured efficient point per card (docs/bench-log.md: the 5090 at 308 W for 122 MH/s, the 9070 XT at 199 W for
/// 18 MH/s), the tuner's ceiling. Rule (main, 7 October 2026, after PC 2 dropped nine minutes after the tuner asked its
/// 5090 for 575 W with proving on the same card): the tuner never requests more than the card's measured efficient point
/// unless Power control is on and the user raised the cap above the default; a card not in this table is held at the
/// cap it runs at. A point below the vendor's minimum limit clamps up to that minimum (the 5090 floors at 400 W).
pub const EFFICIENT_W: &[(&str, f64)] = &[("RTX 5090", 308.0), ("RX 9070 XT", 199.0)];
/// The app's cap when the user chose none (engine.rs requested_watts); a cap above it is one the user raised.
pub const DEFAULT_CAP_PCT: u32 = 80;
/// The measured efficient watts of a card by its name, when the table has it.
pub fn efficient_watts(name: &str) -> Option<f64> {
let n = name.to_ascii_uppercase();
EFFICIENT_W.iter().find(|(k, _)| n.contains(&k.to_ascii_uppercase())).map(|(_, w)| *w)
}
/// The tuner's power ceiling in watts for a card: its measured efficient point, or the cap it runs at when the table has
/// no entry; the user's own cap instead when Power control is on and that cap is above the default (they raised it).
pub fn power_ceiling(limits: &Limits, name: &str, before: Point, power_control: bool) -> f64 {
let cap = Limits { power_ceiling_w: 0.0, ..limits.clone() }.watts_for(if before.power_pct == 0 { DEFAULT_CAP_PCT } else { before.power_pct });
let raised = power_control && before.power_pct > DEFAULT_CAP_PCT;
match efficient_watts(name) {
Some(e) if !raised => e.min(cap),
_ => cap,
}
}
/// A point may lose this much rate against the fastest point and still win on MH per watt (the manifest can change it).
pub const RATE_TOLERANCE_PCT: f64 = 1.0;
/// A step whose hottest GPU reading reaches this is marked hot and cannot win (the engine aborts at 90).
@ -55,9 +80,20 @@ pub struct Limits {
/// clocks.max.mem); 0 = no memory knob (AMD through ADLX on RDNA 4 exposes none)
pub mem_default_mhz: u32,
pub mem_max_mhz: u32,
/// the tuner's ceiling (power_ceiling), 0 = none: no step asks for more watts than this
pub power_ceiling_w: f64,
}
impl Limits {
/// The ceiling as the vendor allows it: never below the card's minimum limit; 0 when there is none.
pub fn ceiling(&self) -> f64 {
if self.power_ceiling_w <= 0.0 { 0.0 } else { self.power_ceiling_w.max(self.power_min_w) }
}
/// The percent of the default that the ceiling is, for the first step of the power ladder.
pub fn ceiling_pct(&self) -> u32 {
let c = self.ceiling();
if c <= 0.0 || self.power_default_w <= 0.0 { 100 } else { (c / self.power_default_w * 100.0).round().clamp(1.0, 100.0) as u32 }
}
pub fn clock_floor(&self) -> u32 {
if self.clock_min_mhz > 0 {
self.clock_min_mhz
@ -88,6 +124,10 @@ impl Limits {
if self.power_max_w > 0.0 {
w = w.min(self.power_max_w);
}
let c = self.ceiling();
if c > 0.0 {
w = w.min(c);
}
w.round()
}
}
@ -276,7 +316,9 @@ impl Plan {
pub fn full(limits: &Limits, before: Point, tolerance_pct: f64) -> Plan {
let mut power = Vec::new();
if limits.power_default_w > 0.0 {
for pct in POWER_STEPS_PCT {
// the ceiling first (its own percent), then the ladder; every step above it clamps to it and is dropped as a duplicate
let top = if limits.ceiling() > 0.0 { vec![limits.ceiling_pct()] } else { Vec::new() };
for pct in top.into_iter().chain(POWER_STEPS_PCT) {
let w = limits.watts_for(pct);
if power.last().map(|s: &Step| (s.watts - w).abs() < 0.5).unwrap_or(false) {
continue;
@ -1051,7 +1093,71 @@ mod tests {
/// PC 1's RTX 5090 (nvidia-smi, 4 and 5 October 2026): default 575 W, min 400 W, max 600 W; clocks.max.gr is
/// read at the first tune (3,090 MHz is the shape used here, not a measurement).
fn l5090() -> Limits {
Limits { power_default_w: 575.0, power_min_w: 400.0, power_max_w: 600.0, clock_max_mhz: 3090, clock_min_mhz: 0, mem_default_mhz: 13801, mem_max_mhz: 14001 }
Limits { power_default_w: 575.0, power_min_w: 400.0, power_max_w: 600.0, clock_max_mhz: 3090, clock_min_mhz: 0, mem_default_mhz: 13801, mem_max_mhz: 14001, power_ceiling_w: 0.0 }
}
fn with_ceiling(name: &str, before: Point, power_control: bool) -> Limits {
let mut l = l5090();
l.power_ceiling_w = power_ceiling(&l, name, before, power_control);
l
}
fn max_watts(plan: &Plan) -> f64 {
let mut rows = Vec::new();
let mut top: f64 = 0.0;
while let Some(s) = plan.next(&rows) {
top = top.max(s.watts);
rows.push(row_at(s.point, s.watts, 100.0));
if rows.len() > 40 { break; }
}
top
}
/// PC 2, 7 October 2026: the tuner asked the 5090 for 100% (575 W) while proving ran on the same card; the PC dropped
/// nine minutes later. The known-failed shape first: without a ceiling the full plan opens at 575 W.
#[test]
fn without_a_ceiling_the_full_plan_asks_the_5090_for_575_w() {
let plan = Plan::full(&l5090(), Point { clock_mhz: 0, power_pct: 70, mem_mhz: 0 }, 1.0);
assert_eq!(plan.next(&[]).unwrap().watts, 575.0);
}
#[test]
fn the_tuner_never_asks_above_the_measured_efficient_point() {
// the 5090's measured point is 308 W; the vendor's minimum is 400 W, so the ceiling clamps up to 400
let l = with_ceiling("NVIDIA GeForce RTX 5090", Point { clock_mhz: 0, power_pct: 70, mem_mhz: 0 }, false);
assert_eq!(l.power_ceiling_w, 308.0);
assert_eq!(l.ceiling(), 400.0);
let plan = Plan::full(&l, Point { clock_mhz: 0, power_pct: 70, mem_mhz: 0 }, 1.0);
let first = plan.next(&[]).unwrap();
assert_eq!((first.watts, first.point.power_pct, first.kind), (400.0, 70, Kind::Power), "one power step at the ceiling, with its own percent");
assert_eq!(plan.len(), 1 + 6, "one power step (every ladder step clamps to 400 W) and the six clock steps");
assert!(max_watts(&plan) <= 400.0, "no step above the ceiling");
// the confirm plan and the climb honour it too: a fleet prior at 100% is clamped
let prior = Point { clock_mhz: 0, power_pct: 100, mem_mhz: 0 };
assert!(max_watts(&Plan::confirm(&l, prior, Point { clock_mhz: 0, power_pct: 70, mem_mhz: 0 }, 1.0)) <= 400.0);
assert!(max_watts(&Plan::climb(&l, prior, Goal::Efficiency, 1.0)) <= 400.0);
}
#[test]
fn power_control_on_with_a_cap_the_user_raised_lifts_the_ceiling_to_that_cap() {
let l = with_ceiling("NVIDIA GeForce RTX 5090", Point { clock_mhz: 0, power_pct: 90, mem_mhz: 0 }, true);
assert_eq!(l.ceiling(), 518.0, "the user's 90% (518 W), never the vendor's 575 W");
assert!(max_watts(&Plan::full(&l, Point { clock_mhz: 0, power_pct: 90, mem_mhz: 0 }, 1.0)) <= 518.0);
// Power control on at the default cap is not a raise: the measured point holds
let d = with_ceiling("NVIDIA GeForce RTX 5090", Point { clock_mhz: 0, power_pct: DEFAULT_CAP_PCT, mem_mhz: 0 }, true);
assert_eq!(d.ceiling(), 400.0);
// a raised cap with Power control OFF does not count
let off = with_ceiling("NVIDIA GeForce RTX 5090", Point { clock_mhz: 0, power_pct: 90, mem_mhz: 0 }, false);
assert_eq!(off.ceiling(), 400.0);
}
#[test]
fn a_card_not_in_the_table_is_held_at_the_cap_it_runs_at() {
let l = with_ceiling("NVIDIA GeForce RTX 4070", Point { clock_mhz: 0, power_pct: 70, mem_mhz: 0 }, false);
assert_eq!(l.ceiling(), 403.0, "70% of 575 W, the card's own cap");
let zero = with_ceiling("NVIDIA GeForce RTX 4070", Point::default(), false);
assert_eq!(zero.ceiling(), 460.0, "no chosen cap: the app's default 80%");
assert_eq!(efficient_watts("AMD Radeon RX 9070 XT"), Some(199.0));
assert_eq!(efficient_watts("gfx1036"), None);
}
fn row_at(p: Point, watts: f64, mhs: f64) -> Row {
@ -1258,7 +1364,7 @@ mod tests {
fn a_fault_during_a_step_reverts_it_and_the_run_goes_on() {
let t0 = Instant::now();
let timing = Timing { settle: Duration::from_secs(2), hold: Duration::from_secs(4), apply: Duration::from_secs(30) };
let limits = Limits { power_default_w: 300.0, power_min_w: 150.0, power_max_w: 300.0, clock_max_mhz: 0, clock_min_mhz: 0, mem_default_mhz: 0, mem_max_mhz: 0 };
let limits = Limits { power_default_w: 300.0, power_min_w: 150.0, power_max_w: 300.0, clock_max_mhz: 0, clock_min_mhz: 0, mem_default_mhz: 0, mem_max_mhz: 0, power_ceiling_w: 0.0 };
let plan = Plan::full(&limits, Point { clock_mhz: 0, power_pct: 80, mem_mhz: 0 }, 1.0);
let mut run = Run::new(0, "0", "c", plan, 240.0, false, timing, t0);
let mut t = t0;

View file

@ -719,6 +719,9 @@ pub struct Engine {
amd_telemetry_retry_at: Instant,
power_busy: bool,
power_restore_pending: bool,
/// a cap the card did not take: how many times in a row, and when to ask again (cap_refusal_plan)
cap_refusals: u32,
cap_retry_at: Option<Instant>,
/// an elevated step handed to the window host: (command line, what, requested watts per device, since)
power_via_host: Option<(String, String, std::collections::HashMap<String, f64>, Instant)>,
/// the manifest's consensus override changed: restart the node at a safe moment
@ -880,6 +883,8 @@ impl Engine {
amd_telemetry_retry_at: now,
power_busy: false,
power_restore_pending: false,
cap_refusals: 0,
cap_retry_at: None,
power_via_host: None,
node_override_restart: false,
node_override_unusable: false,
@ -960,6 +965,16 @@ impl Engine {
let v = crate::detect::node_version(&self.bins.node);
self.st().node.version = v.clone();
self.shared.log(&self.shared.upload_header());
// MF-11 (7 October 2026): the first act after an update is the read-back line, so the intake shows the new
// version up before anything else happens; the shipper's rollout table reads it per machine
if let Some(from) = self.ota.updated_from() {
self.shared.log(&format!("update-return: app {VERSION} up after the update from {from} (node {v}, machine {})", self.shared.runtime.id8()));
}
// a PC that lost power or hard-reset is a fault row at its next start, not a silence (PC 2, 7 October 2026)
if let Some(line) = crate::bootcheck::unexpected_restart() {
self.shared.log(&format!("FAULT pc-restart: {line}"));
self.shared.event("error", &format!("This PC restarted without a shutdown: {line}. Mining resumed; if it happens again check the power supply and the wall."));
}
// which intake and which downloads folder this build reports to and checks (fingerprints, never the values;
// rotation phase 2 reads this line from every machine's upload: docs/plans/rotation-phase-2.md)
self.shared.log(&self.shared.packaged.describe());
@ -1289,6 +1304,22 @@ impl Engine {
Err(e) => e.clone(),
};
self.shared.event("error", &format!("GPU power cap NOT applied ({why}): {}. Retry from the card tile.", missing.join("; ")));
// never left as "cap NOT applied" (main's rule, 7 October 2026): asked again on the ladder, then a FAULT line home
self.cap_refusals += 1;
match cap_refusal_plan(self.cap_refusals) {
Some(wait) => {
self.cap_retry_at = Some(Instant::now() + wait);
self.shared.log(&format!("power cap: refusal {} ({why}); asking again in {} s", self.cap_refusals, wait.as_secs()));
}
None => {
self.cap_retry_at = None;
let power_control = self.shared.settings.lock().unwrap().power_control;
self.shared.log(&format!("FAULT power-cap: {} ({why}); refused {} times, the card runs at the limit it reports; Power control is {}", missing.join("; "), self.cap_refusals, if power_control { "on" } else { "off" }));
}
}
} else if !applied.is_empty() {
self.cap_refusals = 0;
self.cap_retry_at = None;
}
if applied.is_empty() && missing.is_empty() {
self.shared.log(&format!("power cap: nothing to read back for {what}"));
@ -1578,6 +1609,14 @@ impl Engine {
self.shared.log(&format!("tune: request {seq} refused: {}", short(&text.replace('\n', " "), 300)));
if self.sweep.as_ref().map(|r| r.seq == seq).unwrap_or(false) {
self.sweep_abort("the card refused the setting (see the log)");
} else if self.sweep.is_none() {
// the restore after a stopped tune was refused (PC 2, 7 October 2026: request 0 back to 70% never ran):
// the card may sit at the vendor default; the cap path asks again and reports a FAULT on its ladder
self.shared.log(&format!("FAULT power-cap: the restore after the tune was refused ({}); re-applying the cap", short(&text.replace('\n', " "), 120)));
for c in self.st().mining.cards.iter_mut().filter(|c| c.vendor == "nvidia") {
c.power_applied = false;
}
self.apply_power_limits("tune restore refused");
}
}
},
@ -2982,11 +3021,19 @@ impl Engine {
let power_control = probe.direct || self.shared.settings.lock().unwrap().power_control || (cfg!(windows) && self.power_task_registered());
// AMD's power limit is a percent offset from the default (ADLX): the plan's watts scale becomes a percent
// scale (default 100, floor 100 + plimit_min, ceiling 100 + plimit_max), tune_apply sends pct - 100
let limits = if c.vendor == "amd" && probe.direct && probe.plimit_max >= probe.plimit_min && probe.plimit_min > -100.0 {
crate::ember::Limits { power_default_w: 100.0, power_min_w: 100.0 + probe.plimit_min, power_max_w: 100.0 + probe.plimit_max, clock_max_mhz: probe.clock_max_mhz, clock_min_mhz: probe.clock_min_mhz, mem_default_mhz: 0, mem_max_mhz: 0 }
let mut limits = if c.vendor == "amd" && probe.direct && probe.plimit_max >= probe.plimit_min && probe.plimit_min > -100.0 {
crate::ember::Limits { power_default_w: 100.0, power_min_w: 100.0 + probe.plimit_min, power_max_w: 100.0 + probe.plimit_max, clock_max_mhz: probe.clock_max_mhz, clock_min_mhz: probe.clock_min_mhz, mem_default_mhz: 0, mem_max_mhz: 0, power_ceiling_w: 0.0 }
} else {
crate::ember::Limits { power_default_w: c.power_default_w, power_min_w: c.power_min_w, power_max_w: c.power_max_w, clock_max_mhz: probe.clock_max_mhz, clock_min_mhz: probe.clock_min_mhz, mem_default_mhz: probe.mem_default_mhz, mem_max_mhz: probe.mem_max_mhz }
crate::ember::Limits { power_default_w: c.power_default_w, power_min_w: c.power_min_w, power_max_w: c.power_max_w, clock_max_mhz: probe.clock_max_mhz, clock_min_mhz: probe.clock_min_mhz, mem_default_mhz: probe.mem_default_mhz, mem_max_mhz: probe.mem_max_mhz, power_ceiling_w: 0.0 }
};
// the tuner's ceiling (main's rule, 7 October 2026, PC 2): the card's measured efficient point, or the cap it runs
// at, unless Power control is on in Settings and the user raised the cap above the default; watts only (NVIDIA)
if c.vendor != "amd" || !probe.direct {
let settings_power_control = self.shared.settings.lock().unwrap().power_control;
let at = crate::ember::Point { clock_mhz: c.clock_cap_mhz, power_pct: c.power_pct, mem_mhz: c.mem_cap_mhz };
limits.power_ceiling_w = crate::ember::power_ceiling(&limits, &c.name, at, settings_power_control);
self.shared.log(&format!("tune: ceiling {:.0} W for {} ({}; Power control {})", limits.ceiling(), c.name, crate::ember::efficient_watts(&c.name).map(|w| format!("measured efficient point {w:.0} W")).unwrap_or_else(|| "no measured point, the cap it runs at".into()), if settings_power_control { "on" } else { "off" }));
}
let control = match c.vendor.as_str() {
"nvidia" => crate::ember::control_reason("nvidia", &limits, &c.device, power_control, false),
"amd" => crate::ember::control_reason("amd", &limits, &c.device, power_control, probe.direct && probe.amd_ordinal >= 0),
@ -3595,6 +3642,13 @@ impl Engine {
self.last_awake = now;
crate::platform::keep_awake_tick();
}
if self.cap_retry_at.map(|t| now >= t).unwrap_or(false) && !self.power_busy && self.sweep.is_none() && self.sweep_pending.is_none() {
self.cap_retry_at = None;
for c in self.st().mining.cards.iter_mut().filter(|c| c.vendor == "nvidia" && !c.power_applied) {
c.power_applied = false;
}
self.apply_power_limits("retry after a refusal");
}
if self.running && self.node.is_some() && !self.miners.is_empty() && now.duration_since(self.merge_view_at) >= Duration::from_secs(30) {
self.merge_view_at = now;
let shared = self.shared.clone();
@ -5334,6 +5388,17 @@ fn point_words(p: &crate::ember::Point) -> String {
if p.clock_mhz > 0 { format!("{} MHz at {}%", p.clock_mhz, p.power_pct) } else { format!("{}% (clock unlocked)", p.power_pct) }
}
/// After a cap the card did not take: when to ask again (2 min, then 10 min), and None on the third refusal, when the
/// engine writes the FAULT line instead and waits for a setting change or the card tile (main's rule, 7 October 2026:
/// a refused limit is re-applied or reported, never left as "cap NOT applied").
pub fn cap_refusal_plan(refusals: u32) -> Option<Duration> {
match refusals {
0 | 1 => Some(Duration::from_secs(120)),
2 => Some(Duration::from_secs(600)),
_ => None,
}
}
fn requested_watts(c: &CardState) -> f64 {
let pct = if c.power_pct == 0 { 80 } else { c.power_pct.clamp(crate::sweep::MIN_PCT, 100) };
let mut w = c.power_default_w * pct as f64 / 100.0;
@ -5528,6 +5593,15 @@ mod resume_tests {
#[cfg(test)]
mod tests {
/// A refused cap is asked again at 2 and 10 minutes, then reported as a FAULT line (never left as "cap NOT applied")
#[test]
fn a_refused_cap_climbs_the_retry_ladder_then_faults() {
assert_eq!(super::cap_refusal_plan(1), Some(std::time::Duration::from_secs(120)));
assert_eq!(super::cap_refusal_plan(2), Some(std::time::Duration::from_secs(600)));
assert_eq!(super::cap_refusal_plan(3), None, "the third refusal is the FAULT line");
assert_eq!(super::cap_refusal_plan(7), None);
}
#[test]
fn power_control_off_builds_no_elevated_command() {
// the decision (the project lead, 5 October 2026): off = the app never asks; the elevated PC sweep job is the exception

View file

@ -248,11 +248,15 @@ impl Jobs {
active: None,
needs_logged: std::collections::HashSet::new(),
fingerprint: manifest::fingerprint(manifest::OTA_PUBLIC_KEY_HEX),
wake: Arc::new(WakeCtl { on: AtomicBool::new(allowed) }),
wake: Arc::new(WakeCtl::new(allowed, &shared.runtime.id8())),
wake_pending: false,
last_published: String::new(),
};
j.publish(shared);
// the ping names the last job this machine ran, across restarts (the newest record in jobs-state.json)
if let Some(last) = j.ledger.records.values().max_by_key(|r| r.started_at) {
j.wake.set_last_job(&last.id);
}
if !j.url.is_empty() {
let wake_url = wake_url_of(std::env::var("IGNEUM_APP_JOBS_WAKE_URL").ok());
if !wake_url.is_empty() {
@ -568,6 +572,7 @@ impl Jobs {
return None;
}
shared.event("info", &format!("job {} ({}) starts: {}", job.id, job.kind, job.label()));
self.wake.set_last_job(&job.id);
let ctl = Arc::new(Ctl::default());
let needs_miners_stopped = job.kind == "shard-benchmark" || (job.kind == "run" && job.bool_param("stop_miners_first"));
let cards_off: Vec<String> = if job.kind == "run" { job.list_param("cards_off") } else { vec![] };
@ -827,9 +832,43 @@ fn upload_file(shared: &Arc<Shared>, job: &Job, path: &Path, label_prefix: &str)
// busy-loops: a reply that came back early is followed by the rest of a 10 s floor, a failing endpoint backs off
// 5, 15, then 60 s, and an empty stamp (nothing published yet) waits a full hold.
/// Shared with the waker thread: it polls only while remote jobs are allowed.
/// Shared with the waker thread: it polls only while remote jobs are allowed. The ping (MF-11, 0.3.21): every wake
/// request names this machine (its id8, no secret), the app version and the last job it ran, so the relay can show a
/// machine whose job channel has gone quiet ("silent since <time>, last job <name>") the moment 15 minutes pass with no
/// poll; a dead app polls nothing, and before this nothing on the console said so until the next upload gap was noticed.
pub struct WakeCtl {
on: AtomicBool,
machine: String,
last_job: Mutex<String>,
}
impl WakeCtl {
pub fn new(on: bool, machine: &str) -> WakeCtl {
WakeCtl { on: AtomicBool::new(on), machine: machine.to_string(), last_job: Mutex::new(String::new()) }
}
pub fn set_last_job(&self, id: &str) {
*self.last_job.lock().unwrap() = id.to_string();
}
/// The query fragment of the ping: machine=<id8>&v=<version>[&job=<id>]; values are the app's own, URL-safe by shape.
pub fn ping(&self) -> String {
ping_query(&self.machine, crate::engine::VERSION, &self.last_job.lock().unwrap())
}
}
/// machine and job ids as the relay reads them: id8 is 8 hex; a job id is the publisher's `[\w.-]` name; anything else
/// is dropped from the query rather than escaped (the relay clips and validates on its side too).
pub fn ping_query(machine: &str, version: &str, last_job: &str) -> String {
let safe = |s: &str, max: usize| -> String { s.chars().filter(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_' | ':')).take(max).collect() };
let m = safe(machine, 16);
if m.is_empty() {
return String::new();
}
let mut q = format!("machine={m}&v={}", safe(version, 32));
let j = safe(last_job, 80);
if !j.is_empty() {
q.push_str(&format!("&job={j}"));
}
q
}
/// The stamp logic, free of I/O for the tests.
@ -892,17 +931,24 @@ fn wake_url_of(env: Option<String>) -> String {
}
}
fn wake_query(url: &str, since: &str) -> String {
if since.is_empty() {
url.to_string()
} else {
format!("{url}{}since={since}", if url.contains('?') { '&' } else { '?' })
fn wake_query(url: &str, since: &str, ping: &str) -> String {
let mut out = url.to_string();
let mut sep = if url.contains('?') { '&' } else { '?' };
if !since.is_empty() {
out.push(sep);
out.push_str(&format!("since={since}"));
sep = '&';
}
if !ping.is_empty() {
out.push(sep);
out.push_str(ping);
}
out
}
/// One long-poll. Ok carries the relay's stamp (empty when it holds none).
fn wake_request(url: &str, since: &str) -> Result<String, String> {
let full = wake_query(url, since);
fn wake_request(url: &str, since: &str, ping: &str) -> Result<String, String> {
let full = wake_query(url, since, ping);
let max_time = (WAKE_HOLD_S + 13).to_string();
let (code, out) = run_capture(Command::new(crate::platform::tool("curl")).args(["-fsS", "--max-time", &max_time, &full]), Duration::from_secs(WAKE_HOLD_S + 20));
if code != Some(0) {
@ -923,7 +969,7 @@ fn wake_loop(shared: Arc<Shared>, url: String, ctl: Arc<WakeCtl>) {
continue;
}
let t0 = Instant::now();
match wake_request(&url, &st.stamp) {
match wake_request(&url, &st.stamp, &ctl.ping()) {
Ok(stamp) => {
let (r, recovered) = st.reply(&stamp);
if recovered {
@ -1690,9 +1736,16 @@ mod tests {
assert_eq!(wake_url_of(None), WAKE_URL);
assert_eq!(wake_url_of(Some(String::new())), "");
assert_eq!(wake_url_of(Some(" http://127.0.0.1:4180/wake ".into())), "http://127.0.0.1:4180/wake");
assert_eq!(wake_query("https://r/wake", ""), "https://r/wake");
assert_eq!(wake_query("https://r/wake", "2026-10-05T11:02:17Z.5e7b56f5"), "https://r/wake?since=2026-10-05T11:02:17Z.5e7b56f5");
assert_eq!(wake_query("https://r/api/wake?x=1", "S"), "https://r/api/wake?x=1&since=S");
assert_eq!(wake_query("https://r/wake", "", ""), "https://r/wake");
assert_eq!(wake_query("https://r/wake", "2026-10-05T11:02:17Z.5e7b56f5", ""), "https://r/wake?since=2026-10-05T11:02:17Z.5e7b56f5");
assert_eq!(wake_query("https://r/api/wake?x=1", "S", ""), "https://r/api/wake?x=1&since=S");
// the ping (MF-11): the machine, the version and the last job ride on every wake request
assert_eq!(wake_query("https://r/wake", "", "machine=1ccfe586&v=0.3.21"), "https://r/wake?machine=1ccfe586&v=0.3.21");
assert_eq!(wake_query("https://r/wake", "S", "machine=1ccfe586&v=0.3.21&job=update-now-0319"), "https://r/wake?since=S&machine=1ccfe586&v=0.3.21&job=update-now-0319");
assert_eq!(ping_query("1ccfe586", "0.3.21", ""), "machine=1ccfe586&v=0.3.21");
assert_eq!(ping_query("1ccfe586", "0.3.21", "update-now-0319-1ccfe586"), "machine=1ccfe586&v=0.3.21&job=update-now-0319-1ccfe586");
assert_eq!(ping_query("", "0.3.21", "x"), "", "no machine, no ping");
assert_eq!(ping_query("1ccfe586", "0.3.21", "a b&c=d"), "machine=1ccfe586&v=0.3.21&job=abcd", "only the safe characters travel");
}
}

View file

@ -49,6 +49,7 @@ mod chainfacts;
mod card;
mod drivertable;
mod drivers;
mod bootcheck;
use std::io::{BufRead, Write};
use std::sync::mpsc::channel;

View file

@ -380,6 +380,12 @@ impl Updater {
let rolled_back = v.get("rolled_back").and_then(|x| x.as_bool()).unwrap_or(false);
let deferred = v.get("deferred").and_then(|x| x.as_bool()).unwrap_or(false);
let ver = v.get("version").and_then(|x| x.as_str()).unwrap_or("").to_string();
// 0.3.21: the helper says how the app came back (ok, rolled-back, relaunched, installer-failed) and after how long
let ret = v.get("return").and_then(|x| x.as_str()).unwrap_or("");
let ready_s = v.get("ready_s").and_then(|x| x.as_i64()).unwrap_or(-1);
if !ret.is_empty() {
shared.log(&format!("update-return: the helper reports '{ret}' for {ver}{}", if ready_s >= 0 { format!(", an engine answered after {ready_s} s") } else { ", no engine answered inside its window".to_string() }));
}
if !ok && deferred {
// the installer never ran (nobody answered the administrator prompt): not a failure, it tries again
shared.log(&format!("OTA: the update to {ver} was deferred before this start ({err}); it tries again in this machine's slot"));
@ -426,6 +432,12 @@ impl Updater {
self.pending.as_ref().map(|p| p.starts >= 3).unwrap_or(false)
}
/// The version this engine replaced, on the first start after an update (MF-11: the read-back line the engine logs
/// as its first act, "app <version> up after the update from <from>"); None on any other start.
pub fn updated_from(&self) -> Option<String> {
self.pending.as_ref().filter(|p| p.starts == 1 && p.to == self.current).map(|p| p.from.clone())
}
// ---- state for the dashboard -------------------------------------------------------------------------------------
fn publish(&self, shared: &Arc<Shared>) {
@ -951,6 +963,10 @@ impl Updater {
c.args(["-NoProfile", "-ExecutionPolicy", "Bypass", "-WindowStyle", "Hidden", "-File"]).arg(&script).args([
"-Mode", "apply", "-EnginePid", &std::process::id().to_string(), "-Installer", &staged.display().to_string(), "-Version", &to, "-Result", &result.display().to_string(), "-InstallDir", &install_dir.display().to_string(), "-Sha256", &entry.sha256,
]);
// 0.3.21 (MF-11): the helper owns the return. It keeps the exe set beside the app, launches the app itself after
// the installer, polls the new engine's api/state, restores the kept set when nothing answers, and posts one line
// to the intake either way (the key it needs is in igneum-app.json beside the exe; nothing on the command line).
c.args(["-Previous", &previous_dir_for(&install_dir).display().to_string(), "-Machine", &shared.runtime.id8(), "-Intake", &shared.packaged.log_intake_url, "-AppDir", &self.app_dir.display().to_string()]);
let per_user = !under_program_files(&install_dir);
shared.log(&format!("update: starting the installer first, the miners keep running: {} ({})", script.display(), if per_user { "per-user install, no administrator prompt" } else { "install in Program Files: Windows asks for administrator approval" }));
if !per_user {
@ -1002,6 +1018,7 @@ impl Updater {
let mut c = Command::new(crate::platform::tool("powershell"));
c.args(["-NoProfile", "-ExecutionPolicy", "Bypass", "-WindowStyle", "Hidden", "-File"]).arg(&script).args([
"-Sha256", &sha, "-Mode", "rollback", "-EnginePid", &std::process::id().to_string(), "-Installer", &p.previous_installer, "-Version", &p.to, "-Result", &result.display().to_string(), "-InstallDir", &install_dir.display().to_string(),
"-Previous", &previous_dir_for(&install_dir).display().to_string(), "-Machine", &shared.runtime.id8(), "-Intake", &shared.packaged.log_intake_url, "-AppDir", &self.app_dir.display().to_string(),
]);
spawn_detached(&mut c)?;
Ok(())
@ -1089,6 +1106,189 @@ fn installer_name_for(version: &str) -> String {
format!("Igneum-Miner-Setup-{version}.exe")
}
/// Windows: the folder the helper keeps the running exe set in before the installer runs, beside the app
/// (`<install dir>.previous`, so `...\Programs\Igneum Miner.previous`). A rollback copies it back over the install folder.
#[allow(dead_code)]
fn previous_dir_for(install_dir: &Path) -> PathBuf {
let name = install_dir.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_else(|| "Igneum Miner".into());
install_dir.with_file_name(format!("{name}.previous"))
}
// ---- the return after a Windows install (MF-11, 0.3.21) ------------------------------------------------------------
//
// PC 2 took the 0.3.19 update-now at 10:34Z on 7 October 2026 and was silent from 10:46Z. Until 0.3.21 the Windows helper's
// job ended when the installer exited 0: the installer's own [Run] entry relaunched the app, nobody checked that an engine
// answered, the window host never restarted an engine that died, and a second launch deferred to a surviving host through
// its single-instance mutex. The sequence below is what the helper does from the installer's exit on, written once here
// so a test can drive it with injected exit codes and answers, and mirrored line for line by WIN_HELPER's PowerShell.
/// How long the helper waits for an engine to answer api/state after a launch, and how often it asks.
pub const RETURN_READY_S: u64 = 120;
pub const RETURN_POLL_S: u64 = 3;
/// One step of the helper after the installer exits.
#[derive(Debug, Clone, PartialEq)]
pub enum ReturnStep {
/// start igneum-app.exe --launch from the install folder (a detached process; the helper outlives the old engine)
Launch,
/// poll app.url + api/state for RETURN_READY_S; `want` is the version that must answer ("" = any engine)
WaitReady { want: String },
/// quit through the API, then end what is left by name (the installer's own stop order)
StopAll,
/// copy the kept exe set (`<install dir>.previous`) back over the install folder
RestorePrevious,
/// the result file for the next engine and the one intake line
Done { ok: bool, rolled_back: bool, fault: bool, how: &'static str },
}
/// What a WaitReady step saw: the version that answered, or nobody.
pub type Answer = Option<String>;
/// The helper's sequence from the installer's exit code on. `answers` is consulted once per WaitReady, in order (the test
/// injects them; the PowerShell asks the engine). `previous_kept` says whether the exe set was copied aside before the
/// installer ran.
pub fn return_sequence(installer_exit: i32, version: &str, previous_kept: bool, mut answers: impl FnMut(usize) -> Answer) -> Vec<ReturnStep> {
let mut steps = vec![ReturnStep::Launch];
let want = if installer_exit == 0 { version.to_string() } else { String::new() };
steps.push(ReturnStep::WaitReady { want: want.clone() });
match answers(0) {
Some(v) if want.is_empty() || v == want => {
steps.push(if installer_exit == 0 { ReturnStep::Done { ok: true, rolled_back: false, fault: false, how: "ok" } } else { ReturnStep::Done { ok: false, rolled_back: false, fault: true, how: "installer-failed" } });
return steps;
}
_ => {}
}
// nothing (or the wrong engine) answered inside the window: back to the kept version
steps.push(ReturnStep::StopAll);
if previous_kept {
steps.push(ReturnStep::RestorePrevious);
steps.push(ReturnStep::Launch);
steps.push(ReturnStep::WaitReady { want: String::new() });
let how = if answers(1).is_some() { "rolled-back" } else { "rolled-back-silent" };
steps.push(ReturnStep::Done { ok: false, rolled_back: true, fault: true, how });
} else {
steps.push(ReturnStep::Launch);
steps.push(ReturnStep::Done { ok: false, rolled_back: false, fault: true, how: "relaunched" });
}
steps
}
/// The helper before 0.3.21, for the record: the installer's exit code alone decided the result and nothing was asked
/// of the new engine (the known-failed shape of MF-11).
pub fn legacy_return_sequence(installer_exit: i32) -> Vec<ReturnStep> {
if installer_exit == 0 { vec![ReturnStep::Done { ok: true, rolled_back: false, fault: false, how: "" }] } else { vec![ReturnStep::Launch, ReturnStep::Done { ok: false, rolled_back: false, fault: false, how: "" }] }
}
#[cfg(test)]
mod return_tests {
use super::*;
fn done(steps: &[ReturnStep]) -> &ReturnStep {
steps.last().unwrap()
}
fn answers(list: &[Answer]) -> impl FnMut(usize) -> Answer + '_ {
move |i| list.get(i).cloned().flatten()
}
/// Known-failed first: the helper before 0.3.21 reported ok on the installer's exit 0 with nobody answering.
#[test]
fn the_legacy_helper_reports_ok_with_no_engine_up() {
let steps = legacy_return_sequence(0);
assert_eq!(steps, vec![ReturnStep::Done { ok: true, rolled_back: false, fault: false, how: "" }]);
assert!(!steps.iter().any(|s| matches!(s, ReturnStep::WaitReady { .. })), "nothing was asked of the new engine");
assert!(!steps.iter().any(|s| matches!(s, ReturnStep::Launch)), "the launch was the installer's, not the helper's");
}
#[test]
fn installer_ok_and_the_new_engine_answers_is_ok() {
let steps = return_sequence(0, "0.3.21", true, answers(&[Some("0.3.21".into())]));
assert_eq!(steps, vec![ReturnStep::Launch, ReturnStep::WaitReady { want: "0.3.21".into() }, ReturnStep::Done { ok: true, rolled_back: false, fault: false, how: "ok" }]);
}
#[test]
fn installer_ok_and_nobody_answers_restores_the_previous_exe_set() {
let steps = return_sequence(0, "0.3.21", true, answers(&[None, Some("0.3.20".into())]));
assert_eq!(steps, vec![
ReturnStep::Launch, ReturnStep::WaitReady { want: "0.3.21".into() }, ReturnStep::StopAll, ReturnStep::RestorePrevious, ReturnStep::Launch, ReturnStep::WaitReady { want: String::new() },
ReturnStep::Done { ok: false, rolled_back: true, fault: true, how: "rolled-back" },
]);
}
#[test]
fn the_old_engine_still_answering_after_exit_0_is_not_the_new_one() {
// the installer said 0 but never replaced the running engine (files in use): the old version answers, the window
// ends in a rollback to the kept set, which is the same version, and a FAULT line says so
let steps = return_sequence(0, "0.3.21", true, answers(&[Some("0.3.20".into()), Some("0.3.20".into())]));
assert!(matches!(done(&steps), ReturnStep::Done { ok: false, rolled_back: true, fault: true, .. }));
}
#[test]
fn nobody_answers_twice_is_still_reported() {
let steps = return_sequence(0, "0.3.21", true, answers(&[None, None]));
assert_eq!(*done(&steps), ReturnStep::Done { ok: false, rolled_back: true, fault: true, how: "rolled-back-silent" });
}
#[test]
fn without_a_kept_set_the_helper_relaunches_what_is_there_and_reports() {
let steps = return_sequence(0, "0.3.21", false, answers(&[None]));
assert_eq!(steps, vec![ReturnStep::Launch, ReturnStep::WaitReady { want: "0.3.21".into() }, ReturnStep::StopAll, ReturnStep::Launch, ReturnStep::Done { ok: false, rolled_back: false, fault: true, how: "relaunched" }]);
}
#[test]
fn a_failed_installer_relaunches_the_old_version_and_reports_a_fault() {
// exit 5 (cancelled) or 8 (files in use, a restart wanted): any engine answering is the old one, kept, with a FAULT line
for code in [1, 5, 8] {
let steps = return_sequence(code, "0.3.21", true, answers(&[Some("0.3.20".into())]));
assert_eq!(steps, vec![ReturnStep::Launch, ReturnStep::WaitReady { want: String::new() }, ReturnStep::Done { ok: false, rolled_back: false, fault: true, how: "installer-failed" }], "exit {code}");
}
let steps = return_sequence(5, "0.3.21", true, answers(&[None, Some("0.3.20".into())]));
assert!(matches!(done(&steps), ReturnStep::Done { ok: false, rolled_back: true, fault: true, .. }));
}
#[test]
fn every_sequence_launches_before_it_waits_and_ends_in_a_done() {
for code in [0, 1, 5, 8] {
for kept in [true, false] {
for a in [vec![None, None], vec![Some("0.3.21".to_string()), None], vec![None, Some("0.3.20".to_string())]] {
let steps = return_sequence(code, "0.3.21", kept, answers(&a));
assert_eq!(steps[0], ReturnStep::Launch);
assert!(matches!(steps[1], ReturnStep::WaitReady { .. }));
assert!(matches!(done(&steps), ReturnStep::Done { .. }));
let fault = matches!(done(&steps), ReturnStep::Done { fault: true, .. });
let ok = matches!(done(&steps), ReturnStep::Done { ok: true, .. });
assert!(ok != fault, "a result is ok or a fault, never neither: {steps:?}");
}
}
}
}
/// The PowerShell mirrors the sequence: every step has its line, in order, and the installer is told the helper relaunches.
#[test]
fn the_windows_helper_carries_every_step() {
let h = WIN_HELPER;
let at = |s: &str| h.find(s).unwrap_or_else(|| panic!("WIN_HELPER lacks '{s}'"));
assert!(at("/IGNOTA=2") > 0, "the installer must not relaunch (IGNOTA=1 is the old helpers' path)");
assert!(h.contains(&format!("$ReadyS = {RETURN_READY_S}")) && h.contains(&format!("$PollS = {RETURN_POLL_S}")));
let robocopy_keep = at("robocopy $InstallDir $Previous");
let installer = at("Start-Process -FilePath $Installer"); // console: a test marker, not a spawn (the helper line above it carries the comment)
let launch = at("function Launch()");
let wait = at("function WaitReady(");
let stop = at("function StopAll()");
let restore = at("robocopy $Previous $InstallDir");
let report = at("function Report(");
assert!(launch < installer && wait < installer && stop < installer && report < installer, "the functions are defined before the installer runs");
assert!(robocopy_keep < installer && restore < installer, "the keep and the restore are functions defined before the installer line");
assert!(at("$kept = KeepPrevious") < installer, "the exe set is kept before the installer runs");
assert!(at("Comeback $code $kept") > installer, "the return runs after the installer");
for marker in ["'ok'", "'installer-failed'", "'rolled-back'", "'rolled-back-silent'", "'relaunched'"] {
assert!(h.contains(marker), "the helper never writes return={marker}");
}
assert!(h.contains("FAULT update-return:"), "the fault line");
assert!(h.contains("update-return: ok"), "the ok line");
assert!(h.contains("api/state"), "readiness is the engine's own answer");
assert!(!h.contains("-IntakeKey"), "no key travels on a command line (R4.3.8)");
}
}
fn curl(args: &[&str], limit: Duration) -> Result<(), String> {
let mut c = Command::new(crate::platform::tool("curl"));
c.args(args);
@ -1324,60 +1524,180 @@ case "$MODE" in
esac
"#;
#[cfg(windows)]
/// The Windows helper. Not cfg-gated so the return test above can read it on every platform.
#[allow(dead_code)]
const WIN_HELPER: &str = r#"# Igneum Miner update helper, written by the engine (src/ota.rs). Not for running by hand.
# powershell -File ota-apply.ps1 -Mode apply|rollback -EnginePid <pid> -Installer <setup exe> -Version <v> -Result <json> -InstallDir <folder> -Sha256 <hex>
# powershell -File ota-apply.ps1 -Mode apply|rollback -EnginePid <pid> -Installer <setup exe> -Version <v> -Result <json> -InstallDir <folder> -Sha256 <hex> -Previous <folder> -Machine <id8> -Intake <url> -AppDir <app data>
# The installer runs FIRST, while the engine keeps mining (4 October 2026: two unattended PCs sat stopped at an
# administrator prompt nobody could click). A per-user installer (0.3.3 and later, PrivilegesRequired=lowest) needs no
# prompt; an older administrator installer raises one through ShellExecute. Only when the installer actually runs does
# its PrepareToInstall step stop the engine (api/quit: miners first, then the node), replace the files and relaunch
# the app (/IGNOTA=1). A declined, timed-out or unanswered prompt leaves the engine running: the result says
# deferred:true and the engine shows "waits for the next time someone is at this PC". The old app is relaunched only
# when the engine is gone and the install did not happen.
param([string]$Mode, [int]$EnginePid, [string]$Installer, [string]$Version, [string]$Result, [string]$InstallDir, [string]$Sha256 = '')
# its PrepareToInstall step stop the engine (api/quit: miners first, then the node) and replace the files. A declined,
# timed-out or unanswered prompt leaves the engine running: the result says deferred:true and the engine shows "waits for
# the next time someone is at this PC".
# From 0.3.21 (MF-11, 7 October 2026) this helper owns the return: it keeps the running exe set beside the app before the
# installer runs, starts the app itself afterwards (/IGNOTA=2 tells the installer not to), waits for an engine to answer
# api/state with the new version, restores the kept set when nothing answers inside the window, and posts one line to
# the log intake either way (the key is read from igneum-app.json beside the exe, never from the command line). The
# sequence is src/ota.rs return_sequence(), tested there with injected exit codes; this file mirrors it step for step.
param([string]$Mode, [int]$EnginePid, [string]$Installer, [string]$Version, [string]$Result, [string]$InstallDir, [string]$Sha256 = '', [string]$Previous = '', [string]$Machine = '', [string]$Intake = '', [string]$AppDir = '')
$log = Join-Path (Split-Path -Parent $Result) 'ota-apply.log'
if (-not $AppDir) { $AppDir = Split-Path -Parent $Result }
$ReadyS = 120
$PollS = 3
function Log([string]$t) { Add-Content -Path $log -Value ("{0} {1}" -f (Get-Date -Format s), $t) }
function Done([bool]$ok, [string]$err, [bool]$rb, [bool]$deferred) {
$o = @{ ok = $ok; version = $Version; error = $err; rolled_back = $rb; deferred = $deferred; at = [int][double](Get-Date -UFormat %s) }
function Done([bool]$ok, [string]$err, [bool]$rb, [bool]$deferred, [string]$ret, [int]$readyS) {
$o = @{ ok = $ok; version = $Version; error = $err; rolled_back = $rb; deferred = $deferred; at = [int][double](Get-Date -UFormat %s); 'return' = $ret; ready_s = $readyS }
($o | ConvertTo-Json -Compress) | Set-Content -Path $Result -Encoding ASCII
}
function EngineAlive() { return [bool](Get-Process -Id $EnginePid -ErrorAction SilentlyContinue) }
function Relaunch() {
if (EngineAlive) { return }
$exe = Join-Path $InstallDir 'igneum-app.exe'
# console: igneum-app.exe is a windows-subsystem program (no console); -WindowStyle Hidden would hide the window host it opens
if (Test-Path $exe) { Log 'engine gone and nothing installed: starting the old app again'; Start-Process -FilePath $exe -ArgumentList '--launch' -WorkingDirectory $InstallDir | Out-Null }
function AppUrl() {
$f = Join-Path $AppDir 'app.url'
if (Test-Path $f) { return (Get-Content $f -Raw).Trim() }
return ''
}
Log "$Mode : engine $EnginePid installer '$Installer' version $Version (the engine keeps mining until the installer runs)"
if (-not (Test-Path $Installer)) { Log 'installer missing'; Done $false 'the downloaded installer is missing' $false $false; exit 1 }
function AppVersion() {
# the engine's own answer: GET <app.url>api/state and its version field (the URL file is rewritten by every start)
$u = AppUrl
if (-not $u) { return '' }
try { $r = Invoke-RestMethod -Uri ($u + 'api/state') -TimeoutSec 5 -UseBasicParsing; return [string]$r.version } catch { return '' }
}
function WaitReady([string]$want, [int]$limitS) {
# the seconds until an engine answered with the wanted version (any version when $want is empty); -1 when none did
$t0 = Get-Date
while (((Get-Date) - $t0).TotalSeconds -lt $limitS) {
$v = AppVersion
if ($v -and (($want -eq '') -or ($v -eq $want))) { return [int]((Get-Date) - $t0).TotalSeconds }
Start-Sleep -Seconds $PollS
}
return -1
}
function Launch() {
$exe = Join-Path $InstallDir 'igneum-app.exe'
if (-not (Test-Path $exe)) { Log ("nothing to start: " + $exe + " is missing"); return $false }
Log ("starting " + $exe + " --launch")
# console: igneum-app.exe is a windows-subsystem program (no console); -WindowStyle Hidden would hide the window host it opens
Start-Process -FilePath $exe -ArgumentList '--launch' -WorkingDirectory $InstallDir | Out-Null
return $true
}
function StopAll() {
# the quit through the API first (miners, then the node), then whatever is left by name: the installer's own order
$u = AppUrl
if ($u) { try { Invoke-WebRequest -Uri ($u + 'api/quit') -Method POST -Body '{}' -ContentType 'application/json' -UseBasicParsing -TimeoutSec 5 | Out-Null } catch { } }
$deadline = (Get-Date).AddSeconds(40)
while ((Get-Date) -lt $deadline) {
if (@(Get-Process -Name 'igneum-app', 'igneumd' -ErrorAction SilentlyContinue).Count -eq 0) { break }
Start-Sleep -Milliseconds 500
}
foreach ($n in @('Igneum Miner', 'igneum-app', 'igneum-miner', 'igneum-worker-cuda', 'igneum-worker-opencl', 'igneumd')) {
Get-Process -Name $n -ErrorAction SilentlyContinue | ForEach-Object { Stop-Process -Id $_.Id -Force -ErrorAction SilentlyContinue }
}
Start-Sleep -Seconds 1
}
function Report([string]$line) {
# one line to the log intake, label fault-win-<id8>, as site/api/log.mjs expects; the key is the one the installed
# app carries (igneum-app.json beside the exe, or the kept copy); nothing to post with is logged, never fatal
Log $line
if (-not $Intake -or -not $Machine) { return }
$cfg = Join-Path $InstallDir 'igneum-app.json'
if (-not (Test-Path $cfg) -and $Previous) { $cfg = Join-Path $Previous 'igneum-app.json' }
if (-not (Test-Path $cfg)) { Log 'no igneum-app.json to read the intake key from; the line stays in this log'; return }
try {
$key = [string](Get-Content $cfg -Raw | ConvertFrom-Json).log_intake_key
if (-not $key) { Log 'igneum-app.json carries no intake key'; return }
$o = @{ label = ('fault-win-' + $Machine); machine = ($env:COMPUTERNAME + '-' + $Machine); run_id = ('update-return-' + $Version); lines = ("IGNEUM-APP version=" + $Version + " machine=" + $Machine + " platform=windows node=?`n" + $line) }
$bytes = [Text.Encoding]::UTF8.GetBytes(($o | ConvertTo-Json -Compress))
Invoke-RestMethod -Method Post -Uri $Intake -Headers @{ 'x-igneum-key' = $key } -ContentType 'application/json; charset=utf-8' -Body $bytes -TimeoutSec 30 | Out-Null
Log 'intake line posted'
} catch { Log ('intake post failed: ' + $_.Exception.Message) }
}
function KeepPrevious() {
# the running exe set beside the app, what a rollback restores; packs, build and dist are rebuilt or unneeded
if (-not $Previous) { return $false }
try {
& robocopy $InstallDir $Previous /MIR /XD packs build dist /R:1 /W:1 /NFL /NDL /NJH /NJS /NP | Out-Null
if ($LASTEXITCODE -lt 8 -and (Test-Path (Join-Path $Previous 'igneum-app.exe'))) { Log ("previous version kept at " + $Previous); return $true }
Log ("robocopy could not keep the previous version (exit " + $LASTEXITCODE + ")")
} catch { Log ('could not keep the previous version: ' + $_.Exception.Message) }
return $false
}
function RestorePrevious() {
if (-not $Previous -or -not (Test-Path (Join-Path $Previous 'igneum-app.exe'))) { return $false }
try {
& robocopy $Previous $InstallDir /MIR /XD packs build dist /R:2 /W:2 /NFL /NDL /NJH /NJS /NP | Out-Null
if ($LASTEXITCODE -lt 8) { Log 'previous version restored over the install folder'; return $true }
Log ("robocopy could not restore the previous version (exit " + $LASTEXITCODE + ")")
} catch { Log ('could not restore the previous version: ' + $_.Exception.Message) }
return $false
}
function Comeback([int]$code, [bool]$kept) {
# src/ota.rs return_sequence(): Launch, WaitReady, then Done or StopAll, RestorePrevious, Launch, WaitReady, Done
if (-not (EngineAlive)) { Launch | Out-Null } else { Log 'the engine is still up after the installer (it did not stop it)' }
$want = ''
if ($code -eq 0) { $want = $Version }
$ready = WaitReady $want $ReadyS
if ($ready -ge 0) {
if ($code -eq 0) {
Done $true '' $false $false 'ok' $ready
Report ("update-return: ok " + $Version + " up in " + $ready + " s")
exit 0
}
Done $false ("the installer exited with code " + $code + " (see ota-setup.log); the previous version answers again") $false $false 'installer-failed' $ready
Report ("FAULT update-return: the installer of " + $Version + " exited with code " + $code + "; the previous version answered again after " + $ready + " s")
exit 1
}
Log ("no engine answered api/state with '" + $want + "' inside " + $ReadyS + " s; back to the previous version")
StopAll
if ($kept -and (RestorePrevious)) {
Launch | Out-Null
$r2 = WaitReady '' $ReadyS
if ($r2 -ge 0) {
Done $false ("Igneum Miner " + $Version + " did not answer within " + $ReadyS + " s; the previous version was restored") $true $false 'rolled-back' $r2
Report ("FAULT update-return: " + $Version + " did not answer inside " + $ReadyS + " s after the install; the previous exe set was restored and answers after " + $r2 + " s")
exit 1
}
Done $false ("Igneum Miner " + $Version + " did not answer within " + $ReadyS + " s; the previous version was restored but did not answer either") $true $false 'rolled-back-silent' -1
Report ("FAULT update-return: " + $Version + " did not answer inside " + $ReadyS + " s; the previous exe set was restored and did not answer inside " + $ReadyS + " s either; a hand start is needed on this PC")
exit 1
}
Launch | Out-Null
Done $false ("Igneum Miner " + $Version + " did not answer within " + $ReadyS + " s; no kept version to restore; what is installed was started again") $false $false 'relaunched' -1
Report ("FAULT update-return: " + $Version + " did not answer inside " + $ReadyS + " s and no previous exe set was kept; what is installed was started again")
exit 1
}
Log "$Mode : engine $EnginePid installer '$Installer' version $Version previous '$Previous' (the engine keeps mining until the installer runs)"
if (-not (Test-Path $Installer)) { Log 'installer missing'; Done $false 'the downloaded installer is missing' $false $false '' -1; exit 1 }
# the installer is hashed again right before it runs (R4.3.5)
if (-not $Sha256) { Log 'no sha256 given'; Done $false 'no sha256 for the installer; not run' $false $false; exit 1 }
if (-not $Sha256) { Log 'no sha256 given'; Done $false 'no sha256 for the installer; not run' $false $false '' -1; exit 1 }
$have = (Get-FileHash -Path $Installer -Algorithm SHA256).Hash.ToLower()
if ($have -ne $Sha256.ToLower()) { Log "sha256 mismatch: $have"; Remove-Item -Path $Installer -Force -ErrorAction SilentlyContinue; Done $false 'the installer changed since it was verified; not run' $false $false; exit 1 }
if ($have -ne $Sha256.ToLower()) { Log "sha256 mismatch: $have"; Remove-Item -Path $Installer -Force -ErrorAction SilentlyContinue; Done $false 'the installer changed since it was verified; not run' $false $false '' -1; exit 1 }
Log 'installer sha256 verified'
$kept = $false
if ($Mode -eq 'apply') { $kept = KeepPrevious }
$setupLog = Join-Path (Split-Path -Parent $Result) 'ota-setup.log'
$setupArgs = @('/VERYSILENT', '/SUPPRESSMSGBOXES', '/NORESTART', '/CLOSEAPPLICATIONS', '/IGNOTA=1', ('/LOG="' + $setupLog + '"'))
$setupArgs = @('/VERYSILENT', '/SUPPRESSMSGBOXES', '/NORESTART', '/CLOSEAPPLICATIONS', '/IGNOTA=2', ('/LOG="' + $setupLog + '"'))
$code = -1
try {
# no -Verb RunAs: a per-user installer just runs; an administrator installer makes Windows ask, and a declined or
# timed-out prompt comes back here as an exception with the engine still mining
# console: the Inno Setup installer is a GUI program (no console), /VERYSILENT shows nothing
$p = Start-Process -FilePath $Installer -ArgumentList $setupArgs -Wait -PassThru
if ($p.ExitCode -eq 0) {
if ($Mode -eq 'rollback') { Done $false "Igneum Miner $Version did not stay up twice; the previous version was reinstalled" $true $false }
else { Done $true '' $false $false }
Log 'installer exit 0'
exit 0
}
Log ("installer exit " + $p.ExitCode)
Done $false ("the installer exited with code " + $p.ExitCode + " (see ota-setup.log)") $false $false
Relaunch
exit 1
$code = $p.ExitCode
} catch {
$msg = $_.Exception.Message
Log ("installer did not run: " + $msg)
Log 'OTA: waiting for administrator approval; the engine keeps mining; the update waits for the next time someone is at this PC'
Done $false ("waiting for administrator approval (" + $msg + ")") $false $true
Relaunch
Done $false ("waiting for administrator approval (" + $msg + ")") $false $true '' -1
if (-not (EngineAlive)) { Launch | Out-Null }
exit 1
}
Log ("installer exit " + $code)
if ($Mode -eq 'rollback') {
# the kept installer of the previous version ran: the same return, reported as the rollback it is
if (-not (EngineAlive)) { Launch | Out-Null }
$r = WaitReady '' $ReadyS
Done $false ("Igneum Miner " + $Version + " did not stay up twice; the previous version was reinstalled") $true $false 'rolled-back' $r
Report ("FAULT update-return: " + $Version + " did not stay up twice; the previous version was reinstalled (installer exit " + $code + ", answered after " + $r + " s)")
exit 1
}
Comeback $code $kept
"#;

View file

@ -13,7 +13,7 @@ pub fn tool(name: &str) -> PathBuf {
let sys = format!("{root}\\System32");
let p = match name {
"powershell" => format!("{sys}\\WindowsPowerShell\\v1.0\\powershell.exe"),
"cmd" | "curl" | "reg" | "icacls" | "taskkill" | "w32tm" | "net" | "tar" | "wsl" => format!("{sys}\\{name}.exe"),
"cmd" | "curl" | "reg" | "icacls" | "taskkill" | "w32tm" | "net" | "tar" | "wsl" | "wevtutil" | "robocopy" => format!("{sys}\\{name}.exe"),
"nvidia-smi" => {
let pf = std::env::var("ProgramFiles").unwrap_or_else(|_| "C:\\Program Files".into());
let a = format!("{pf}\\NVIDIA Corporation\\NVSMI\\nvidia-smi.exe");

View file

@ -34,6 +34,14 @@ using namespace Microsoft::WRL;
#define ID_TRAY_PAUSE 1002
#define ID_TRAY_QUIT 1003
#define ID_QUIT_TIMER 7
// MF-11 (7 October 2026): an engine that stops without a quit is started again (10 s, then 60 s after three in ten
// minutes); a second "Igneum Miner.exe" that finds this window asks it to do so now (WM_ENGINE_RESTART), instead of
// showing a window that says "the engine stopped" with nothing mining behind it.
#define ID_RESTART_TIMER 8
#define WM_ENGINE_RESTART (WM_APP + 3)
#define RESTART_SOON_MS 10000
#define RESTART_SLOW_MS 60000
#define RESTART_WINDOW_MS 600000
#define IDI_APP 1
static HWND g_hwnd = nullptr;
@ -52,6 +60,8 @@ static bool g_paused = false, g_quitting = false, g_exited = false, g_webviewOk
static NOTIFYICONDATAW g_nid = {};
static std::wstring g_trayTitle = L"Igneum Miner";
static ULONGLONG g_quitStarted = 0;
static int g_restarts = 0; // engine restarts inside the current window
static ULONGLONG g_restartWindowStart = 0; // when that window opened
static std::wstring widen(const std::string& s) {
if (s.empty()) return L"";
@ -261,6 +271,47 @@ static bool startEngine() {
return true;
}
// The engine's handles, closed before another engine is started (the reader thread has already left: it posts the
// "gone" line only after the pipe closed).
static void closeEngine() {
if (g_engineIn) { CloseHandle(g_engineIn); g_engineIn = nullptr; }
if (g_engineOut) { CloseHandle(g_engineOut); g_engineOut = nullptr; }
if (g_engine) { CloseHandle(g_engine); g_engine = nullptr; }
}
// Starts the engine again after it stopped on its own. Three restarts inside ten minutes come 10 s apart; from the
// fourth they come a minute apart, for ever: a miner that sits stopped is a miner lost (plug, tune, play).
static void scheduleRestart() {
ULONGLONG now = GetTickCount64();
if (g_restartWindowStart == 0 || now - g_restartWindowStart > RESTART_WINDOW_MS) { g_restartWindowStart = now; g_restarts = 0; }
g_restarts++;
UINT delay = g_restarts <= 3 ? RESTART_SOON_MS : RESTART_SLOW_MS;
wchar_t buf[160];
swprintf_s(buf, L"The engine stopped. Starting it again in %u s (restart %d).", delay / 1000, g_restarts);
g_status = buf;
setTray(L"Igneum Miner: starting the engine again");
repaintStatus();
SetTimer(g_hwnd, ID_RESTART_TIMER, delay, nullptr);
}
static void restartEngineNow() {
KillTimer(g_hwnd, ID_RESTART_TIMER);
if (g_quitting || !g_exited) return;
closeEngine();
g_exited = false;
g_url.clear();
if (startEngine()) {
g_status = L"";
setTray(L"Igneum Miner: starting");
repaintStatus();
} else {
g_exited = true;
g_status = L"igneum-app.exe is missing next to this program. Run the installer again.";
repaintStatus();
scheduleRestart();
}
}
static void showTrayMenu() {
HMENU m = CreatePopupMenu();
AppendMenuW(m, MF_STRING, ID_TRAY_OPEN, L"Open Igneum Miner");
@ -302,9 +353,9 @@ static LRESULT CALLBACK WndProc(HWND hwnd, UINT msg, WPARAM wp, LPARAM lp) {
if (wp == 1) {
g_exited = true;
if (g_quitting) { DestroyWindow(hwnd); return 0; }
g_status = L"The engine stopped. Close this window and open Igneum Miner again.";
setTray(L"Igneum Miner: stopped");
repaintStatus();
// not a quit of ours: the engine died, or a local caller (the installer, a job) asked it to stop and nothing
// will start it again; this window does (MF-11)
scheduleRestart();
return 0;
}
std::string* line = (std::string*)lp;
@ -330,7 +381,21 @@ static LRESULT CALLBACK WndProc(HWND hwnd, UINT msg, WPARAM wp, LPARAM lp) {
delete line;
return 0;
}
case WM_ENGINE_RESTART:
// a second "Igneum Miner.exe" (the update helper, the Start Menu, the relay agent's start-app) found this window:
// an engine that is gone starts now, not in its backoff
if (g_exited && !g_quitting) restartEngineNow();
return 0;
case WM_QUERYENDSESSION:
// the Restart Manager (an installer's /CLOSEAPPLICATIONS) or a sign-out: this window closes for real, it does not
// hide to the tray (WM_CLOSE below is the user's X)
beginQuit();
return TRUE;
case WM_ENDSESSION:
if (wp) { if (g_engine && !g_exited) sendEngine("quit"); }
return 0;
case WM_TIMER:
if (wp == ID_RESTART_TIMER) { restartEngineNow(); return 0; }
if (wp == ID_QUIT_TIMER) {
DWORD code = 0;
bool gone = !g_engine || (GetExitCodeProcess(g_engine, &code) && code != STILL_ACTIVE);
@ -435,7 +500,7 @@ int WINAPI wWinMain(HINSTANCE hInst, HINSTANCE, PWSTR, int) {
HANDLE once = CreateMutexW(nullptr, TRUE, L"Local\\IgneumMinerWindow");
if (GetLastError() == ERROR_ALREADY_EXISTS) {
HWND other = FindWindowW(L"IgneumMinerWindow", nullptr);
if (other) { ShowWindow(other, SW_SHOW); SetForegroundWindow(other); }
if (other) { PostMessageW(other, WM_ENGINE_RESTART, 0, 0); ShowWindow(other, SW_SHOW); SetForegroundWindow(other); }
return 0;
}
CoInitializeEx(nullptr, COINIT_APARTMENTTHREADED);
@ -473,7 +538,7 @@ int WINAPI wWinMain(HINSTANCE hInst, HINSTANCE, PWSTR, int) {
TranslateMessage(&msg);
DispatchMessageW(&msg);
}
if (g_engine) { CloseHandle(g_engine); }
closeEngine();
CloseHandle(once);
CoUninitialize();
return 0;

View file

@ -85,6 +85,8 @@ Name: "{autodesktop}\Igneum Miner"; Filename: "{app}\igneum-app.exe"; Parameters
; Started as the signed-in user, not as administrator (the data lands in that user's %LOCALAPPDATA%).
Filename: "{app}\igneum-app.exe"; Parameters: "--launch"; Description: "Start Igneum Miner now"; Flags: postinstall nowait skipifsilent runasoriginaluser
; The over-the-air updater (packaging/ota, src/ota.rs) runs this installer /VERYSILENT /IGNOTA=1 and the app must come back by itself.
; /IGNOTA=2 (helpers from 0.3.21 on, MF-11): the helper starts the app itself, polls the new engine and restores the kept
; exe set when nothing answers; this entry stays silent so the two never race for the single-instance window.
Filename: "{app}\igneum-app.exe"; Parameters: "--launch"; Flags: nowait runasoriginaluser; Check: OtaRelaunch
[UninstallRun]
@ -95,7 +97,8 @@ Filename: "powershell.exe"; Parameters: "-NoProfile -ExecutionPolicy Bypass -Fil
Type: filesandordirs; Name: "{app}"
[Code]
// /IGNOTA=1: the app's own updater started this install; relaunch the app when the files are in.
// /IGNOTA=1: the app's own updater (before 0.3.21) started this install; relaunch the app when the files are in.
// /IGNOTA=2: the 0.3.21 helper started it and relaunches the app itself (src/ota.rs WIN_HELPER): no relaunch here.
function OtaRelaunch: Boolean;
begin
Result := ExpandConstant('{param:IGNOTA|0}') = '1';