Build job: a PC builds the node and the app engine for Linux and Windows inside WSL2, mining untouched
New job kind `build` (jobs.rs, jobbuild.rs, jobrun.rs run_build): free-space check on both sides (20 GB), the build-inputs zip by sha256, setup inside the distro as root (mingw-w64 posix, clang for bindgen, protoc, zstd, the Windows rust target; idempotent), sources extracted with the target dir persisting under /root/igneum-build, cargo build --release native and for x86_64-pc-windows-gnu, cargo test for the manifest's packages, binaries zstd-compressed and sent to the relay (fn=upload, Blob PUT, fn=drop; 50 MB each) with sha256 in RESULT lines, STAGE lines with UTC times, a 40-minute default budget and per-stage caps, the Linux side killed on a cap. The app's runner stays serial (one Active at a time), so a build never overlaps a shard job; nothing stops the miners. From this version an unknown job kind is skipped by the app (parse_lenient) instead of rejecting the whole file; the signer stays strict. Mac side: packaging/windows/push-build-inputs.sh packs a fork worktree, app/igneum-app, brand/icons and proto-cuda with a manifest (branch, commit, dirty, builds, tests) and the sha256; publish-jobs.sh add --kind build; tools/build-job.mjs packs, publishes, watches, fetches, checks both sha256 per file and the PE header of every exe (plus verify-exe.py on igneum-app.exe), and places the binaries where push-inputs.sh, make-payload.sh and the cloud-devnet scripts look. relay.mjs drop <file> --body carries the body. Tested on the Mac: 33 app tests (6 new) and the signer's 21; cargo check for x86_64-pc-windows-gnu; the packer (7.9 MB zip, no target dirs); the publisher against a scratch folder with the rebuilt signer, the old signer refusing the kind, a bad job refused at signing; the fetch path against the live relay with a real exe (sha256 and PE pass, a wrong sha256 refused; test items deleted). Not run on a PC: the job itself. docs/plans/build-job.md has the first job for PC 1 and the rollout order (0.3.4 must be on the PCs before a build job is published). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
igneum-josh
parent
0f44edd3de
commit
4e94f27467
11 changed files with 1544 additions and 17 deletions
574
app/igneum-app/src/jobbuild.rs
Normal file
574
app/igneum-app/src/jobbuild.rs
Normal file
|
|
@ -0,0 +1,574 @@
|
|||
//! The `build` job (the model is src/jobs.rs, the runner src/jobrun.rs): a Windows PC builds the node and the app
|
||||
//! engine for Linux and Windows inside its WSL2 Ubuntu, as root, with nothing from Josh. Josh's ask, 4 October 2026
|
||||
//! evening ("efficiency"): every Windows build went through a GitHub runner at 15 to 25 minutes a round and every
|
||||
//! Linux binary was cross-compiled on the Mac under the build lock; the two RTX 5090 PCs sit idle on the CPU side.
|
||||
//!
|
||||
//! What this module holds is the pure part, so it is unit-tested on the Mac: the job's parameters, the plan read
|
||||
//! from the inputs manifest (what to build, what to test), the bash stage scripts that run inside the distro, the
|
||||
//! per-stage time caps, and the parsers for what comes back (free space, the pack stage's outputs file, the relay's
|
||||
//! JSON replies). The runner (jobrun.rs, `run_build`) does the process work: fetch, wsl.exe per stage, upload.
|
||||
//!
|
||||
//! Stages, in order (each a RESULT or STAGE line with a UTC time in the report, each under its own cap):
|
||||
//! fetch the build-inputs zip by https, sha256 checked (src/jobrun.rs fetch_file), the manifest read out of it
|
||||
//! setup apt packages (mingw, clang for bindgen, protoc, zstd), rustup target x86_64-pc-windows-gnu; idempotent
|
||||
//! extract /root/igneum-build/src replaced by the zip's sources (the target dir /root/igneum-build/target persists)
|
||||
//! linux cargo build --release per unit, native
|
||||
//! windows cargo build --release --target x86_64-pc-windows-gnu per unit, mingw-w64 (posix threads), static libgcc
|
||||
//! test cargo test --release for the packages the manifest names (Linux, native)
|
||||
//! pack zstd per binary, sha256 of both forms, build-outputs.json, copied to the job folder on the Windows side
|
||||
//! upload every .zst and the outputs file to the relay (fn=upload client token, PUT to Blob, fn=drop); 50 MB each
|
||||
//! Mining is never stopped: the build is CPU work under `nice`; the app's job runner is serial, so a build never
|
||||
//! overlaps a shard benchmark (src/jobrun.rs: one `Active` at a time, queued jobs wait).
|
||||
//!
|
||||
//! The zip's layout (packaging/windows/push-build-inputs.sh): igneum-build-inputs/{manifest.json, node/ (the fork
|
||||
//! worktree without target dirs), app/igneum-app/, brand/icons/, proto-cuda/ (without nvrtc/redist)}. The manifest:
|
||||
//! { "created_at", "node": {"branch","commit","dirty","source"}, "repo": {...}, "app_version",
|
||||
//! "builds": [{"dir":"node","packages":["kaspad","igneum-miner"],"features":["kaspad/igneum-pow"],"bins":["igneumd","igneum-miner"],"targets":["linux","windows"]},
|
||||
//! {"dir":"app/igneum-app","packages":["igneum-app"],"bins":["igneum-app"],"targets":["linux","windows"],"optional_on":["linux"]}],
|
||||
//! "tests": [{"dir":"app/igneum-app","packages":["igneum-app"]}, {"dir":"node","packages":["igneum-miner"]}] }
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use crate::jobs::{self, Job};
|
||||
use serde_json::Value;
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::Duration;
|
||||
|
||||
pub const RELAY_URL_DEFAULT: &str = "https://relay.igneum.network";
|
||||
/// Everything of the build lives here inside the distro: target/ (persists), src/ (per job), out/<id>/ (outputs).
|
||||
pub const WSL_BASE: &str = "/root/igneum-build";
|
||||
pub const ZIP_ROOT: &str = "igneum-build-inputs";
|
||||
pub const OUTPUTS_FILE: &str = "build-outputs.json";
|
||||
pub const DEFAULT_DISTRO: &str = "Ubuntu-24.04";
|
||||
pub const DEFAULT_WSL_USER: &str = "root";
|
||||
pub const DEFAULT_NICE: u64 = 19;
|
||||
/// The relay's cap for one Blob upload (relay/lib/relay.mjs MAX_BLOB).
|
||||
pub const MAX_UPLOAD_BYTES: u64 = 50 * 1024 * 1024;
|
||||
/// Debian packages the build needs; installed only when `dpkg -s` says they are missing.
|
||||
pub const APT_COMMON: &[&str] = &["build-essential", "pkg-config", "libssl-dev", "clang", "libclang-dev", "cmake", "unzip", "zstd", "protobuf-compiler", "ca-certificates", "curl", "git"];
|
||||
pub const APT_WINDOWS: &[&str] = &["gcc-mingw-w64-x86-64", "g++-mingw-w64-x86-64", "binutils-mingw-w64-x86-64", "mingw-w64-x86-64-dev"];
|
||||
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct BuildParams {
|
||||
pub targets: Vec<String>,
|
||||
pub budget_min: u64,
|
||||
pub stage_min: BTreeMap<String, u64>,
|
||||
pub min_free_gb: u64,
|
||||
pub tests: bool,
|
||||
pub relay_url: String,
|
||||
pub distro: String,
|
||||
pub wsl_user: String,
|
||||
pub nice: u64,
|
||||
/// cargo -j; 0 = cargo's default (every core)
|
||||
pub cargo_jobs: u64,
|
||||
}
|
||||
|
||||
impl BuildParams {
|
||||
pub fn from_job(job: &Job) -> BuildParams {
|
||||
let mut targets = job.list_param("targets");
|
||||
if targets.is_empty() {
|
||||
targets = jobs::BUILD_TARGETS.iter().map(|s| s.to_string()).collect();
|
||||
}
|
||||
targets.dedup();
|
||||
let mut stage_min = BTreeMap::new();
|
||||
if let Some(o) = job.params.get("stage_minutes").and_then(|v| v.as_object()) {
|
||||
for (k, v) in o {
|
||||
if let Some(n) = v.as_u64() {
|
||||
stage_min.insert(k.clone(), n.max(1));
|
||||
}
|
||||
}
|
||||
}
|
||||
let relay = job.str_param("relay_url");
|
||||
let distro = job.str_param("distro");
|
||||
let user = job.str_param("wsl_user");
|
||||
BuildParams {
|
||||
targets,
|
||||
budget_min: job.timeout_minutes(),
|
||||
stage_min,
|
||||
min_free_gb: job.u64_param("min_free_gb").unwrap_or(jobs::DEFAULT_BUILD_MIN_FREE_GB),
|
||||
tests: job.params.get("tests").and_then(|v| v.as_bool()).unwrap_or(true),
|
||||
relay_url: if relay.is_empty() { RELAY_URL_DEFAULT.to_string() } else { relay.trim_end_matches('/').to_string() },
|
||||
distro: if distro.is_empty() { DEFAULT_DISTRO.to_string() } else { distro },
|
||||
wsl_user: if user.is_empty() { DEFAULT_WSL_USER.to_string() } else { user },
|
||||
nice: job.u64_param("nice").unwrap_or(DEFAULT_NICE).min(19),
|
||||
cargo_jobs: job.u64_param("cargo_jobs").unwrap_or(0),
|
||||
}
|
||||
}
|
||||
pub fn wants(&self, target: &str) -> bool {
|
||||
self.targets.iter().any(|t| t == target)
|
||||
}
|
||||
}
|
||||
|
||||
/// A stage's cap: its own minutes when the job names them, else what is left of the total; never over what is left.
|
||||
pub fn stage_cap(p: &BuildParams, stage: &str, remaining: Duration) -> Duration {
|
||||
match p.stage_min.get(stage) {
|
||||
Some(m) => Duration::from_secs(m * 60).min(remaining),
|
||||
None => remaining,
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the plan from the inputs manifest ------------------------------------------------------------------------------
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Default)]
|
||||
pub struct Unit {
|
||||
/// path inside the zip root ("node", "app/igneum-app")
|
||||
pub dir: String,
|
||||
pub packages: Vec<String>,
|
||||
pub features: Vec<String>,
|
||||
/// the binaries cargo leaves in <target dir>/release (and <target dir>/x86_64-pc-windows-gnu/release with .exe)
|
||||
pub bins: Vec<String>,
|
||||
pub targets: Vec<String>,
|
||||
/// targets where a failure is reported but does not fail the job (the engine on Linux)
|
||||
pub optional_on: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Default)]
|
||||
pub struct TestUnit {
|
||||
pub dir: String,
|
||||
pub packages: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Default)]
|
||||
pub struct Manifest {
|
||||
pub created_at: String,
|
||||
pub node_branch: String,
|
||||
pub node_commit: String,
|
||||
pub node_dirty: bool,
|
||||
pub app_version: String,
|
||||
pub builds: Vec<Unit>,
|
||||
pub tests: Vec<TestUnit>,
|
||||
}
|
||||
|
||||
fn strings(v: Option<&Value>) -> Vec<String> {
|
||||
v.and_then(|a| a.as_array()).map(|a| a.iter().filter_map(|x| x.as_str()).map(|s| s.trim().to_string()).filter(|s| !s.is_empty()).collect()).unwrap_or_default()
|
||||
}
|
||||
|
||||
/// A plain relative directory inside the zip root: no "..", no leading slash, no drive, one or more components.
|
||||
fn plain_dir(s: &str) -> bool {
|
||||
jobs::safe_rel_path(s).is_some()
|
||||
}
|
||||
|
||||
fn plain_name(s: &str) -> bool {
|
||||
!s.is_empty() && s.len() <= 80 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_' || c == '.' || c == '/') && !s.starts_with('.') && !s.contains("..")
|
||||
}
|
||||
|
||||
/// Reads manifest.json out of the zip. Refuses anything that would not be a plain cargo invocation inside the
|
||||
/// extracted tree (a dir with "..", a package name with spaces), because these strings go into a shell script.
|
||||
pub fn parse_manifest(text: &str) -> Result<Manifest, String> {
|
||||
let v: Value = serde_json::from_str(text).map_err(|e| format!("manifest.json is not JSON: {e}"))?;
|
||||
let s = |k: &str| v.get(k).and_then(|x| x.as_str()).unwrap_or("").trim().to_string();
|
||||
let node = v.get("node").cloned().unwrap_or(Value::Null);
|
||||
let ns = |k: &str| node.get(k).and_then(|x| x.as_str()).unwrap_or("").trim().to_string();
|
||||
let mut m = Manifest { created_at: s("created_at"), node_branch: ns("branch"), node_commit: ns("commit"), node_dirty: node.get("dirty").and_then(|x| x.as_bool()).unwrap_or(false), app_version: s("app_version"), ..Default::default() };
|
||||
let builds = v.get("builds").and_then(|b| b.as_array()).ok_or("manifest.json has no \"builds\" list")?;
|
||||
for (i, b) in builds.iter().enumerate() {
|
||||
let dir = b.get("dir").and_then(|x| x.as_str()).unwrap_or("").trim().to_string();
|
||||
if !plain_dir(&dir) {
|
||||
return Err(format!("builds[{i}].dir '{dir}' is not a plain relative path"));
|
||||
}
|
||||
let u = Unit { dir, packages: strings(b.get("packages")), features: strings(b.get("features")), bins: strings(b.get("bins")), targets: { let t = strings(b.get("targets")); if t.is_empty() { jobs::BUILD_TARGETS.iter().map(|s| s.to_string()).collect() } else { t } }, optional_on: strings(b.get("optional_on")) };
|
||||
if u.packages.is_empty() {
|
||||
return Err(format!("builds[{i}] ({}) names no packages", u.dir));
|
||||
}
|
||||
if u.bins.is_empty() {
|
||||
return Err(format!("builds[{i}] ({}) names no bins", u.dir));
|
||||
}
|
||||
for x in u.packages.iter().chain(u.features.iter()).chain(u.bins.iter()) {
|
||||
if !plain_name(x) {
|
||||
return Err(format!("builds[{i}] ({}): '{x}' is not a plain name", u.dir));
|
||||
}
|
||||
}
|
||||
for t in &u.targets {
|
||||
if !jobs::BUILD_TARGETS.contains(&t.as_str()) {
|
||||
return Err(format!("builds[{i}] ({}): target '{t}' is unknown", u.dir));
|
||||
}
|
||||
}
|
||||
m.builds.push(u);
|
||||
}
|
||||
if m.builds.is_empty() {
|
||||
return Err("manifest.json names nothing to build".into());
|
||||
}
|
||||
for (i, t) in v.get("tests").and_then(|b| b.as_array()).map(|a| a.to_vec()).unwrap_or_default().iter().enumerate() {
|
||||
let dir = t.get("dir").and_then(|x| x.as_str()).unwrap_or("").trim().to_string();
|
||||
if !plain_dir(&dir) {
|
||||
return Err(format!("tests[{i}].dir '{dir}' is not a plain relative path"));
|
||||
}
|
||||
let packages = strings(t.get("packages"));
|
||||
if packages.iter().any(|p| !plain_name(p)) {
|
||||
return Err(format!("tests[{i}] ({dir}): a package name is not plain"));
|
||||
}
|
||||
if !packages.is_empty() {
|
||||
m.tests.push(TestUnit { dir, packages });
|
||||
}
|
||||
}
|
||||
Ok(m)
|
||||
}
|
||||
|
||||
// ---- the stage scripts (bash, run as `wsl -d <distro> -u root -- bash <script on /mnt/c>`) ---------------------------
|
||||
|
||||
/// What every stage script starts with: the same PATH the working shard job used (run-20261004-173115: cargo and
|
||||
/// the toolchain live under /root; a login shell from a process without a console need not source ~/.cargo/env),
|
||||
/// the persistent target dir, no interactive apt.
|
||||
pub fn prelude(p: &BuildParams, job_id: &str) -> String {
|
||||
let mut s = String::new();
|
||||
s.push_str("#!/usr/bin/env bash\n");
|
||||
s.push_str("set -uo pipefail\n");
|
||||
s.push_str("export HOME=/root\n");
|
||||
s.push_str("export CARGO_HOME=/root/.cargo RUSTUP_HOME=/root/.rustup\n");
|
||||
s.push_str("CUDA_DIR=$(ls -d /usr/local/cuda-12.* 2>/dev/null | sort -V | tail -1)\n");
|
||||
s.push_str("export PATH=\"/root/.cargo/bin:${CUDA_DIR:+$CUDA_DIR/bin:}/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"\n");
|
||||
s.push_str("export DEBIAN_FRONTEND=noninteractive\n");
|
||||
s.push_str(&format!("B={WSL_BASE}\n"));
|
||||
s.push_str(&format!("SRC=$B/src/{ZIP_ROOT}\n"));
|
||||
s.push_str(&format!("OUT=$B/out/{job_id}\n"));
|
||||
s.push_str("export CARGO_TARGET_DIR=$B/target\n");
|
||||
s.push_str("export CARGO_NET_RETRY=5 CARGO_TERM_COLOR=never CARGO_INCREMENTAL=0\n");
|
||||
s.push_str(&format!("NICE=\"nice -n {}\"\n", p.nice));
|
||||
s.push_str(&format!("JOBS=\"{}\"\n", if p.cargo_jobs > 0 { format!("-j {}", p.cargo_jobs) } else { String::new() }));
|
||||
s.push_str("now() { date -u +%Y-%m-%dT%H:%M:%SZ; }\n");
|
||||
s.push_str("mkdir -p \"$B\" \"$OUT\"\n");
|
||||
s
|
||||
}
|
||||
|
||||
/// The environment of the Windows target build: Ubuntu's mingw-w64 (posix threads, so libstdc++ has std::thread
|
||||
/// for rocksdb), bindgen's clang pointed at the mingw headers (librocksdb-sys), static libgcc and libstdc++ so the
|
||||
/// exe carries no mingw DLLs (proto-cuda/windows-node/cross-build.sh does the same with Homebrew's toolchain).
|
||||
pub fn windows_env() -> String {
|
||||
let mut s = String::new();
|
||||
s.push_str("export CC_x86_64_pc_windows_gnu=x86_64-w64-mingw32-gcc-posix\n");
|
||||
s.push_str("export CXX_x86_64_pc_windows_gnu=x86_64-w64-mingw32-g++-posix\n");
|
||||
s.push_str("export AR_x86_64_pc_windows_gnu=x86_64-w64-mingw32-ar\n");
|
||||
s.push_str("export CARGO_TARGET_X86_64_PC_WINDOWS_GNU_LINKER=x86_64-w64-mingw32-gcc-posix\n");
|
||||
s.push_str("export CARGO_TARGET_X86_64_PC_WINDOWS_GNU_RUSTFLAGS=\"-C link-arg=-static -C link-arg=-static-libgcc\"\n");
|
||||
s.push_str("export IGNEUM_WINDRES=x86_64-w64-mingw32-windres\n");
|
||||
s.push_str("LLVM_LIB=$(ls -d /usr/lib/llvm-*/lib 2>/dev/null | sort -V | tail -1)\n");
|
||||
s.push_str("export LIBCLANG_PATH=\"${LLVM_LIB:-/usr/lib/llvm-18/lib}\"\n");
|
||||
s.push_str("export BINDGEN_EXTRA_CLANG_ARGS_x86_64_pc_windows_gnu=\"--target=x86_64-w64-mingw32 --sysroot=/usr/x86_64-w64-mingw32 -I/usr/x86_64-w64-mingw32/include\"\n");
|
||||
s
|
||||
}
|
||||
|
||||
/// The stage script for `setup`: packages the build needs when dpkg says they are missing, rustup when cargo is
|
||||
/// missing, the Windows target when wanted. Every step idempotent; every outcome a RESULT line.
|
||||
pub fn setup_script(p: &BuildParams, job_id: &str) -> String {
|
||||
let mut s = prelude(p, job_id);
|
||||
s.push_str("echo \"STAGE setup start $(now)\"\n");
|
||||
s.push_str("need=\"\"\n");
|
||||
let mut pkgs: Vec<&str> = APT_COMMON.to_vec();
|
||||
if p.wants("windows") {
|
||||
pkgs.extend_from_slice(APT_WINDOWS);
|
||||
}
|
||||
s.push_str(&format!("for pkg in {}; do dpkg -s \"$pkg\" >/dev/null 2>&1 || need=\"$need $pkg\"; done\n", pkgs.join(" ")));
|
||||
s.push_str("if [ -n \"$need\" ]; then echo \"installing:$need\"; apt-get update -qq && apt-get install -y -qq --no-install-recommends $need || { echo \"RESULT setup apt failed for:$need\"; exit 2; }; else echo \"apt packages present\"; fi\n");
|
||||
s.push_str("if ! command -v cargo >/dev/null 2>&1; then echo \"installing rustup (minimal)\"; curl -fsSL https://sh.rustup.rs | sh -s -- -y --profile minimal --no-modify-path || { echo \"RESULT setup rustup failed\"; exit 2; }; fi\n");
|
||||
if p.wants("windows") {
|
||||
s.push_str("rustup target list --installed 2>/dev/null | grep -qx x86_64-pc-windows-gnu || rustup target add x86_64-pc-windows-gnu || { echo \"RESULT setup rustup target x86_64-pc-windows-gnu failed\"; exit 2; }\n");
|
||||
s.push_str("command -v x86_64-w64-mingw32-gcc-posix >/dev/null 2>&1 || { echo \"RESULT setup mingw gcc (posix) missing after install\"; exit 2; }\n");
|
||||
s.push_str("command -v x86_64-w64-mingw32-windres >/dev/null 2>&1 || { echo \"RESULT setup mingw windres missing after install\"; exit 2; }\n");
|
||||
}
|
||||
s.push_str("command -v protoc >/dev/null 2>&1 || { echo \"RESULT setup protoc missing after install\"; exit 2; }\n");
|
||||
s.push_str("command -v zstd >/dev/null 2>&1 || { echo \"RESULT setup zstd missing after install\"; exit 2; }\n");
|
||||
if p.wants("windows") {
|
||||
s.push_str("echo \"RESULT setup ok $(cargo --version) | $(rustc --version) | mingw $(x86_64-w64-mingw32-gcc-posix --version 2>/dev/null | head -1 || echo none) | $(now)\"\n");
|
||||
} else {
|
||||
s.push_str("echo \"RESULT setup ok $(cargo --version) | $(rustc --version) | $(now)\"\n");
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
/// `extract`: the zip (copied from the job folder on /mnt/c) replaces $B/src; the target dir stays.
|
||||
pub fn extract_script(p: &BuildParams, job_id: &str, zip_wsl: &str) -> String {
|
||||
let mut s = prelude(p, job_id);
|
||||
s.push_str("echo \"STAGE extract start $(now)\"\n");
|
||||
s.push_str(&format!("ZIP='{}'\n", zip_wsl.replace('\'', "'\\''")));
|
||||
s.push_str("[ -f \"$ZIP\" ] || { echo \"RESULT extract zip missing at $ZIP\"; exit 2; }\n");
|
||||
s.push_str("rm -rf \"$B/src\" && mkdir -p \"$B/src\" && cp \"$ZIP\" \"$B/inputs.zip\" || { echo \"RESULT extract cannot copy the zip into $B\"; exit 2; }\n");
|
||||
s.push_str("unzip -q -o \"$B/inputs.zip\" -d \"$B/src\" || { echo \"RESULT extract unzip failed\"; exit 2; }\n");
|
||||
s.push_str("[ -f \"$SRC/manifest.json\" ] || { echo \"RESULT extract no manifest.json under $SRC\"; exit 2; }\n");
|
||||
s.push_str("echo \"RESULT extract ok $(find \"$B/src\" -type f | wc -l) files, $(du -sh \"$B/src\" | cut -f1) at $(now)\"\n");
|
||||
s
|
||||
}
|
||||
|
||||
fn cargo_unit_args(u: &Unit) -> String {
|
||||
let mut a = String::new();
|
||||
for p in &u.packages {
|
||||
a.push_str(&format!(" -p {p}"));
|
||||
}
|
||||
if !u.features.is_empty() {
|
||||
a.push_str(&format!(" --features {}", u.features.join(",")));
|
||||
}
|
||||
a
|
||||
}
|
||||
|
||||
/// `linux` or `windows`: every unit of the manifest that wants the target, one cargo build each, the binaries
|
||||
/// copied to $OUT/<target>/. A unit that is optional on this target reports and goes on; any other failure ends
|
||||
/// the stage with its exit code.
|
||||
pub fn build_script(p: &BuildParams, job_id: &str, m: &Manifest, target: &str) -> String {
|
||||
let mut s = prelude(p, job_id);
|
||||
let windows = target == "windows";
|
||||
if windows {
|
||||
s.push_str(&windows_env());
|
||||
}
|
||||
s.push_str(&format!("echo \"STAGE {target} start $(now)\"\n"));
|
||||
s.push_str(&format!("mkdir -p \"$OUT/{target}\"\n"));
|
||||
s.push_str("rc_all=0\n");
|
||||
for u in m.builds.iter().filter(|u| u.targets.iter().any(|t| t == target)) {
|
||||
let optional = u.optional_on.iter().any(|t| t == target);
|
||||
let rel = if windows { "x86_64-pc-windows-gnu/release" } else { "release" };
|
||||
let tflag = if windows { " --target x86_64-pc-windows-gnu" } else { "" };
|
||||
s.push_str(&format!("echo \"STAGE {target} {dir} start $(now)\"\n", dir = u.dir));
|
||||
s.push_str(&format!("t0=$(date +%s); rc=1\n"));
|
||||
s.push_str(&format!("if cd \"$SRC/{dir}\"; then $NICE cargo build --release $JOBS{args}{tflag} 2>&1; rc=$?; else echo \"no $SRC/{dir}\"; rc=2; fi\n", dir = u.dir, args = cargo_unit_args(u)));
|
||||
s.push_str(&format!("echo \"RESULT {target} {dir} build exit $rc $(( $(date +%s) - t0 )) s at $(now)\"\n", dir = u.dir));
|
||||
s.push_str("if [ \"$rc\" = 0 ]; then\n");
|
||||
for b in &u.bins {
|
||||
let name = if windows { format!("{b}.exe") } else { b.clone() };
|
||||
s.push_str(&format!(" if [ -f \"$CARGO_TARGET_DIR/{rel}/{name}\" ]; then cp -f \"$CARGO_TARGET_DIR/{rel}/{name}\" \"$OUT/{target}/{name}\"; echo \"RESULT {target} {name} $(stat -c %s \"$OUT/{target}/{name}\") bytes sha256 $(sha256sum \"$OUT/{target}/{name}\" | cut -c1-64)\"; else echo \"RESULT {target} {name} missing after the build\"; rc=3; fi\n"));
|
||||
}
|
||||
s.push_str("fi\n");
|
||||
if optional {
|
||||
s.push_str(&format!("[ \"$rc\" = 0 ] || echo \"RESULT {target} {dir} optional on {target}: not fatal\"\n", dir = u.dir));
|
||||
} else {
|
||||
s.push_str("[ \"$rc\" = 0 ] || rc_all=$rc\n");
|
||||
}
|
||||
}
|
||||
s.push_str(&format!("echo \"STAGE {target} done $(now) exit $rc_all\"\n"));
|
||||
s.push_str("exit $rc_all\n");
|
||||
s
|
||||
}
|
||||
|
||||
/// `test`: cargo test --release for the manifest's test units (native). Failures are reported per unit and the
|
||||
/// stage's exit is the first failure; the runner packs and uploads the binaries regardless and marks the job failed.
|
||||
pub fn test_script(p: &BuildParams, job_id: &str, m: &Manifest) -> String {
|
||||
let mut s = prelude(p, job_id);
|
||||
s.push_str("echo \"STAGE test start $(now)\"\n");
|
||||
s.push_str("rc_all=0\n");
|
||||
for t in &m.tests {
|
||||
let pk: String = t.packages.iter().map(|p| format!(" -p {p}")).collect();
|
||||
s.push_str(&format!("t0=$(date +%s); rc=1\n"));
|
||||
s.push_str(&format!("if cd \"$SRC/{dir}\"; then $NICE cargo test --release $JOBS{pk} 2>&1; rc=$?; else echo \"no $SRC/{dir}\"; rc=2; fi\n", dir = t.dir));
|
||||
s.push_str(&format!("echo \"RESULT test {dir} [{names}] exit $rc $(( $(date +%s) - t0 )) s at $(now)\"\n", dir = t.dir, names = t.packages.join(" ")));
|
||||
s.push_str("[ \"$rc\" = 0 ] || [ \"$rc_all\" != 0 ] || rc_all=$rc\n");
|
||||
}
|
||||
if m.tests.is_empty() {
|
||||
s.push_str("echo \"RESULT test nothing named in the manifest\"\n");
|
||||
}
|
||||
s.push_str("echo \"STAGE test done $(now) exit $rc_all\"\n");
|
||||
s.push_str("exit $rc_all\n");
|
||||
s
|
||||
}
|
||||
|
||||
/// `pack`: zstd per binary (linux: <name>.linux.zst, windows: <name>.exe.zst), sha256 of both forms,
|
||||
/// build-outputs.json, the lot copied to the job folder on the Windows side for the upload.
|
||||
pub fn pack_script(p: &BuildParams, job_id: &str, m: &Manifest, out_wsl: &str) -> String {
|
||||
let mut s = prelude(p, job_id);
|
||||
s.push_str("echo \"STAGE pack start $(now)\"\n");
|
||||
s.push_str(&format!("DEST='{}'\n", out_wsl.replace('\'', "'\\''")));
|
||||
s.push_str("PACK=$OUT/pack; rm -rf \"$PACK\"; mkdir -p \"$PACK\" \"$DEST\" || { echo \"RESULT pack cannot make $DEST\"; exit 2; }\n");
|
||||
s.push_str("n=0; first=1\n");
|
||||
s.push_str(&format!("printf '{{\"job\":\"%s\",\"packed_at\":\"%s\",\"node_branch\":\"%s\",\"node_commit\":\"%s\",\"app_version\":\"%s\",\"files\":[' \"{job_id}\" \"$(now)\" '{nb}' '{nc}' '{av}' > \"$PACK/{OUTPUTS_FILE}\"\n", nb = m.node_branch.replace('\'', ""), nc = m.node_commit.replace('\'', ""), av = m.app_version.replace('\'', "")));
|
||||
s.push_str("for target in linux windows; do\n");
|
||||
s.push_str(" [ -d \"$OUT/$target\" ] || continue\n");
|
||||
s.push_str(" for f in \"$OUT/$target\"/*; do\n");
|
||||
s.push_str(" [ -f \"$f\" ] || continue\n");
|
||||
s.push_str(" name=$(basename \"$f\"); case \"$target\" in linux) z=\"$name.linux.zst\";; *) z=\"$name.zst\";; esac\n");
|
||||
s.push_str(" zstd -q -f -T0 -12 \"$f\" -o \"$PACK/$z\" || { echo \"RESULT pack zstd failed on $name\"; exit 2; }\n");
|
||||
s.push_str(" sum=$(sha256sum \"$f\" | cut -c1-64); zsum=$(sha256sum \"$PACK/$z\" | cut -c1-64); bytes=$(stat -c %s \"$f\"); zbytes=$(stat -c %s \"$PACK/$z\")\n");
|
||||
s.push_str(" [ $first = 1 ] || printf ',' >> \"$PACK/build-outputs.json\"; first=0\n");
|
||||
s.push_str(" printf '{\"name\":\"%s\",\"target\":\"%s\",\"bytes\":%s,\"sha256\":\"%s\",\"zst\":\"%s\",\"zst_bytes\":%s,\"zst_sha256\":\"%s\"}' \"$name\" \"$target\" \"$bytes\" \"$sum\" \"$z\" \"$zbytes\" \"$zsum\" >> \"$PACK/build-outputs.json\"\n");
|
||||
s.push_str(" echo \"RESULT output $target $name $bytes bytes sha256 $sum zst $z $zbytes bytes sha256 $zsum\"\n");
|
||||
s.push_str(" n=$((n+1))\n");
|
||||
s.push_str(" done\n");
|
||||
s.push_str("done\n");
|
||||
s.push_str("printf ']}\\n' >> \"$PACK/build-outputs.json\"\n");
|
||||
s.push_str("[ $n -gt 0 ] || { echo \"RESULT pack nothing to pack\"; exit 3; }\n");
|
||||
s.push_str("rm -f \"$DEST\"/*.zst \"$DEST\"/build-outputs.json; cp -f \"$PACK\"/* \"$DEST\"/ || { echo \"RESULT pack cannot copy to $DEST\"; exit 2; }\n");
|
||||
s.push_str("rm -rf \"$OUT/linux\" \"$OUT/windows\" \"$B/inputs.zip\"\n");
|
||||
s.push_str("echo \"RESULT pack ok $n files, $(du -sh \"$PACK\" | cut -f1), target dir $(du -sh \"$CARGO_TARGET_DIR\" 2>/dev/null | cut -f1) at $(now)\"\n");
|
||||
s.push_str("echo \"STAGE pack done $(now) exit 0\"\n");
|
||||
s
|
||||
}
|
||||
|
||||
/// The free-space probe inside the distro: GB available where the build lives (the ext4 vhdx; df reports its
|
||||
/// virtual size, so the Windows drive is checked too by the runner).
|
||||
pub fn free_gb_script() -> &'static str {
|
||||
"mkdir -p /root/igneum-build; df -BG --output=avail /root/igneum-build | tail -1 | tr -dc '0-9'"
|
||||
}
|
||||
|
||||
/// What to run inside the distro after a cap ends wsl.exe: the Linux side outlives it.
|
||||
pub fn kill_script() -> &'static str {
|
||||
"pkill -f 'cargo build' ; pkill -f 'cargo test' ; pkill -x rustc ; pkill -x cc1plus ; pkill -x zstd ; true"
|
||||
}
|
||||
|
||||
// ---- parsers for what comes back ------------------------------------------------------------------------------------
|
||||
|
||||
/// The first whole number in a probe's output ("123" from "123\n", or from " 123G").
|
||||
pub fn parse_free_gb(s: &str) -> Option<u64> {
|
||||
let digits: String = s.chars().skip_while(|c| !c.is_ascii_digit()).take_while(|c| c.is_ascii_digit()).collect();
|
||||
digits.parse().ok()
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Default)]
|
||||
pub struct Output {
|
||||
pub name: String,
|
||||
pub target: String,
|
||||
pub bytes: u64,
|
||||
pub sha256: String,
|
||||
pub zst: String,
|
||||
pub zst_bytes: u64,
|
||||
pub zst_sha256: String,
|
||||
}
|
||||
|
||||
/// build-outputs.json from the pack stage.
|
||||
pub fn parse_outputs(text: &str) -> Result<Vec<Output>, String> {
|
||||
let v: Value = serde_json::from_str(text).map_err(|e| format!("{OUTPUTS_FILE} is not JSON: {e}"))?;
|
||||
let files = v.get("files").and_then(|f| f.as_array()).ok_or(format!("{OUTPUTS_FILE} has no files list"))?;
|
||||
let mut out = Vec::new();
|
||||
for f in files {
|
||||
let s = |k: &str| f.get(k).and_then(|x| x.as_str()).unwrap_or("").to_string();
|
||||
let n = |k: &str| f.get(k).and_then(|x| x.as_u64()).unwrap_or(0);
|
||||
let o = Output { name: s("name"), target: s("target"), bytes: n("bytes"), sha256: s("sha256"), zst: s("zst"), zst_bytes: n("zst_bytes"), zst_sha256: s("zst_sha256") };
|
||||
if o.name.is_empty() || o.zst.is_empty() || o.sha256.len() != 64 || o.zst_sha256.len() != 64 {
|
||||
return Err(format!("{OUTPUTS_FILE}: entry {:?} is incomplete", o.name));
|
||||
}
|
||||
out.push(o);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// A string field of a JSON reply (the relay's `token`, `put_url`, `api_version`, the Blob PUT's `url`).
|
||||
pub fn json_str(text: &str, key: &str) -> Option<String> {
|
||||
let v: Value = serde_json::from_str(text.trim()).ok()?;
|
||||
v.get(key).and_then(|x| match x { Value::String(s) => Some(s.clone()), Value::Number(n) => Some(n.to_string()), _ => None })
|
||||
}
|
||||
|
||||
/// The relay's item id from a `drop` reply.
|
||||
pub fn json_u64(text: &str, key: &str) -> Option<u64> {
|
||||
let v: Value = serde_json::from_str(text.trim()).ok()?;
|
||||
v.get(key).and_then(|x| x.as_u64())
|
||||
}
|
||||
|
||||
/// The title of a relay item that carries one output, which tools/build-job.mjs searches for.
|
||||
pub fn upload_title(job_id: &str, file: &str) -> String {
|
||||
format!("build-job {job_id} {file}")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
const MANIFEST: &str = r#"{"created_at":"2026-10-04T20:00:00Z","node":{"branch":"devnet-v4","commit":"3bfe346f","dirty":false,"source":"vendor/igneum-node-v4"},"repo":{"commit":"0f44edd","branch":"build-job","dirty":true},"app_version":"0.3.4",
|
||||
"builds":[{"dir":"node","packages":["kaspad","igneum-miner"],"features":["kaspad/igneum-pow"],"bins":["igneumd","igneum-miner"],"targets":["linux","windows"]},
|
||||
{"dir":"app/igneum-app","packages":["igneum-app"],"bins":["igneum-app"],"optional_on":["linux"]}],
|
||||
"tests":[{"dir":"app/igneum-app","packages":["igneum-app"]},{"dir":"node","packages":["igneum-miner"]},{"dir":"node","packages":[]}]}"#;
|
||||
|
||||
fn job(params: Value) -> Job {
|
||||
let text = json!({"jobs":[{"id":"build-20261004-200000","kind":"build","expires_at":"2026-10-06T15:00:00Z","target":{"machine_ids":["ae432dc7"],"platform":"windows","requires":["wsl"]},"params":params}]}).to_string();
|
||||
jobs::parse(&text).unwrap().jobs.remove(0)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn params_defaults_and_overrides() {
|
||||
let base = json!({"zip_url":"https://dl.igneum.network/dl/t/build-inputs.zip","sha256":"a".repeat(64),"size":1});
|
||||
let p = BuildParams::from_job(&job(base.clone()));
|
||||
assert_eq!(p.targets, vec!["linux", "windows"]);
|
||||
assert_eq!(p.budget_min, 40);
|
||||
assert_eq!(p.min_free_gb, 20);
|
||||
assert!(p.tests);
|
||||
assert_eq!(p.relay_url, RELAY_URL_DEFAULT);
|
||||
assert_eq!((p.distro.as_str(), p.wsl_user.as_str(), p.nice, p.cargo_jobs), ("Ubuntu-24.04", "root", 19, 0));
|
||||
assert!(p.stage_min.is_empty());
|
||||
let mut o = base.as_object().unwrap().clone();
|
||||
o.insert("targets".into(), json!(["windows", "windows"]));
|
||||
o.insert("budget_minutes".into(), json!(120));
|
||||
o.insert("stage_minutes".into(), json!({"linux": 30, "upload": 5}));
|
||||
o.insert("min_free_gb".into(), json!(30));
|
||||
o.insert("tests".into(), json!(false));
|
||||
o.insert("relay_url".into(), json!("https://relay.example/"));
|
||||
o.insert("nice".into(), json!(5));
|
||||
o.insert("cargo_jobs".into(), json!(8));
|
||||
let p = BuildParams::from_job(&job(Value::Object(o)));
|
||||
assert_eq!(p.targets, vec!["windows"]);
|
||||
assert!(p.wants("windows") && !p.wants("linux"));
|
||||
assert_eq!(p.budget_min, 120);
|
||||
assert_eq!(p.stage_min.get("linux"), Some(&30));
|
||||
assert_eq!(p.min_free_gb, 30);
|
||||
assert!(!p.tests);
|
||||
assert_eq!(p.relay_url, "https://relay.example");
|
||||
assert_eq!((p.nice, p.cargo_jobs), (5, 8));
|
||||
// the cap: a stage's own minutes, never over what is left of the budget
|
||||
assert_eq!(stage_cap(&p, "linux", Duration::from_secs(3600)), Duration::from_secs(1800));
|
||||
assert_eq!(stage_cap(&p, "linux", Duration::from_secs(600)), Duration::from_secs(600));
|
||||
assert_eq!(stage_cap(&p, "windows", Duration::from_secs(600)), Duration::from_secs(600));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn manifest_parses_and_refuses_shell_unsafe_names() {
|
||||
let m = parse_manifest(MANIFEST).unwrap();
|
||||
assert_eq!((m.node_branch.as_str(), m.node_commit.as_str(), m.app_version.as_str(), m.node_dirty), ("devnet-v4", "3bfe346f", "0.3.4", false));
|
||||
assert_eq!(m.builds.len(), 2);
|
||||
assert_eq!(m.builds[0].features, vec!["kaspad/igneum-pow"]);
|
||||
assert_eq!(m.builds[1].targets, vec!["linux", "windows"]); // default when absent
|
||||
assert_eq!(m.builds[1].optional_on, vec!["linux"]);
|
||||
assert_eq!(m.tests.len(), 2); // the empty one is dropped
|
||||
assert!(parse_manifest("nope").unwrap_err().contains("JSON"));
|
||||
assert!(parse_manifest(r#"{"builds":[]}"#).unwrap_err().contains("nothing"));
|
||||
assert!(parse_manifest(&MANIFEST.replace("\"dir\":\"node\",\"packages\":[\"kaspad\"", "\"dir\":\"../node\",\"packages\":[\"kaspad\"")).unwrap_err().contains("plain"));
|
||||
assert!(parse_manifest(&MANIFEST.replace("\"kaspad\"", "\"kaspad; rm -rf /\"")).unwrap_err().contains("plain name"));
|
||||
assert!(parse_manifest(&MANIFEST.replace("\"bins\":[\"igneum-app\"]", "\"bins\":[]")).unwrap_err().contains("bins"));
|
||||
assert!(parse_manifest(&MANIFEST.replace("\"linux\",\"windows\"", "\"linux\",\"amiga\"")).unwrap_err().contains("amiga"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stage_scripts_carry_the_plan() {
|
||||
let p = BuildParams::from_job(&job(json!({"zip_url":"https://x/a.zip","sha256":"b".repeat(64),"cargo_jobs":6,"nice":10})));
|
||||
let m = parse_manifest(MANIFEST).unwrap();
|
||||
let id = "build-20261004-200000";
|
||||
let pre = prelude(&p, id);
|
||||
assert!(pre.contains("B=/root/igneum-build\n") && pre.contains("export CARGO_TARGET_DIR=$B/target"));
|
||||
assert!(pre.contains("OUT=$B/out/build-20261004-200000"));
|
||||
assert!(pre.contains("NICE=\"nice -n 10\"") && pre.contains("JOBS=\"-j 6\""));
|
||||
assert!(!pre.contains('\r'));
|
||||
let setup = setup_script(&p, id);
|
||||
assert!(setup.contains("gcc-mingw-w64-x86-64") && setup.contains("protobuf-compiler") && setup.contains("rustup target add x86_64-pc-windows-gnu"));
|
||||
let only_linux = BuildParams { targets: vec!["linux".into()], ..p.clone() };
|
||||
assert!(!setup_script(&only_linux, id).contains("mingw"));
|
||||
let lin = build_script(&p, id, &m, "linux");
|
||||
assert!(lin.contains("cargo build --release $JOBS -p kaspad -p igneum-miner --features kaspad/igneum-pow 2>&1"));
|
||||
assert!(lin.contains("cd \"$SRC/app/igneum-app\""));
|
||||
assert!(lin.contains("optional on linux: not fatal"));
|
||||
assert!(!lin.contains("--target x86_64-pc-windows-gnu"));
|
||||
assert!(lin.contains("RESULT linux igneumd $(stat"));
|
||||
let win = build_script(&p, id, &m, "windows");
|
||||
assert!(win.contains("--target x86_64-pc-windows-gnu") && win.contains("x86_64-w64-mingw32-gcc-posix") && win.contains("link-arg=-static"));
|
||||
assert!(win.contains("x86_64-pc-windows-gnu/release/igneumd.exe") && win.contains("igneum-app.exe"));
|
||||
assert!(!win.contains("optional on windows"));
|
||||
let t = test_script(&p, id, &m);
|
||||
assert!(t.contains("cargo test --release $JOBS -p igneum-app 2>&1") && t.contains("-p igneum-miner"));
|
||||
let pk = pack_script(&p, id, &m, "/mnt/c/Users/Admin/AppData/Local/igneum/app/jobs/build-20261004-200000/out");
|
||||
assert!(pk.contains("zstd -q -f -T0 -12") && pk.contains("DEST='/mnt/c/Users/Admin/AppData/Local/igneum/app/jobs/build-20261004-200000/out'"));
|
||||
assert!(pk.contains("\"node_commit\":\"%s\"") && pk.contains("'3bfe346f'"));
|
||||
let ex = extract_script(&p, id, "/mnt/c/it's/build-inputs.zip");
|
||||
assert!(ex.contains("ZIP='/mnt/c/it'\\''s/build-inputs.zip'"));
|
||||
assert!(ex.contains("unzip -q -o"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parsers() {
|
||||
assert_eq!(parse_free_gb("123\n"), Some(123));
|
||||
assert_eq!(parse_free_gb(" 57G\r\n"), Some(57));
|
||||
assert_eq!(parse_free_gb("Avail\n0\n"), Some(0));
|
||||
assert_eq!(parse_free_gb("no disk"), None);
|
||||
let text = r#"{"job":"b","packed_at":"2026-10-04T20:30:00Z","files":[{"name":"igneumd","target":"linux","bytes":46883304,"sha256":"a1b2c3d4a1b2c3d4a1b2c3d4a1b2c3d4a1b2c3d4a1b2c3d4a1b2c3d4a1b2c3d4","zst":"igneumd.linux.zst","zst_bytes":15000000,"zst_sha256":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"}]}"#;
|
||||
let o = parse_outputs(text).unwrap();
|
||||
assert_eq!(o.len(), 1);
|
||||
assert_eq!((o[0].name.as_str(), o[0].target.as_str(), o[0].bytes, o[0].zst.as_str(), o[0].zst_bytes), ("igneumd", "linux", 46883304, "igneumd.linux.zst", 15000000));
|
||||
assert!(parse_outputs(&text.replace("igneumd.linux.zst", "")).unwrap_err().contains("incomplete"));
|
||||
assert!(parse_outputs("{}").unwrap_err().contains("files"));
|
||||
assert_eq!(json_str(r#"{"ok":true,"token":"t.x","put_url":"https://vercel.com/api/blob/?pathname=a","api_version":"11"}"#, "put_url"), Some("https://vercel.com/api/blob/?pathname=a".into()));
|
||||
assert_eq!(json_str(r#"{"api_version":11}"#, "api_version"), Some("11".into()));
|
||||
assert_eq!(json_str("garbage", "url"), None);
|
||||
assert_eq!(json_u64(r#"{"ok":true,"id":4711,"kind":"file"}"#, "id"), Some(4711));
|
||||
assert_eq!(upload_title("build-1", "igneumd.exe.zst"), "build-job build-1 igneumd.exe.zst");
|
||||
}
|
||||
}
|
||||
|
|
@ -7,7 +7,9 @@
|
|||
//! https and sha256 into the app data dir), collect (files by glob or a command's output to the log intake),
|
||||
//! restart (miners, node or the app), update-now (the updater checks and installs at once), shard-benchmark
|
||||
//! (miners stopped, GPU idle, the prove package fetched, prove-shard.sh inside WSL under a time cap, every
|
||||
//! RESULT and STAGE line and the results/*.json to the intake, miners back)
|
||||
//! RESULT and STAGE line and the results/*.json to the intake, miners back), build (the node and the app
|
||||
//! engine for Linux and Windows inside WSL2 as root, mining untouched, outputs zstd-compressed to the relay;
|
||||
//! the plan and the stage scripts are src/jobbuild.rs)
|
||||
//! -> every job reports to the log intake as run_id job-<id>-<machine id8>: the first line is a JSON summary
|
||||
//! (SUMMARY {...}), then the captured output; long jobs report every 5 minutes while they run
|
||||
//! -> the dashboard shows the running job (strip) and the history (Settings), and the switch "Allow remote jobs
|
||||
|
|
@ -16,6 +18,7 @@
|
|||
//! IGNEUM_APP_JOBS_FIRST_SECS the first delay.
|
||||
|
||||
use crate::engine::{Cmd, Shared};
|
||||
use crate::jobbuild as jb;
|
||||
use crate::jobs::{self, Eligibility, Job, Ledger};
|
||||
use crate::manifest;
|
||||
use serde_json::{json, Value};
|
||||
|
|
@ -278,7 +281,10 @@ impl Jobs {
|
|||
let machine_id = shared.runtime.machine_id.clone();
|
||||
let shared2 = shared.clone();
|
||||
std::thread::spawn(move || {
|
||||
let r = fetch_jobs(&url, &dir).map(|f| {
|
||||
let r = fetch_jobs(&url, &dir).map(|(f, skipped)| {
|
||||
for id in &skipped {
|
||||
shared2.log(&format!("job {id}: its kind is unknown to this version ({}); skipped, it waits for an app update", crate::engine::VERSION));
|
||||
}
|
||||
let now = crate::platform::unix_now();
|
||||
let platform = manifest::platform_name();
|
||||
let probes: Mutex<std::collections::HashMap<String, bool>> = Mutex::new(std::collections::HashMap::new());
|
||||
|
|
@ -469,6 +475,7 @@ impl Jobs {
|
|||
"fetch" => run_fetch(&job, &sink, &dir, &data_root, &shared2.runtime.app_dir, &ctl),
|
||||
"collect" => run_collect(&shared2, &job, &sink, &data_root, &ctl),
|
||||
"shard-benchmark" => run_shard_benchmark(&shared2, &job, &sink, &data_root, &jobs_url, &ctl, started),
|
||||
"build" => run_build(&shared2, &job, &sink, &dir, &ctl, started),
|
||||
k => Err(format!("kind {k} is not run on this side")),
|
||||
};
|
||||
let finished = crate::platform::unix_now();
|
||||
|
|
@ -591,7 +598,8 @@ fn curl(args: &[&str], limit: Duration) -> Result<(), String> {
|
|||
if code == Some(0) { Ok(()) } else { Err(if t.is_empty() { format!("curl exit {code:?}") } else { t.lines().last().unwrap_or("curl failed").to_string() }) }
|
||||
}
|
||||
|
||||
fn fetch_jobs(url: &str, dir: &Path) -> Result<jobs::JobsFile, String> {
|
||||
/// The jobs file and its signature, verified; jobs of a kind this version does not know come back as skipped ids.
|
||||
fn fetch_jobs(url: &str, dir: &Path) -> Result<(jobs::JobsFile, Vec<String>), String> {
|
||||
let jf = dir.join("jobs.json.new");
|
||||
let sf = dir.join("jobs.json.sig.new");
|
||||
let _ = std::fs::remove_file(&jf);
|
||||
|
|
@ -600,10 +608,10 @@ fn fetch_jobs(url: &str, dir: &Path) -> Result<jobs::JobsFile, String> {
|
|||
curl(&["-fsSL", "--max-time", "20", "-o", &sf.display().to_string(), &format!("{url}.sig")], Duration::from_secs(25)).map_err(|e| format!("jobs signature: {e}"))?;
|
||||
let bytes = std::fs::read(&jf).map_err(|e| e.to_string())?;
|
||||
let sig = std::fs::read_to_string(&sf).map_err(|e| e.to_string())?;
|
||||
let f = jobs::verify_and_parse(&bytes, sig.trim(), manifest::OTA_PUBLIC_KEY_HEX)?;
|
||||
let (f, skipped) = jobs::verify_and_parse_lenient(&bytes, sig.trim(), manifest::OTA_PUBLIC_KEY_HEX)?;
|
||||
let _ = std::fs::rename(&jf, dir.join("jobs.json"));
|
||||
let _ = std::fs::rename(&sf, dir.join("jobs.json.sig"));
|
||||
Ok(f)
|
||||
Ok((f, skipped))
|
||||
}
|
||||
|
||||
/// What the toolchain probe runs inside the distro: the cargo and sp1 paths set by hand (a login shell from a
|
||||
|
|
@ -1181,6 +1189,247 @@ fn run_shard_benchmark(shared: &Arc<Shared>, job: &Job, sink: &Sink, data_root:
|
|||
Ok(done)
|
||||
}
|
||||
|
||||
// ---- kind: build ---------------------------------------------------------------------------------------------------------
|
||||
|
||||
/// One stage inside the distro: the script written to the job folder (LF only), run as
|
||||
/// `wsl -d <distro> -u <user> -- bash /mnt/c/.../stage-<name>.sh` under the stage's cap and the abort flag, every
|
||||
/// line into the sink, STAGE lines with UTC times around it. A cap ends wsl.exe and then the Linux side.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn build_stage(shared: &Arc<Shared>, job: &Job, sink: &Sink, ctl: &Ctl, p: &jb::BuildParams, dir: &Path, stage: &str, script: &str, started: u64, overall: Instant) -> Result<Ran, String> {
|
||||
let path = dir.join(format!("stage-{stage}.sh"));
|
||||
std::fs::write(&path, script.replace("\r\n", "\n")).map_err(|e| format!("cannot write the {stage} script: {e}"))?;
|
||||
let wsl_path = jobs::to_wsl_path(&path.display().to_string()).ok_or("the job folder has no drive letter; WSL cannot see it")?;
|
||||
let left = overall.saturating_duration_since(Instant::now());
|
||||
if left < Duration::from_secs(30) {
|
||||
sink.line(&format!("STAGE {stage} skipped {}: the budget is used up", jobs::format_time(crate::platform::unix_now())));
|
||||
return Ok(Ran { code: None, timed_out: true });
|
||||
}
|
||||
let cap = jb::stage_cap(p, stage, left);
|
||||
sink.line(&format!("STAGE {stage} start {} cap {} min", jobs::format_time(crate::platform::unix_now()), cap.as_secs() / 60));
|
||||
let t0 = Instant::now();
|
||||
let mut cmd = Command::new(crate::platform::tool("wsl"));
|
||||
cmd.args(["-d", &p.distro, "-u", &p.wsl_user, "--", "bash", &wsl_path]);
|
||||
cmd.current_dir(dir);
|
||||
let ran = run_streamed(&mut cmd, sink, ctl, cap, shared, job, started, &format!("build: {stage}"))?;
|
||||
if ran.code.is_none() {
|
||||
let _ = run_capture(Command::new(crate::platform::tool("wsl")).args(["-d", &p.distro, "-u", &p.wsl_user, "--", "bash", "-c", jb::kill_script()]), Duration::from_secs(30));
|
||||
}
|
||||
sink.line(&format!("STAGE {stage} {} {} {} s exit {}", if ran.timed_out { "timeout" } else { "end" }, jobs::format_time(crate::platform::unix_now()), t0.elapsed().as_secs(), ran.code.map(|c| c.to_string()).unwrap_or_else(|| "none".into())));
|
||||
Ok(ran)
|
||||
}
|
||||
|
||||
/// One output to the relay: fn=upload for a client token (the intake key is allowed for upload and drop; round 4,
|
||||
/// X23 keeps it away from run and task posts), the bytes PUT straight to Vercel Blob, then fn=drop with the Blob
|
||||
/// URL so the file is an item the Mac finds by its title (jobbuild::upload_title). Returns (item id, blob url).
|
||||
fn relay_upload(shared: &Arc<Shared>, p: &jb::BuildParams, path: &Path, title: &str, body: &str) -> Result<(u64, String), String> {
|
||||
let key = shared.packaged.log_intake_key.clone();
|
||||
if key.is_empty() {
|
||||
return Err("this build carries no intake key; nothing can be uploaded".into());
|
||||
}
|
||||
let size = std::fs::metadata(path).map(|m| m.len()).map_err(|e| e.to_string())?;
|
||||
if size > jb::MAX_UPLOAD_BYTES {
|
||||
return Err(format!("{size} bytes is over the relay's {} MB cap", jb::MAX_UPLOAD_BYTES / 1024 / 1024));
|
||||
}
|
||||
let name = path.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default();
|
||||
let api = format!("{}/api/relay?fn=", p.relay_url);
|
||||
let key_header = format!("x-igneum-key: {key}");
|
||||
let post = |fn_name: &str, body: &str, limit: u64| -> Result<String, String> {
|
||||
let (code, out) = run_capture(Command::new(crate::platform::tool("curl")).args(["-sS", "--max-time", &limit.to_string(), "-X", "POST", &format!("{api}{fn_name}"), "-H", "Content-Type: application/json", "-H", &key_header, "--data-binary", body]), Duration::from_secs(limit + 10));
|
||||
if code != Some(0) {
|
||||
return Err(format!("relay {fn_name}: curl exit {code:?}: {}", short_out(&out)));
|
||||
}
|
||||
if out.contains("\"ok\":false") {
|
||||
return Err(format!("relay {fn_name}: {}", short_out(&out)));
|
||||
}
|
||||
Ok(out)
|
||||
};
|
||||
let t = post("upload", &json!({ "name": name, "size": size }).to_string(), 60)?;
|
||||
let token = jb::json_str(&t, "token").ok_or("relay upload: no token in the reply")?;
|
||||
let put_url = jb::json_str(&t, "put_url").ok_or("relay upload: no put_url in the reply")?;
|
||||
let api_version = jb::json_str(&t, "api_version").unwrap_or_else(|| "11".into());
|
||||
let (code, out) = run_capture(
|
||||
Command::new(crate::platform::tool("curl")).args(["-sS", "--max-time", "900", "-X", "PUT", &put_url, "-H", &format!("authorization: Bearer {token}"), "-H", &format!("x-api-version: {api_version}"), "-H", "x-add-random-suffix: 1", "-H", "x-content-type: application/octet-stream", "--data-binary", &format!("@{}", path.display())]),
|
||||
Duration::from_secs(910),
|
||||
);
|
||||
if code != Some(0) {
|
||||
return Err(format!("blob PUT: curl exit {code:?}: {}", short_out(&out)));
|
||||
}
|
||||
let url = jb::json_str(&out, "url").ok_or_else(|| format!("blob PUT: no url in the reply: {}", short_out(&out)))?;
|
||||
if !url.contains(".public.blob.vercel-storage.com/") {
|
||||
return Err(format!("blob PUT: unexpected url {}", short_out(&url)));
|
||||
}
|
||||
let from = format!("{}-{}", shared.runtime.host, shared.runtime.id8());
|
||||
let d = post("drop", &json!({ "from": from, "to": "mac", "kind": "file", "title": title, "body": body, "file_name": name, "file_url": url, "size": size }).to_string(), 60)?;
|
||||
let id = jb::json_u64(&d, "id").ok_or("relay drop: no id in the reply")?;
|
||||
Ok((id, url))
|
||||
}
|
||||
|
||||
/// `build`: the plan is in src/jobbuild.rs. Here: the free-space check (the Windows drive and the distro), the
|
||||
/// inputs zip (sha256 checked) and its manifest, then setup, extract, the targets, the tests, pack and upload,
|
||||
/// each as a stage under its cap. The miners are never touched. Required failures mark the job failed after every
|
||||
/// stage that can still run has run, so the binaries of a good target reach the Mac even when another failed.
|
||||
fn run_build(shared: &Arc<Shared>, job: &Job, sink: &Sink, jobs_dir: &Path, ctl: &Ctl, started: u64) -> Result<Done, String> {
|
||||
if !cfg!(windows) {
|
||||
return Err("build runs on a Windows PC with WSL2 (the Linux build happens inside the distro)".into());
|
||||
}
|
||||
let p = jb::BuildParams::from_job(job);
|
||||
let dir = jobs_dir.join(&job.id);
|
||||
let _ = std::fs::create_dir_all(&dir);
|
||||
let overall = Instant::now() + Duration::from_secs(p.budget_min * 60);
|
||||
let now = || jobs::format_time(crate::platform::unix_now());
|
||||
let wsl = crate::platform::tool("wsl");
|
||||
sink.line(&format!("STAGE plan {} targets {} budget {} min, stage caps {:?}, free floor {} GB, tests {}, nice {}, distro {} as {}, relay {}", now(), p.targets.join("+"), p.budget_min, p.stage_min, p.min_free_gb, p.tests, p.nice, p.distro, p.wsl_user, p.relay_url));
|
||||
sink.line("the miners keep mining: this job stops nothing");
|
||||
let mut stages: serde_json::Map<String, Value> = serde_json::Map::new();
|
||||
let mut failures: Vec<String> = Vec::new();
|
||||
|
||||
// ---- check: free space on the Windows drive (the vhdx grows into it) and inside the distro
|
||||
sink.stage("checking free space");
|
||||
let drive = dir.display().to_string().trim_start_matches("\\\\?\\").chars().next().filter(|c| c.is_ascii_alphabetic()).unwrap_or('C');
|
||||
let ps = format!("[math]::Floor(([IO.DriveInfo]::new('{drive}')).AvailableFreeSpace/1GB)");
|
||||
let (wc, wo) = run_capture(Command::new(crate::platform::tool("powershell")).args(["-NoProfile", "-ExecutionPolicy", "Bypass", "-Command", &ps]), Duration::from_secs(40));
|
||||
let win_free = if wc == Some(0) { jb::parse_free_gb(&wo) } else { None };
|
||||
let (lc, lo) = run_capture(Command::new(&wsl).args(["-d", &p.distro, "-u", &p.wsl_user, "--", "bash", "-c", jb::free_gb_script()]), Duration::from_secs(120));
|
||||
let wsl_free = if lc == Some(0) { jb::parse_free_gb(&lo) } else { None };
|
||||
sink.line(&format!("RESULT check {} drive {drive}: {} GB free, {} /root: {} GB free, floor {} GB", now(), win_free.map(|g| g.to_string()).unwrap_or_else(|| format!("unknown (powershell exit {wc:?}: {})", short_out(&wo))), p.distro, wsl_free.map(|g| g.to_string()).unwrap_or_else(|| format!("unknown (wsl exit {lc:?}: {})", short_out(&lo))), p.min_free_gb));
|
||||
let Some(wf) = win_free else { return Err("cannot read the free space of the Windows drive".into()) };
|
||||
let Some(lf) = wsl_free else { return Err(format!("{} does not answer the free-space probe; is WSL installed and the distro present?", p.distro)) };
|
||||
if wf < p.min_free_gb || lf < p.min_free_gb {
|
||||
return Err(format!("not enough free space: drive {drive} {wf} GB, distro {lf} GB, floor {} GB", p.min_free_gb));
|
||||
}
|
||||
if ctl.aborted() {
|
||||
return Err("aborted".into());
|
||||
}
|
||||
|
||||
// ---- fetch: the inputs zip and its manifest
|
||||
sink.stage("fetching the build inputs");
|
||||
let t0 = Instant::now();
|
||||
let zip = dir.join("build-inputs.zip");
|
||||
let size = fetch_file(&job.str_param("zip_url"), &zip, &job.str_param("sha256"), job.u64_param("size"), sink)?;
|
||||
let member = format!("{}/manifest.json", jb::ZIP_ROOT);
|
||||
let _ = std::fs::remove_dir_all(dir.join(jb::ZIP_ROOT));
|
||||
let (tc, to) = run_capture(Command::new(crate::platform::tool("tar")).args(["-xf", &zip.display().to_string(), "-C", &dir.display().to_string(), &member]), Duration::from_secs(120));
|
||||
if tc != Some(0) {
|
||||
return Err(format!("cannot read {member} out of the zip: tar exit {tc:?}: {}", short_out(&to)));
|
||||
}
|
||||
let manifest_text = std::fs::read_to_string(dir.join(jb::ZIP_ROOT).join("manifest.json")).map_err(|e| format!("manifest.json: {e}"))?;
|
||||
let m = jb::parse_manifest(&manifest_text)?;
|
||||
sink.line(&format!("RESULT fetch {} {size} bytes sha256 ok, node {} {}{}, app {}, {} build unit{}, {} test unit{}, {} s", now(), m.node_branch, m.node_commit, if m.node_dirty { " (dirty worktree)" } else { "" }, m.app_version, m.builds.len(), if m.builds.len() == 1 { "" } else { "s" }, m.tests.len(), if m.tests.len() == 1 { "" } else { "s" }, t0.elapsed().as_secs()));
|
||||
stages.insert("fetch".into(), json!({ "exit": 0, "secs": t0.elapsed().as_secs() }));
|
||||
|
||||
// ---- the stages inside the distro
|
||||
let record = |name: &str, ran: &Ran, required: bool, stages: &mut serde_json::Map<String, Value>, failures: &mut Vec<String>| {
|
||||
stages.insert(name.into(), json!({ "exit": ran.code, "timeout": ran.timed_out }));
|
||||
let ok = ran.code == Some(0);
|
||||
if !ok && required {
|
||||
failures.push(format!("{name} {}", if ran.timed_out { "hit its cap".to_string() } else { format!("exit {}", ran.code.map(|c| c.to_string()).unwrap_or_else(|| "none".into())) }));
|
||||
}
|
||||
ok
|
||||
};
|
||||
sink.stage("setup inside the distro");
|
||||
let ran = build_stage(shared, job, sink, ctl, &p, &dir, "setup", &jb::setup_script(&p, &job.id), started, overall)?;
|
||||
if !record("setup", &ran, true, &mut stages, &mut failures) {
|
||||
return Err(format!("setup failed: {}", failures.join("; ")));
|
||||
}
|
||||
sink.stage("extracting the sources");
|
||||
let zip_wsl = jobs::to_wsl_path(&zip.display().to_string()).ok_or("the zip path has no drive letter")?;
|
||||
let ran = build_stage(shared, job, sink, ctl, &p, &dir, "extract", &jb::extract_script(&p, &job.id, &zip_wsl), started, overall)?;
|
||||
if !record("extract", &ran, true, &mut stages, &mut failures) {
|
||||
return Err(format!("extract failed: {}", failures.join("; ")));
|
||||
}
|
||||
let mut built_any = false;
|
||||
for target in ["linux", "windows"] {
|
||||
if !p.wants(target) || ctl.aborted() {
|
||||
continue;
|
||||
}
|
||||
sink.stage(&format!("building for {target}"));
|
||||
let ran = build_stage(shared, job, sink, ctl, &p, &dir, target, &jb::build_script(&p, &job.id, &m, target), started, overall)?;
|
||||
if record(target, &ran, true, &mut stages, &mut failures) {
|
||||
built_any = true;
|
||||
}
|
||||
}
|
||||
if p.tests && !m.tests.is_empty() && !ctl.aborted() {
|
||||
sink.stage("running the tests");
|
||||
let ran = build_stage(shared, job, sink, ctl, &p, &dir, "test", &jb::test_script(&p, &job.id, &m), started, overall)?;
|
||||
record("test", &ran, true, &mut stages, &mut failures);
|
||||
}
|
||||
if ctl.aborted() {
|
||||
return Err("aborted".into());
|
||||
}
|
||||
|
||||
// ---- pack and upload what was built
|
||||
let out_dir = dir.join("out");
|
||||
let _ = std::fs::remove_dir_all(&out_dir);
|
||||
let _ = std::fs::create_dir_all(&out_dir);
|
||||
let mut outputs: Vec<Value> = Vec::new();
|
||||
let mut uploaded = 0usize;
|
||||
let mut uploaded_bytes = 0u64;
|
||||
if built_any {
|
||||
sink.stage("packing the binaries");
|
||||
let out_wsl = jobs::to_wsl_path(&out_dir.display().to_string()).ok_or("the out folder has no drive letter")?;
|
||||
let ran = build_stage(shared, job, sink, ctl, &p, &dir, "pack", &jb::pack_script(&p, &job.id, &m, &out_wsl), started, overall)?;
|
||||
if record("pack", &ran, true, &mut stages, &mut failures) {
|
||||
sink.stage("uploading to the relay");
|
||||
let t0 = Instant::now();
|
||||
let listed = std::fs::read_to_string(out_dir.join(jb::OUTPUTS_FILE)).map_err(|e| format!("{}: {e}", jb::OUTPUTS_FILE))?;
|
||||
let files = jb::parse_outputs(&listed)?;
|
||||
let cap = jb::stage_cap(&p, "upload", overall.saturating_duration_since(Instant::now()).max(Duration::from_secs(120)));
|
||||
let deadline = Instant::now() + cap;
|
||||
let mut all_ok = true;
|
||||
for o in &files {
|
||||
if ctl.aborted() {
|
||||
return Err("aborted".into());
|
||||
}
|
||||
let path = out_dir.join(&o.zst);
|
||||
let mut entry = json!({ "name": o.name, "target": o.target, "bytes": o.bytes, "sha256": o.sha256, "zst": o.zst, "zst_bytes": o.zst_bytes, "zst_sha256": o.zst_sha256 });
|
||||
if Instant::now() >= deadline {
|
||||
sink.line(&format!("RESULT upload {} {} skipped: the upload cap is used up", now(), o.zst));
|
||||
all_ok = false;
|
||||
} else {
|
||||
let body = format!("job {}\nnode {} {}\napp {}\n{} {} {} bytes sha256 {}\n{} {} bytes sha256 {}\n", job.id, m.node_branch, m.node_commit, m.app_version, o.target, o.name, o.bytes, o.sha256, o.zst, o.zst_bytes, o.zst_sha256);
|
||||
match relay_upload(shared, &p, &path, &jb::upload_title(&job.id, &o.zst), &body) {
|
||||
Ok((id, url)) => {
|
||||
uploaded += 1;
|
||||
uploaded_bytes += o.zst_bytes;
|
||||
sink.line(&format!("RESULT upload {} {} relay item {id} {} bytes sha256 {} (unpacked {} {} bytes sha256 {})", now(), o.zst, o.zst_bytes, o.zst_sha256, o.name, o.bytes, o.sha256));
|
||||
entry["relay_id"] = json!(id);
|
||||
entry["url"] = json!(url);
|
||||
}
|
||||
Err(e) => {
|
||||
sink.line(&format!("RESULT upload {} {} FAILED: {e}", now(), o.zst));
|
||||
all_ok = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
outputs.push(entry);
|
||||
}
|
||||
let list_path = out_dir.join(jb::OUTPUTS_FILE);
|
||||
match relay_upload(shared, &p, &list_path, &jb::upload_title(&job.id, jb::OUTPUTS_FILE), &format!("job {}\nnode {} {}\napp {}\n{} files\n", job.id, m.node_branch, m.node_commit, m.app_version, files.len())) {
|
||||
Ok((id, _)) => sink.line(&format!("RESULT upload {} {} relay item {id}", now(), jb::OUTPUTS_FILE)),
|
||||
Err(e) => {
|
||||
sink.line(&format!("RESULT upload {} {} FAILED: {e}", now(), jb::OUTPUTS_FILE));
|
||||
all_ok = false;
|
||||
}
|
||||
}
|
||||
stages.insert("upload".into(), json!({ "exit": if all_ok { 0 } else { 1 }, "secs": t0.elapsed().as_secs(), "files": uploaded }));
|
||||
if !all_ok {
|
||||
failures.push("upload incomplete".into());
|
||||
}
|
||||
}
|
||||
} else {
|
||||
sink.line(&format!("RESULT pack {} skipped: no target built", now()));
|
||||
}
|
||||
let _ = std::fs::remove_file(&zip);
|
||||
|
||||
let mins = (crate::platform::unix_now().saturating_sub(started)) / 60;
|
||||
let summary = format!("node {} {} app {}: {}{}; {uploaded} file{} uploaded ({} MB); {mins} min of {}", m.node_branch, m.node_commit, m.app_version, if failures.is_empty() { "every stage ok".to_string() } else { format!("FAILED: {}", failures.join(", ")) }, if built_any { "" } else { "; nothing built" }, if uploaded == 1 { "" } else { "s" }, uploaded_bytes / 1024 / 1024, p.budget_min);
|
||||
sink.line(&format!("RESULT build {} {} {summary}", now(), if failures.is_empty() { "done" } else { "failed" }));
|
||||
sink.line("the miners were never stopped by this job");
|
||||
let extra = json!({ "node_branch": m.node_branch, "node_commit": m.node_commit, "node_dirty": m.node_dirty, "app_version": m.app_version, "targets": p.targets, "stages": stages, "outputs": outputs, "uploaded_files": uploaded, "relay_url": p.relay_url, "failures": failures });
|
||||
let status = if failures.is_empty() { "done" } else if failures.iter().any(|f| f.contains("hit its cap")) { "timeout" } else { "failed" };
|
||||
Ok(Done { status: status.into(), exit: if failures.is_empty() { 0 } else { 1 }, summary, extra })
|
||||
}
|
||||
|
||||
// ---- kind: restart app -----------------------------------------------------------------------------------------------
|
||||
|
||||
/// A detached helper that starts the app again a few seconds after this engine has gone.
|
||||
|
|
|
|||
|
|
@ -31,8 +31,16 @@
|
|||
//! update-now no params: the over-the-air check runs and a newer version installs at once
|
||||
//! shard-benchmark zip_url, sha256, size, fixtures [shard fixture, block fixtures...], cap_minutes (90), distro
|
||||
//! (Ubuntu-24.04), wsl_user
|
||||
//! build zip_url, sha256, size (the build-inputs zip from packaging/windows/push-build-inputs.sh), targets
|
||||
//! ["linux", "windows"], budget_minutes (40 in total), stage_minutes {setup, fetch, linux, windows,
|
||||
//! test, pack, upload}, min_free_gb (20), tests (true), relay_url (https, where the outputs go),
|
||||
//! distro, wsl_user, nice (19). Mining is never stopped; the build is CPU work inside WSL (src/jobbuild.rs).
|
||||
//! A job never writes outside the app data directory except through an explicit `run` script, which is the
|
||||
//! operator's responsibility.
|
||||
//!
|
||||
//! Unknown kinds: the signer refuses them (`parse`), so a typo never ships; the app skips them (`parse_lenient`) so
|
||||
//! a jobs file that carries a kind this version does not know still runs the kinds it does (0.3.3 and earlier reject
|
||||
//! the whole file, which is why a new kind goes to the PCs in an app update before its first job is published).
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
|
|
@ -42,7 +50,7 @@ use std::collections::BTreeMap;
|
|||
use std::path::{Path, PathBuf};
|
||||
|
||||
pub const JOBS_FILE: &str = "igneum-jobs.json";
|
||||
pub const KINDS: &[&str] = &["run", "fetch", "collect", "restart", "update-now", "shard-benchmark"];
|
||||
pub const KINDS: &[&str] = &["run", "fetch", "collect", "restart", "update-now", "shard-benchmark", "build"];
|
||||
/// Requirements the engine knows how to probe (src/jobrun.rs). An unknown requirement is never satisfied.
|
||||
pub const KNOWN_REQUIRES: &[&str] = &["wsl", "wsl-prover", "nvidia"];
|
||||
/// Named folders a `fetch` may write into, all under the app data root.
|
||||
|
|
@ -50,6 +58,11 @@ pub const FETCH_DIRS: &[&str] = &["jobs", "prove", "packs", "updates"];
|
|||
pub const DEFAULT_RUN_TIMEOUT_MIN: u64 = 60;
|
||||
pub const MAX_RUN_TIMEOUT_MIN: u64 = 600;
|
||||
pub const DEFAULT_SHARD_CAP_MIN: u64 = 90;
|
||||
/// `build`: the whole job (fetch, setup, both targets, tests, pack, upload) must fit this unless budget_minutes says more.
|
||||
pub const DEFAULT_BUILD_BUDGET_MIN: u64 = 40;
|
||||
pub const DEFAULT_BUILD_MIN_FREE_GB: u64 = 20;
|
||||
pub const BUILD_TARGETS: &[&str] = &["linux", "windows"];
|
||||
pub const BUILD_STAGES: &[&str] = &["fetch", "setup", "extract", "linux", "windows", "test", "pack", "upload"];
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Default)]
|
||||
pub struct Target {
|
||||
|
|
@ -101,10 +114,14 @@ impl Job {
|
|||
_ => vec![],
|
||||
}
|
||||
}
|
||||
/// `run`: the script's timeout; `shard-benchmark`: the cap. Clamped to MAX_RUN_TIMEOUT_MIN.
|
||||
/// `run`: the script's timeout; `shard-benchmark`: the cap; `build`: the total budget. Clamped to MAX_RUN_TIMEOUT_MIN.
|
||||
pub fn timeout_minutes(&self) -> u64 {
|
||||
let d = if self.kind == "shard-benchmark" { DEFAULT_SHARD_CAP_MIN } else { DEFAULT_RUN_TIMEOUT_MIN };
|
||||
let v = self.u64_param("timeout_minutes").or_else(|| self.u64_param("cap_minutes")).unwrap_or(d);
|
||||
let d = match self.kind.as_str() {
|
||||
"shard-benchmark" => DEFAULT_SHARD_CAP_MIN,
|
||||
"build" => DEFAULT_BUILD_BUDGET_MIN,
|
||||
_ => DEFAULT_RUN_TIMEOUT_MIN,
|
||||
};
|
||||
let v = self.u64_param("timeout_minutes").or_else(|| self.u64_param("cap_minutes")).or_else(|| self.u64_param("budget_minutes")).unwrap_or(d);
|
||||
v.clamp(1, MAX_RUN_TIMEOUT_MIN)
|
||||
}
|
||||
/// The run id under which the machine reports this job to the log intake.
|
||||
|
|
@ -233,10 +250,21 @@ fn parse_target(v: Option<&Value>) -> Result<Target, String> {
|
|||
/// Parses the jobs file (after the signature was checked). Every job is validated; one bad job rejects the file,
|
||||
/// so a typo on the Mac is caught by the signer before anything is published.
|
||||
pub fn parse(text: &str) -> Result<JobsFile, String> {
|
||||
parse_inner(text, false).map(|(f, _)| f)
|
||||
}
|
||||
|
||||
/// The runner's parse: a job whose kind this version does not know is skipped (its id is returned) instead of
|
||||
/// rejecting the file. Everything else is as strict as `parse`.
|
||||
pub fn parse_lenient(text: &str) -> Result<(JobsFile, Vec<String>), String> {
|
||||
parse_inner(text, true)
|
||||
}
|
||||
|
||||
fn parse_inner(text: &str, skip_unknown_kinds: bool) -> Result<(JobsFile, Vec<String>), String> {
|
||||
let v: Value = serde_json::from_str(text).map_err(|e| format!("jobs file is not JSON: {e}"))?;
|
||||
let s = |v: &Value, k: &str| v.get(k).and_then(|x| x.as_str()).unwrap_or("").trim().to_string();
|
||||
let list = v.get("jobs").and_then(|j| j.as_array()).ok_or("jobs file has no \"jobs\" list")?;
|
||||
let mut out = JobsFile { published_at: s(&v, "published_at"), jobs: Vec::new() };
|
||||
let mut skipped = Vec::new();
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
for (i, j) in list.iter().enumerate() {
|
||||
let id = s(j, "id");
|
||||
|
|
@ -248,6 +276,10 @@ pub fn parse(text: &str) -> Result<JobsFile, String> {
|
|||
}
|
||||
let kind = s(j, "kind");
|
||||
if !KINDS.contains(&kind.as_str()) {
|
||||
if skip_unknown_kinds {
|
||||
skipped.push(id);
|
||||
continue;
|
||||
}
|
||||
return Err(format!("job {id}: kind '{kind}' is unknown (known: {})", KINDS.join(", ")));
|
||||
}
|
||||
let expires_at = s(j, "expires_at");
|
||||
|
|
@ -266,7 +298,7 @@ pub fn parse(text: &str) -> Result<JobsFile, String> {
|
|||
validate_params(&job).map_err(|e| format!("job {id}: {e}"))?;
|
||||
out.jobs.push(job);
|
||||
}
|
||||
Ok(out)
|
||||
Ok((out, skipped))
|
||||
}
|
||||
|
||||
fn https_ok(url: &str) -> bool {
|
||||
|
|
@ -339,18 +371,69 @@ pub fn validate_params(job: &Job) -> Result<(), String> {
|
|||
}
|
||||
}
|
||||
}
|
||||
"build" => {
|
||||
if !https_ok(&job.str_param("zip_url")) {
|
||||
return Err("build: params.zip_url is not https (the build-inputs zip)".into());
|
||||
}
|
||||
if !sha_ok(&job.str_param("sha256")) {
|
||||
return Err("build: params.sha256 of the build-inputs zip is not 64 hex characters".into());
|
||||
}
|
||||
for t in job.list_param("targets") {
|
||||
if !BUILD_TARGETS.contains(&t.as_str()) {
|
||||
return Err(format!("build: target '{t}' is not one of {}", BUILD_TARGETS.join(", ")));
|
||||
}
|
||||
}
|
||||
if let Some(b) = job.params.get("budget_minutes") {
|
||||
match b.as_u64() {
|
||||
Some(n) if (1..=MAX_RUN_TIMEOUT_MIN).contains(&n) => {}
|
||||
_ => return Err(format!("build: budget_minutes must be 1 to {MAX_RUN_TIMEOUT_MIN}")),
|
||||
}
|
||||
}
|
||||
if let Some(m) = job.params.get("stage_minutes") {
|
||||
let o = m.as_object().ok_or("build: stage_minutes is not an object of stage: minutes")?;
|
||||
for (k, v) in o {
|
||||
if !BUILD_STAGES.contains(&k.as_str()) {
|
||||
return Err(format!("build: stage_minutes has unknown stage '{k}' (stages: {})", BUILD_STAGES.join(", ")));
|
||||
}
|
||||
if !matches!(v.as_u64(), Some(n) if n >= 1) {
|
||||
return Err(format!("build: stage_minutes.{k} must be a whole number of minutes, at least 1"));
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(g) = job.params.get("min_free_gb") {
|
||||
if g.as_u64().is_none() {
|
||||
return Err("build: min_free_gb must be a whole number of GB".into());
|
||||
}
|
||||
}
|
||||
let relay = job.str_param("relay_url");
|
||||
if !relay.is_empty() && !https_ok(&relay) {
|
||||
return Err("build: relay_url is not https".into());
|
||||
}
|
||||
if let Some(n) = job.params.get("nice") {
|
||||
if !matches!(n.as_u64(), Some(v) if v <= 19) {
|
||||
return Err("build: nice must be 0 to 19".into());
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => return Err(format!("kind '{}' is unknown", job.kind)),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Verifies the detached signature over the exact bytes, then parses.
|
||||
/// Verifies the detached signature over the exact bytes, then parses (strict: the signer's check).
|
||||
pub fn verify_and_parse(bytes: &[u8], sig_hex: &str, pub_hex: &str) -> Result<JobsFile, String> {
|
||||
manifest::verify_signature(bytes, sig_hex, pub_hex).map_err(|_| "jobs file signature does not verify".to_string())?;
|
||||
let text = std::str::from_utf8(bytes).map_err(|_| "jobs file is not UTF-8")?;
|
||||
parse(text)
|
||||
}
|
||||
|
||||
/// The runner's variant: the same signature check, unknown kinds skipped (their ids come back).
|
||||
pub fn verify_and_parse_lenient(bytes: &[u8], sig_hex: &str, pub_hex: &str) -> Result<(JobsFile, Vec<String>), String> {
|
||||
manifest::verify_signature(bytes, sig_hex, pub_hex).map_err(|_| "jobs file signature does not verify".to_string())?;
|
||||
let text = std::str::from_utf8(bytes).map_err(|_| "jobs file is not UTF-8")?;
|
||||
parse_lenient(text)
|
||||
}
|
||||
|
||||
// ---- targeting ----------------------------------------------------------------------------------------------------
|
||||
|
||||
pub fn targets_machine(t: &Target, machine_id: &str) -> bool {
|
||||
|
|
@ -685,6 +768,58 @@ mod tests {
|
|||
assert!(j("run", r#"{"script":"ls","shell":"zsh"}"#).unwrap_err().contains("shell"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_params_are_checked() {
|
||||
let j = |params: &str| parse(&format!(r#"{{"jobs":[{{"id":"b","kind":"build","expires_at":"2026-10-06T15:00:00Z","target":{{"machine_ids":["ae432dc7"],"platform":"windows","requires":["wsl"]}},"params":{params}}}]}}"#));
|
||||
let sha = "c".repeat(64);
|
||||
let ok = format!(r#"{{"zip_url":"https://dl.igneum.network/dl/t/build-inputs.zip","sha256":"{sha}","size":12345}}"#);
|
||||
let f = j(&ok).unwrap();
|
||||
let b = &f.jobs[0];
|
||||
assert_eq!(b.timeout_minutes(), DEFAULT_BUILD_BUDGET_MIN);
|
||||
assert!(b.list_param("targets").is_empty());
|
||||
assert!(j(r#"{}"#).unwrap_err().contains("zip_url"));
|
||||
assert!(j(r#"{"zip_url":"http://x/a.zip","sha256":"aa"}"#).unwrap_err().contains("https"));
|
||||
assert!(j(r#"{"zip_url":"https://x/a.zip","sha256":"aa"}"#).unwrap_err().contains("sha256"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"targets":["linux","amiga"]"#)).unwrap_err().contains("amiga"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"budget_minutes":0"#)).unwrap_err().contains("budget_minutes"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"budget_minutes":601"#)).unwrap_err().contains("budget_minutes"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"stage_minutes":{"lunch":5}"#)).unwrap_err().contains("lunch"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"stage_minutes":{"linux":"ten"}"#)).unwrap_err().contains("stage_minutes.linux"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"stage_minutes":[5]"#)).unwrap_err().contains("object"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"min_free_gb":"lots""#)).unwrap_err().contains("min_free_gb"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"relay_url":"ftp://relay""#)).unwrap_err().contains("relay_url"));
|
||||
assert!(j(&ok.replace("12345", r#"1,"nice":25"#)).unwrap_err().contains("nice"));
|
||||
let full = j(&ok.replace("12345", r#"1,"targets":["windows"],"budget_minutes":120,"stage_minutes":{"linux":30,"windows":40},"min_free_gb":25,"tests":false,"relay_url":"https://relay.igneum.network","nice":10"#)).unwrap();
|
||||
let b = &full.jobs[0];
|
||||
assert_eq!(b.timeout_minutes(), 120);
|
||||
assert_eq!(b.list_param("targets"), vec!["windows"]);
|
||||
assert_eq!(b.u64_param("min_free_gb"), Some(25));
|
||||
assert!(!b.params.get("tests").and_then(|v| v.as_bool()).unwrap_or(true));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_kinds_are_refused_by_the_signer_and_skipped_by_the_runner() {
|
||||
let (sk, pk) = key();
|
||||
let text = SAMPLE.replace("],\"published_at\"", r#",{"id":"future-1","kind":"teleport","expires_at":"2026-10-06T15:00:00Z","target":{"machine_ids":"all"}}],"published_at""#);
|
||||
assert!(text.contains("teleport"), "the sample must carry the unknown kind");
|
||||
assert!(parse(&text).unwrap_err().contains("teleport"));
|
||||
let (f, skipped) = parse_lenient(&text).unwrap();
|
||||
assert_eq!(f.jobs.len(), 2);
|
||||
assert_eq!(skipped, vec!["future-1".to_string()]);
|
||||
// the signature still has to verify, and a bad job of a known kind still rejects the file
|
||||
let sig = manifest::hex_encode(&sk.sign(text.as_bytes()).to_bytes());
|
||||
let (f2, s2) = verify_and_parse_lenient(text.as_bytes(), &sig, &pk).unwrap();
|
||||
assert_eq!((f2.jobs.len(), s2.len()), (2, 1));
|
||||
assert!(verify_and_parse_lenient(text.replace("future-1", "future-2").as_bytes(), &sig, &pk).is_err());
|
||||
let bad = text.replace("\"kind\":\"collect\"", "\"kind\":\"restart\"");
|
||||
assert!(parse_lenient(&bad).unwrap_err().contains("restart"));
|
||||
// a duplicate id is a duplicate even when one of them is unknown
|
||||
let dup = text.replace("future-1", "collect-1");
|
||||
assert!(parse_lenient(&dup).unwrap_err().contains("twice"));
|
||||
// the strict parse is unchanged for a clean file
|
||||
assert_eq!(parse_lenient(SAMPLE).unwrap().1.len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn signature_verifies_and_tampering_fails() {
|
||||
let (sk, pk) = key();
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ mod ota;
|
|||
mod update;
|
||||
mod jobs;
|
||||
mod jobrun;
|
||||
mod jobbuild;
|
||||
mod prover;
|
||||
|
||||
use std::io::{BufRead, Write};
|
||||
|
|
|
|||
|
|
@ -482,7 +482,7 @@
|
|||
}
|
||||
// ---------- remote jobs (src/jobrun.rs): one strip while a job runs and after it, the settings block ----------
|
||||
function jobKey(j) { if (!j) return ''; return j.active ? ('run:' + j.id) : (j.last && j.last.id ? ('done:' + j.last.id + ':' + j.last.status) : ''); }
|
||||
function jobTitle(j) { return j.title && j.title !== j.kind ? j.title : (j.kind === 'shard-benchmark' ? 'shard benchmark' : j.kind); }
|
||||
function jobTitle(j) { return j.title && j.title !== j.kind ? j.title : (j.kind === 'shard-benchmark' ? 'shard benchmark' : j.kind === 'build' ? 'build (node and app, mining continues)' : j.kind); }
|
||||
function jobLine(j, now) {
|
||||
if (j.active) {
|
||||
var m = Math.max(0, Math.floor((now - j.started_at) / 60));
|
||||
|
|
|
|||
149
docs/plans/build-job.md
Normal file
149
docs/plans/build-job.md
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
# The `build` job: a PC builds the node and the app, nobody at the keyboard
|
||||
|
||||
4 October 2026, evening. Josh's ask ("efficiency"): every Windows node and app build went through a GitHub runner at
|
||||
15 to 25 minutes a round, and every Linux binary was cross-compiled on this Mac under the build lock. The two
|
||||
RTX 5090 PCs (PC 1 `ae432dc7`, PC 2 `1ccfe586`) run Igneum Miner 0.3.3 with the signed job channel and each has a
|
||||
WSL2 Ubuntu 24.04 owned by the app (root, cargo and the SP1 toolchain under /root from the shard jobs). So the build
|
||||
goes to them: a `build` job, mining untouched.
|
||||
|
||||
## What is built (branch `build-job`)
|
||||
|
||||
| Piece | Where | State |
|
||||
|---|---|---|
|
||||
| Job kind `build`: params, validation, the 40-minute default budget, `parse_lenient` (an unknown kind is skipped by the app instead of rejecting the file; the signer stays strict) | `app/igneum-app/src/jobs.rs` | unit tests on the Mac |
|
||||
| The plan: parameters, the inputs manifest (what to build, what to test; shell-unsafe names refused), the bash stage scripts (setup, extract, linux, windows, test, pack), per-stage caps, parsers for free space, the outputs file and the relay replies | `app/igneum-app/src/jobbuild.rs` (new) | unit tests on the Mac |
|
||||
| The runner: free-space check on the Windows drive and inside the distro, fetch with sha256, the manifest out of the zip, one `wsl.exe` call per stage under its cap (kill of the Linux side on a cap), zstd outputs uploaded to the relay (fn=upload, Blob PUT, fn=drop), RESULT and STAGE lines with UTC times, closing SUMMARY with `outputs[]` | `app/igneum-app/src/jobrun.rs` (`run_build`, `build_stage`, `relay_upload`) | compiles for macOS and `x86_64-pc-windows-gnu`; not run on a PC |
|
||||
| Packer: the fork worktree, `app/igneum-app`, `brand/icons`, `proto-cuda` (no redist) into `build-inputs.zip` with `.sha256` and `.json` (branch, commit, dirty, date, builds, tests) on the downloads host | `packaging/windows/push-build-inputs.sh` (new) | run against a scratch folder: 7.9 MB, 1937 files, no target dirs |
|
||||
| Publisher: `publish-jobs.sh add --kind build` (platform windows, requires `wsl`, `--budget-minutes`, `--stage-minutes`, `--targets`, `--no-tests`, `--min-free-gb`, `--relay-url`, `--nice`, `--cargo-jobs`) | `packaging/ota/publish-jobs.sh` | see "Tested" |
|
||||
| Mac tool: pack + publish + watch + fetch; sha256 of the zst and of the unpacked file against the PC's RESULT lines; PE header check of every exe (MZ, PE, x86-64, PE32+, .text, over 1 MB); `verify-exe.py --version` on igneum-app.exe; placement where `push-inputs.sh`, `make-payload.sh` and the cloud-devnet scripts look | `tools/build-job.mjs` (new) | fetch path run against the live relay: a real exe round-tripped, a wrong sha256 refused |
|
||||
| `relay.mjs drop <file> --body` carries the body (the fetch fallback reads the sha256 from it) | `tools/relay.mjs` | run live |
|
||||
| Dashboard label, README row | `app/igneum-app/ui/app.js`, `packaging/ota/README.md` | |
|
||||
|
||||
### What the job does on the PC
|
||||
|
||||
| Stage | What | Cap |
|
||||
|---|---|---|
|
||||
| check | free GB on the drive that holds the app data (PowerShell `DriveInfo`) and under `/root/igneum-build` inside the distro (`df`); both must be at or over `min_free_gb` (20) | 2 min |
|
||||
| fetch | `build-inputs.zip` by https, sha256 and size as signed in the job; `manifest.json` read out of it with `tar` | curl, 60 min |
|
||||
| setup | `apt-get install` of what `dpkg -s` says is missing (build-essential, clang and libclang for bindgen, protobuf-compiler, zstd, unzip, `gcc-mingw-w64-x86-64`, `g++-mingw-w64-x86-64`, `binutils-mingw-w64-x86-64`, `mingw-w64-x86-64-dev`), rustup when cargo is missing, `rustup target add x86_64-pc-windows-gnu` | stage cap |
|
||||
| extract | `/root/igneum-build/src` replaced by the zip; `/root/igneum-build/target` (CARGO_TARGET_DIR) persists, so the ~500 dependency crates compile once | stage cap |
|
||||
| linux | per manifest unit: `nice -n 19 cargo build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow`, then the app crate (optional on Linux: a failure is a RESULT line, not a job failure) | stage cap |
|
||||
| windows | the same with `--target x86_64-pc-windows-gnu`, `CC/CXX = x86_64-w64-mingw32-gcc-posix/g++-posix`, `LIBCLANG_PATH` and `BINDGEN_EXTRA_CLANG_ARGS` for librocksdb-sys, `-C link-arg=-static -C link-arg=-static-libgcc`, `IGNEUM_WINDRES` for the coin icon | stage cap |
|
||||
| test | `cargo test --release` per manifest test unit (default `igneum-app`, `igneum-miner`); a failure marks the job failed but the binaries still ship | stage cap |
|
||||
| pack | `zstd -T0 -12` per binary (`igneumd.linux.zst`, `igneumd.exe.zst`, ...), sha256 of both forms, `build-outputs.json`, copied to the job folder on the Windows side | stage cap |
|
||||
| upload | each `.zst` under 50 MB to the relay: `fn=upload` with the intake key (allowed for upload and drop; round 4 X23 keeps the key away from run and task posts), PUT to Vercel Blob, `fn=drop` to `mac` titled `build-job <id> <file>` with the sha256 lines in the body | stage cap |
|
||||
|
||||
Every stage prints `STAGE <name> start <utc> cap N min` and `STAGE <name> end|timeout <utc> N s exit M`; every
|
||||
binary a `RESULT <target> <name> <bytes> bytes sha256 <hex>` line and later `RESULT output ...` with the zst sha256
|
||||
and `RESULT upload ... relay item <id>`. The console's Jobs tab shows these as they arrive; the job is final only on
|
||||
the app's closing SUMMARY, whose `outputs[]` carries the relay item ids and both sha256 per file.
|
||||
|
||||
Guard rails, as asked:
|
||||
- The whole job runs under `budget_minutes` (default 40; `MAX_RUN_TIMEOUT_MIN` 600). Each stage takes
|
||||
`stage_minutes.<stage>` when given, never more than what is left of the budget. A cap ends `wsl.exe` and then
|
||||
`cargo`, `rustc`, `cc1plus` and `zstd` inside the distro.
|
||||
- 20 GB free on both sides before anything is fetched.
|
||||
- The app's runner is serial: `Jobs.tick` starts a queued job only when `self.active.is_none()`
|
||||
(`src/jobrun.rs`), so a build never overlaps a shard benchmark; the second job waits in the queue.
|
||||
- Nothing stops the miners: `needs_miners_stopped` is true only for `shard-benchmark` and a `run` with
|
||||
`stop_miners_first`. The report says so twice ("the miners keep mining", "the miners were never stopped").
|
||||
|
||||
## The order of events (the main session does this)
|
||||
|
||||
The app on both PCs is 0.3.3, whose parser rejects the WHOLE jobs file when any job has a kind it does not know
|
||||
("one bad job rejects the file"). So the `build` kind must reach the PCs before the first build job is published:
|
||||
|
||||
1. Merge `build-job`, bump the app to 0.3.4 (`app/igneum-app/Cargo.toml`, `resources/igneum-app.rc` x2, `app/windows/version.h`, as the 0.3.3 commit did) and cut it. This last round still goes through the GitHub runner (`fetch-ci-artifacts.sh`, `publish-manifest.sh`). From 0.3.4 on, an unknown kind is skipped, so the next new kind needs no such dance.
|
||||
2. Rebuild the signer in the checkout that publishes: `cd app/igneum-app && cargo build --release --bin igneum-ota-sign` (the old binary refuses `kind 'build' is unknown`).
|
||||
3. Confirm 0.3.4 on PC 1 in the console (Machines tab, app version) or `node tools/jobs.mjs status` after an `update-now` job.
|
||||
4. Publish the first job (below) and watch.
|
||||
|
||||
## The first job: PC 1, which is idle on jobs
|
||||
|
||||
packaging/windows/push-build-inputs.sh --node vendor/igneum-node-v4
|
||||
packaging/ota/publish-jobs.sh add --kind build --target ae432dc7 \
|
||||
--title "First PC build: node devnet-v4 and app, Linux and Windows" \
|
||||
--budget-minutes 150 --stage-minutes '{"setup":20,"linux":50,"windows":50,"test":20,"upload":15}' --deploy
|
||||
|
||||
or in one go, watching and fetching included:
|
||||
|
||||
node tools/build-job.mjs run --node vendor/igneum-node-v4 --target ae432dc7 --budget-minutes 150 \
|
||||
--stage-minutes '{"setup":20,"linux":50,"windows":50,"test":20,"upload":15}'
|
||||
|
||||
The first job is cold: the distro has no mingw, no clang, no `/root/igneum-build/target`, so the node's ~500
|
||||
crates compile for both targets. The Mac's cross-builds of the same tree take 8 to 15 minutes per target with 4 to 6
|
||||
jobs at nice 19 (`infra/cross/build-linux.sh`, `proto-cuda/windows-node/cross-build.sh`, 4 October); the PC's CPU is
|
||||
unknown to me (approximate: comparable), and the ext4 vhdx is slower than the Mac's SSD. Hence 150 minutes for the
|
||||
first job, the 40-minute default for the warm ones after it.
|
||||
|
||||
What success looks like, in order, on the console's Jobs tab (or `node tools/build-job.mjs watch <id>`):
|
||||
|
||||
| Line | Means |
|
||||
|---|---|
|
||||
| `RESULT check <utc> drive C: N GB free, Ubuntu-24.04 /root: M GB free, floor 20 GB` | both sides over 20 GB |
|
||||
| `RESULT fetch <utc> 79xxxxx bytes sha256 ok, node devnet-v4 3bfe346f, app 0.3.4, 2 build units, 2 test units` | the zip is the signed one |
|
||||
| `RESULT setup ok cargo 1.x | rustc 1.x | mingw x86_64-w64-mingw32-gcc-posix (GCC) 13.x` | toolchain in place (first run installs; later runs say "apt packages present") |
|
||||
| `RESULT linux node build exit 0 NNN s`, `RESULT linux igneumd NNN bytes sha256 ...`, `RESULT linux igneum-miner ...` | the Linux node |
|
||||
| `RESULT linux app/igneum-app build exit 0` or `... optional on linux: not fatal` | the engine on Linux, best effort |
|
||||
| `RESULT windows node build exit 0 NNN s`, `RESULT windows igneumd.exe ...`, `igneum-miner.exe`, `igneum-app.exe` | the Windows binaries |
|
||||
| `RESULT test app/igneum-app [igneum-app] exit 0`, `RESULT test node [igneum-miner] exit 0` | tests |
|
||||
| `RESULT output ...` x5, `RESULT upload <utc> igneumd.exe.zst relay item N ...` x5 | on the relay |
|
||||
| `RESULT build <utc> done node devnet-v4 3bfe346f app 0.3.4: every stage ok; 5 files uploaded (NN MB); NN min of 150` | final |
|
||||
| SUMMARY: `status done, exit 0` | the job is closed; `uploaded_files` 5 in the extras |
|
||||
|
||||
Then on the Mac: `node tools/build-job.mjs fetch <id>` prints one line per file ("matches the PC PE ok ...",
|
||||
"coin icon and version block ok" for igneum-app.exe) and places them:
|
||||
|
||||
| File | Lands in |
|
||||
|---|---|
|
||||
| `igneumd.exe`, `igneum-miner.exe` | `vendor/igneum-node/target-integration/x86_64-pc-windows-gnu/release/` (what `push-inputs.sh` and `make-payload.sh` read) |
|
||||
| `igneum-app.exe` | `app/igneum-app/target/x86_64-pc-windows-gnu/release/` (`make-payload.sh`'s default) |
|
||||
| `igneumd`, `igneum-miner`, `igneum-app` (Linux) | `infra/cross/out/` with `version.txt` (where `build-linux.sh` leaves them for the cloud devnet) |
|
||||
|
||||
Also check: the Machines tab still shows PC 1 mining through the whole job (hash rate, accepted blocks), and the
|
||||
Relay tab shows five `build-job <id> ...` file items from `DESKTOP-KMCV30N-ae432dc7` to `mac`.
|
||||
|
||||
Failure signatures to expect on a first run, and what they mean:
|
||||
|
||||
| Line | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `check`: distro free space unknown, "does not answer" | WSL not reachable from the app's account (PC 2's 4 Oct `getpwnam` class) | the job's `wsl_user`/`distro` params; the engine log's `probe wsl` lines |
|
||||
| `RESULT setup apt failed for: gcc-mingw-w64-x86-64 ...` | apt mirror or package names on 24.04 | `run` job with `apt-cache policy gcc-mingw-w64-x86-64` |
|
||||
| windows node build: `undefined reference to pthread_...` or libstdc++ errors | the mingw thread model (win32 vs posix); the script names the `-posix` compilers on purpose | `run` job: `x86_64-w64-mingw32-g++-posix --version`, `ls /usr/x86_64-w64-mingw32/lib/libwinpthread.a` |
|
||||
| windows node build: `bindgen` cannot find `stddef.h` | `BINDGEN_EXTRA_CLANG_ARGS` sysroot | the sysroot path on the PC (`/usr/x86_64-w64-mingw32`) |
|
||||
| app windows build: `no windres found` | `binutils-mingw-w64-x86-64` missing | setup stage output |
|
||||
| `RESULT upload ... over the relay's 50 MB cap` | `igneumd.exe` zst over 50 MB (today's Mac cross-build: 50.5 MB exe, zst approximate 16 MB; fine) | split or a bigger cap in `relay/lib/relay.mjs` |
|
||||
| `STAGE windows timeout` | the cold build under the cap | re-add with a bigger `stage_minutes.windows`; the target dir keeps what compiled |
|
||||
|
||||
A failed job still uploads whatever target built (the pack and upload stages run when any target succeeded), and
|
||||
the Mac tool fetches those; the job status says `failed` with the stage in `failures[]`.
|
||||
|
||||
## What was tested on the Mac, and what was not
|
||||
|
||||
Tested:
|
||||
- `cargo test` of the app crate: 33 tests pass (the ones that existed plus 6 new: build params refused and accepted,
|
||||
unknown kinds refused by the signer and skipped by the runner with the signature still checked, the plan from the
|
||||
manifest with shell-unsafe names refused, the stage scripts carrying the plan, the parsers); the signer binary's
|
||||
21 pass.
|
||||
- `cargo check --target x86_64-pc-windows-gnu` of the app crate (the Windows runner code compiles).
|
||||
- The packer against a scratch folder; the zip inspected (manifest, Cargo files, icon present; no target dirs).
|
||||
- The publisher: `add --kind build` against a scratch `--dest` with the rebuilt signer (signs, verifies, lists;
|
||||
the job carries budget 150, the five stage caps, the zip's sha256 and size, target `ae432dc7`, platform windows,
|
||||
requires `wsl`). The signer built before this branch refuses the same file with "kind 'build' is unknown", which
|
||||
is exactly what a 0.3.3 app would do: hence the rollout order above. A build job with a bad sha256 is refused at
|
||||
signing.
|
||||
- The Mac tool's fetch path against the live relay: a real `igneum-app.exe` zstd-compressed, posted as a build
|
||||
output with the sha256 lines, fetched, decompressed, both sha256 matched, PE header and `verify-exe.py` passed; a
|
||||
second post with a wrong sha256 refused (exit 1). Both test items deleted from the relay afterwards.
|
||||
- The PE check on today's real `igneumd.exe` and `igneum-app.exe` (ok) and on a Linux binary (refused).
|
||||
|
||||
Not tested (only a PC can):
|
||||
- The job itself: wsl.exe output through the sink per stage, the free-space probes, the apt install on 24.04, the
|
||||
mingw posix toolchain with rocksdb, bindgen against `/usr/x86_64-w64-mingw32`, the static link, build times
|
||||
against the caps, `taskkill` plus the in-distro `pkill` on a cap, the relay upload from Windows curl (the PUT with
|
||||
`--data-binary @file`), the 50 MB limit against the real zst sizes.
|
||||
- `cargo test` of `igneum-miner` inside the distro (what tests it has, how long).
|
||||
- The dashboard strip with a `build` job running.
|
||||
- Whether the engine builds on Linux at all (optional on purpose).
|
||||
|
||||
The main session publishes the first job and cuts 0.3.4; this branch stops here.
|
||||
|
|
@ -157,13 +157,14 @@ Ubuntu-24.04), `nvidia`; an unknown one is never met and the job waits until it
|
|||
| `restart` | miners, node (the miners follow) or the app (a detached helper opens it again) | `what` |
|
||||
| `update-now` | the updater checks and installs a newer version at once | |
|
||||
| `shard-benchmark` | miners stopped, GPU under 5%, the prove zip fetched fresh, `prove-shard.sh` inside WSL under the cap, RESULT and STAGE lines and `results/*.json` uploaded, miners back | `zip_url`, `sha256`, `size`, `fixtures`, `cap_minutes` (90), `distro`, `wsl_user` |
|
||||
| `build` | mining untouched; free space checked (20 GB on the drive and in the distro), the build-inputs zip fetched (sha256), setup inside WSL2 as root (mingw-w64, clang, protoc, zstd, the Windows rust target; idempotent), sources extracted, `cargo build --release` native and for `x86_64-pc-windows-gnu`, `cargo test` for the manifest's packages, binaries zstd-compressed and sent to the relay (50 MB each) with sha256 in RESULT lines; the target dir under `/root/igneum-build` persists between jobs. `src/jobbuild.rs`, `packaging/windows/push-build-inputs.sh`, `tools/build-job.mjs` | `zip_url`, `sha256`, `size`, `targets` (linux, windows), `budget_minutes` (40), `stage_minutes` {stage: min}, `min_free_gb` (20), `tests` (true), `relay_url`, `nice` (19), `cargo_jobs`, `distro`, `wsl_user` |
|
||||
|
||||
Reports: every job uploads to the log intake as run_id `job-<id>-<machine id8>` with the label `job-<kind>`; the
|
||||
first line is `SUMMARY {json}` (status, exit, times, the RESULT lines, per-kind extras), then the captured output.
|
||||
Long jobs upload a running report every 5 minutes; collected files and result files are separate labels
|
||||
(`file-<name>`, `result-<name>`, `prove-log`) under the same run_id.
|
||||
|
||||
Safety: the file is rejected when the signature or any job's `expires_at` or params fail; a job id runs once per
|
||||
Safety: the file is rejected when the signature or any job's `expires_at` or params fail (since 0.3.4 a job of a kind the app does not know is skipped instead of rejecting the file; 0.3.3 and earlier reject the whole file, so a new kind reaches the PCs in an app update before its first job is published); a job id runs once per
|
||||
machine (a crash mid-job counts); nothing writes outside the app data folder except an explicit `run` script,
|
||||
which is the operator's responsibility; the dashboard shows the running job and a history (id, kind, started, exit,
|
||||
report uploaded); Settings has "Allow remote jobs from Igneum (signed)", ON by default on this devnet build, with
|
||||
|
|
@ -175,6 +176,7 @@ Publishing and reading:
|
|||
|
||||
packaging/ota/publish-jobs.sh add --kind shard-benchmark --target 1ccfe586 --title "Shard proof run on the 5090" --deploy
|
||||
packaging/ota/publish-jobs.sh add --kind run --target ae432dc7,1ccfe586 --script fix.ps1 --title "..." [--elevated] [--stop-miners]
|
||||
packaging/ota/publish-jobs.sh add --kind build --target ae432dc7 --deploy (after packaging/windows/push-build-inputs.sh; or node tools/build-job.mjs run)
|
||||
packaging/ota/publish-jobs.sh list | remove <id> | sign
|
||||
node tools/jobs.mjs | status | <id> [--all] | watch <id>
|
||||
|
||||
|
|
|
|||
|
|
@ -15,6 +15,11 @@
|
|||
# packaging/ota/publish-jobs.sh add --kind update-now --target all
|
||||
# packaging/ota/publish-jobs.sh add --kind shard-benchmark --target 1ccfe586 [--zip ~/Desktop/igneum-prove-wsl2.zip] \
|
||||
# [--fixtures "block-338-shard1 block-341-shards2 block-344-shards4"] [--cap-minutes 90] [--distro Ubuntu-24.04] [--wsl-user josh]
|
||||
# packaging/ota/publish-jobs.sh add --kind build --target ae432dc7 [--zip <dl>/build-inputs.zip] [--targets linux,windows] \
|
||||
# [--budget-minutes 40] [--stage-minutes '{"linux":20,"windows":20}'] [--min-free-gb 20] [--no-tests] [--relay-url https://...] \
|
||||
# [--nice 19] [--cargo-jobs 0] [--distro Ubuntu-24.04] [--wsl-user root]
|
||||
# (the zip comes from packaging/windows/push-build-inputs.sh; the default is the one in the downloads folder;
|
||||
# tools/build-job.mjs does pack + publish + watch + fetch in one go)
|
||||
# common: --target <id8 or id16, comma list, or all> [--platform windows|mac|any] [--requires wsl-prover,nvidia | none]
|
||||
# (shard-benchmark defaults to --requires wsl-prover; --requires none or "" publishes with no requirement)
|
||||
# [--id custom-id] [--title "..."] [--expires-hours 48] [--deploy]
|
||||
|
|
@ -41,6 +46,7 @@ SCRIPT="" SHELL_KIND="" ELEVATED=0 STOP_MINERS=0 TIMEOUT_MIN=""
|
|||
FILE="" URL="" SHA="" SIZE="" DIR="" TO="" EXTRACT=0 EXTRACT_DIR="" FRESH=0
|
||||
GLOBS=() COMMAND="" WHAT=""
|
||||
ZIP="" FIXTURES="" CAP_MIN="" DISTRO="" WSL_USER="" REMOVE_ID=""
|
||||
TARGETS="" BUDGET_MIN="" STAGE_MIN="" MIN_FREE_GB="" TESTS=1 RELAY_URL="" NICE="" CARGO_JOBS=""
|
||||
case "$CMD" in
|
||||
remove) REMOVE_ID="${1:-}"; [ -n "$REMOVE_ID" ] || { echo "remove <id>" >&2; exit 2; }; shift ;;
|
||||
esac
|
||||
|
|
@ -75,6 +81,14 @@ while [ $# -gt 0 ]; do
|
|||
--cap-minutes) CAP_MIN="$2"; shift 2 ;;
|
||||
--distro) DISTRO="$2"; shift 2 ;;
|
||||
--wsl-user) WSL_USER="$2"; shift 2 ;;
|
||||
--targets) TARGETS="$2"; shift 2 ;;
|
||||
--budget-minutes) BUDGET_MIN="$2"; shift 2 ;;
|
||||
--stage-minutes) STAGE_MIN="$2"; shift 2 ;;
|
||||
--min-free-gb) MIN_FREE_GB="$2"; shift 2 ;;
|
||||
--no-tests) TESTS=0; shift ;;
|
||||
--relay-url) RELAY_URL="$2"; shift 2 ;;
|
||||
--nice) NICE="$2"; shift 2 ;;
|
||||
--cargo-jobs) CARGO_JOBS="$2"; shift 2 ;;
|
||||
--deploy) DEPLOY=1; shift ;;
|
||||
--no-deploy) DEPLOY=0; shift ;;
|
||||
--base-url) BASE="$2"; shift 2 ;;
|
||||
|
|
@ -82,7 +96,7 @@ while [ $# -gt 0 ]; do
|
|||
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
case "$CMD" in add|list|remove|sign) ;; *) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;; esac
|
||||
case "$CMD" in add|list|remove|sign) ;; *) sed -n '2,33p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;; esac
|
||||
|
||||
[ -f "$KEY" ] || { echo "no $KEY (see packaging/ota/README.md, Keys)" >&2; exit 1; }
|
||||
[ -f "$PUB" ] || { echo "no $PUB" >&2; exit 1; }
|
||||
|
|
@ -193,7 +207,33 @@ if a[7]: d["wsl_user"]=a[7]
|
|||
print(json.dumps(d))' "$ZURL" "$ZSHA" "$ZSIZE" "$FIXTURES" "$CAP_MIN" "$DISTRO" "$WSL_USER")"
|
||||
[ -n "$TITLE" ] || TITLE="shard benchmark on the GPU"
|
||||
;;
|
||||
*) echo "unknown kind $KIND (run, fetch, collect, restart, update-now, shard-benchmark)" >&2; exit 2 ;;
|
||||
build)
|
||||
[ -n "$ZIP" ] || ZIP="$DEST/build-inputs.zip"
|
||||
[ -f "$ZIP" ] || { echo "build: no $ZIP; run packaging/windows/push-build-inputs.sh first (or --zip <file>)" >&2; exit 2; }
|
||||
read -r ZURL ZSHA ZSIZE < <(hosted_file "$ZIP")
|
||||
[ -n "$PLATFORM" ] || PLATFORM=windows
|
||||
[ "$REQUIRES_SET" = 1 ] || REQUIRES=wsl
|
||||
PARAMS="$(python3 -c 'import json,sys; a=sys.argv; d={"zip_url": a[1], "sha256": a[2], "size": int(a[3])}
|
||||
if a[4]: d["targets"]=[t.strip() for t in a[4].split(",") if t.strip()]
|
||||
if a[5]: d["budget_minutes"]=int(a[5])
|
||||
if a[6]: d["stage_minutes"]=json.loads(a[6])
|
||||
if a[7]: d["min_free_gb"]=int(a[7])
|
||||
if a[8]=="0": d["tests"]=False
|
||||
if a[9]: d["relay_url"]=a[9]
|
||||
if a[10]: d["nice"]=int(a[10])
|
||||
if a[11]: d["cargo_jobs"]=int(a[11])
|
||||
if a[12]: d["distro"]=a[12]
|
||||
if a[13]: d["wsl_user"]=a[13]
|
||||
print(json.dumps(d))' "$ZURL" "$ZSHA" "$ZSIZE" "$TARGETS" "$BUDGET_MIN" "$STAGE_MIN" "$MIN_FREE_GB" "$TESTS" "$RELAY_URL" "$NICE" "$CARGO_JOBS" "$DISTRO" "$WSL_USER")"
|
||||
if [ -z "$TITLE" ]; then
|
||||
BR="$(python3 -c 'import json,sys
|
||||
try:
|
||||
m=json.load(open(sys.argv[1])); print(m.get("node",{}).get("branch",""), m.get("node",{}).get("commit",""))
|
||||
except Exception: print("")' "${ZIP%.zip}.json" 2>/dev/null || true)"
|
||||
TITLE="build node and app${BR:+ ($BR)}"
|
||||
fi
|
||||
;;
|
||||
*) echo "unknown kind $KIND (run, fetch, collect, restart, update-now, shard-benchmark, build)" >&2; exit 2 ;;
|
||||
esac
|
||||
[ -n "$PLATFORM" ] || PLATFORM=any
|
||||
[ -n "$ID" ] || ID="$KIND-$(date -u +%Y%m%d-%H%M%S)"
|
||||
|
|
|
|||
135
packaging/windows/push-build-inputs.sh
Executable file
135
packaging/windows/push-build-inputs.sh
Executable file
|
|
@ -0,0 +1,135 @@
|
|||
#!/usr/bin/env bash
|
||||
# Publishes build-inputs.zip: the sources a `build` job (app/igneum-app/src/jobbuild.rs) compiles on a PC inside its
|
||||
# WSL2 Ubuntu, so neither the GitHub runner nor the Mac's build lock is needed for the node and the app engine.
|
||||
# The zip goes to the downloads folder (dl/<token>/) next to payload-inputs.zip, with its sha256 and a manifest of
|
||||
# what is inside, exactly as push-inputs.sh does; the job carries the zip's sha256 under the signature, so only the
|
||||
# signed hash is trusted, never the host. Nothing secret goes in: the jobs file is public.
|
||||
#
|
||||
# packaging/windows/push-build-inputs.sh [--node <fork worktree, default vendor/igneum-node-v4>]
|
||||
# [--node-tests "igneum-miner"] [--app-tests "igneum-app"] [--no-app] [--no-deploy] [--out <zip>]
|
||||
#
|
||||
# What goes in (under igneum-build-inputs/):
|
||||
# manifest.json created_at, node {branch, commit, dirty, source}, repo {branch, commit, dirty}, app_version,
|
||||
# builds [{dir, packages, features, bins, targets, optional_on}], tests [{dir, packages}]
|
||||
# node/ the fork worktree: everything but target*/ and .git (the working tree, dirty or not; the
|
||||
# manifest says so, and the job's RESULT lines carry it)
|
||||
# app/igneum-app/ the engine crate (src, ui, resources, build.rs, Cargo.lock), without target/
|
||||
# brand/icons/ igneum.ico and the icon sources (build.rs links the coin icon on the Windows target)
|
||||
# proto-cuda/ the worker sources (without nvrtc/redist, 90 MB of NVIDIA DLLs the job does not need)
|
||||
# Outputs the job returns: igneumd, igneum-miner (Linux and Windows), igneum-app (Windows; Linux when it builds).
|
||||
#
|
||||
# Reads: ~/.config/igneum/dl-token, ~/.config/igneum/dlsite-dir (IGNEUM_DLSITE overrides), ~/.config/igneum/vercel
|
||||
# for the deploy. Then: packaging/ota/publish-jobs.sh add --kind build --target ae432dc7 --deploy, or
|
||||
# node tools/build-job.mjs run, which does both and brings the binaries back.
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
ROOT="$(cd "$HERE/../.." && pwd)"
|
||||
NODE_SRC="$ROOT/vendor/igneum-node-v4"
|
||||
NODE_TESTS="${IGNEUM_BUILD_NODE_TESTS:-igneum-miner}"
|
||||
APP_TESTS="${IGNEUM_BUILD_APP_TESTS:-igneum-app}"
|
||||
WITH_APP=1
|
||||
DEPLOY=1
|
||||
OUT=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--node) NODE_SRC="$2"; shift 2 ;;
|
||||
--node-tests) NODE_TESTS="$2"; shift 2 ;;
|
||||
--app-tests) APP_TESTS="$2"; shift 2 ;;
|
||||
--no-app) WITH_APP=0; shift ;;
|
||||
--no-deploy) DEPLOY=0; shift ;;
|
||||
--out) OUT="$2"; shift 2 ;;
|
||||
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
case "$NODE_SRC" in /*) ;; *) NODE_SRC="$ROOT/$NODE_SRC" ;; esac
|
||||
[ -f "$NODE_SRC/Cargo.toml" ] || { echo "no node source at $NODE_SRC (a vendor/igneum-node-* worktree)" >&2; exit 1; }
|
||||
[ -f "$ROOT/app/igneum-app/Cargo.toml" ] || { echo "no app crate at $ROOT/app/igneum-app" >&2; exit 1; }
|
||||
[ -f "$ROOT/brand/icons/igneum.ico" ] || { echo "no $ROOT/brand/icons/igneum.ico (python3 brand/icons/make-icons.py)" >&2; exit 1; }
|
||||
command -v rsync >/dev/null || { echo "rsync is needed" >&2; exit 1; }
|
||||
command -v zip >/dev/null || { echo "zip is needed" >&2; exit 1; }
|
||||
|
||||
TOKEN_FILE="$HOME/.config/igneum/dl-token"
|
||||
DLSITE="${IGNEUM_DLSITE:-}"
|
||||
[ -n "$DLSITE" ] || { [ -f "$HOME/.config/igneum/dlsite-dir" ] && DLSITE="$(tr -d '[:space:]' < "$HOME/.config/igneum/dlsite-dir")"; } || true
|
||||
if [ -z "$OUT" ]; then
|
||||
[ -f "$TOKEN_FILE" ] || { echo "no $TOKEN_FILE (the downloads token); or give --out <zip>" >&2; exit 1; }
|
||||
TOKEN="$(tr -d '[:space:]' < "$TOKEN_FILE")"
|
||||
[ -n "$DLSITE" ] && [ -d "$DLSITE/dl/$TOKEN" ] || { echo "no downloads folder: set IGNEUM_DLSITE or ~/.config/igneum/dlsite-dir (it must hold dl/<token>/)" >&2; exit 1; }
|
||||
DEST="$DLSITE/dl/$TOKEN"
|
||||
OUT="$DEST/build-inputs.zip"
|
||||
else
|
||||
TOKEN=""
|
||||
DEST="$(cd "$(dirname "$OUT")" && pwd)"
|
||||
OUT="$DEST/$(basename "$OUT")"
|
||||
DEPLOY=0
|
||||
fi
|
||||
|
||||
NODE_BRANCH="$(git -C "$NODE_SRC" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)"
|
||||
NODE_COMMIT="$(git -C "$NODE_SRC" rev-parse --short HEAD 2>/dev/null || echo unknown)"
|
||||
NODE_DIRTY=false; [ -z "$(git -C "$NODE_SRC" status --porcelain 2>/dev/null)" ] || NODE_DIRTY=true
|
||||
REPO_BRANCH="$(git -C "$ROOT" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)"
|
||||
REPO_COMMIT="$(git -C "$ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)"
|
||||
REPO_DIRTY=false; [ -z "$(git -C "$ROOT" status --porcelain -- app/igneum-app brand/icons proto-cuda 2>/dev/null)" ] || REPO_DIRTY=true
|
||||
APP_VERSION="$(sed -n 's/^version = "\(.*\)"/\1/p' "$ROOT/app/igneum-app/Cargo.toml" | head -1)"
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
STAGE="$TMP/igneum-build-inputs"
|
||||
mkdir -p "$STAGE/node" "$STAGE/app" "$STAGE/brand"
|
||||
echo "packing the node fork $NODE_SRC ($NODE_BRANCH $NODE_COMMIT$([ "$NODE_DIRTY" = true ] && echo ', dirty'))"
|
||||
rsync -a --exclude 'target' --exclude 'target-*' --exclude 'target*' --exclude '.git' --exclude '.DS_Store' --exclude '*.vhdx' "$NODE_SRC/" "$STAGE/node/"
|
||||
if [ "$WITH_APP" = 1 ]; then
|
||||
echo "packing app/igneum-app ($APP_VERSION) and brand/icons"
|
||||
rsync -a --exclude 'target' --exclude '.DS_Store' "$ROOT/app/igneum-app/" "$STAGE/app/igneum-app/"
|
||||
rsync -a --exclude '.DS_Store' "$ROOT/brand/icons/" "$STAGE/brand/icons/"
|
||||
fi
|
||||
echo "packing proto-cuda (without nvrtc/redist)"
|
||||
rsync -a --exclude 'nvrtc/redist' --exclude '.DS_Store' --exclude '*.exe' --exclude '*.dll' "$ROOT/proto-cuda/" "$STAGE/proto-cuda/"
|
||||
|
||||
python3 - "$STAGE/manifest.json" "$NODE_BRANCH" "$NODE_COMMIT" "$NODE_DIRTY" "${NODE_SRC#"$ROOT"/}" "$REPO_BRANCH" "$REPO_COMMIT" "$REPO_DIRTY" "$APP_VERSION" "$WITH_APP" "$NODE_TESTS" "$APP_TESTS" <<'PY'
|
||||
import json, sys, datetime
|
||||
out, nb, nc, nd, ns, rb, rc, rd, av, with_app, node_tests, app_tests = sys.argv[1:13]
|
||||
builds = [{"dir": "node", "packages": ["kaspad", "igneum-miner"], "features": ["kaspad/igneum-pow"], "bins": ["igneumd", "igneum-miner"], "targets": ["linux", "windows"]}]
|
||||
tests = []
|
||||
if node_tests.strip():
|
||||
tests.append({"dir": "node", "packages": node_tests.split()})
|
||||
if with_app == "1":
|
||||
builds.append({"dir": "app/igneum-app", "packages": ["igneum-app"], "bins": ["igneum-app"], "targets": ["linux", "windows"], "optional_on": ["linux"]})
|
||||
if app_tests.strip():
|
||||
tests.append({"dir": "app/igneum-app", "packages": app_tests.split()})
|
||||
m = {
|
||||
"created_at": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
"node": {"branch": nb, "commit": nc, "dirty": nd == "true", "source": ns},
|
||||
"repo": {"branch": rb, "commit": rc, "dirty": rd == "true"},
|
||||
"app_version": av if with_app == "1" else "",
|
||||
"builds": builds,
|
||||
"tests": tests,
|
||||
}
|
||||
json.dump(m, open(out, "w"), indent=2, sort_keys=True)
|
||||
PY
|
||||
|
||||
rm -f "$OUT"
|
||||
(cd "$TMP" && zip -qr "$OUT" "igneum-build-inputs" -x '*.DS_Store')
|
||||
SUM="$(shasum -a 256 "$OUT" | awk '{print $1}')"
|
||||
SIZE="$(stat -f %z "$OUT")"
|
||||
printf '%s\n' "$SUM" > "${OUT%.zip}.sha256"
|
||||
cp "$STAGE/manifest.json" "${OUT%.zip}.json"
|
||||
rm -rf "$TMP"
|
||||
echo "$(basename "$OUT"): $SIZE bytes, sha256 $SUM"
|
||||
cat "${OUT%.zip}.json"
|
||||
|
||||
if [ "$DEPLOY" = 1 ]; then
|
||||
echo "deploying $DLSITE"
|
||||
(cd "$DLSITE" && npx --yes vercel@latest --global-config "$HOME/.config/igneum/vercel" deploy --prod --yes 2>&1 | grep -v "$TOKEN" || true)
|
||||
LIVE="$(mktemp)"
|
||||
code="$(curl -s -o "$LIVE" -w '%{http_code}' "https://dl.igneum.network/dl/$TOKEN/build-inputs.sha256")"
|
||||
echo "https://dl.igneum.network/dl/<token>/build-inputs.sha256 -> HTTP $code"
|
||||
[ "$code" = 200 ] && [ "$(tr -d '[:space:]' < "$LIVE")" = "$SUM" ] || { echo "the live sha256 is not reachable or is not this zip's; check the deploy output" >&2; rm -f "$LIVE"; exit 1; }
|
||||
rm -f "$LIVE"
|
||||
echo "live: build-inputs.zip verified by sha256"
|
||||
elif [ -n "$TOKEN" ]; then
|
||||
echo "not deployed (--no-deploy): cd $DLSITE && npx --yes vercel@latest --global-config ~/.config/igneum/vercel deploy --prod --yes"
|
||||
else
|
||||
echo "written to $OUT (not the downloads folder; nothing deployed)"
|
||||
fi
|
||||
echo "Next: packaging/ota/publish-jobs.sh add --kind build --target ae432dc7 --deploy (or node tools/build-job.mjs run)"
|
||||
echo "Note: a build job pins this zip by sha256; publishing a new zip while a job is still queued makes that job fail its sha256 check."
|
||||
242
tools/build-job.mjs
Executable file
242
tools/build-job.mjs
Executable file
|
|
@ -0,0 +1,242 @@
|
|||
#!/usr/bin/env node
|
||||
// Mac side of the `build` job (app/igneum-app/src/jobbuild.rs): a PC builds the node and the app engine for Linux
|
||||
// and Windows inside its WSL2 Ubuntu and sends the binaries to the relay; this tool packs the sources, publishes
|
||||
// the signed job, watches the report, brings the binaries back, checks every sha256 and the Windows PE headers,
|
||||
// and puts them where the packaging scripts look.
|
||||
// node tools/build-job.mjs run [--node vendor/igneum-node-v4] [--target ae432dc7] [--budget-minutes 40]
|
||||
// [--stage-minutes '{"linux":20}'] [--targets linux,windows] [--no-tests]
|
||||
// [--title "..."] [--no-place] [--out dir] pack + publish + watch + fetch
|
||||
// node tools/build-job.mjs publish [the same flags] pack + publish, prints the id
|
||||
// node tools/build-job.mjs watch <job id> STAGE and RESULT lines as they land
|
||||
// node tools/build-job.mjs fetch <job id> [--out dir] [--no-place] download, zstd -d, sha256, PE check, place
|
||||
// node tools/build-job.mjs verify <exe...> the PE check alone
|
||||
// Where the binaries go (--no-place keeps them in --out only):
|
||||
// igneumd.exe, igneum-miner.exe -> $IGNEUM_WIN_RELEASE or vendor/igneum-node/target-integration/x86_64-pc-windows-gnu/release/
|
||||
// (packaging/windows/push-inputs.sh and make-payload.sh read them there)
|
||||
// igneum-app.exe -> app/igneum-app/target/x86_64-pc-windows-gnu/release/ (make-payload.sh's IGNEUM_APP_EXE default)
|
||||
// igneumd, igneum-miner, igneum-app (Linux) -> infra/cross/out/ with version.txt (where infra/cross/build-linux.sh leaves them)
|
||||
// Reads ~/.config/igneum/env (DATABASE_URL, the intake, as tools/jobs.mjs), relay-token, log-intake-key, dl-token.
|
||||
// Needs zstd and python3 on the PATH. No dependencies.
|
||||
import { readFileSync, writeFileSync, mkdirSync, existsSync, copyFileSync, chmodSync, statSync } from 'node:fs';
|
||||
import { homedir, tmpdir } from 'node:os';
|
||||
import { join, resolve, dirname, basename } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
process.stdout.on('error', e => { if (e.code === 'EPIPE') process.exit(0); throw e; });
|
||||
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
|
||||
const cfg = n => { try { return readFileSync(join(homedir(), '.config', 'igneum', n), 'utf8').trim(); } catch { return ''; } };
|
||||
const RELAY_BASE = (cfg('relay-url') || 'https://relay.igneum.network').replace(/\/$/, '');
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
const flags = {}; const pos = [];
|
||||
const BOOL = new Set(['no-tests', 'no-place', 'no-deploy', 'help']);
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a.startsWith('--')) { const k = a.slice(2); const next = argv[i + 1]; if (!BOOL.has(k) && next !== undefined && !next.startsWith('--')) { flags[k] = next; i++; } else flags[k] = true; }
|
||||
else pos.push(a);
|
||||
}
|
||||
const cmd = pos[0] || 'help';
|
||||
const usage = () => { console.log(readFileSync(fileURLToPath(import.meta.url), 'utf8').split('\n').slice(1, 20).map(l => l.replace(/^\/\/ ?/, '')).join('\n')); };
|
||||
|
||||
// ---- the intake (Neon HTTP SQL, as tools/jobs.mjs and tools/logs.mjs) ----------------------------------------------------
|
||||
function db() {
|
||||
const m = /^DATABASE_URL=(.*)$/m.exec(cfg('env'));
|
||||
if (!m) { console.error('DATABASE_URL not found in ~/.config/igneum/env'); process.exit(1); }
|
||||
const url = m[1].trim().replace(/^['"]|['"]$/g, '');
|
||||
const host = new URL(url).hostname.replace('-pooler', '');
|
||||
return async (query, params = []) => {
|
||||
const r = await fetch(`https://${host}/sql`, { method: 'POST', headers: { 'Neon-Connection-String': url, 'Content-Type': 'application/json' }, body: JSON.stringify({ query, params }) });
|
||||
const j = await r.json();
|
||||
if (!r.ok) throw new Error(j.message || JSON.stringify(j));
|
||||
return j.rows;
|
||||
};
|
||||
}
|
||||
const summaryOf = lines => { const l = (lines || '').split('\n')[0]; if (!l.startsWith('SUMMARY ')) return null; try { return JSON.parse(l.slice(8)); } catch { return null; } };
|
||||
const FINAL = new Set(['done', 'failed', 'timeout', 'aborted']);
|
||||
|
||||
/// The newest report per machine for the job: { machine, summary, lines[] }.
|
||||
async function reports(sql, id) {
|
||||
const rows = await sql(`SELECT DISTINCT ON (run_id) run_id, machine, received_at, lines FROM miner_logs WHERE run_id LIKE $1 AND label LIKE 'job-%' ORDER BY run_id, received_at DESC`, [`job-${id}-%`]);
|
||||
return rows.map(r => ({ run_id: r.run_id, machine: r.machine, received_at: r.received_at, summary: summaryOf(r.lines), lines: (r.lines || '').split('\n') }));
|
||||
}
|
||||
|
||||
async function watch(id, quiet = false) {
|
||||
const sql = db();
|
||||
const seen = new Set();
|
||||
let waited = 0;
|
||||
for (;;) {
|
||||
const rs = await reports(sql, id);
|
||||
for (const r of rs) {
|
||||
for (const l of r.lines) {
|
||||
const t = l.trimEnd();
|
||||
if (!/^(STAGE|RESULT|BUILD FAILED)/.test(t) || seen.has(t)) continue;
|
||||
seen.add(t); if (!quiet) console.log(`${r.machine}: ${t}`);
|
||||
}
|
||||
}
|
||||
const finals = rs.filter(r => r.summary && FINAL.has(r.summary.status));
|
||||
if (rs.length && finals.length === rs.length) {
|
||||
for (const r of finals) console.log(`${r.machine}: ${r.summary.status} exit ${r.summary.exit} after ${r.summary.duration_s} s: ${r.summary.summary}`);
|
||||
return finals;
|
||||
}
|
||||
waited += 20;
|
||||
if (!quiet && waited % 120 === 0) console.log(`${rs.length ? 'still running' : 'nothing from the PC yet (it polls the jobs file every 10 min)'}; ${Math.floor(waited / 60)} min waited`);
|
||||
await new Promise(r => setTimeout(r, 20000));
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the relay (file download by item id, feed search by title) ------------------------------------------------------
|
||||
function relayApi() {
|
||||
const token = cfg('relay-token'); const key = cfg('log-intake-key');
|
||||
if (!token && !key) { console.error('no ~/.config/igneum/relay-token and no log-intake-key'); process.exit(1); }
|
||||
const api = `${RELAY_BASE}/r/${token || '-'}/api/`;
|
||||
const headers = key ? { 'x-igneum-key': key } : {};
|
||||
return {
|
||||
async get(fn, q) { const r = await fetch(api + fn + (q ? '?' + new URLSearchParams(q) : ''), { headers }); const j = await r.json(); if (!r.ok || j.ok === false) throw new Error(j.error || `http ${r.status}`); return j; },
|
||||
async download(id, to) { const r = await fetch(`${api}file?id=${id}`, { headers, redirect: 'follow' }); if (!r.ok) throw new Error(`download of relay item ${id}: http ${r.status}`); writeFileSync(to, Buffer.from(await r.arrayBuffer())); return statSync(to).size; },
|
||||
};
|
||||
}
|
||||
const sha256 = p => createHash('sha256').update(readFileSync(p)).digest('hex');
|
||||
|
||||
/// A Windows exe: MZ, a PE signature, x86-64, a .text section, at least 1 MB. Prints what it finds.
|
||||
export function peCheck(path, minBytes = 1024 * 1024) {
|
||||
const d = readFileSync(path);
|
||||
const problems = [];
|
||||
if (d.length < minBytes) problems.push(`only ${d.length} bytes (under ${minBytes})`);
|
||||
if (d.length < 0x40 || d.toString('latin1', 0, 2) !== 'MZ') { problems.push('no MZ header'); return { ok: false, problems }; }
|
||||
const pe = d.readUInt32LE(0x3c);
|
||||
if (pe + 24 > d.length || d.toString('latin1', pe, pe + 4) !== 'PE\0\0') { problems.push('no PE signature'); return { ok: false, problems }; }
|
||||
const machine = d.readUInt16LE(pe + 4);
|
||||
if (machine !== 0x8664) problems.push(`machine 0x${machine.toString(16)} is not x86-64 (0x8664)`);
|
||||
const nsec = d.readUInt16LE(pe + 6); const optSize = d.readUInt16LE(pe + 20);
|
||||
const magic = d.readUInt16LE(pe + 24);
|
||||
if (magic !== 0x20b) problems.push(`optional header magic 0x${magic.toString(16)} is not PE32+`);
|
||||
const subsystem = optSize >= 70 ? d.readUInt16LE(pe + 24 + 68) : 0;
|
||||
const names = [];
|
||||
for (let i = 0; i < nsec; i++) { const s = pe + 24 + optSize + 40 * i; if (s + 8 > d.length) break; names.push(d.toString('latin1', s, s + 8).replace(/\0+$/, '')); }
|
||||
if (!names.includes('.text')) problems.push('no .text section');
|
||||
return { ok: !problems.length, problems, bytes: d.length, machine, subsystem, sections: names };
|
||||
}
|
||||
|
||||
// ---- fetch: the outputs of a finished job -----------------------------------------------------------------------------------
|
||||
async function fetchOutputs(id, finals) {
|
||||
const out = resolve(flags.out || join(tmpdir(), 'igneum-build-job', id));
|
||||
mkdirSync(out, { recursive: true });
|
||||
const relay = relayApi();
|
||||
// the SUMMARY carries outputs[] with relay ids; the feed is the fallback (titles "build-job <id> <file>")
|
||||
let outputs = [];
|
||||
let meta = {};
|
||||
for (const f of finals || []) { if (f.summary && Array.isArray(f.summary.outputs) && f.summary.outputs.length) { outputs = f.summary.outputs; meta = f.summary; break; } }
|
||||
if (!outputs.length) {
|
||||
const feed = await relay.get('feed', { limit: 500 });
|
||||
for (const it of feed.items || []) {
|
||||
const m = new RegExp(`^build-job ${id.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')} (.+)$`).exec(it.title || '');
|
||||
if (!m || !it.has_file || m[1] === 'build-outputs.json') continue;
|
||||
const body = it.body || '';
|
||||
const unpacked = /^(linux|windows) (\S+) (\d+) bytes sha256 ([0-9a-f]{64})$/m.exec(body);
|
||||
const packed = /^(\S+\.zst) (\d+) bytes sha256 ([0-9a-f]{64})$/m.exec(body);
|
||||
outputs.push({ name: unpacked ? unpacked[2] : m[1].replace(/\.zst$/, '').replace(/\.linux$/, ''), target: unpacked ? unpacked[1] : (m[1].endsWith('.exe.zst') ? 'windows' : 'linux'), bytes: unpacked ? Number(unpacked[3]) : 0, sha256: unpacked ? unpacked[4] : '', zst: m[1], zst_bytes: packed ? Number(packed[2]) : it.size, zst_sha256: packed ? packed[3] : '', relay_id: it.id });
|
||||
const nb = /^node (\S+) (\S+)$/m.exec(body); if (nb) { meta.node_branch = nb[1]; meta.node_commit = nb[2]; }
|
||||
const av = /^app (\S+)$/m.exec(body); if (av) meta.app_version = av[1];
|
||||
}
|
||||
}
|
||||
if (!outputs.length) { console.error(`no outputs for job ${id}: nothing in the SUMMARY and nothing titled "build-job ${id} ..." on the relay`); process.exit(1); }
|
||||
const placed = []; let bad = 0;
|
||||
for (const o of outputs) {
|
||||
if (!o.relay_id) { console.log(`${o.zst}: not uploaded (no relay item); skipped`); bad++; continue; }
|
||||
const zst = join(out, o.zst);
|
||||
const got = await relay.download(o.relay_id, zst);
|
||||
if (o.zst_sha256 && sha256(zst) !== o.zst_sha256) { console.log(`${o.zst}: sha256 of the download does not match the RESULT line (${got} bytes); refused`); bad++; continue; }
|
||||
const plain = join(out, o.target === 'windows' ? o.name : `${o.name}`);
|
||||
const r = spawnSync('zstd', ['-d', '-q', '-f', zst, '-o', plain], { stdio: 'inherit' });
|
||||
if (r.status !== 0) { console.log(`${o.zst}: zstd -d failed (exit ${r.status}); is zstd installed? (brew install zstd)`); bad++; continue; }
|
||||
const sum = sha256(plain);
|
||||
if (o.sha256 && sum !== o.sha256) { console.log(`${o.name} (${o.target}): sha256 after decompression ${sum.slice(0, 16)}... is not the PC's ${o.sha256.slice(0, 16)}...; refused`); bad++; continue; }
|
||||
if (!o.sha256) console.log(`${o.name} (${o.target}): WARNING no sha256 from the PC for this file (no SUMMARY and no body on the relay item); only the download and the PE check stand`);
|
||||
if (o.target === 'linux') { try { chmodSync(plain, 0o755); } catch {} }
|
||||
let note = '';
|
||||
if (o.target === 'windows') {
|
||||
const pe = peCheck(plain);
|
||||
if (!pe.ok) { console.log(`${o.name}: PE check FAILED: ${pe.problems.join('; ')}`); bad++; continue; }
|
||||
note = ` PE ok (${pe.sections.join(' ')}, subsystem ${pe.subsystem})`;
|
||||
if (o.name === 'igneum-app.exe') {
|
||||
const v = spawnSync('python3', [join(ROOT, 'packaging/windows/resources/verify-exe.py'), ...(meta.app_version ? ['--version', meta.app_version] : []), plain], { encoding: 'utf8' });
|
||||
if (v.status !== 0) { console.log(`${o.name}: verify-exe.py FAILED:\n${(v.stdout || '') + (v.stderr || '')}`); bad++; continue; }
|
||||
note += ', coin icon and version block ok';
|
||||
}
|
||||
}
|
||||
console.log(`${o.name} (${o.target}): ${statSync(plain).size} bytes, sha256 ${sum.slice(0, 16)}...${o.sha256 ? ' matches the PC' : ' (unchecked)'}${note}`);
|
||||
if (!flags['no-place']) {
|
||||
const dest = placeFor(o);
|
||||
mkdirSync(dirname(dest), { recursive: true });
|
||||
copyFileSync(plain, dest);
|
||||
if (o.target === 'linux') { try { chmodSync(dest, 0o755); } catch {} }
|
||||
placed.push(dest);
|
||||
}
|
||||
}
|
||||
if (placed.some(p => p.startsWith(join(ROOT, 'infra/cross/out')))) {
|
||||
const v = `igneumd ${meta.node_commit || ''}\nbuilt by build job ${id} on a PC (WSL2, native x86_64 Linux) from ${meta.node_commit || '?'} (${meta.node_branch || '?'}), app ${meta.app_version || '?'}\n${meta.node_commit || ''}\n`;
|
||||
writeFileSync(join(ROOT, 'infra/cross/out/version.txt'), v);
|
||||
placed.push(join(ROOT, 'infra/cross/out/version.txt'));
|
||||
}
|
||||
console.log(`\n${outputs.length - bad} of ${outputs.length} outputs verified${bad ? `, ${bad} refused` : ''}; downloads in ${out}`);
|
||||
for (const p of placed) console.log(` placed ${p.replace(ROOT + '/', '')}`);
|
||||
if (placed.length) console.log('Next: packaging/windows/push-inputs.sh (the Windows payload inputs), then the installer round; Linux binaries: infra/cross/out/');
|
||||
return bad === 0;
|
||||
}
|
||||
|
||||
function placeFor(o) {
|
||||
if (o.target === 'windows') {
|
||||
if (o.name === 'igneum-app.exe') return join(ROOT, 'app/igneum-app/target/x86_64-pc-windows-gnu/release/igneum-app.exe');
|
||||
return join(process.env.IGNEUM_WIN_RELEASE || join(ROOT, 'vendor/igneum-node/target-integration/x86_64-pc-windows-gnu/release'), o.name);
|
||||
}
|
||||
return join(ROOT, 'infra/cross/out', o.name);
|
||||
}
|
||||
|
||||
// ---- publish: pack, then the signed job ---------------------------------------------------------------------------------------
|
||||
function publish() {
|
||||
const pack = ['packaging/windows/push-build-inputs.sh'];
|
||||
if (flags.node) pack.push('--node', flags.node);
|
||||
if (flags['no-deploy']) pack.push('--no-deploy');
|
||||
console.log(`$ ${pack.join(' ')}`);
|
||||
const r = spawnSync('bash', pack, { cwd: ROOT, stdio: 'inherit' });
|
||||
if (r.status !== 0) { console.error('push-build-inputs.sh failed'); process.exit(1); }
|
||||
const add = ['packaging/ota/publish-jobs.sh', 'add', '--kind', 'build', '--target', flags.target || 'ae432dc7'];
|
||||
if (flags['budget-minutes']) add.push('--budget-minutes', String(flags['budget-minutes']));
|
||||
if (flags['stage-minutes']) add.push('--stage-minutes', String(flags['stage-minutes']));
|
||||
if (flags.targets) add.push('--targets', String(flags.targets));
|
||||
if (flags['no-tests']) add.push('--no-tests');
|
||||
if (flags['min-free-gb']) add.push('--min-free-gb', String(flags['min-free-gb']));
|
||||
if (flags.title) add.push('--title', String(flags.title));
|
||||
if (flags.id) add.push('--id', String(flags.id));
|
||||
if (!flags['no-deploy']) add.push('--deploy');
|
||||
console.log(`$ ${add.join(' ')}`);
|
||||
const a = spawnSync('bash', add, { cwd: ROOT, encoding: 'utf8' });
|
||||
process.stdout.write(a.stdout || ''); process.stderr.write(a.stderr || '');
|
||||
if (a.status !== 0) { console.error('publish-jobs.sh failed'); process.exit(1); }
|
||||
const m = /^added: (\S+) build/m.exec(a.stderr || '');
|
||||
if (!m) { console.error('could not read the job id from the publisher output'); process.exit(1); }
|
||||
console.log(`job ${m[1]} published${flags['no-deploy'] ? ' (not deployed)' : ''}`);
|
||||
return m[1];
|
||||
}
|
||||
|
||||
if (cmd === 'help' || flags.help) { usage(); process.exit(0); }
|
||||
if (cmd === 'verify') {
|
||||
if (pos.length < 2) { console.error('verify <exe...>'); process.exit(1); }
|
||||
let ok = true;
|
||||
for (const p of pos.slice(1)) { const r = peCheck(resolve(p)); console.log(`${p}: ${r.ok ? `ok, ${r.bytes} bytes, sections ${r.sections.join(' ')}, subsystem ${r.subsystem}` : 'FAIL: ' + r.problems.join('; ')}`); ok = ok && r.ok; }
|
||||
process.exit(ok ? 0 : 1);
|
||||
}
|
||||
if (cmd === 'publish') { publish(); process.exit(0); }
|
||||
if (cmd === 'watch') { if (!pos[1]) { console.error('watch <job id>'); process.exit(1); } await watch(pos[1]); process.exit(0); }
|
||||
if (cmd === 'fetch') { if (!pos[1]) { console.error('fetch <job id>'); process.exit(1); } const finals = await reports(db(), pos[1]).then(rs => rs.filter(r => r.summary && FINAL.has(r.summary.status))); process.exit((await fetchOutputs(pos[1], finals)) ? 0 : 1); }
|
||||
if (cmd === 'run') {
|
||||
const id = publish();
|
||||
if (flags['no-deploy']) { console.log('not deployed, so nothing to watch'); process.exit(0); }
|
||||
console.log(`watching job ${id} (the PC polls every 10 minutes; Settings > remote jobs > Check now on the PC runs it at once)`);
|
||||
const finals = await watch(id);
|
||||
if (!finals.some(f => f.summary.status === 'done' || (f.summary.outputs || []).length)) { console.error('the job produced nothing to fetch'); process.exit(1); }
|
||||
process.exit((await fetchOutputs(id, finals)) ? 0 : 1);
|
||||
}
|
||||
console.error(`unknown command ${cmd}`); usage(); process.exit(2);
|
||||
|
|
@ -3,7 +3,7 @@
|
|||
// node tools/relay.mjs the feed, newest first (last 50)
|
||||
// node tools/relay.mjs list [N] [--machine X] more of the feed
|
||||
// node tools/relay.mjs read <id> print an item; its file is downloaded to --out (default $TMPDIR/igneum-relay)
|
||||
// node tools/relay.mjs drop "<text>" | <file> post a note or a file from the Mac [--to PC1] [--title "..."]
|
||||
// node tools/relay.mjs drop "<text>" | <file> post a note or a file from the Mac [--to PC1] [--title "..."] [--body "..." with a file]
|
||||
// node tools/relay.mjs task <machine> "title" [file] [--body "..."] a task for a person or a Claude session on that PC
|
||||
// node tools/relay.mjs run <machine> "title" <script.ps1|.sh> [--elevated] [--reboot-continue] a script the igneum-agent runs
|
||||
// node tools/relay.mjs watch [--since <id>] poll every 10 s and print new items (results included)
|
||||
|
|
@ -96,7 +96,7 @@ try {
|
|||
else if (cmd === 'drop') {
|
||||
const what = pos[1]; if (!what) throw new Error('drop "<text>" or drop <file>');
|
||||
const o = { from: flags.from || 'Mac', to: flags.to || 'all', title: flags.title || '' };
|
||||
if (existsSync(what) && statSync(what).isFile()) Object.assign(o, await uploadFile(what), { kind: 'file' }); else { o.body = what; o.kind = flags.kind || 'text'; }
|
||||
if (existsSync(what) && statSync(what).isFile()) Object.assign(o, await uploadFile(what), { kind: 'file', body: typeof flags.body === 'string' ? flags.body : '' }); else { o.body = what; o.kind = flags.kind || 'text'; }
|
||||
const r = await api('drop', { body: o }); console.log(`sent #${r.id} (${r.kind})`);
|
||||
}
|
||||
else if (cmd === 'task' || cmd === 'run') {
|
||||
|
|
|
|||
Loading…
Reference in a new issue