igneum/docs/plans/build-job.md

The build job: a PC builds the node and the app, nobody at the keyboard

4 October 2026, evening. the project lead's ask ("efficiency"): every Windows node and app build went through a GitHub runner at 15 to 25 minutes a round, and every Linux binary was cross-compiled on this Mac under the build lock. The two RTX 5090 PCs (PC 1 ae432dc7, PC 2 1ccfe586) run Igneum Miner 0.3.3 with the signed job channel and each has a WSL2 Ubuntu 24.04 owned by the app (root, cargo and the SP1 toolchain under /root from the shard jobs). So the build goes to them: a build job, mining untouched.

What is built (branch build-job)

Piece	Where	State
Job kind build: params, validation, the 40-minute default budget, parse_lenient (an unknown kind is skipped by the app instead of rejecting the file; the signer stays strict)	app/igneum-app/src/jobs.rs	unit tests on the Mac
The plan: parameters, the inputs manifest (what to build, what to test; shell-unsafe names refused), the bash stage scripts (setup, extract, linux, windows, test, pack), per-stage caps, parsers for free space, the outputs file and the relay replies	app/igneum-app/src/jobbuild.rs (new)	unit tests on the Mac
The runner: free-space check on the Windows drive and inside the distro, fetch with sha256, the manifest out of the zip, one wsl.exe call per stage under its cap (kill of the Linux side on a cap), zstd outputs uploaded to the relay (fn=upload, Blob PUT, fn=drop), RESULT and STAGE lines with UTC times, closing SUMMARY with outputs[]	app/igneum-app/src/jobrun.rs (run_build, build_stage, relay_upload)	compiles for macOS and x86_64-pc-windows-gnu; not run on a PC
Packer: the fork worktree, app/igneum-app, brand/icons, proto-cuda (no redist) into build-inputs.zip with .sha256 and .json (branch, commit, dirty, date, builds, tests) on the downloads host	packaging/windows/push-build-inputs.sh (new)	run against a scratch folder: 7.9 MB, 1937 files, no target dirs
Publisher: publish-jobs.sh add --kind build (platform windows, requires wsl, --budget-minutes, --stage-minutes, --targets, --no-tests, --min-free-gb, --relay-url, --nice, --cargo-jobs)	packaging/ota/publish-jobs.sh	see "Tested"
Mac tool: pack + publish + watch + fetch; sha256 of the zst and of the unpacked file against the PC's RESULT lines; PE header check of every exe (MZ, PE, x86-64, PE32+, .text, over 1 MB); verify-exe.py --version on igneum-app.exe; placement where push-inputs.sh, make-payload.sh and the cloud-devnet scripts look	tools/build-job.mjs (new)	fetch path run against the live relay: a real exe round-tripped, a wrong sha256 refused
relay.mjs drop <file> --body carries the body (the fetch fallback reads the sha256 from it)	tools/relay.mjs	run live
Dashboard label, README row	app/igneum-app/ui/app.js, packaging/ota/README.md

What the job does on the PC

Stage	What	Cap
check	free GB on the drive that holds the app data (PowerShell DriveInfo) and under /root/igneum-build inside the distro (df); both must be at or over min_free_gb (20)	2 min
fetch	build-inputs.zip by https, sha256 and size as signed in the job; manifest.json read out of it with tar	curl, 60 min
setup	apt-get install of what dpkg -s says is missing (build-essential, clang and libclang for bindgen, protobuf-compiler, zstd, unzip, gcc-mingw-w64-x86-64, g++-mingw-w64-x86-64, binutils-mingw-w64-x86-64, mingw-w64-x86-64-dev), rustup when cargo is missing, rustup target add x86_64-pc-windows-gnu	stage cap
extract	/root/igneum-build/src replaced by the zip; /root/igneum-build/target (CARGO_TARGET_DIR) persists, so the ~500 dependency crates compile once	stage cap
linux	per manifest unit: nice -n 19 cargo build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow, then the app crate (optional on Linux: a failure is a RESULT line, not a job failure)	stage cap
windows	the same with --target x86_64-pc-windows-gnu, CC/CXX = x86_64-w64-mingw32-gcc-posix/g++-posix, LIBCLANG_PATH and BINDGEN_EXTRA_CLANG_ARGS for librocksdb-sys, -C link-arg=-static -C link-arg=-static-libgcc, IGNEUM_WINDRES for the coin icon	stage cap
test	cargo test --release per manifest test unit (default igneum-app, igneum-miner); a failure marks the job failed but the binaries still ship	stage cap
pack	zstd -T0 -12 per binary (igneumd.linux.zst, igneumd.exe.zst, ...), sha256 of both forms, build-outputs.json, copied to the job folder on the Windows side	stage cap
upload	each .zst under 50 MB to the relay: fn=upload with the intake key (allowed for upload and drop; round 4 X23 keeps the key away from run and task posts), PUT to Vercel Blob, fn=drop to mac titled build-job <id> <file> with the sha256 lines in the body	stage cap

Every stage prints STAGE <name> start <utc> cap N min and STAGE <name> end|timeout <utc> N s exit M; every binary a RESULT <target> <name> <bytes> bytes sha256 <hex> line and later RESULT output ... with the zst sha256 and RESULT upload ... relay item <id>. The console's Jobs tab shows these as they arrive; the job is final only on the app's closing SUMMARY, whose outputs[] carries the relay item ids and both sha256 per file.

Guard rails, as asked:

• The whole job runs under budget_minutes (default 40; MAX_RUN_TIMEOUT_MIN 600). Each stage takes stage_minutes.<stage> when given, never more than what is left of the budget. A cap ends wsl.exe and then cargo, rustc, cc1plus and zstd inside the distro.
• 20 GB free on both sides before anything is fetched.
• The app's runner is serial: Jobs.tick starts a queued job only when self.active.is_none() (src/jobrun.rs), so a build never overlaps a shard benchmark; the second job waits in the queue.
• Nothing stops the miners: needs_miners_stopped is true only for shard-benchmark and a run with stop_miners_first. The report says so twice ("the miners keep mining", "the miners were never stopped").

The order of events (the main session does this)

The app on both PCs is 0.3.3, whose parser rejects the WHOLE jobs file when any job has a kind it does not know ("one bad job rejects the file"). So the build kind must reach the PCs before the first build job is published:

1. Merge build-job, bump the app to 0.3.4 (app/igneum-app/Cargo.toml, resources/igneum-app.rc x2, app/windows/version.h, as the 0.3.3 commit did) and cut it. This last round still goes through the GitHub runner (fetch-ci-artifacts.sh, publish-manifest.sh). From 0.3.4 on, an unknown kind is skipped, so the next new kind needs no such dance.
2. Rebuild the signer in the checkout that publishes: cd app/igneum-app && cargo build --release --bin igneum-ota-sign (the old binary refuses kind 'build' is unknown).
3. Confirm 0.3.4 on PC 1 in the console (Machines tab, app version) or node tools/jobs.mjs status after an update-now job.
4. Publish the first job (below) and watch.

The first job: PC 1, which is idle on jobs

packaging/windows/push-build-inputs.sh --node vendor/igneum-node-v4
packaging/ota/publish-jobs.sh add --kind build --target ae432dc7 \
    --title "First PC build: node devnet-v4 and app, Linux and Windows" \
    --budget-minutes 150 --stage-minutes '{"setup":20,"linux":50,"windows":50,"test":20,"upload":15}' --deploy

or in one go, watching and fetching included:

node tools/build-job.mjs run --node vendor/igneum-node-v4 --target ae432dc7 --budget-minutes 150 \
    --stage-minutes '{"setup":20,"linux":50,"windows":50,"test":20,"upload":15}'

The first job is cold: the distro has no mingw, no clang, no /root/igneum-build/target, so the node's ~500 crates compile for both targets. The Mac's cross-builds of the same tree take 8 to 15 minutes per target with 4 to 6 jobs at nice 19 (infra/cross/build-linux.sh, proto-cuda/windows-node/cross-build.sh, 4 October); the PC's CPU is unknown to me (approximate: comparable), and the ext4 vhdx is slower than the Mac's SSD. Hence 150 minutes for the first job, the 40-minute default for the warm ones after it.

What success looks like, in order, on the console's Jobs tab (or node tools/build-job.mjs watch <id>):

Line	Means
RESULT check <utc> drive C: N GB free, Ubuntu-24.04 /root: M GB free, floor 20 GB	both sides over 20 GB
RESULT fetch <utc> 79xxxxx bytes sha256 ok, node devnet-v4 3bfe346f, app 0.3.4, 2 build units, 2 test units	the zip is the signed one
`RESULT setup ok cargo 1.x	rustc 1.x
RESULT linux node build exit 0 NNN s, RESULT linux igneumd NNN bytes sha256 ..., RESULT linux igneum-miner ...	the Linux node
RESULT linux app/igneum-app build exit 0 or ... optional on linux: not fatal	the engine on Linux, best effort
RESULT windows node build exit 0 NNN s, RESULT windows igneumd.exe ..., igneum-miner.exe, igneum-app.exe	the Windows binaries
RESULT test app/igneum-app [igneum-app] exit 0, RESULT test node [igneum-miner] exit 0	tests
RESULT output ... x5, RESULT upload <utc> igneumd.exe.zst relay item N ... x5	on the relay
RESULT build <utc> done node devnet-v4 3bfe346f app 0.3.4: every stage ok; 5 files uploaded (NN MB); NN min of 150	final
SUMMARY: status done, exit 0	the job is closed; uploaded_files 5 in the extras

Then on the Mac: node tools/build-job.mjs fetch <id> prints one line per file ("matches the PC PE ok ...", "coin icon and version block ok" for igneum-app.exe) and places them:

File	Lands in
igneumd.exe, igneum-miner.exe	vendor/igneum-node/target-integration/x86_64-pc-windows-gnu/release/ (what push-inputs.sh and make-payload.sh read)
igneum-app.exe	app/igneum-app/target/x86_64-pc-windows-gnu/release/ (make-payload.sh's default)
igneumd, igneum-miner, igneum-app (Linux)	infra/cross/out/ with version.txt (where build-linux.sh leaves them for the cloud devnet)

Also check: the Machines tab still shows PC 1 mining through the whole job (hash rate, accepted blocks), and the Relay tab shows five build-job <id> ... file items from DESKTOP-KMCV30N-ae432dc7 to mac.

Failure signatures to expect on a first run, and what they mean:

Line	Likely cause	Fix
check: distro free space unknown, "does not answer"	WSL not reachable from the app's account (PC 2's 4 Oct getpwnam class)	the job's wsl_user/distro params; the engine log's probe wsl lines
RESULT setup apt failed for: gcc-mingw-w64-x86-64 ...	apt mirror or package names on 24.04	run job with apt-cache policy gcc-mingw-w64-x86-64
windows node build: undefined reference to pthread_... or libstdc++ errors	the mingw thread model (win32 vs posix); the script names the -posix compilers on purpose	run job: x86_64-w64-mingw32-g++-posix --version, ls /usr/x86_64-w64-mingw32/lib/libwinpthread.a
windows node build: bindgen cannot find stddef.h	BINDGEN_EXTRA_CLANG_ARGS sysroot	the sysroot path on the PC (/usr/x86_64-w64-mingw32)
app windows build: no windres found	binutils-mingw-w64-x86-64 missing	setup stage output
RESULT upload ... over the relay's 50 MB cap	igneumd.exe zst over 50 MB (today's Mac cross-build: 50.5 MB exe, zst approximate 16 MB; fine)	split or a bigger cap in relay/lib/relay.mjs
STAGE windows timeout	the cold build under the cap	re-add with a bigger stage_minutes.windows; the target dir keeps what compiled

A failed job still uploads whatever target built (the pack and upload stages run when any target succeeded), and the Mac tool fetches those; the job status says failed with the stage in failures[].

What was tested on the Mac, and what was not

Tested:

• cargo test of the app crate: 33 tests pass (the ones that existed plus 6 new: build params refused and accepted, unknown kinds refused by the signer and skipped by the runner with the signature still checked, the plan from the manifest with shell-unsafe names refused, the stage scripts carrying the plan, the parsers); the signer binary's 21 pass.
• cargo check --target x86_64-pc-windows-gnu of the app crate (the Windows runner code compiles).
• The packer against a scratch folder; the zip inspected (manifest, Cargo files, icon present; no target dirs).
• The publisher: add --kind build against a scratch --dest with the rebuilt signer (signs, verifies, lists; the job carries budget 150, the five stage caps, the zip's sha256 and size, target ae432dc7, platform windows, requires wsl). The signer built before this branch refuses the same file with "kind 'build' is unknown", which is exactly what a 0.3.3 app would do: hence the rollout order above. A build job with a bad sha256 is refused at signing.
• The Mac tool's fetch path against the live relay: a real igneum-app.exe zstd-compressed, posted as a build output with the sha256 lines, fetched, decompressed, both sha256 matched, PE header and verify-exe.py passed; a second post with a wrong sha256 refused (exit 1). Both test items deleted from the relay afterwards.
• The PE check on today's real igneumd.exe and igneum-app.exe (ok) and on a Linux binary (refused).

Not tested (only a PC can):

• The job itself: wsl.exe output through the sink per stage, the free-space probes, the apt install on 24.04, the mingw posix toolchain with rocksdb, bindgen against /usr/x86_64-w64-mingw32, the static link, build times against the caps, taskkill plus the in-distro pkill on a cap, the relay upload from Windows curl (the PUT with --data-binary @file), the 50 MB limit against the real zst sizes.
• cargo test of igneum-miner inside the distro (what tests it has, how long).
• The dashboard strip with a build job running.
• Whether the engine builds on Linux at all (optional on purpose).

The main session publishes the first job and cuts 0.3.4; this branch stops here.