78 KiB
Igneum Miner 0.3.6: the testnet adoption, staged 5 October 2026
Prepared by the integration agent on the morning of 5 October 2026 after the owner's decision. Nothing was deployed,
published or merged to master. The app and document work sits on branch testnet-adopt (pushed to origin), worktree
/Users/joshm/Projects/igneum-wt-adopt; the node work sits on the fork branch release-0.3.6 (local only; the fork
has no remote), worktree vendor/igneum-node-036. The cut is one command (section 4) after the inputs push of
section 3, which must come first.
0. The decision (owner, 5 October 2026)
The proposed testnet identity (docs/testnet/README.md) and the proposed fee floors and prover-gas table
(docs/analysis/base-fee-floor.md, spec 05 section 5.11) are ADOPTED as proposed. The three documents now say so;
the fork's FeeParams::CALIBRATED_V1 and TESTNET_PARAMS are unchanged from the proposal.
One rule the adoption added, so the live devnet is never forked by a node update: the devnet keeps its fee rules until a height switch says otherwise (section 5).
1. What is in
1a. The app repository (branch testnet-adopt, from origin/master 2054ae3 = the 0.3.5 cut)
| Merged | Branch head | What it carries | Conflicts and how they were resolved |
|---|---|---|---|
origin/testnet-prep |
beed743 (3 commits over e22068f) |
docs/testnet/README.md, docs/analysis/base-fee-floor.md, spec 05 section 5.11, docs/plans/history-rewrite.md (G14, not acted on here), G13 signed build inputs (app/igneum-app/src/inputs.rs, igneum-ota-sign sign-inputs / verify-inputs, packaging/windows/push-inputs.sh, inputs-manifest.sh, node-source.pin, test-inputs-signing.sh, fetch-ci-artifacts.sh --sign-manifest, .github/workflows/windows.yml), tools/ci/check-workflow-shell.mjs, the testnet terms on the download section, site/wallet.html, the litepaper's app paragraph, the fast-time profile's fees object |
four generated site files. site/index.html: the dev-fee sentence of 0.3.5 kept, testnet-prep's #testnet-terms card and wallet link kept (both sides of the download section); the inlined journey block is regenerated. site/litepaper.html: 0.3.5's two-paragraph dev-fee text kept (it is the fuller one; testnet-prep's one-sentence version dropped), testnet-prep's MetaMask and Igneum Wallet paragraph kept. site/journey.json: 0.3.5's log kept (the feed is the newest 40 entries; testnet-prep's older entries had already aged out of it). site/sitemap.xml: both /miners and /wallet. Then node site/build.mjs |
On top of the merge, in the same branch:
| Change | Where |
|---|---|
| "proposed" to "adopted 5 October 2026" with the sign-off noted, and the per-network fee rule written in | docs/testnet/README.md, docs/analysis/base-fee-floor.md, docs/spec/05-fees-and-economics.md (section 5.10 and the parameter table) |
fees_v1_activation_daa: 0 added to the 60x fast-time profile (the fork's fast_time_60x_file_is_the_devnet_at_60x test wants every override field present) |
infra/fast-time/override-60x.json |
| this file | docs/plans/release-0.3.6.md |
1b. The node fork (branch release-0.3.6 in vendor/igneum-node-036, from release-0.3.5 20139145)
| Fork merge | Head | What it carries | Conflicts |
|---|---|---|---|
testnet-params (worktree vendor/igneum-node-testnet, forked from finality-fixes 6aa69a45) |
11e86144 | consensus/core/src/fees.rs (FeeParams, PgasTable, CALIBRATED_V1, PROTOTYPE), Params.fees and the override file's fees object, the testnet identity (igneum-testnet-1, chain id 4462, ports 268xx, the frozen genesis, FinalityParams::MAINNET, every switch at 0), the override file refused on the testnet, the execution layer reading the installed set (B_p, S_p, the intrinsic, modexp, the floors) |
consensus/core/src/config/params.rs: both methods kept (apply_env_pow_schedule from G12 and install_fee_params); the M31 comment on max_coinbase_payload_len kept. kaspad/src/daemon.rs: the 0.3.5 start-up block kept (environment schedule on devnet and simnet, install_pow_schedule, the digest line) and install_fee_params with its own print added after it |
On top of the merge, the fee height switch (section 5), in the same branch:
| Change | Where |
|---|---|
FeeParams::DEVNET and SIMNET = PROTOTYPE; TESTNET and MAINNET = CALIBRATED_V1. FeeSchedule { base, v1_activation_daa } with at(daa); install_fee_params(base, switch); fee_params_at(daa) replaces fee_params() |
consensus/core/src/fees.rs |
Params.fees_v1_activation_daa (devnet and simnet u64::MAX, testnet and mainnet 0), OverrideParams.fees_v1_activation_daa, the digest rule of section 5, four new or changed tests |
consensus/core/src/config/params.rs |
the switch and a fees object printed from the override file; the installed schedule printed with the network |
kaspad/src/daemon.rs |
shard_proving_gas_budget_at(daa) |
consensus/core/src/proving.rs |
every reader takes a DAA score; execute_segment picks the set by the block's DAA score and raises the carried base fees to its floors; the inspector carries the block's pgas table (modexp reads it); next_base_fee takes floor and denominator; the pool's add and select and every RPC quote use the tip's DAA score plus one; the shard plan uses the segment's; one new executor test |
igneum/exec/src/{config,executor,pgas,pool,proving,rpc,service}.rs |
On top of that, two more fork commits:
| Commit | What |
|---|---|
| cf369022 | DEV_FEE_ADDRESS = 0x7F45d7d7272e57639BeBb739A60B05bB2CD4C126, the release dev-fee payout address (testnet and mainnet; Josh, 5 October 2026, EIP-55 checksum verified, on file at ~/.config/igneum/dev-fee-release.json); DEV_FEE_ADDRESS_DEVNET unchanged; the placeholder test replaced by the_release_address_pays_the_fee_outside_the_devnet (cargo test -p igneum-miner dev_fee: 5 passed). docs/design/miner-dev-fee.md updated on this branch |
| 2b6d23ef | the merge of fork miner-latency (section 2), the final tip of release-0.3.6 |
1c. What is out
| Branch | Why |
|---|---|
the app repository's local miner-latency (dd47ff3, worktree igneum-wt-latency: tools/miner-latency/run.mjs, docs/plans/miner-latency.md, two bench-log entries; never pushed to origin) |
not part of this task; it conflicts with testnet-adopt only in docs/bench-log.md (both appended entries) and can be merged at any time. The harness was run from that worktree for section 2 |
2. The miner-latency decision: IN
Fork miner-latency (0f88b6d6, vendor/igneum-node-latency, one commit over devnet-v4: the NewBlockTemplate
subscription in the miner, node-side template prewarm, workers switching without draining the batch, --poll-templates
and --job-ms) was merged on a side branch first (release-0.3.6-latency, worktree vendor/igneum-node-036-lat,
c07093ce) and landed on release-0.3.6 (2b6d23ef) only after both gates passed on the merged tree:
| Gate | Command | Result |
|---|---|---|
| Miner tests | cargo test -p igneum-miner on c07093ce |
ok: 15 passed, 0 failed |
| 3-node fast-time run | node tools/miner-latency/run.mjs --mode subscribe (from igneum-wt-latency, IGNEUMD and IGNEUM_MINER = the release build of c07093ce in vendor/igneum-node/target-036-lat/release, under the run lock, 08:11 to 08:16Z) |
3 nodes up (peers 2/1/1 throughout), 3 CPU miners at 2 threads for 295 s: 82 + 81 + 80 blocks accepted, 0 rejected; chain blocks 228, merged blue 15, merged red 0, tips mean 1.03 max 2, sinks agree; 236 to 241 template switches per miner, notified p50 14 to 16 ms, template p50 20 to 24 ms, switched p50 46 to 52 ms (p90 118 to 130 ms, max 194 to 372 ms); node prewarm 52 to 56 builds a minute, mean 3 to 4.5 ms; 1 to 2 stale submits per miner; no error, panic or WORKER FAULT line in any miner log |
| The dev fee through the new feed | node tools/dev-fee/run.mjs --secs 180 on the same binaries (two nodes, three stub miners, two at the default fee and one at --dev-fee 0, then igneum-miner payouts; 08:17 to 08:20Z) |
pass: miner-a found=327 fee=4, miner-b found=301 fee=4, control miner-c found=309 fee=0 (0 dev-fee block lines); payouts on both nodes: 938 blocks, the devnet dev address 8 blocks = the miners' counters, 1.27% of the fee-paying miners' blocks against the 1% expectation |
The conflicts (13 hunks in igneum/miner/src/main.rs, the 0.3.5 dev fee and X21 mismatch guard against the latency
rewrite of the template path) were resolved by keeping both sides: the feed takes the dev fee and requests one
template in 100 with it, as the 0.3.5 fetcher did; the continuous CPU loop carries the fee through Work.dev_fee
and counts fee blocks; the found path runs check_found into the mismatch guard; WorkEngine::Real holds the
day-keyed EpochRef of M30. The 3-node run above showed fee=0 on every miner: 243 blocks found at 1 in 100
templates expects about 2.4 fee blocks, and zero has a probability of about 9%, so the dev-fee harness was run as
the third gate; it showed the fee riding through the new feed (8 of 8 fee blocks on chain, the control at 0).
3. Test results, with the command
Every build ran through the main checkout's lock, /Users/joshm/Projects/igneum/tools/lock/with-lock.sh build, at
nice -n 19 with -j 4. The fork's builds used CARGO_TARGET_DIR=vendor/igneum-node/target-036, an APFS clone of
target-release (48 s to clone). The rustup cargo (1.99.0) on ~/.cargo/bin.
3a. The app repository (worktree igneum-wt-adopt)
| Check | Command | Result |
|---|---|---|
| Site | node site/build.mjs |
built: bench, journey (40 entries), index, litepaper, live, evidence, wallet, 404, miners (6 rows) |
| Links | node tools/ci/link-check.mjs |
324 internal links across 8 pages, 0 broken |
| Workflow shell | node tools/ci/check-workflow-shell.mjs |
self-test fires on the fixture; 11 run blocks in 2 workflows, 25 .ps1 files, 0 findings |
| Signed inputs | packaging/windows/test-inputs-signing.sh with igneum-ota-sign built from this tree (cargo build --release -j 4 --bin igneum-ota-sign, 8 s with the cloned target) |
16 passed, 0 failed: the four positive cases, the ten refusals (changed zip byte, changed manifest byte, changed unpacked file, unlisted file, missing file, wrong pin, short pin, another key, the embedded key against the throwaway signature, the two sign-time refusals), and the Mac's real OTA key verified by the key compiled into the app |
| Shell parses | bash -n on push-inputs.sh, fetch-ci-artifacts.sh, inputs-manifest.sh, test-inputs-signing.sh |
ok |
| JavaScript parses | node --check on tools/ci/check-workflow-shell.mjs, tools/ci/link-check.mjs, site/build.mjs |
ok |
PowerShell parse rule (no $var: inside double quotes) |
the 0.3.5 scanner over every tracked .ps1 and the PowerShell strings in ota.rs, jobrun.rs, jobbuild.rs |
the 3 baseline hits of 0.3.5 only (two comments in check-ps51.ps1 that quote the rule; ota.rs line 1190, the Mac helper's bash); testnet-prep changed no .ps1 |
3b. The node fork (worktree vendor/igneum-node-036)
| Fork tip | Command | Result |
|---|---|---|
| a11455e7 (testnet-params merged, the fee switch) | cargo test -p kaspa-consensus-core with IGNEUM_FAST_TIME_FILE=<igneum-wt-adopt>/infra/fast-time/override-60x.json (the fast-time test wants every override field; the main checkout's file predates fees) |
ok: 101 passed, 0 failed, 2 ignored (lib) + 7 (db_compat); among them igneum_testnet_identity, override_params_carry_the_fees, override_params_carry_the_fees_v1_activation, consensus_digest_keeps_the_0_3_5_value_until_the_fee_switch_is_set (pins 9409dedac4bf... for the devnet params), consensus_digest_covers_every_consensus_field_and_nothing_else (the fee switch in its list), test_genesis_hashes, fast_time_60x_file_is_the_devnet_at_60x; 07:41Z |
| a11455e7 | cargo test -p igneum-exec |
ok: 11 passed, 0 failed; among them the_fee_switch_meters_by_the_block_daa_score (a block below the switch meters 200 intrinsic pgas and keeps its base fees; the block at the switch meters 300 and its base fees jump to the v1 floors) and the bomb tests re-sized to fit both tables; 07:43Z |
| a11455e7 | cargo test -p kaspa-consensus --features igneum-pow -- finality |
ok: 8 passed, 0 failed (92 filtered), 125.9 s |
| a11455e7 | cargo test -p kaspa-consensus --features igneum-pow -- difficulty |
ok: 15 passed, 0 failed |
| a11455e7 | cargo test -p igneum-miner |
ok: 12 passed, 0 failed (the guard and dev-fee tests) |
c07093ce (miner-latency merged on the side branch release-0.3.6-latency, worktree vendor/igneum-node-036-lat, CARGO_TARGET_DIR=target-036-lat) |
cargo test -p igneum-miner |
ok: 15 passed, 0 failed (the 12 above and job_size_follows_the_target_ms, cpu_found_nonce_is_matched_to_its_own_work, worker_found_hash_is_matched_to_its_own_template) |
| c07093ce | 3-node fast-time run (section 2) | pass (section 2) |
| cf369022 (the release dev-fee address) | cargo test -p igneum-miner dev_fee |
ok: 5 passed |
| 2b6d23ef (the final tip: cf369022 plus the latency merge) | cargo test -p igneum-miner |
ok: 15 passed, 0 failed |
Not built here: the release binaries for the three platforms (section 4b).
4. The morning cut, in order
4a. The signed inputs come first (G13)
The workflow and push-inputs.sh changed as a pair. From this tree on, .github/workflows/windows.yml fetches
payload-inputs.json.sig from the downloads host and refuses to build until igneum-ota-sign verify-inputs embedded
accepts the manifest's signature, the zip's sha256, every unpacked file and the node commit pinned in
packaging/windows/node-source.pin in the commit it builds. The downloads host holds no signature today (0.3.5
pushed an unsigned inputs.json and a bare .sha256), and the pin in this tree still reads 6aa69a45 (the 0.3.4
node, written by the old script). So the FIRST push of the new workflow would fail at "payload inputs" unless the
inputs are pushed from this tree before the workflow runs:
cd /Users/joshm/Projects/igneum-wt-adopt
gh auth switch --user igneum-josh
packaging/windows/test-inputs-signing.sh # 16 of 16, with the real key at the end
IGNEUM_NODE_SRC=vendor/igneum-node-036 IGNEUM_WIN_RELEASE=<the 0.3.6 Windows release folder> \
packaging/windows/push-inputs.sh --deploy # signs payload-inputs.json, writes node-source.pin
git add packaging/windows/node-source.pin && git commit -m "inputs: pin node <commit> for 0.3.6"
What the workflow needs, and where it comes from:
| Need | Source |
|---|---|
DL_TOKEN repository secret |
already set for 0.3.5 (gh secret set DL_TOKEN < ~/.config/igneum/dl-token); the workflow fetches the zip, the manifest and the signature with it |
| the embedded public key | compiled into the app (app/igneum-app/src/manifest.rs); igneum-ota-sign embedded prints it; push-inputs.sh refuses to sign if ~/.config/igneum/ota-signing-key.pub is not that key |
~/.config/igneum/ota-signing-key (0600) and .pub |
the OTA key the apps already trust; signs the manifest on the Mac |
packaging/windows/node-source.pin |
written by push-inputs.sh, committed with the push; the runner compares it with the manifest's node_source_commit |
| the Windows node exes | IGNEUM_WIN_RELEASE names the folder with igneumd.exe and igneum-miner.exe built from the fork commit the pin names (the 0.3.6 cross-build; proto-cuda/windows-node/cross-build.sh as in 0.3.5) |
Then the update manifest is signed only on request: packaging/windows/fetch-ci-artifacts.sh --sign-manifest <run-id> --deploy after the run is green, which re-verifies the run's inputs artifact against the key and the pin at the
run's commit before publish-manifest.sh runs.
4b. The merge and the ship
From the MAIN checkout, on master, after 4a's pin commit is on testnet-adopt:
cd /Users/joshm/Projects/igneum
gh auth switch --user igneum-josh
git fetch origin && git checkout master && git pull --ff-only
git merge --ff-only origin/testnet-adopt # testnet-adopt contains master 2054ae3
node tools/ship-app.mjs --check # 0.3.5 in all 6 files
node tools/ship-app.mjs 0.3.6 \
--node vendor/igneum-node-036 \
--win-release <the 0.3.6 Windows release folder> \
--mac-release <the 0.3.6 Mac release folder> \
--notes "Testnet identity and the adopted fee table in the node (devnet unchanged until the fee switch), signed build inputs"
The 0.3.6 binaries are NOT built yet (this plan stops at the suites; section 3b says what was and was not built).
Build them from fork 2b6d23ef as 0.3.5's section 4a did: Mac cargo build --release -j 4 -p kaspad -p igneum-miner --features kaspad/igneum-pow with CARGO_TARGET_DIR=vendor/igneum-node/target-036 (the release build in
target-036-lat is of c07093ce, one commit short: no release dev-fee address); Windows through
proto-cuda/windows-node/cross-build.sh; Linux through infra/cross/build-linux.sh. The push to master triggers
windows.yml, which now verifies the inputs of 4a before it builds the app.
5. The devnet rollout of the fee floor: a height switch
The live devnet runs the prototype fee set (B_p 30,000,000, 1 gwei floors, intrinsic 200). A 0.3.6 node on the
devnet runs exactly that until told otherwise, so 0.3.5 and 0.3.6 nodes build and accept the same segments through
the whole rolling update. The adopted set reaches the devnet by fees_v1_activation_daa, the same pattern as
difficulty_v2_activation_daa (33,000, 4 October 2026):
| Step | What | Check |
|---|---|---|
| 1 | Ship 0.3.6 (section 4). Every node updates with the switch at its default, never. The consensus digest does not move (the fee fields enter the digest only once the switch or the set leaves the 0.3.5 state), so 0.3.5 and 0.3.6 peers keep handshaking | Consensus params digest: 9409dedac4bf... on a node with no override file, the same line 0.3.5 printed; the live devnet nodes (override file with the difficulty and finality switches) print their own unchanged value |
| 2 | Wait until every devnet node is on 0.3.6: the console's Machines card and the user agents in the peer list (igneumd/2.1.0-<0.3.6 fork commit>) |
no -20139145 user agent left |
| 3 | Pick the switch height: a DAA score at least 24 hours ahead (86,400 blocks), on a round number, announced in the engineering log and the app's update note | |
| 4 | Publish the switch with the other switches in the update manifest's consensus.override (packaging/ota/publish-manifest.sh --override '{"difficulty_v2_activation_daa": 33000, ..., "fees_v1_activation_daa": N}' --deploy), and add the line to every hand-run node's override file. The digest moves the moment a node restarts with it; a node that has not restarted is refused by those that have, which is why the restart must sweep every node before N |
Calibrated v1 fees from the override file: ... from DAA score N in every node's start-up log; one digest across the peer list |
| 5 | At N: the first chain block at or above N meters with v1 (intrinsic 300, B_p 120,000, S_p 30,000, modexp 10 + 1 per 10 bytes) and its base fees jump to 100 gwei per gas and 10,000 gwei per pgas. Nothing resets; history stays |
igneum_getBudgets returns provingGasLimit 120,000 and the two base fees at the floors; eth_getBlockByNumber of the switch block shows provingBaseFeePerGas 0x9184e72a000 |
| 6 | The prover's mirror of the table (proving/igneum-prove/core/src/config.rs, pgas.rs: prototype values at b7fca5a0) must carry v1 before N, or every shard statement after N differs from the node's plan. This is the one change this plan does not make; it re-cuts every fixture at S_p 30,000 and changes the guest program id |
the fixtures and the guest id in docs/analysis/base-fee-floor.md section 4 |
Why a switch and not a fresh chain: the devnet has 12 cloud nodes, three PCs and outside machines on it, with the
difficulty v2 rollout as the precedent that a height switch over a live chain works. Why not the override file's
fees object: that changes the rules of every block including the past, so a node restarted with it could not
replay its own history.
6. Open after this plan
| Item | State |
|---|---|
--netsuffix default 1 under --testnet |
one line in kaspad/src/args.rs, not done; igneumd --testnet --netsuffix 1 until then |
| the testnet genesis message | reads proposed, not final (the proposal's words, now part of the hashed genesis 52a3e6a9...). Adopted as computed. If the owner wants the words changed, it is one print_genesis_hashes run, new hash and merkle root in genesis.rs, test_genesis_hashes and the README, and it must happen before the first public node, never after |
| the prover's table mirror | section 5 step 6 |
| seed nodes, public RPC, the explorer, the app's testnet build | docs/testnet/README.md section 5, unchanged |
| G14 history rewrite | docs/plans/history-rewrite.md, merged as a plan, not acted on |
| the inputs push (4a) and the 0.3.6 binaries | not done here |
7. What the proving activation taught us (written on master, 5 October 2026, 08:45 BST)
Written 5 October 2026, 08:45 BST, while proving v0 went live on the devnet at DAA 84,100.
Must ship in 0.3.6
| Item | Why | Where |
|---|---|---|
The app sets IGNEUM_PROOF_VERIFIER for its node |
On 5 October no node on the network ran a verifier (verifier: Off on every app node and the seed). Spec 7.7 item 4: a producer without a verifier never includes a record, so proofs were stored and never paid. The Mac app was relaunched by hand with the variable; Windows apps cannot be. |
app/igneum-app/src/engine.rs node spawn: Mac and Linux point at the shipped igneum-prove-host; Windows points at a small wrapper exe that runs the WSL host (wsl -d Ubuntu-24.04 -- /opt/igneum/igneum-prove-host --mode verify ... with wslpath for the proof file). Until the wrapper exists, devnet Windows nodes run IGNEUM_PROOF_VERIFY=trust (devnet only, never testnet). |
Windows payload ships wsl2/bin/igneum-prove-host and igneum-prove-export |
The app's probe looks there first; today a PC needs the 20-minute WSL setup or a hand-installed /opt/igneum. The Linux binaries come from infra/cross/build-linux.sh (CUDA feature needs the CUDA toolchain in the cross image, else ship the CPU build and let setup-wsl.sh build the CUDA one). |
packaging/windows/make-payload.sh, Igneum-Miner.iss |
| The proving tile shows the verifier state | A node that relays but never includes should say so on the tile and on /live. |
app/igneum-app/src/state.rs, site/api/live.mjs (verifier is already in the observer report) |
| Rotation phase 2 | Branch rotation-2 (5317305): --dl-both, tools/logs.mjs --rotation, fresh-repo script. Plan: docs/plans/rotation-phase-2.md. |
|
Testnet parameters behind fees_v1_activation_daa |
Branch testnet-prep and the fork's testnet-params (agent in progress). |
Done (5 October 2026, branch proving-app, app side only; the node is unchanged)
| Item | Done | Commit |
|---|---|---|
The app sets IGNEUM_PROOF_VERIFIER for its node |
app/igneum-app/src/verifier.rs decides once per node start and engine.rs passes it to the igneumd spawn. macOS and Linux: igneum-prove-host next to the engine's binaries (the DMG's Contents/Resources/bin). Windows: the new igneum-prove-verify.exe (src/bin/prove-verify.rs, a bin target of the app crate, shipped by make-payload.sh next to the engine) is set only when its --probe finds a host inside WSL2; it rewrites --proof with wslpath -a, runs the host in the order of src/wslhost.rs and returns its exit code, 2 when there is no host. Trust mode is never the default: the setting proof_verify_trust (Settings, "devnet only") sets IGNEUM_PROOF_VERIFY=trust only when no verifier was found, and changing it restarts the node. After the WSL2 setup runs on a PC, the prover thread asks for one node restart so the verifier is picked up. |
063a9e7 |
| The prover's WSL2 probe checks both layouts | Order: the payload's wsl2/bin, ~/igneum-prove/proving/igneum-prove/target/release/igneum-prove-host (what setup-wsl.sh builds), ~/igneum-prove/target/release/igneum-prove-host, /opt/igneum/igneum-prove-host; one list in src/wslhost.rs, shared with the wrapper. The tile's message names every path it looked at, and says when WSL2 did not answer. |
063a9e7 |
| The proving tile shows the verifier state | The prover thread reads igneum_getProvingStatus().verifier every 30 s whether proving is on or off; /api/state carries proving.verifier (the node's words), verifier_mode (off, trust, command, unknown), verifier_set (what the app passed), verifier_reason, verifier_note and the pool counts. The tile has a verifier row and a note: "This node relays proofs but does not verify them, so it never includes a proof record in its blocks: ", "Devnet only: this node trusts proof records without verifying them", or "This node verifies proof records with igneum-prove-host". site/api/live.mjs already carried verifier; untouched. |
063a9e7 |
The three items are one commit because they share src/prover.rs and src/state.rs. Not done here: the Windows payload's wsl2/bin host binaries (item 2 of the table above, needs the Linux cross-build), rotation phase 2, testnet parameters. The version in app/igneum-app/Cargo.toml is still 0.3.5; the ship script bumps it.
Operational lessons from the activation (5 October 2026)
- Consensus override changes must land on every node at once: a hand node restarted early with a different
proving_v0_activation_daawas refused by every peer (digest handshake) and sat isolated at a lower height for 20 minutes. Order that works: publish the manifest override,update-nowto every app, wait for every app node to log the new parameters, then restart the hand nodes and the seed with the same file. scratchpad/restart-hand-nodes.shdied silently afterigneumd --version(the 0.3.5 binary exits 1 after printing) underset -e; the restart it reported never happened. Every restart script ends by printing the new pids and their start times.- Switching proving on needs no app restart:
POST <app.url>/api/prove {"on":true}(the jobprove-on-pc2-84100does this after installing the CUDA host into/opt/igneumfor the app's WSL user).
Instant jobs (5 October 2026)
Josh: "why is it taking so long for pc2 and pc1s tasks to spin up? can we speed it up?". Before 0.3.6 every app polled
igneum-jobs.json every 10 minutes (CHECK_EVERY_S = 600), so a job published from the Mac waited up to 10 minutes
on each PC. The PCs have no inbound ports and one miner is off the LAN, so the fix is a wake signal the app pulls
over an outbound connection. Branch job-wake.
| Piece | What it does | Where |
|---|---|---|
| Wake endpoint | GET /wake?since=<stamp>: public (the apps hold no token), 30 a minute per IP, held up to 45 s, answers {stamp, at, added, changed, held_ms} the moment the stored stamp differs from since, else the unchanged stamp at the deadline. POST /r/<token>/wake {stamp, added} (the relay's auth) records a stamp. Rows in Neon relay_wake, created by the first POST. maxDuration 60 s. |
relay/api/wake.mjs, relay/lib/wake.mjs, relay/vercel.json, test relay/test/wake.test.mjs (fake database and clock; in CI's site job) |
| App waker | One thread per app long-polls the endpoint with the last stamp; a changed stamp sends Event::Wake and the engine fetches the jobs file at once (signature check unchanged). First reply seeds the stamp. Backoff 5, 15, 60 s while the relay is unreachable, 10 s floor between requests, a full hold when the relay has no stamp yet. One log line per wake, one when it falls back, one when it recovers. IGNEUM_APP_JOBS_WAKE_URL overrides the URL (set and empty: no waker). |
app/igneum-app/src/jobrun.rs (WakeState, wake_loop), unit tests for the stamp and backoff logic |
| Fallback poll | 2 minutes (was 10), the first poll still 40 to 60 s after start. An unchanged file is no longer logged on every poll. | jobrun.rs CHECK_EVERY_S |
| Publisher | After the live file verifies, --deploy POSTs the stamp (published_at plus 8 hex of the file's sha256) and the added id; the token goes in a header file, never on the command line; prints "woke the apps" or a one-line warning. |
packaging/ota/publish-jobs.sh wake_apps |
| Status | node tools/jobs.mjs status shows "woken +N s after the publish" per machine for a job a publish added. |
tools/jobs.mjs |
Expected latency, publish to job start: the relay re-reads the stamp every 2 s inside the hold, the app's curl returns at once, the engine fetches the file (two small downloads) and starts the job on its next tick. About 3 to 6 s when the app is mid-hold, plus up to 10 s if the app was inside its floor after an earlier reply; the 2-minute poll is the ceiling when the relay is down. Numbers below are measured, not estimated.
| Machine | Publish to started_at | Date | Source |
|---|---|---|---|
| PC 1 (ae432dc7) | TODO (measure with node tools/jobs.mjs status after the first 0.3.6 job) |
||
| PC 2 (1ccfe586) | TODO | ||
| Mac | TODO |
Not yet done on this branch: the relay deploy (relay/, by the owner; the /wake route and the 60 s maxDuration
go live with it, the relay_wake table appears on the first POST), the first live publish, and the Windows curl path
of the long-poll (curl.exe 8.x in System32; the 58 s --max-time was reviewed, not run).
8. The cut, 5 October 2026 (release engineer, from 09:30 BST)
Josh at 09:30 BST: "can we push 0.3.6 through?". Worktree /Users/joshm/Projects/igneum-wt-ship036, branch
release-0.3.6 from master 31c1b34. Every build through /Users/joshm/Projects/igneum/tools/lock/with-lock.sh build
at nice -n 19 with -j 4, the rustup cargo 1.99.0 on ~/.cargo/bin. Times are UTC unless marked BST.
8a. The merges, in order
| Merge | Head | Commit | Conflicts and how they were resolved |
|---|---|---|---|
origin/testnet-adopt |
09baf8e |
c9bc6d4 |
docs/spec/05-fees-and-economics.md: master's 5.10 (security budget, E15) kept as 5.10, the adopted fee table from testnet-adopt became 5.11; its cross references in docs/analysis/base-fee-floor.md and this file moved to 5.11. docs/plans/release-0.3.6.md (add/add): testnet-adopt's plan is the body, master's "what the proving activation taught us" follows as section 7. site/index.html, site/journey.json: generated, rebuilt with node site/build.mjs (bench, journey 40 entries, index, litepaper, live, evidence, wallet, 404, miners 6 rows) |
app-ui |
ce31cbb |
356b3e6 |
.github/workflows/ci.yml: both test steps kept (the wake test from master, the notice-strip test from app-ui); node --test app/igneum-app/ui/notices.test.mjs: 6 pass |
proving-app |
010a372 |
ada90aa |
app/igneum-app/src/prover.rs: proving-app's probe taken (the same hidden wsl call as master's 31c1b34, through the shared wslhost::lookup_script). This file: proving-app's Done table kept under section 7 |
rotation-2 |
5317305 |
50920da |
.github/workflows/windows.yml: both header comments; rotation-2's "packaged configuration" step runs first, then master's G13 "payload inputs" step (signature, hashes, node commit); the duplicate dl-token write in the inputs step dropped (the configuration step writes it). node tools/ci/check-workflow-shell.mjs: 12 run blocks, 25 .ps1, 0 findings |
engine.rs, state.rs and ui/app.js auto-merged. job-wake was already in master.
8b. Changes on top of the merges
| Commit | What |
|---|---|
9541543 |
Every process the engine starts on Windows is hidden. The four helpers (detect::run_timeout, jobrun::run_capture, jobrun::run_streamed, procs::spawn) already set CREATE_NO_WINDOW, which is why only 15 of the 89 Command::new sites were visibly wrapped; the direct .output()/.spawn() sites were not: the window host launch (main.rs), icacls and the three reg calls (platform.rs), the setup's cmd /c start (prover.rs), the Mac-only xattr/hdiutil calls (ota.rs, wrapped for a clean audit; quiet is a no-op off Windows). igneum-prove-verify.exe is now a windows-subsystem binary: the fork's igneum/exec/src/proving.rs:498 spawns it with a bare Command::new from a node that has no console, so a console-subsystem wrapper opened a window on every verification. Sites left as they are: inside #[cfg(target_os = "macos")] or #[cfg(target_os = "linux")] blocks (scutil, caffeinate, osascript, open, pkexec, xdg-open) and the test-only bash in manifest.rs |
3811d8e |
Every WSL script runs from a file (coordinator, 09:4x BST; PC 2 on 0.3.5 reported "proving needs the WSL2 setup" with /opt/igneum/igneum-prove-host present: the inline bash -lc "<script>" with [ -x "$f" ] and the "Igneum Miner" path reached bash mangled; the same script from a file found the host, job prove-install-pc2-3 09:12 BST). src/wslhost.rs: script_dir() (%LOCALAPPDATA%\igneum\wsl), write_script (UTF-8, LF, no BOM, #!/bin/bash, unique name per process and call, removed on drop), bash_line (bash [-l] '<file>' '<arg>'..., every word single-quoted, never a double quote or newline), command (on Windows the tail after -- goes on the command line as written, CommandExt::raw_arg, so neither the C runtime's quoting nor the shell inside the distribution re-reads it; arguments reach the script as $1, $2). Converted: the prover probe and run_tool (prover.rs), the setup launch (cmd /c start "" wsl -d <distro> -- bash -l '<setup-wsl.sh>'), the wrapper's probe and run (bin/prove-verify.rs; its wslpath -a round trip dropped for the pure mapping, same exposure), jobrun's six inline scripts (prover-probe, wsl-context, gpu-run, prover-kill, build-kill, free-gb). Unit tests: the line never carries a double quote or a newline; the file has LF endings and no BOM and is removed after use; the wrapper's run script ends in exec "$h" "$@" |
7d7570a |
Igneum Miner 0.3.6: ..., the six version files (bumped with the tool's own table, checked by node tools/ship-app.mjs --check: 0.3.6 in all 6) |
a98be35 |
push-build-inputs.sh: the live sha256 check retries for a minute (8c) |
Not changed: the fork's own Command::new for the verifier (no flags; the wrapper's subsystem makes it moot).
8c. Test results, with the command
| Tip | Command | Result |
|---|---|---|
9541543 (merges + hidden windows) |
cargo test -p igneum-app |
ok: 72 (lib) + 26 (bin) + 5 (prove-verify), 0 failed, 08:36Z |
9541543 |
cargo build --release -p igneum-app |
ok, 1 min 32 s (target cloned by APFS from the main checkout: 5.5 s) |
3811d8e (WSL scripts from files) |
cargo test -p igneum-app |
ok: 75 + 26 + 8, 0 failed, 08:47Z |
3811d8e |
cargo check --target x86_64-pc-windows-gnu -p igneum-app |
ok (raw_arg and the subsystem attribute compile for Windows) |
3811d8e |
cargo build --release -p igneum-app |
ok |
a98be35 |
packaging/windows/test-inputs-signing.sh (signer from this tree) |
16 passed, 0 failed, the Mac's real OTA key verified by the key compiled into the app |
a98be35 |
node tools/ship-app.mjs 0.3.6 --node vendor/igneum-node-036 --branch release-0.3.6 --dl-both --dry-run |
preflight read everything; the only problems were the two Windows exes not yet built (8e) |
a98be35 |
bash -n packaging/windows/push-build-inputs.sh |
ok |
8d. Secrets (values never shown)
| Where | Name | Set | Listing |
|---|---|---|---|
GitHub repo igneum-network/igneum |
LOG_INTAKE_KEY |
08:36:35Z from ~/.config/igneum/log-intake-key (tr -d '[:space:]' | gh secret set) |
gh secret list: DL_TOKEN (04 Oct), DL_TOKEN_NEXT, LOG_INTAKE_KEY, LOG_INTAKE_KEY_NEXT |
| GitHub | LOG_INTAKE_KEY_NEXT |
08:36:36Z from log-intake-key.next |
same |
| GitHub | DL_TOKEN_NEXT |
08:36:37Z from dl-token.next |
same |
Vercel project igneum (team igneum, linked from site/ in the shared checkout) |
LOG_INTAKE_KEY_NEXT (production) |
08:3xZ from log-intake-key.next |
vercel env ls --scope igneum: DATABASE_URL, LOG_INTAKE_KEY, LOG_INTAKE_KEY_NEXT (the function reads it at the next production deploy, which the push to master is) |
8e. Node binaries from fork 2b6d23ef
| Platform | Path | sha256 | Size | Build |
|---|---|---|---|---|
| Mac arm64 igneumd | vendor/igneum-node/target-036/release/igneumd (copied to vendor/igneum-node-036/target-integration/release/) |
64138a1760ebcd5809582b26c7ba2c1e1c1f94f2a8e217287e2d71f97aa4a40b | 40,968,080 | CARGO_TARGET_DIR=vendor/igneum-node/target-036 cargo build --release -j 4 -p kaspad -p igneum-miner --features kaspad/igneum-pow, 8 min 26 s, done 08:45:58Z; strings: 2b6d23ef present, 20139145 absent; igneum-miner --help lists --dev-fee, --poll-templates, --job-ms |
| Mac arm64 igneum-miner | same folder | ef438be2eaaaedfa767408a319ec5b9dd5178d51e264e618d5a8207f1044b195 | 8,514,864 | same |
| Windows x86-64 igneumd.exe, igneum-miner.exe; Linux igneumd, igneum-miner, igneum-app | PC 1 build job (8f) | see 8f |
8f. The PC build job (Windows and Linux)
IGNEUM_WIN_RELEASE=<main checkout>/vendor/igneum-node-036/target-integration/x86_64-pc-windows-gnu/release node tools/build-job.mjs run --node vendor/igneum-node-036 --target ae432dc7 --targets linux,windows --budget-minutes 45 --title "0.3.6 build on PC 1" from the worktree root (a vendor symlink to the main checkout's vendor/ makes the relative paths resolve; vendor/ is ignored by git). take3 (release-0.3.5, 20139145) had finished at 08:30:50Z: every stage ok, 6 files uploaded, 5 min of 45.
| Attempt | Published | What happened |
|---|---|---|
| 1 | 08:47:41Z | push-build-inputs.sh packed the fork (2b6d23ef) and the app (0.3.6), deployed the downloads folder (build-inputs.zip 8,206,587 bytes, e46ae8d74b892097296f3fc3b4a44ce2b0960f6d3578cf00eccb1fed0c82fcbc) and then read the live build-inputs.sha256 once, straight after "Aliased": the edge still served the previous file, the check failed and nothing was published. A curl a minute later served e46ae8d7. Fix a98be35: six tries, 10 s apart (publish-jobs.sh already retried) |
| 2 | 08:51:35Z | build-20261005-085135 published and the apps woken (stamp 2026-10-05T08:51:35Z.852c054f). PC 1 runs 0.3.5, which has no waker (job-wake landed after the 0.3.5 cut): the 10-minute poll applies unless Check now is pressed. Started 08:54:43Z. FAILED: linux node exit 101 after 112 s, windows node exit 101 after 129 s; the app built on both (5 s and 6 s), the test stage passed (igneum-miner, igneum-app), nothing uploaded; 4 min of 45. The uploaded report holds only STAGE and RESULT lines, so the full app/jobs/build-20261005-085135/job.log (98,325 bytes) was fetched with a collect job (collect-20261005-090521, published 09:05:21Z, run 09:08:57Z, woken +193 s). The errors: kaspad/src/daemon.rs no field fees_v1_activation_daa on type OverrideParams, no field fees on type Params, no method install_fee_params (8 errors); kaspa-rpc-service no method prewarm_block_template on MiningManagerProxy. The zip's sources are right (params.rs carries the field 26 times, mining/src/manager.rs prewarm_block_template 5 times). Root cause: PC 1 keeps /root/igneum-build/target between jobs, unzip restores the Mac's mtimes (params.rs 07:38Z, executor.rs 07:42Z), and take3 had built 0.3.5 at 08:25 to 08:30Z, so cargo judged kaspa-consensus-core and igneum-exec fresh (neither printed "Compiling") and linked the new kaspad and kaspa-rpc-service against the cached 0.3.5 crates. Fix: master ea9794d (push-build-inputs.sh stamps every staged file before zipping), cherry-picked as c26d6be; and the PC side, 021a715 (jobbuild.rs extract stage: find "$B/src" -type f -exec touch {} + after the unzip, the zip root and the sibling igneum-pow, with a unit test on the order unzip, stamp, manifest check), which the 0.3.6 app carries so a packer regression cannot repeat this |
| 3 | 09:12:54Z | build-20261005-091254 (zip 7c63df808331de5893ef93e5d3f2be65d59998668d1f99325751c5fbed033066, 8,206,958 bytes, the stamped tree; the live sha256 check passed on try 1), budget 60 min for the full rebuild, apps woken (stamp 2026-10-05T09:12:54Z.4fa446f5). Started 09:14:00Z (the 10-minute poll), done 09:20:33Z, 393 s, every stage ok: linux 154 s, windows 187 s, test 20 s (igneum-miner and igneum-app, the app's 74 + 25 + 8), pack 2 s, 6 files uploaded (38 MB). node tools/build-job.mjs run kept printing "still running" after the SUMMARY said done (its watcher missed the final state; CLAUDE.md's rule on watchers), so the outputs were fetched with node tools/build-job.mjs fetch build-20261005-091254: 6 of 6 verified against the PC's sha256 lines, the three Windows PE headers checked, placed under vendor/igneum-node-036/target-integration/x86_64-pc-windows-gnu/release/, app/igneum-app/target/x86_64-pc-windows-gnu/release/ and infra/cross/out/ |
| Binary (fork 2b6d23ef, PC 1 job build-20261005-091254) | sha256 | Size |
|---|---|---|
| Windows x86-64 igneumd.exe | d08404c20397fefcc02cd56cad0dd4d29b427a468fd89e3189435f7c3431cd7a | 49,971,712 |
| Windows x86-64 igneum-miner.exe | 8bdb4c6e64190b73ae88cd893c3c2efd8fb0e226b43c0c240c1d545206ed7147 | 10,725,888 |
| Windows x86-64 igneum-app.exe (the PC's build; the installer's engine is the GitHub runner's) | 880e7c81acc1c1a532b8c1b244c70c40406f2b8c4e1420392a2dd8edd7f236ed | 2,834,944 |
Linux x86-64 igneumd (infra/cross/out/, for the seed and HiveOS, not in the app) |
c24fd2c5e8c4f976e2b3abc55873bca29ae6b97b47046c946f779d3a04947025 | 48,733,480 |
| Linux x86-64 igneum-miner | 038fcf6b133d0af36d17e28fe822c5bcc7055429958dc280e75459c4347f9a53 | 9,607,440 |
| Linux x86-64 igneum-app | db73814f87c07ee3cada1fec5d728062ae8a8fa3a318fdd94b99f1e6113134fb | 2,297,160 |
PC 2's cold build of the same source gave the same igneum-miner (038fcf6b...) and igneum-app (db73814f...) but another igneumd (d9d227a9... against c24fd2c5...): the daemon's build is not reproducible across the two machines (unverified why; build paths or timestamps are the usual causes). Not acted on.
Standing rule from Josh during the cut (CLAUDE.md f378aa0): builds and test suites run on the PCs, the Mac builds only
the macOS binaries. So the fork's suites and the app's tests went to PC 2 as a second build job:
build-20261005-090600 (09:06:00Z, target 1ccfe586, --targets linux, budget 60 min, its own zip
build-inputs-tests.zip f60713b133ec7b1b61895bf51f2e82d765e96719c2c1b2bcf1e51aa79c52de45 packed with
--node-tests "kaspa-consensus-core igneum-exec kaspa-pow kaspa-consensus igneum-miner" --app-tests "igneum-app"; the
build job cannot skip the build, so PC 2 builds the Linux node first and then runs cargo test --release per package).
PC 2 has no build cache, so its zip did not need the stamp. The test stage cannot pass --features igneum-pow or a
filter, so kaspa-consensus runs its whole suite without the feature; the feature-gated finality and difficulty runs
are the adopt agent's results in 3b. The app tests had already run on this Mac before the rule arrived (8c).
PC 2 result (build-20261005-090600, started 09:09:09Z, 399 s): Linux node build ok in 219 s from a cold cache
(igneumd d9d227a93148cf766969cd2856a4b73d0daa13e855f3e4d02f5b06f33fe4428e 48,733,480; igneum-miner
038fcf6b133d0af36d17e28fe822c5bcc7055429958dc280e75459c4347f9a53 9,607,440), Linux app ok in 7 s. Tests, 141 s for
the node packages: kaspa-consensus-core, igneum-exec, kaspa-pow, igneum-miner all passed; kaspa-consensus
92 passed, 2 failed, 3 ignored. App tests ok: 74 (main) + 25 (ota-sign) + 8 (prove-verify), 0 failed, 5 s.
| Failing test | Where | Why |
|---|---|---|
processes::finality::tests::frozen_table_holds_a_side_without_the_other_keys_for_one_window |
the test helper mine, finality.rs:1588: validate_and_insert_block(...).await.unwrap() |
PowCacheQueueFull("pow cache build queue full (4 waiting, 2 building) for epoch edc4fa84... day 1243883") |
processes::finality::tests::reorg_past_an_unlocked_checkpoint_re_determines_it_and_verifies_the_pending_certificate |
the same line | the same error |
The rejection is M30's PoW cache build queue (0.3.5, fud-consensus: at most 2 caches building and 4 waiting per
process). The whole kaspa-consensus suite runs its tests in parallel on PC 2 and more than six of them build a
cache at the same moment; the Mac runs of 3b filtered to the finality module (8 tests) and never queued that many.
Whether it is pre-existing is being settled the way the coordinator asked: two more PC 2 jobs with
--node-tests kaspa-consensus and nothing else, build-20261005-091739 on release-0.3.5 20139145
(zip build-inputs-t035.zip d1dd841d...) and build-20261005-091739-036 on 2b6d23ef (build-inputs-t036.zip
c01a7525...), published 09:17:39Z and 09:18:06Z (the second add within the same second had collided with the
first's generated id; --id fixed it).
| Job | Tree | Result |
|---|---|---|
build-20261005-091739 |
release-0.3.5 20139145 (the SUMMARY names it) | started 09:19:09Z, 203 s: Linux node built in 78 s, cargo test --release -p kaspa-consensus: 94 passed, 0 failed, 3 ignored, 110 s. The whole suite alone passes on 0.3.5 |
build-20261005-091739-036 |
2b6d23ef | NOT a valid run: both zips were packed at 09:17Z, before the first job built 0.3.5 into PC 2's target dir at 09:20Z, so this job's stamps were older than that build and cargo kept the 0.3.5 crates (the same class as 8f attempt 2, now from the two zips of one pair): Linux node exit 101 after 16 s, and its "94 passed" in 2 s came from the cached 0.3.5 test binaries |
So what is known: the two finality tests pass on 0.3.5 with the suite alone, pass on 2b6d23ef when the finality
module runs alone (3b, the Mac), and fail on 2b6d23ef only in the five-package parallel run on a cold cache
(build-20261005-090600), where the error is M30's cache queue (PowCacheQueueFull, 4 waiting, 2 building), a
per-process limit that a parallel suite exceeds. The owner's call (coordinator, 09:2x BST): a test-isolation
problem, not a consensus regression; recorded here and in the ledger (tests get a per-process PoW cache
directory, 0.3.7); the ship is not blocked. A clean kaspa-consensus-alone run on 2b6d23ef is still owed and goes
after the cut (pack the zip after the previous PC 2 job has built, or wait for the 0.3.6 app's extract stamp).
8h2. The DMG
NODE=<fork>/target-integration/release/igneumd MINER=.../igneum-miner tools/lock/with-lock.sh build packaging/mac/build-dmg.sh
from the worktree, 09:14 to 09:16Z: intake key: log-intake-key.next (32 chars, fingerprint 477bb0ef),
manifest: dl-token.next (11 chars, fingerprint ed9c4d2e), the two fingerprints the rotation plan requires. That first
DMG was 21,038,829 bytes against 0.3.5's 40,027,384: the 0.3.5 DMG (mounted read-only) ships
igneum-prove-host (42,095,744) and igneum-prove-export (2,117,456) in Contents/Resources/bin, which the
worktree lacks (proving/igneum-prove/target is not in git; the main checkout's copy is another build, 55,286,800
bytes). Shipping the Mac node without its verifier would undo 0.3.6's first item, so the two binaries were taken
from the 0.3.5 DMG itself, the files the Mac node runs today (host
4dc6b1c44ccdc83079cbcc5408f9593de9af2680992fea25a00bb58d58c26b84, export
2d1d70f9dcb7724e3e91db55be401a81df7c459b370cc0f4450b178ab7720cd0), and the DMG rebuilt with PROVE_HOST and
PROVE_EXPORT pointing at them (09:18 to 09:19Z, the same two fingerprints printed). The ship tool's dmg step takes
a DMG newer than the bump as built.
| DMG | sha256 | Size |
|---|---|---|
packaging/mac/dist/Igneum-Miner-0.3.6.dmg (build 202610050918) |
fddf3580353d87e0b4f9a910894c507eb5b1cab7b500f4cb9d653db34a7af311 | 40,156,607 |
8g. The digest check (step 4)
vendor/igneum-node/target-036/release/igneumd --devnet --override-params-file=/tmp/igneum-devnet/override-v3.json --appdir=<scratch>/digest-036/appdir --rpclisten=127.0.0.1:60985 --listen=127.0.0.1:60986 --nodnsseed --nologfiles --yes for 20 s (08:48:06Z to 08:48:28Z, then killed; nothing else touched; the override file reads {"difficulty_v2_activation_daa": 33000, "proving_v0_activation_daa": 84100}):
Proving v0 from the override file: provers paid from DAA score 84100
Difficulty rule v2 from the override file: active from DAA score 33000
Fees on igneum-devnet: pgas table v0, B_p 30000000 pgas, S_p 7500000 pgas, floors 1000000000 wei per gas and 1000000000 wei per pgas; calibrated v1 from DAA score never
Consensus params digest: f10a4eab4b1f4f341d54b0b0221161d1122fac302bca90d6b61d14939b69fbd6 (exchanged in the p2p handshake; a peer with another digest is refused)
igneumd/2.1.0-2b6d23ef
The digest equals the live one (difficulty 33000, proving 84100); the fee switch stays "never". Two warnings in the log, neither about the node: UPnP found no free port, and the eth_ RPC port 26790 was held by the live node 1.
8h. The live manifest before the cut (read 08:3xZ from dl/<token>/igneum-app-latest.json)
| Field | Value |
|---|---|
| version, channel, published_at | 0.3.5, devnet, 2026-10-05T07:20:49Z |
| consensus.override | {"difficulty_v2_activation_daa": 33000, "proving_v0_activation_daa": 84100} |
| consensus.activation_height, deadline_note | 84100, "proving v0" |
| fees_v1_activation_daa | absent |
| tuning | absent |
| mac | dmg 2dad2b26..., 40,027,384 |
| windows | inno-setup 0184abec..., 44,447,899 |
publish-manifest.sh carries consensus.override and tuning over from the folder's manifest when not given; activation_height and deadline_note only come from flags, so the ship command passes --activation-height 84100 --deadline-note "proving v0" to keep the consensus object identical field by field.
8i. The ship
The push: Josh pushed release-0.3.6 at 5483322 (09:29Z). windows.yml runs only on master pushes, so the run was
dispatched on the branch: gh workflow run windows.yml --ref release-0.3.6 → run 37290313983, started 09:29:00Z,
green 09:33:40Z (both jobs: the PowerShell and batch parse checks; engine, window host, payload, installer, smoke
run; the G13 inputs step verified the signed payload inputs of 8f against the pin).
Preflight refused the real run on one count, "10 commits behind origin/master" (master had moved to 2766b26: site
pages, litepaper v0.2, the PCs build rule, the packer stamp), so origin/master was merged into the branch (8a79f35)
as the coordinator allowed: site/build.mjs takes master's PRODUCT table and PAGES; master's /wallet is the
product page and the testnet-prep MetaMask page moved to /metamask (in PAGES and the sitemap, the two links on the
home page retargeted); the testnet terms card stays on the home page under master's miner section; litepaper takes
master's Ember wording; node site/build.mjs then node tools/ci/link-check.mjs: 475 links across 10 pages, 0
broken. Those commits touch no app or packaging file, so the installer built at 5483322 is the release; the ship
state file (~/.cache/igneum/ship/0.3.6.json) got sha = 5483322, what the tool's own commit step would have
recorded, and the run resumed with --from ci.
node tools/ship-app.mjs 0.3.6 --node vendor/igneum-node-036 --branch release-0.3.6 --dl-both \
--activation-height 84100 --deadline-note "proving v0" --notes "Instant jobs, a proof verifier for the node, one notice strip, lower miner latency, packaged configuration, hidden helper windows; node 2b6d23ef: testnet identity and the fee table behind a height switch, signed build inputs" \
--from ci
| Step | Result | Time |
|---|---|---|
| preflight | ok (tree 8a79f35 clean, 0.3.6 in all 6, fork 2b6d23ef, both exes, both Mac binaries, both folders, gh igneum-josh, live inputs 2b6d23ef) |
09:31:48Z |
| ci | green after 2 min of polling | 09:34Z |
| fetch | Igneum-Miner-Setup-0.3.6.exe 18.6 MB, igneum-windows-app.zip 26.0 MB (OTA_SKIP=1 CONSOLE_SKIP=1 fetch-ci-artifacts.sh 37290313983) |
|
| dmg | already (8h2) | |
| copy | the DMG into the OLD folder | |
| mirror | 10 files into the NEXT folder, sha256-checked | |
| manifest | consensus.override carried over from the folder's 0.3.5 manifest; both manifests signed (key 8f186e37...), verified locally, same fields |
|
| deploy | one deploy of the downloads folder | 09:34Z |
The live manifests, read 09:35:00Z, compared field by field with the 0.3.5 record of 8h:
| Field | OLD folder | NEXT folder |
|---|---|---|
| version, channel, min_supported_version | 0.3.6, devnet, 0.3.0 | same |
| published_at | 2026-10-05T09:34:06Z | 2026-10-05T09:34:07Z |
| consensus | identical to the live 0.3.5 object: override {difficulty_v2_activation_daa 33000, proving_v0_activation_daa 84100}, activation_height 84100, deadline_note "proving v0" | identical |
| fees_v1_activation_daa | absent | absent |
| tuning | absent | absent |
| mac | dmg fddf3580353d87e0b4f9a910894c507eb5b1cab7b500f4cb9d653db34a7af311, 40,156,607, URL in its own folder | same bytes, URL in the NEXT folder |
| windows | inno-setup ca0fb9197bee6e3a867a33f4e6bcd2c570c9136444e3587ac311d23bfde61531, 19,536,899, URL in its own folder | same bytes, URL in the NEXT folder |
igneum-windows-app.zip cf91659a64afceb0fb4cdb29e7691b5057f1ec264d6fd6d07cd725b23e2cdddc. The 0.3.6 installer is
19.5 MB against 0.3.5's 44.4 MB: the payload no longer carries what the PC builds itself (to confirm from the CI
payload listing; noted, not verified here).
| verify | both folders: the three files HEAD 200 with the local sizes, GET sha256 ok, the manifest signature ok (the first HEAD of the DMG in the OLD folder answered 404 straight after the deploy; the retry 15 s later matched: edge propagation, the same class as 8f attempt 1) | 09:35Z | | console | item #346 "Igneum Miner 0.3.6 shipped (mac+windows)", then sync-dl | 09:35Z |
The tool ended with exit 0 at 09:35:07Z: "manifest 0.3.6 published 2026-10-05T09:34:06Z; every app checks within the hour".
8j. The machines after the publish (manifest live 09:34:06Z)
Watch: node tools/console.mjs machines and node tools/logs.mjs --rotation every 2 minutes from 09:35Z.
Baseline 09:32Z: all five machines (Mac d937c69d, PC 1 ae432dc7, PC 2 1ccfe586, Sam's Mac 3a9bf309, PC 37ba0461)
on app 0.3.5, node 2.1.0-20139145; rotation 0 of 6 on the new key and folder.
| Machine | Seen on 0.3.6 | What its log shows |
|---|---|---|
| PC 1 ae432dc7 (Windows, the first over-the-air update through the fixed helper) | 09:37:31Z watch tick (the first tick after the restart); the 0.3.6 engine's first line is at 09:35:17Z, 71 s after the manifest went live | run win-ae432dc7-20261005-093517: IGNEUM-APP version=0.3.6 machine=ae432dc7 platform=windows; [ok] updated to Igneum Miner 0.3.6 from 0.3.5; config: intake ... key 477bb0ef (packaged); manifest .../dl/<token>/igneum-app-latest.json folder ed9c4d2e (packaged) (the NEXT key and folder: rotation 1 of 6); node proof verifier: command (...\igneum-prove-verify.exe); igneumd started (pid 26572); update check: 0.3.6 is current; wake: listening at https://relay.igneum.network/wake, and at 09:36:47Z update to 0.3.6 complete (from 0.3.5); keeping the previous version for a rollback; 09:37:15Z a wake stamp change fetched the jobs file within seconds (the instant-jobs path live on Windows). The 0.3.5 engine's own download, stage and helper lines were not uploaded before it quit (its last app-log upload was 09:30:14Z; the node log kept uploading to 09:35:14Z): fetched afterwards with a collect job (below). Then the node: the console showed node 2.1.0-20139145 until 09:44Z, then 2.1.0 with no commit and 0.0 MH/s: the new igneumd.exe does not start (section 9) |
| PC 2 1ccfe586 (Windows) | 09:39:51Z watch tick; engine restart at 09:38:40Z (unix 1791193120: [ok] updated to Igneum Miner 0.3.6 from 0.3.5), 4 min 34 s after the manifest went live |
run win-1ccfe586-20261005-093840: the same header and config: fingerprints (477bb0ef, ed9c4d2e); igneumd started (pid 26476) 09:38:53Z, then [error] igneumd exited with code -1073741511 after 6 s; restarting (0xC0000139, STATUS_ENTRYPOINT_NOT_FOUND), again after 3 s, again after 95 s, and so on: the watchdog restarts it every few seconds; update to 0.3.6 complete (from 0.3.5); keeping the previous version for a rollback 09:40:10Z. Mining 0.0 MH/s from the restart (the app marks the update complete on its own health, not the node's: a 0.3.7 item) |
| Mac d937c69d | 09:39:51Z watch tick; engine restart 09:37:35Z (unix 1791193055), 3 min 29 s after the publish | run mac-d937c69d-20261005-093735: [ok] updated to Igneum Miner 0.3.6 from 0.3.5, fingerprints 477bb0ef/ed9c4d2e, igneumd started (pid 81496) from the bundle's bin, node proof verifier: command (.../igneum-prove-host) and ten seconds later node proof verifier reported: command (the first app node on the network with a verifier), update to 0.3.6 complete 09:39:05Z; node 2.1.0-2b6d23ef on the console from 09:39:51Z, mining on |
| Sam's Mac 3a9bf309 | not by 09:48Z (silent 6 min at that tick; its 0.3.5 app checks on its own hourly slot) | |
| PC 37ba0461 | not by 09:48Z (0.3.5, hourly slot) |
Rotation (logs.mjs --rotation): 3 of 6 on the new key and folder from 09:39:51Z (the three machines above), 3 not yet.
9. Incident 09:4x BST and the 0.3.7 hotfix
Both PCs took 0.3.6 over the air (the Windows OTA path works: PC 1's engine restarted 71 s after the manifest went live, PC 2 by 09:39Z), and on both the new igneumd.exe refused to start: "Entry Point Not Found: ZNKSt25__codecvt_utf8_utf16_baseIwE10do_unshiftERiPcS2_RS2 could not be located in igneumd.exe". Mining was down on both PCs; the Mac (d937c69d) ran 0.3.6 with node 2b6d23ef from 09:39Z. The coordinator republished 0.3.5 in the OLD folder (0.3.5 machines stay put; the NEXT folder kept 0.3.6) and Josh approved a DLL swap job (fix-runtime-036) and 0.3.7 through the pipeline.
Cause, read with x86_64-w64-mingw32-objdump -p: every igneumd.exe so far imports libstdc++-6.dll (0.3.5's Mac
cross-build, 197 symbols; the PC build, 198), -C link-arg=-static notwithstanding. 0.3.5 shipped Mac-built exes with
the Mac toolchain's DLL (GCC 16.2.0): a match. 0.3.6 shipped PC-built exes (Ubuntu 24.04's mingw, GCC 13) with the same
Mac DLL, and GCC 16's libstdc++ no longer exports seven symbols the GCC 13 exe imports (five
std::__codecvt_utf8_utf16_base<wchar_t> methods, basic_stringbuf::seekpos, and one more). The -static flag and
the toolchain rule were both assumed, neither checked: the lesson is the gate below.
0.3.7 = release-0.3.6 tip + (commit 7c... "Igneum Miner 0.3.7"):
| Change | Where | Check |
|---|---|---|
| The gate: every symbol an exe imports from a shipped lib*.dll must be exported by that DLL | packaging/windows/check-runtime-dlls.sh, run by push-inputs.sh before signing and by make-payload.sh before the installer is packed (skips with a note where objdump is missing, the runner) |
known-bad (the 0.3.6 PC exes with the Mac DLL): FAILED, 7 missing, the morning's symbol first; known-good (0.3.5's Mac exes with the same DLL): 197 of 197 exported, pass |
| DLLs from the toolchain that linked the exes | push-inputs.sh takes lib*.dll next to the exes first, then the Mac toolchain (make-payload.sh already did); jobbuild.rs windows stage copies the PC toolchain's libstdc++-6.dll, libgcc_s_seh-1.dll (gcc -print-file-name) and libwinpthread-1.dll into the pack with RESULT lines; build-job.mjs accepts the small DLL PEs and places them next to the exes |
unit test on the stage script; the PC path itself runs only once the 0.3.7 app is on PC 1 (the stage scripts come from the app on the PC) |
-static-libstdc++ tried |
proto-cuda/windows-node/cross-build.sh |
measured on the 0.3.7 cross-build below: does the import disappear? |
| version 0.3.7 | the six files | ship-app.mjs --check |
The 0.3.7 Windows exes come from the Mac cross-build (the 0.3.5 way, Homebrew mingw-w64 GCC 16.2.0, target dir
cloned from target-release-win), paired with the same toolchain's DLLs; the PC-built pairing would need the GCC 13
DLLs, which the 0.3.6 app's build job cannot upload. The DMG is rebuilt for the 0.3.7 engine with the same node
binaries (2b6d23ef) and the 0.3.5 prover. The ship publishes to BOTH folders (OLD folder too: the 0.3.5 machines go
straight to 0.3.7).
| 0.3.7 artefact | sha256 | Size | Built |
|---|---|---|---|
packaging/mac/dist/Igneum-Miner-0.3.7.dmg (engine 0.3.7, node 2b6d23ef, the 0.3.5 prover; fingerprints 477bb0ef key, ed9c4d2e folder) |
8ee96b3b054f45721ea08d0e00dfae79fc127fdfa503e05565d117b2c71cd9e2 | 40,156,739 | 09:46 to 09:49Z under the lock |
Windows igneumd.exe, igneum-miner.exe (Mac cross-build of 2b6d23ef with -static-libstdc++ added, CARGO_TARGET_DIR=vendor/igneum-node/target-036-win, cloned from target-release-win; every crate rebuilt because the RUSTFLAGS changed) |
see below | from 09:43Z |
The helpers' own logs (app/ota-apply.log, collect job collect-20261005-093948, published 09:39:48Z, run by the
three 0.3.6 machines within four minutes; Sam's Mac had not fetched it by 09:50Z):
| Machine | Helper line | Then |
|---|---|---|
| PC 1 ae432dc7 | 2026-10-05T10:35:10 apply : engine 24000 installer '...\updates\Igneum-Miner-Setup-0.3.6.exe' version 0.3.6 (the engine keeps mining until the installer runs), installer sha256 verified (BST; 09:35:10Z, 64 s after the manifest went live) |
the installer ran, the 0.3.6 engine's first line at 09:35:17Z: the whole Windows path, check to restart, took 71 s and the helper launched with CREATE_NO_WINDOW alone did run (the 0.3.0 to 0.3.4 stall is gone) |
| PC 2 1ccfe586 | 2026-10-05T10:38:34 apply : engine 14028 installer '...Igneum-Miner-Setup-0.3.6.exe' version 0.3.6, installer sha256 verified (09:38:34Z; its log also holds the 0.3.4 attempt of 4 October: sha256 mismatch, the stall of that day) |
engine restart 09:38:40Z |
| Mac d937c69d | 2026-10-05T09:37:30Z apply: engine 63267 host 63263 ... version 0.3.6, staged bundle digest verified, swapped; opening /Applications/Igneum Miner.app, 0.3.6 is running |
engine restart 09:37:35Z |
Correction from the PCs (coordinator, 09:5x BST): with the matching GCC 13 DLLs swapped in, the PC-built igneumd.exe
(1dcac192...) prints --version and satisfies all 182 libstdc++ imports, yet exits within 2 s of "Logs to console
only" on PC 1 on the real database and on a scratch appdir (job probe-node-pc1-036), no stderr, empty exit code: a
loader or abort, not a Rust panic. So the PC-built Windows node is unusable at runtime, not only mislinked, and does
not ship. 0.3.7's igneumd.exe and igneum-miner.exe are the Mac cross-build (2b6d23ef, this Mac's mingw GCC 16.2.0)
with that toolchain's DLLs, packaged exactly as 0.3.5 was. Both PCs are being rolled back to 0.3.5 with the silent
installer (job rollback-035-pcs) and will read the OLD folder, so 0.3.7 goes to both folders.
Open item for 0.3.8: why the PC-built node dies at start. Reproduce under WSL with wine or on PC 2 with a scratch run;
compare Ubuntu's mingw (GCC 13, posix threads) with the Mac's (GCC 16); check -static against rocksdb's thread
model. Until then the PC build job builds Linux binaries and tests; the Windows node comes from the Mac cross-build.
Watch end, 09:59:38Z (every 2 minutes from 09:35:12Z): Mac d937c69d on 0.3.6 with node 2b6d23ef and mining throughout from 09:39Z; PC 1 and PC 2 on app 0.3.6 from 09:37Z and 09:39Z with the node dead (section 9), 0.0 MH/s; Sam's Mac and PC 37ba0461 never left 0.3.5 within the window (hourly slots; Sam's Mac silent from 09:41Z). Rotation peaked at 3 of 6 (09:39 to 09:54Z) and read 1 of 6 at 09:56:30Z, when the PCs' latest uploads carried the old key and folder again: the coordinator's 0.3.5 rollback (job rollback-035-pcs) landing. The console's app column for the PCs lags their next report.
The 0.3.7 Windows exes (Mac cross-build, proto-cuda/windows-node/cross-build.sh vendor/igneum-node-036 4,
CARGO_TARGET_DIR=vendor/igneum-node/target-036-win, 25 min 32 s, 09:43 to 10:08Z, every crate rebuilt because the
RUSTFLAGS changed):
| File | sha256 | Size | lib*.dll imports |
|---|---|---|---|
| igneumd.exe | 98a40af986eb2b722f69203ca264ccc3720649682a3832672bf6a1beb70896a9 | 50,773,504 | libstdc++-6.dll (197 symbols); -C link-arg=-static-libstdc++ changed nothing, as the script's own comment on -static-libstdc++ warned: the gcc driver rustc links through ignores it. The flag stays as a record; the DLL pairing is the rule |
| igneum-miner.exe | eb77bf93d0949aa4e38cd2c428c562e991ab96302b6b7bcf8badc8f077d75ce0 | 10,778,624 | none |
Both embed 2b6d23ef. The gate with this Mac's toolchain DLLs (GCC 16.2.0): igneumd.exe, all 197 imported symbols
exported (5,994 exports), pass. The PC-built pair (d08404c2..., 8bdb4c6e...) is kept in the session scratchpad
(pc-built-036/) for the 0.3.8 item and is no longer under target-integration.
9a. The 0.3.7 ship
Josh pushed release-0.3.6 at c528ce8 (10:10Z). gh workflow run windows.yml --ref release-0.3.6 → run 37294921529,
10:11:01Z, green 10:16Z (the G13 step verified the 0.3.7 inputs of 57d2ad32...). Dry run clean (0 behind origin/master
this time). The real run, node tools/ship-app.mjs 0.3.7 --node vendor/igneum-node-036 --branch release-0.3.6 --dl-both --activation-height 84100 --deadline-note "proving v0" --notes "Hotfix for Windows nodes: ..." --from ci:
| Step | Result |
|---|---|
| preflight | ok, tree c528ce8 clean |
| ci | green after 5 min of polling |
| fetch | Igneum-Miner-Setup-0.3.7.exe 18.9 MB, igneum-windows-app.zip 26.3 MB |
| dmg | already (8ee96b3b..., built 09:46 to 09:49Z) |
| copy, mirror | DMG into the OLD folder; 7 files into the NEXT folder |
| manifest | override carried over from the OLD folder's republished 0.3.5 manifest (the same object the NEXT folder's 0.3.6 held; both read before the step); both signed, verified, same fields |
| deploy | 10:16Z |
| verify | 3 files and the manifest match in both folders |
| console | item #350 "Igneum Miner 0.3.7 shipped (mac+windows)"; exit 0 |
The live manifests, read 10:17:22Z, compared field by field with the 0.3.5 record (8h):
| Field | OLD folder | NEXT folder |
|---|---|---|
| version, channel, min_supported_version, published_at | 0.3.7, devnet, 0.3.0, 2026-10-05T10:16:29Z | the same, 2026-10-05T10:16:29Z |
| consensus | identical to 0.3.5's: override {33000, 84100}, activation_height 84100, deadline_note "proving v0" | identical |
| fees_v1_activation_daa, tuning | absent, absent | absent, absent |
| mac | dmg 8ee96b3b054f45721ea08d0e00dfae79fc127fdfa503e05565d117b2c71cd9e2, 40,156,739, URL in its folder | same bytes, URL in the NEXT folder |
| windows | inno-setup 2bacba64628b02e5f03e54007e794a30438a966238c75293a2905e1b5c5e2557, 19,784,297, URL in its folder | same bytes, URL in the NEXT folder |
update-now-20261005-101725 to all, published 10:17:25Z (mirrored to both folders by the publisher), the apps woken
(stamp 2026-10-05T10:17:25Z.726198ee): the 0.3.5 PCs check hourly otherwise.
Baseline 10:11:52Z, before the publish: Mac d937c69d app 0.3.6 node 2b6d23ef DAA 92597 22.4 MH/s; PC 1 ae432dc7 app 0.3.5 node 2.1.0 (the rolled-back 0.3.5 node) DAA 92578 112.1 MH/s; PC 2 1ccfe586 app 0.3.5 DAA 92578 122.3 MH/s; Sam's Mac 3a9bf309 silent 28 min (app 0.3.5, 0.0 MH/s at its last report); PC 37ba0461 app 0.3.5 node 20139145 DAA 92583 2.3 MH/s. A machine counts as updated only when its node reports a DAA score and the miner a hash rate on 0.3.7.
The update-now job reached only the OLD folder: publish-jobs.sh on this branch writes dl/<token>/igneum-jobs.json
and has no mirror step (checked 10:21Z: the NEXT folder still served the 09:55:44Z file, 48 jobs, without it; the
0.3.6 Mac, packaged on the NEXT folder, could not see the job). The jobs file and its signature were copied into the
NEXT folder by hand (the ship's own mirror step does the same) and the folder deployed once; the NEXT folder served
the 10:17:25Z file with the job from 10:22:2xZ. Item: publish-jobs.sh --deploy must write both folders while the
rotation lasts (rotation-phase-2 plan section 4).
update-now-20261005-101725 ran on: Mac d937c69d 10:23:57Z (94 s after the NEXT folder's jobs file went live, on its
2-minute poll), PC 37ba0461 10:24:21Z, PC 2 1ccfe586 10:27:04Z, PC 1 ae432dc7 10:27:13Z (the 0.3.5 PCs on their
10-minute poll); Sam's Mac 3a9bf309 silent since 09:41Z, not reached. Each answered "update check and install asked".
| Machine | On 0.3.7 (node DAA and hash rate reported) | Its log |
|---|---|---|
| Mac d937c69d | engine restart 10:24:20Z (unix 1791195860), 23 s after the job, 3 min 51 s after the publish; node synced at 93390 blocks with 4 peers at +30 s; mining from +36 s (1.9 MH/s while the kernel race ran its 14 variants, the full rate after) | run mac-d937c69d-20261005-102419: IGNEUM-APP version=0.3.7, [ok] updated to Igneum Miner 0.3.7 from 0.3.6, config: ... key 477bb0ef ... folder ed9c4d2e (packaged), igneumd started (pid 99175), node proof verifier: command (.../igneum-prove-host), node proof verifier reported: command at +10 s |
| PC 2 1ccfe586 (Windows, 0.3.5 to 0.3.7) | engine 10:27:13Z (unix 1791196033), 9 s after the job's ask; node started 10:27:16Z, node proof verifier reported: command +10 s, node synced at 93544 blocks with 2 peers at +30 s, 118.74 MH/s at +60 s; console 10:28:14Z: node 2b6d23ef, DAA 93582, 122.3 MH/s |
run win-1ccfe586-20261005-102712: [ok] updated to Igneum Miner 0.3.7 from 0.3.5, fingerprints 477bb0ef/ed9c4d2e, no exited with code line (the 0.3.6 loop is gone) |
| PC 37ba0461 (Windows, the US laptop, 0.3.5 to 0.3.7) | engine 10:24:59Z (1791195899), 38 s after its ask at 10:24:21Z; node started 10:25:04Z, synced at 93449 blocks at +60 s, mining; update to 0.3.7 complete 10:26:27Z; console 10:28:14Z: node 2b6d23ef, DAA 93568, 1.8 MH/s (its usual 2.3) |
run win-37ba0461-20261005-102457; node proof verifier: off (the WSL2 prover is not installed ...), as designed on a machine without the WSL prover |
| PC 1 ae432dc7 (Windows, 0.3.5 to 0.3.7) | engine 10:27:21Z (1791196041), 8 s after its ask; node started 10:27:25Z, node proof verifier reported: command +10 s, node synced at 93552 blocks with 3 peers at +30 s, mining |
run win-ae432dc7-20261005-102720; the hash-rate line follows at +60 s (next watch tick) |
| Sam's Mac 3a9bf309 | not reached: silent since 09:41Z (last report app 0.3.5, 0.0 MH/s); the update-now job and the manifest wait for it |
Console 10:30:51Z, every reachable machine on 0.3.7 with node 2.1.0-2b6d23ef, a DAA score and a hash rate: Mac d937c69d DAA 93724, 22.8 MH/s; PC 1 ae432dc7 DAA 93724, 114.3 MH/s; PC 2 1ccfe586 DAA 93715, 121.2 MH/s; PC 37ba0461 DAA 93696, 1.8 MH/s. Sam's Mac 3a9bf309 silent (47 min). Publish to the last of the four updated: 14 min 22 s, of which the update-now job's reach on the 0.3.5 PCs (10-minute poll) was 9 min 48 s. The watch was stopped here.
10. Open after the cut
| Item | State |
|---|---|
| Why the PC-built Windows node dies at start even with matching DLLs (section 9) | ANSWERED and FIXED (housekeeping, 5 October 2026; fork commit on housekeeping after 7003055b: database/build.rs, database/rocks-probe). The real exit: Windows Application event 1000, Faulting application name: igneumd.exe ... Faulting module name: unknown ... Exception code: 0xc0000005 ... Fault offset: 0x0000000000000000: an access violation at instruction pointer 0, a call through a null function pointer (job probe-exit-pc1-hk3, the PC exe d08404c2 with the GCC 13 DLLs, cmd /c ... & echo EXIT %ERRORLEVEL%; --version exits 0, igneum-miner.exe runs; no stderr, no Rust panic, no file written under the appdir). Narrowed on PC 1: igneumd.exe built WITHOUT igneum-pow dies identically (job build-hk-winprobe-1, exe cf21bebe; run run-hk-winprobe-2), and rocks-probe.exe (a new fork crate that does the node's rocksdb opens one step at a time) printed STEP dir created and died before STEP rocksdb Options built: the first call into librocksdb, rocksdb_options_create, before any thread, open or exception. Cause: the exe linked with -static carries a static libgcc and winpthread but still imports libstdc++-6.dll (the cc crate links the C++ runtime as -lstdc++, rustc puts -Wl,-Bdynamic in front of it, and the gcc driver ignores -static-libstdc++); with Ubuntu's mingw-w64 (GCC 13.2, mingw-w64 11, msvcrt) that mixed link jumps to 0 at the first C++ call, with Homebrew's (GCC 16.2, mingw-w64 14, UCRT) it happens to work. Not the thread model, not panic=abort, not the stack size, not the feature. Fix (the fork's build settings): database/build.rs copies the toolchain's own libstdc++.a ($CXX -print-file-name) into an OUT_DIR search directory as libstdc++.dll.a and libstdc++.a and adds it with rustc-link-search, so -lstdc++ keeps its place in the link line and resolves to the static archive; cargo refuses the two env spellings (CXXSTDLIB=:libstdc++.a: "library name must not be empty"; dylib:+verbatim=libstdc++.a: "overriding linking modifiers from command line is not supported", jobs build-hk-winprobe-2 and -3). The exes of this target now import NO mingw DLL (igneumd 50,883,584 bytes df59dadb..., igneum-miner 3860d81e..., rocks-probe 55bb5851...); the installer's DLL gate and jobbuild.rs's DLL copy stay for an older fork. The Mac cross-build takes the same path (same CXX_x86_64_pc_windows_gnu). Acceptance (job run-hk-accept-1, PC 1, 16:23:44Z, no mingw DLL next to the exes on purpose): rocks-probe printed every step to STEP all ok (21 files in its db), RESULT run node60: still running after 60 s (alive for the whole wait); killing it with gRPC, P2P and the exec genesis up on a scratch appdir (14 files written), RESULT event 1000: none; igneum-miner.exe key-hash probe prints the hash. (The one warning in that run, cannot bind the eth_ JSON-RPC server on 127.0.0.1:26790, is PC 1's live node holding the fixed port; not the probe's business.) So the PC build job can build the Windows node again once a release fork carries this commit; the Mac cross-build rule is no longer needed for that reason |
publish-jobs.sh --deploy writes one folder; PC 2's 0.3.7 app refused one jobs file (jobs file signature does not verify, 13:19:41Z, release-0.3.8 plan section 11) |
ANSWERED (housekeeping, 5 October 2026, commit on branch housekeeping). Cause: the app fetched igneum-jobs.json and then igneum-jobs.json.sig in two requests (jobrun.rs fetch_jobs), and the edge serves the previous deployment for some seconds after a deploy, per object (the same afternoon a publish needed 4 live-check tries, 15 s, before the edge served the new file). Two requests a moment apart can therefore return a file from one deployment and a signature from the other: a pair that does not belong together, which the key correctly refuses. The mirror step (bdde87a) was not the cause: it copies after the signer's read-back and one deploy ships both folders. Fix: ONE object, igneum-jobs.signed.json ({"file":"<the canonical jobs text>","format":"igneum-jobs-signed-1","sig":"<hex>"}), made by the signer (igneum-ota-sign envelope-jobs, which refuses a pair that does not verify) and read back by it (verify-signed-jobs) before anything moves into place; the app fetches that one object (0.3.9 fetch_jobs, Cache-Control: no-cache, the pair only when no envelope is published); publish-jobs.sh writes all three files, mirrors all three, and after a deploy verifies the envelope and the pair in EVERY folder it wrote (verify_live walks folders()); tools/jobs.mjs reads the envelope; ship-app.mjs carries it. Tests: jobs.rs signed_envelope_binds_file_and_signature (round trip, a file with another publish's signature refused at wrapping and at reading with the words the app logs, a tampered inner byte, another key, each shape error named; 27 signer tests pass), packaging/ota/test-publish-jobs.sh (24 checks with the real key in a --dest folder: the three files, the envelope's text IS the plain file byte for byte, the stale pair refused by the signer and as a hand-made envelope, sign rewrites all three). PC 2 job build-hk-tests-1 ran the app suite with this change: igneum-app 75 + 26 + 8, 0 failed. Until the 0.3.9 apps are out, a refusal of this kind is harmless: the next poll (2 minutes) or wake fetches a consistent pair. Note for the publishers: until master carries this, a publish from an older checkout rewrites the pair and leaves the envelope stale; the new script's verify names that ("differs from the local one (the envelope)") and a 0.3.9 app would read the stale envelope, so every publisher moves to the merged script before 0.3.9 ships |
| The app marks "update complete" on its own health | it should wait for the node's first DAA score, so a dead node rolls back (the 0.3.6 PCs looped for 20 minutes) |
| Two finality tests under the five-package parallel suite | ANSWERED (housekeeping, 5 October 2026, fork commit 7003055b on housekeeping from 2b6d23ef). Cause: the PoW engine is one per process (kaspa_pow::igneum::engine(), a OnceLock), its cache build queue is capped at 2 building + 4 waiting (M15/M30, a node property: what peers can make one node do), and cargo test -p kaspa-consensus runs its 94 tests in parallel in ONE process, each mining on its own day; more than six cold 256 MiB builds at once had the header processor refuse two finality tests with PowCacheQueueFull. The engine is in play only when igneum-miner is in the same cargo invocation (its feature unifies kaspa-pow/igneum-pow onto kaspa-consensus), which is why the suite alone passed and the five-package run did not. Not a per-process cache directory: the caches are in memory, there is no directory. Fix: kaspa_pow::igneum::set_unbounded_build_queue(true), a process-wide switch that TestConsensus::new and with_db set, so every caller in a test process waits for a build slot (the miner's own path) and nobody is refused; the node never sets it. Tests: build_queue_is_bounded (the default, unchanged) and build_queue_is_unbounded_for_test_processes (the same race with the switch on: 0 refused, every caller built), serialised on one mutex so they never read each other's setting. Evidence: PC 2 job build-hk-tests-1 (15:53:42Z, --node-tests "kaspa-consensus-core igneum-exec kaspa-pow kaspa-consensus igneum-miner" --app-tests igneum-app, tools/build-job.mjs now forwards both flags): RESULT test node [kaspa-consensus-core igneum-exec kaspa-pow kaspa-consensus igneum-miner] exit 0 41 s, kaspa-consensus 94 passed 0 failed 3 ignored with frozen_table_holds_a_side_without_the_other_keys_for_one_window ... ok and reorg_past_an_unlocked_checkpoint_re_determines_it_and_verifies_the_pending_certificate ... ok, kaspa-pow 13 passed including both queue tests, RESULT test app/igneum-app [igneum-app] exit 0 5 s (75 + 26 + 8). The owed kaspa-consensus-alone run is covered: the whole five-package set is green on the fixed tree |
| igneumd not reproducible across PC 1 and PC 2 (8f) | unverified why |
| Sam's Mac | silent since 09:41Z; takes 0.3.7 on its next check |
| The 0.3.6 app-side fixes carried into 0.3.7 and not yet exercised on a PC | the PC's windows stage shipping its DLLs (jobbuild.rs), the stamp after extract, the WSL scripts from files on a real probe (the Mac verifier path is proven; the Windows wrapper path only by its unit tests and the node's verifier reported: command on both PCs, which means the wrapper's probe answered) |