46 KiB
Igneum Miner 0.3.11: program class v3 (Counter ASIC 2.0) and proving v1 on the devnet, 5 October 2026
Release engineer, from 22:39 UTC, on the coordinator's instruction under the project lead's delegation ("Counter ASIC 2.0 fully deployed",
"deploy what is absolute best" for proving v1). Worktree /Users/joshm/Projects/igneum-wt-ship0311, branch release-0.3.11,
assembled by the Counter ASIC coordinator from master b38f3de (the 0.3.10 merge) and taken over at its merge tip b968ee0 so there
is one ship, not two. Fork worktree vendor/igneum-node-0311 UNDER the release tree (the node links ../../../../igneum-pow, the
release tree's crate), branch release-0.3.11-node at 89dfcb95 (on 21d4c73c = 0.3.10's node, with proving-v1 ece42979 and the
digest re-pin). The 0.3.10 recipe (release-0.3.10.md) throughout; every Mac build under the main checkout's lock; every PC
job and publish from this worktree's tools (the signed envelope, the per-job zip names). Times are UTC.
1. What 0.3.11 carries
| Change | Where | State |
|---|---|---|
Program class v3 (Counter ASIC 2.0): the era draw, the cache growth rule, the mixer x8, behind program_class_v3_activation_daa (keys on the epoch: program_class_v3_first_epoch); the workers carry the class, era and attempt rules on the serve protocol and refuse a pack of the wrong class or era |
main ca2-v3 fa3c932 (code 49c7e78; igneum-pow, the workers, the fast-time scripts, the plans); fork ca2-v3-node 89dfcb95 |
merged (c9b0b2c) |
| Counter ASIC 2.0 docs, spec, site, evidence, the rollout plan and its gates (G1, G2, G3, G4, G4b, G6 green; G5 = the one-commit workers of this cut) | main ca2-coord 076c0ab then 57844e9 (C34) |
merged (18605f4, 5cedcd4) |
Proving v1 (spec 7.8): the aggregated segment record, the chain rule and the unproven rule behind proving_v1_activation_daa, with proving_v1_segment_blocks 8, proving_v1_unproven_daa 600, proving_v1_aggregator_share_bps 1000; the app's prover loop (the CPU path refused under 32 GB with the reason, the root-socket cleanup, the PermissionDenied line naming the cause); the resume fix (every stopped card re-armed, its pack re-exported, checked 90 s later); every worker gets --prepare-packs in the platform's path form (the Mac's Metal worker takes a class v3 day from the prepared pack) |
main proving-v1 22c2363 (its agent: the final code tip; c36dfea after it is docs only and waits); fork proving-v1 ece42979 inside 89dfcb95 |
merged (5cbb796) |
CI: bash-body-check.sh (inline bash bodies in PowerShell job scripts parse), kit-path-check.sh (a run job tests its fetched kit before use, C32), prover-socket-check.sh (every root prover playbook unlinks the GPU server's socket) |
main bash-body-check e3bd761 |
merged (fe1ecdb; ci.yml keeps the signer-pipe step, both new steps and proving-v1's socket step) |
| The consequences ledger, the proving-methods analysis, the ASIC-resistance history | consequences 99fd988, proving-methods e7e0db7, asic-history 9e4af7f |
merged (5dffb1b, 0f5bfc3, b968ee0) |
| The pinned proving guests | unchanged (no prover drain) |
Changelog line (the coordinator's words): "Igneum Miner 0.3.11: program class v3 (the era draw, the cache growth rule, the mixer x8) from epoch N4/3600 and proving v1 from DAA N5; the resume fix; the Metal worker takes v3 from a prepared pack".
2. The branch
| Commit | What |
|---|---|
| b968ee0 | the Counter ASIC coordinator's merge tip (above), taken over at 22:40Z; its checks on the Mac: igneum-pow 53 + 4 + 19 + 7, the app 113 + 27 + 8, 0 failed |
| 21173c4 | Igneum Miner 0.3.11: the six version files (--check: 0.3.11 in all 6) |
| cc72f4a | CI on the merged tree, three findings fixed: the identity check's hostname pattern MacBook matched prose in card-lifetime-2026-10-05.md and proving-methods.md (reworded "Apple laptop"); the kit-path check flagged tools/proving-v1/pc2-{memory-miner-on,memory-sweep,sp-curve}.ps1 for a bare jobs\ literal in WslPath (Join-Path ...) (now Test-Path on the kit root, then WslPath $kitFile); the socket check flagged tools/amd-prove/pc1-cpu-prove.ps1 and its -sp sibling, which the addendum said were allow-listed and were not (allowed, the CPU path starts no GPU server). The other agent's resolution had kept both ci.yml sides and bash-body-check's 24-line socket check; one slip of mine (a git show :3: redirect after that resolution had already committed) truncated that script to 0 lines in the working tree for a minute and was restored from HEAD |
| 23bc2b2 | packaging/mac/packaged-config.sh: the nine-field object (section 4) in the packaged line, --test passes (C34: a fresh install must start on the fleet's digest) |
| a94416e | the plan draft committed to the tree before the push (the reviewer's C37) |
| after a94416e | C38 (the reviewer through the Counter ASIC coordinator): the evidence rows, the litepaper's chip bullet, chip-model-v3.md and the rollout plan cite files on four Counter ASIC branches that never reached the tree. Taken as docs plus standalone sources under one gate: the diff against 23bc2b2 over every input a built artefact reads (igneum-pow/src, app/, proto-cuda/nvrtc/{packfile.h,worker.cpp,cuda_api.h,build-windows.sh}, proto-cuda/{host.cu,build.bat,windows-app}, proto-opencl/{host.c,cl_dynamic.h,build.*}, proto-metal, packaging, proving, vendor, .github) must stay empty apart from files no script compiles or copies. Merged: ca2-analysis ee42d7c (sram-mirror.md, int8-matrix-family.md, the dot4 probe sources: proto-metal/dot4-probe.swift is standalone, the DMG script compiles main.swift alone), ca2-epoch e95e8b5 (epoch-length.md), prover-floor cfe3d80 (prover-floor.md, tools/prover-floor/ scripts, a playbook, proving/prover-floor/sp1-gpu-6.8.1-floor.patch, which nothing reads: the next cut's packaging row); docs/bench-log.md conflicted each time (append-only: both sides kept). ca2-soundness a465881 conflicted in igneum-pow/tests/scratch.rs (add/add) and proto-metal/packbench.swift, code the mixer merge already carries, so its merge was aborted and docs/analysis/scratch-soundness.md taken alone. So G5 holds: the workers, the Metal worker and the DMG built at 23bc2b2 correspond to the tip's built inputs |
Every check at cc72f4a: identity 0 hits over 220 files, copied-sources, pinned-guests, signer-pipe, bash-body 15 bodies in 28 files, kit-path 14 of 14 kits checked, prover-socket, no-conflict-markers, workflow shell 0 findings over 35 .ps1, relay tests 17, UI tests 23.
3. Builds and tests (the 0.3.10 recipe)
| What | Command | Result |
|---|---|---|
| The fork's Mac node, 89dfcb95 | CARGO_TARGET_DIR=vendor/igneum-node/target-0311 cargo build --release -j 4 -p kaspad -p igneum-miner --features kaspad/igneum-pow from vendor/igneum-node-0311 (under the release tree), under the lock; the target dir cloned by APFS from target-0310 |
22:46:16 to 22:49:5xZ (3 min 19 s, incremental): igneumd bd7f043c453f4b3e9575e678912b71115227b09789d5ec0f367b8278f031d043 (41,386,016, 89dfcb95 in its strings); copied into the fork worktree's target-integration/release/ |
| The seed's Linux node (glibc 2.36 target, zig) | NODE_SRC=<abs fork> TARGET_DIR=<abs> OUT_DIR=<abs> infra/cross/build-linux.sh (the 0.3.10 fix: absolute paths; cargo clean -p kaspa-build-info first), under the lock |
22:46:35 to 22:49:59Z (3 min 20 s): igneumd 63cf490d42483d3aa4e525eba9b4e4b68cae9090c32f409e745858defc381c52 (47,913,832, GLIBC_2.34 at most, 89dfcb95 in its strings), igneum-miner a136d622... (9,861,280) |
| The two Windows workers (G5: one commit) | proto-cuda/nvrtc/build-windows.sh under the lock, tree 23bc2b2 |
22:46:43 to 22:46:50Z: igneum-worker-cuda.exe 2b3b8c92885442179f6bf2907c6f3eb453dc4a19908d90fd05981a09b7c2674c (1,536,512), igneum-worker-opencl.exe edc4a75da3b93d814caa69fd635010780d63d5b622ec24c3741d433c584f91e3 (478,208); both carry the resource block; both differ from 0.3.10's pair (85cc357b..., afa73a32...): the class, era and attempt rules are in them |
| The app | cargo build --release in app/igneum-app (target cloned from the 0.3.10 worktree), then cargo test --release -p igneum-app, under the lock |
22:47Z: igneum-app 0.3.11; tests ok 113 (lib) + 27 (ota-sign) + 8 (prove-verify), 0 failed |
igneum-pow |
cargo test --release in igneum-pow, under the lock |
22:47:33Z: ok 53 + 4 + 19 + 7, 0 failed (the class v3 vectors, the era draw, the mixer x8, the scratch soundness) |
| The prover host and export (the pin unchanged) | cargo build --release -p igneum-prove-export -p igneum-prove-host in proving/igneum-prove (the worktree needed the vendor links: vendor/igneum-node-exec and 45 others symlinked to the main checkout's, beside the real igneum-node-0311 worktree) |
22:48Z: --mode id shard 0x2b1a81cb..., aggregator 0x474678f3... (the 0.3.9 pin: no prover drain); the nine real fixtures --mode native all ok |
| PC 1 build job (the node and the app, Linux and Windows) | IGNEUM_WIN_RELEASE=<fork>/target-integration/x86_64-pc-windows-gnu/release node tools/build-job.mjs run --node vendor/igneum-node-0311 --target ae432dc7 --targets linux,windows --no-tests from this worktree, its own zip build-inputs-20261005224625-29789.zip |
job build-20261005-224654, published 22:46:54Z (PC 1 given by the Counter ASIC coordinator at 22:4xZ: Ember's collect job closed, the AMD sweep off tonight). (pending) |
| PC 2 combined job | IGNEUM_WIN_RELEASE=<fork>/target-integration/x86_64-pc-windows-gnu/release node tools/build-job.mjs run --node vendor/igneum-node-0311 --target 1ccfe586 --targets linux,windows --node-tests "kaspa-consensus kaspa-consensus-core igneum-exec kaspa-pow igneum-miner kaspa-p2p-flows" --app-tests igneum-app from this worktree (its own zip build-inputs-20261005230716-50058.zip); PC 1's job removed from the jobs file so nothing double-places the exes |
job build-20261005-230745, published 23:07:45Z (PC 2 woken), the first job on PC 2's re-set schedule; started 23:09Z, done 23:16:49Z (469 s): the Linux stage, the Windows stage 264 s, the test stage RESULT test node [kaspa-consensus kaspa-consensus-core igneum-exec kaspa-pow igneum-miner kaspa-p2p-flows] exit 0 54 s and RESULT test app/igneum-app [igneum-app] exit 0 6 s; 9 outputs verified and placed: igneumd.exe be8e83c07aeae5eb6842768071735289f3ef149ed9a7088592d8c54a4c252c08 (51,321,856), igneum-miner.exe 1ba1a249e2a21d52087a81f61f37cd2e1e3273c7ec8809ef9cfaa98f015895ce (10,987,520), igneum-app.exe ab104cc0... (3,065,344, the PC's; the installer's engine is the runner's), Linux igneumd d7a2715e... (49,193,960, glibc 2.39, HiveOS), igneum-miner 09d05ff6..., igneum-app 222f30c0... |
| PC 2 suites | the combined job above (the PC 1 job of the row above never started: PC 1's app is down, section 3a) | six node suites exit 0 in 54 s, the app suite exit 0 in 6 s (23:16:49Z) |
| The Linux workers for HiveOS | infra/cross/build-workers-linux.sh (zig, glibc 2.36 target) under the lock, 23:10Z |
igneum-worker-cuda 4aaff27fcb26bc5fe98d2b311b9f95f099c59414a556af32080b82b41e8360db (6,755,568), igneum-worker-opencl 82d90890be36f9b795b218794062e388bfd9c6f7524fe671074cdec84c906259 (306,000), ELF x86-64 dynamic; untested on a GPU host, as the script says |
| The HiveOS package | NODE_OUT=<the zig node> WORKERS_OUT=<those workers> VERSION=0.3.11 packaging/hive/make-hive-package.sh, 23:11:30Z, then packaging/ota/publish-public.sh --hive into dl/public/ (the ship's deploy carries it) |
igneum-hive-0.3.11.tar.gz c606a17043c013c411086b4abd0f38a227aa8a7db9d5934157760a3da5a5edc9 (24,496,653); no override inside (section 5) |
| The DMG | NODE=<fork>/target-integration/release/igneumd MINER=... packaging/mac/build-dmg.sh under the lock, 22:50:0x to 22:50:51Z |
Igneum-Miner-0.3.11.dmg b7e81d4f6f3af9f9179e29faa72c844b795df757dfb2e4cd1d56cd454e78e1e7 (41,592,041): engine 0.3.11, node 89dfcb95 Mac arm64, the 0.3.9 prover host and export, igneum-bench (the Metal worker) rebuilt from this tree (667,145 to 555,808 bytes, signed ad hoc), fingerprints 477bb0ef and ed9c4d2e; read back from the mounted image: Contents/Resources/igneum-app.json carries the nine-field node_override_params with N4 = N5 = 154,800 (C34) |
3a. PC 1's app down, and the relay task that hit the wrong machine (22:31 to 23:05Z)
PC 1's installed 0.3.10 app quit at 22:31:06Z (its last upload 22:31:08Z, run win-ae432dc7-20261005-214041; Ember's tune job on it
reported "aborted (the app is quitting)"; the quit's sender is C35 for the consequences reviewer) and did not come back, so PC 1 mined
nothing, its build job build-20261005-224654 could not start and no update-now could reach it. Three relay tasks (#241 at 22:52:31Z, #243 at
22:55:10Z, #245 at 23:03:18Z) went to the relay machine named "PC1" to relaunch the app, the second ending an engine that answered nothing on
/api/state and the third ending every igneum process before the launch, as the app's own updater does.
They hit the wrong machine. The relay's "PC1" is the 1ccfe586 box, the console's PC 2: both PCs carry the hostname DESKTOP-KMCV30N, the only
relay agent runs on the 1ccfe586 box and was named PC1 when the clients were set up, and the relay's PC2 entry reads "never seen". The
intake proves it: PC 2 got two new engine runs, win-1ccfe586-20261005-225528 and -230330, at the exact times of #243 and #245, while PC 1
has no run after 21:40:41Z; #243's "hung" engine, pid 26696 from 21:49:40Z, was PC 2's healthy 0.3.10 engine (my probe's "no answer" on
/api/state was its own fault, no token), and the node, three miners and three workers it found were PC 2's own (a 5090 and the iGPU; PC 1
would have shown the 9070 XT too). So PC 2, the box the night's measurements run on, was force-restarted at 22:55:28Z and 23:03:30Z, which
killed the aggregation-cost agent's job 3 (re-run owed, 20 min) and ended whatever followed; its app came back each time (after #245: pid 30484,
responding, node, three miners and both workers up, the card climbing at 23:04Z). PC 1 is exactly as it was: engine down since 22:31:06Z,
no relay agent on that box, unreachable tonight; it waits for the project lead in the morning and takes 0.3.11 through the manifest at its relaunch; the
fleet runs short its 141 MH/s until then. Told the Counter ASIC coordinator at 23:05Z; it re-set PC 2's schedule (this cut's combined build
and suite job first, then the prover-floor pair, the aggregation-cost re-run, "PC 2 clear", the M16 job) and ruled that no relay task goes to
"PC1" from anyone without its word. Every relay "PC1" reading tonight was PC 2 (the AMD agent's "9070 XT absent" probes read a box that has
no 9070 XT; the hardware events are being corrected). The relay machine should be renamed PC2 (node tools/relay.mjs name <hostname> <name>)
and the PC 1 box get its own agent (section 11). The Mac cross-build fallback for the Windows exes was announced and not started: the
combined PC 2 job replaced it within the minute.
C35's cause (Ember Tune, 00:2xZ): the second engine its measurement job starts on PC 1 reports 0.3.9 (the ember-tune branch's Cargo
version), sat under the manifest's min_supported_version, took 0.3.10 as urgent (over auto_update = false in its copied settings), ran
ota-apply.ps1 at 22:31:05Z, and the per-user installer's PrepareToInstall quit the INSTALLED app at 22:31:06Z, which then hung on the pipe
its orphaned grandchildren held. For this plan's bookkeeping: PC 1's 0.3.10 came through my update-now at 21:40:41Z (release-0.3.10.md
section 8); the 22:31:05Z run was a second install of 0.3.10 over 0.3.10 by that engine, outside the rollout order, and it is what took
PC 1 down. The guard is ember-tune e600e63 (section 10).
4. The override object, N4 and N5, the digests
The object every node runs after the publish (the four live fields plus the five new ones):
{"difficulty_v2_activation_daa":33000,"proving_v0_activation_daa":84100,"fees_v1_activation_daa":210000,"finality_v3_activation_daa":135200,"program_class_v3_activation_daa":154800,"proving_v1_activation_daa":154800,"proving_v1_segment_blocks":8,"proving_v1_unproven_daa":600,"proving_v1_aggregator_share_bps":1000}
N4 and N5 are fixed BEFORE the DMG and the installer are built, because the packaged line (C34) must equal the manifest's object: DAA 136,967 at 22:45Z (PC 1's card) at 0.965 blocks/s puts the publish (about 23:40Z) near 140,200; tip + 14,400 is near 154,600; the first multiple of 3,600 at or above it is 154,800 (N4's rule), which N5 takes too. At the publish the floor N - DAA >= 10,800 is checked; it holds until DAA 144,000 (about 00:45Z); past that the line is re-pinned and the DMG and installer rebuilt.
The two digest readings on the 0.3.11 Mac node (bd7f043c..., ports 60975/60976, 22 s each, under run):
| Override file | Lines | Digest |
|---|---|---|
| none (the rolling-upgrade value: both new activations at never) | igneumd/2.1.0-89dfcb95 |
c562d70e1428c9789823cc40067623b4767f7c555ce7ff4ea11c1498f013ef6c, EQUAL to the node agent's pinned test on 79bd8e10 (22:49:59Z) |
| the nine-field object above | Program class v3 from the override file: active from epoch 43 (DAA score 154800 rounded up to the epoch boundary at 154800, epochs of 3600 DAA), Proving v1 from the override file: segment records paid from DAA score 154800, 8 blocks a segment, unproven after 600 DAA, aggregator share 1000 bps, Calibrated v1 fees ... 210000, Finality rule v3 ... 135200 |
0139ab9dc2992d449ec787d8f021974933631eb55740ab4b6ce9d5c226e72888 (22:50:23Z): the value every node must print after the publish; the sweep in section 9 reads it on every node |
| the fleet's live four-field file (what every node runs today) | Calibrated v1 fees ... 210000, Finality rule v3 ... 135200, no class v3 or proving v1 line (both at never) | 4d8f8bb668828a3dcf7b783b995f3d3ebfde32a092dd1dbd5bf4373c5c65a62c (23:13:54Z): the 0.3.11 BINARY alone flips the digest (the 0.3.10 node prints 1f4b4425... with the same file, release-0.3.10.md 9a): the new fields enter the digest even at never. This is the digest of step 1 (the reviewer's C40, the Counter ASIC rollout plan's section 4 step 1); both had assumed c562d70e..., which is the no-file case |
Three digests on the 0.3.11 binary, so two sweeps: step 1 moves every node to the binary (4d8f8bb6...) and step 2 moves every node to the nine-field object (0139ab9d...). A node on either side of a sweep is refused by the other side (the handshake), so each sweep is one window, as the fee switch's 8 min 47 s was.
5. The rollout order: two publishes, two sweeps (the reviewer's C39 and C40, the Counter ASIC coordinator's rule, the fee-switch shape)
Why two: the app writes the manifest's consensus.override at the manifest TAKE (ota.rs 661, write_override) and restarts its node with
it at the next safe window, whatever binary is installed; a 0.3.10 igneumd refuses a file with program_class_v3_activation_daa or the
proving v1 fields (OverrideParams is deny_unknown_fields) and dies at start, and the update then waits for a synced node (engine.rs
2211) until the slot minute or the 1,800-block rule forces it. One manifest with 0.3.11 AND the nine fields would take every 0.3.10 node
down at its next safe window: the 0.3.5 class the fee-switch plan named. And the 0.3.11 binary alone flips the digest (section 4), so the
binary move is itself a sweep.
| Step | What | Digest after |
|---|---|---|
| 1a | the observer, node 1 and the seed on the 0.3.11 binaries with the four-field file: IGNEUMD=<fork>/target-integration/release/igneumd IGNEUMD_COMMIT=89dfcb95 infra/devnet/restart-hand-nodes.sh '<the four-field object>', then IGNEUMD_LINUX=<the zig build> IGNEUMD_LINUX_SHA256=63cf490d... infra/devnet/restart-seed.sh '<the same>'; the apps still on 0.3.10 are refused by them from this moment until each updates |
4d8f8bb6... on the three |
| 1b | the manifest: 0.3.11 with consensus carried over UNCHANGED (--activation-height 135200 --deadline-note "finality v3", the four-field object, exactly as 0.3.10 shipped), --public (the HiveOS package rides along) |
|
| 1c | update-now: the Mac (its card follows node 1) and the laptop first; PC 2 only on the Counter ASIC coordinator's "PC 2 clear"; PC 1 is down and unreachable (section 3a) and takes 0.3.11 through the manifest at its morning relaunch, refused until then. The watch: every app logs 0.3.11 and its node a DAA score at 4d8f8bb6...; every worker starts clean on the first try with the class-aware pair; the Mac's Metal worker takes the class v3 day from the prepared pack; C32: the agents whose kits sit on PC 2 republish their fetches after its update | 4d8f8bb6... on every reporting node |
| 2a | the floor: 154,800 minus the tip's DAA at least 10,800 (holds until DAA 144,000, about 00:45Z); past it N4 = N5 re-pinned to the first multiple of 3,600 at or above tip + 14,400, the packaged line, the DMG and the installer rebuilt; the DAA read sent to the Counter ASIC coordinator before 2b | |
| 2b | the hand nodes' and the seed's files switched to the nine-field object and restarted (the same two scripts), the manifest republished with --override '<the nine-field object>' --activation-height 154800 --deadline-note "program class v3 + proving v1", update-now (the same order; PC 2 on "clear" again), the sweep |
0139ab9d... on every node |
| 3 | the plan's final sections, the merge to master (the live observer must not read stale: public-api-check's other arm), the push, the report with per-machine times |
The HiveOS package carries NO override: packaging/hive/h-run.sh line 31 starts the rig's node with --devnet --appdir --rpclisten --listen and
the peers, no --override-params-file, and no HiveOS package has ever carried one, so a rig's bundled node runs on genesis params and is refused
by every devnet peer (the HiveOS path is untested on a GPU host since 4 October). "Republish with the new override" therefore needs an
h-run.sh change (the file written from the Flight Sheet's extra config, as PEERS= is), which is the next cut's; tonight's package carries
the class-aware binaries only (section 11).
6. The push and CI
git push -u origin release-0.3.11 at 3b0262f (23:18:29Z, the credential helper; the pre-push hook's site flip restored), gh workflow run windows.yml --ref release-0.3.11 -> run 37387737179, acquired at once (GitHub operational again), green 23:23:11Z (the parse job 23:18:39 to
23:19:25Z; engine, window host, payload, installer, smoke run 23:19:31 to 23:23:11Z, the G13 step against the 89dfcb95 inputs). The main
ci.yml run on 3b0262f (37387751432) failed in one step, the igneum-census release build (the class v3 fields missing from the census's own
initialisers, fetch's new Layout argument, no Scratch/Hot match arms; pow tests, simulators and site jobs passed); the coordinator fixed it
in this worktree as 2a62735 (igneum-census/src/main.rs only; git diff --stat 3b0262f 2a62735 -- app packaging igneum-pow proto-cuda proto-opencl proto-metal is empty, so the Windows artefacts of 37387737179 stand) and its run 37388453875 is green (pow tests and census
build, simulators, site). The 0.3.11 CI verdict is therefore run 37388453875 on 2a62735; the Windows build is run 37387737179 on 3b0262f,
the same app sources; the merge to master goes from 2a62735.
| File | sha256 | Size |
|---|---|---|
| Igneum-Miner-0.3.11.dmg | b7e81d4f6f3af9f9179e29faa72c844b795df757dfb2e4cd1d56cd454e78e1e7 | 41,592,041 |
| Igneum-Miner-Setup-0.3.11.exe | 84a21443f78598597f33cef307fa162e53c680dd90d29f02ceaf79ac5e231ee4 | 50,065,009 |
| igneum-windows-app.zip | ed0cf2a75a1de8897de33ddcdcdb4ea844eca6b38627d6b91c562700c9fbe5eb | 72,070,333 |
| igneum-hive-0.3.11.tar.gz (dl/public) | c606a17043c013c411086b4abd0f38a227aa8a7db9d5934157760a3da5a5edc9 | 24,496,653 |
7. The rollout, step 1 (the binary sweep to 4d8f8bb6...)
Baseline 23:19:40Z: tip DAA 139,642; the observer and node 1 on 21d4c73c at 1f4b4425...; the Mac app 0.3.10 (attached to node 1); PC 2 0.3.10 at 120.8 MH/s; PC 37ba0461 back on 0.3.10 at 2.3 MH/s; PC 1 down since 22:31:06Z (section 3a); Sam's Mac quit since 20:47Z.
| Step | Time | Result |
|---|---|---|
| 1a the observer | 23:23:54Z (pid 61754) | igneumd/2.1.0-89dfcb95, the four-field file, digest 4d8f8bb668828a3dcf7b783b995f3d3ebfde32a092dd1dbd5bf4373c5c65a62c |
| 1a node 1 | 23:24:06Z (pid 61864, caffeinate 61866) | the same |
| 1a the seed | 23:24:25Z (MainPID 125853) | igneumd/2.1.0-89dfcb95, the same lines; the a24ab01a... no: the 21d4c73c binary kept as igneumd.prev-035 (the script's name) |
1b the ship (--from ci) |
23:24:44Z | preflight ok (tree 2a62735 clean, 0.3.11 in all 6, fork 89dfcb95, gh igneum-labs, live inputs 89dfcb95 built 23:17:51Z); ci, fetch, dmg, copy already; consensus.override: carried over from the current manifest (the four-field object), activation_height 135200, deadline_note "finality v3"; manifest 0.3.11 mac+windows signed (key 8f186e37...) and in dl/public/; one deploy; verify: the token folder's manifest 0.3.11, signature ok, mac b7e81d4f..., windows 84a21443...; the public folder's igneum-downloads.json not yet the local bytes at the edge (the 0.3.10 class), resumed from verify for the console item |
| 1c update-now, the Mac and the laptop | 23:28:06Z (update-now-0311-d937c69d-37ba0461, apps woken) |
the Mac: the job ran 23:28:48Z, Igneum Miner 0.3.11 is available: downloading (41 MB) 23:28:49Z, engine restart 23:29:05Z (run mac-d937c69d-20261005-232905), 59 s after the job; [ok] updated to Igneum Miner 0.3.11 from 0.3.10; a node already answers on 127.0.0.1:26610; using it: the app attaches to node 1 (89dfcb95, 4d8f8bb6...), so its card follows node 1; cards: Apple M5 Max [apple, off]: its Metal miner was off before and after, so the prepared-pack check has no subject on the Mac tonight. The laptop (PC 37ba0461): (pending) |
| the console | 23:30Z | item #366 "Igneum Miner 0.3.11 shipped (mac+windows)" (--from console after the public index settled at the edge; the ship's own verify had refused it as 0.3.10's did) |
| the old side during the window | from 23:24Z | PC 2 and the laptop at 0 peers (refused by the new side) until each updates. The new side (node 1, the observer, the seed, the Mac app attached to node 1 with its miner off) has NO miner on it, so its chain STALLED at DAA 139,751 from about 23:29Z (the observer's /api/stats at 23:31:42Z: 139,751, age 1.8 s) until a miner joins it; the old side's fork grows only while a miner mines there, and PC 2's 0.0 MH/s from 23:29Z is the prover-floor sweep stopping its miners for its run (floor-sweep-2, started 23:21:17Z), not the refusal. Put to the Counter ASIC coordinator at 23:31Z with the numbers; its call: "PC 2 clear" the minute the sweep closes (about 23:36Z) and at 23:45Z at the latest, the restore job after the update, because an app restart under the sweep kills its job tree and its restore never runs; the laptop's 2 MH/s restarts the new side's chain when its install ends. The lesson for the next cut's plan: step 1a (the hand nodes and the seed first) moves the hub to a side with no hash until the first miner updates; the first update-now should go to a miner within the same minute |
| the Mac's miner (C42) | 23:39:10Z | the new side had NO miner: the Mac app's mining was a persisted settings.paused (cleared only by Resume; its STATUS lines read "0.00 MH/s, paused" since the update), node 1 runs no miner, the fleet's hash was on the refused old side. POST <app.url>/api/resume (the token path) answered ok at 23:39:10Z; the Metal worker compiled the program inline at 23:39:11Z ("not prepared": the prepared-pack path did not engage on this start, a G4b finding, section 11), raced 14 variants, 3.15 MH/s wall at 23:39:48Z; the new side's first blocks accepted 23:39:49, 23:39:51, 23:40:00Z. The stall: about 23:29Z to 23:39:49Z, 11 minutes of stopped DAA clock on the side every node ends up on. New check for step 1 of every digest-flipping cut: the new side has at least one miner before the first update-now |
| 1c update-now, PC 2 | 23:39:44Z (update-now-0311-1ccfe586, on the Counter ASIC coordinator's "PC 2 clear": floor-sweep-2's first point had hung 18 min, nothing in flight to protect; its restore-and-diagnose job is the first PC 2 job after the update). PC 2 fetched the woken file at 23:40:13Z and logged 1 new for this machine, 1 queued: update-now is itself a job and the app runs jobs one after another, so it waits behind the hung floor-sweep-2 (started 23:21:17Z, cap 30 min, freed about 23:51:17Z); PC 2 reads `0.00 MH/s, waiting |
node 139753 blocks, 0 peers, syncing` every 30 s meanwhile (refused by the new side, its miners stopped by the sweep). The class, for the next cut's plan: an update-now cannot pre-empt a running job; a hung job's cap sets the update's time |
| the laptop (PC 37ba0461) | silent since 23:28:28Z | its last upload, "2.25 MH/s, mining | node 139757 blocks, 0 peers" at 23:28:28Z, is 22 s after update-now-0311-d937c69d-37ba0461 was published; no job line reached the intake before the silence. Its 0.3.10 install kept it silent 55 minutes (21:41 to 22:36Z), so this is its install in progress until shown otherwise; publish 2 does not wait on it (a 2 MH/s machine whose 0.3.11 reads the nine fields when it returns; on 0.3.10 its node would die on the file until the forced apply, the C39 case for one machine) |
| PC 1 | unreachable tonight (section 3a); refused by every peer on 1f4b4425 until its morning relaunch takes 0.3.11 through the manifest |
8. The rollout, step 2 (the object sweep to 0139ab9d...)
| Step | Time | Result |
|---|---|---|
| 2a the floor | 23:55:42Z | tip DAA 140,706; 154,800 - 140,706 = 14,094 >= 10,800, so N4 = N5 = 154,800 stand (the floor holds until DAA 144,000); no re-pin, no rebuild |
| 2b the observer | 23:56:59Z (pid 97249) | /tmp/igneum-devnet/override-v3.json switched to the nine-field object; igneumd/2.1.0-89dfcb95, Program class v3 from the override file: active from epoch 43 (DAA score 154800 ...), Proving v1 from the override file: ... 154800, 8 blocks a segment, unproven after 600 DAA, aggregator share 1000 bps, digest 0139ab9dc2992d449ec787d8f021974933631eb55740ab4b6ce9d5c226e72888 |
| 2b node 1 | 23:57:12Z (pid 97417) | the same lines and digest; it refused the seed (still 4d8f8bb6...) at 23:57:13Z and registered it at 23:57:42Z (protocol version 15), 29 s after the seed's own restart |
| 2b the seed | 23:57:30Z (MainPID 126124) | /etc/igneum/override-v3.json the nine-field object; the same binary 63cf490d..., the same lines, digest 0139ab9d... |
| 2b the manifest | 23:57:49Z | publish-manifest.sh --version 0.3.11 --override '<the nine-field object>' --activation-height 154800 --deadline-note "program class v3 + proving v1" --notes '<section 1>' --public --deploy; the live manifest's consensus read back byte-identical at the edge (the nine fields, activation_height 154800) |
| 2b update-now, the Mac and the laptop | 00:00:30Z (update-now-0311-switch-d937c69d-37ba0461, apps woken) |
the Mac ran it 00:01:06Z: 0.3.11 is current, consensus parameters from the signed manifest: {... nine fields ...}, consensus override changed (.../Igneum/app/override.json); the node restarts with it at a safe moment; the Mac's node card is node 1 (external: pid 0, starts 0, consensus_digest empty), already restarted by hand at 23:57:12Z, so there was nothing for the app to restart and its digest is node 1's. The laptop (PC 37ba0461) was not on the air (below) |
| 2b update-now, PC 2 | 00:01:10Z (update-now-0311-switch-1ccfe586, on the Counter ASIC coordinator's "PC 2 clear": floor-restore-1 closed 23:57:29Z) |
PC 2 fetched the woken file 00:01:41Z, ran the job the same second (0.3.11 is current, consensus override changed (C:\Users\Admin\AppData\Local\igneum\app\override.json); the node restarts with it at a safe moment), its node restarted at 00:01:42Z (Consensus params digest: 0139ab9d..., igneumd/2.1.0), the first STATUS 00:02:01Z "0.00 MH/s, waiting, node 141065 blocks, 1 peers, synced", mining at 00:02:31Z, 106.48 MH/s with 830 accepted this run and 0 faults at 00:06:31Z; the run id stays win-1ccfe586-20261005-235130 (the node restarted, not the engine) |
| the window | 23:57:42 to 00:01:42Z | PC 2 (on 4d8f8bb6...) refused the seed (on 0139ab9d...) at 23:57:42Z and 23:58:12Z and mined on the old side alone; node 1 refused the seed once (23:57:13Z). The new side (the observer, node 1, the seed) had no miner on it either: the Mac's miner was off node 1 from 23:57:14Z (the next row), so the new side only relayed PC 2's old-side blocks it had already accepted (PoW accepted ... daa 140757 at 23:57:32Z was the last) and waited; the two sides rejoined when PC 2's node restarted at 00:01:42Z and PC 2's hash carried the chain. The observer's tip read 140,757 at 00:02:22Z and 141,645 at 00:10:39Z (about 1.7 blocks/s, the catch-up after the rejoin), no stall as in step 1 |
| the Mac's miner (a miner finding) | 23:57:14Z to 00:08:34Z | the miner (igneum-miner mine grpc://127.0.0.1:26610) lost node 1 at node 1's 23:57:12Z restart and NEVER reconnected: 5,317 submit error ... Not connected to server and template error: Not connected to server lines, its TEMPLATES line frozen at templates=2350 with subscribed=true while fetch_errors climbed (143 at 23:59:34Z, 595 at 00:05:07Z), the card's line "the node is not answering; the miner retries", the STATUS line still "26 MH/s, mining" with the accepted count frozen at 18,117 (773 this run). The retries are template and submit calls on a dead gRPC channel; nothing re-subscribes. Recovery through a job: publish-jobs.sh add --kind restart --target d937c69d --what miners (restart-miners-0311-d937c69d, 00:08:01Z); the Mac ran it 00:08:31Z (miners restarted), the new miner (pid 8567) started 00:08:33Z, its first block was accepted 00:08:34Z, the kernel race settled at 26.9 MH/s 00:09:09Z. Lost: 11 min 20 s of 26 MH/s. In step 1 this was masked: node 1's 23:24:06Z restart was followed by the Mac's own engine restart (the update) at 23:29:05Z, and the Mac was paused anyway |
| PC 2 at the epoch boundary | 23:52:22Z | 40 s after PC 2's first 0.3.11 pack the hour turned (f4d9d3d8... to cb5b51cc...): hourly program changed: the pair was not prepared (unexpected seeds); the worker compiles inline, 80 s of worker fault: seed mismatch: the worker holds another program; preparing the current pair epoch cb5b51cc... day 2, the worker on the new epoch at 23:53:42Z (113 MH/s), one did not come back within 180 s line for the old worker instance at 23:56:01Z. The restart landed inside the minute before the boundary, before the next epoch's prepare had run; the attempt-aware path (program pack checked ... attempt 0) recovered it without a job. Also at 23:52:14Z: prover: block 74094 shard 0: ... Failed to create the CUDA prover impl: ... PermissionDenied (a GPU-server socket ...), the 0.3.10 socket class again on the first shard after the update; shards after it proved (block 93665 shard 0 paid 0.9446373 IGN at 00:06:09Z) |
| the laptop (PC 37ba0461) | absent | no upload since 23:28:28Z (section 7) and no card on the console at 00:08Z; both update-now jobs wait for it in the jobs file; its 0.3.10 app takes 0.3.11 and the nine-field object through the manifest when it is next on the air |
| PC 1 (ae432dc7), Sam's Mac (3a9bf309) | pending their relaunch | PC 1 down since 22:31:06Z (section 3a), Sam's Mac quit since 20:47Z on 0.3.9; each takes the manifest at its relaunch: the OLD app writes the nine-field object at the manifest take (ota.rs 661) and its old node, if restarted with that file before the 0.3.11 install lands, dies on the unknown fields (deny_unknown_fields, the reviewer's C39); the install then starts the 0.3.11 node on the file, so the worst case is one node death inside the update, to be read in the morning's intake |
9. The digest sweep
| Node | Binary | Digest | Since |
|---|---|---|---|
the observer (/tmp/igneum-devnet/observer-v4, rpc 26640, json 28640) |
igneumd/2.1.0-89dfcb95 (bd7f043c...) |
0139ab9dc2992d449ec787d8f021974933631eb55740ab4b6ce9d5c226e72888 | 23:56:59Z |
node 1 (/tmp/igneum-devnet/node1, rpc 26610) |
the same | 0139ab9d... | 23:57:12Z |
the seed (188.245.5.161, igneumd.service) |
igneumd/2.1.0-89dfcb95 (63cf490d..., glibc 2.36 target) |
0139ab9d... | 23:57:30Z |
PC 2 (1ccfe586, devnet-v4, rpc 26610) |
the 0.3.11 installer's igneumd.exe be8e83c0... (PC 2's own build job) |
0139ab9d... | 00:01:42Z |
| the Mac (d937c69d) | attached to node 1 (no node of its own) | node 1's | 23:57:12Z |
| the laptop (37ba0461) | 0.3.10 node, 1f4b4425... at its last upload | pending | silent since 23:28:28Z |
| PC 1 (ae432dc7) | 0.3.10 node, 1f4b4425... | pending | down since 22:31:06Z |
| Sam's Mac (3a9bf309) | 0.3.9 node a24ab01a | pending | quit since 20:47Z |
The chain at 00:10:39Z: tip DAA 141,645 on the observer (age 1.4 s), node 1 with 2 peers (the seed and PC 2's side), PC 2 with 1 peer at 106 to 109 MH/s, the Mac at 26 MH/s, every live node on 0139ab9d... Epoch 43 (DAA 154,800) is about 3 h 45 min out at 0.965 blocks/s (about 03:55Z); the class v3 and proving v1 lines of every node name it.
10. The next cut
| Branch | What | Why not 0.3.11 |
|---|---|---|
ember-tune e600e63 |
the C35 guard: Engine.no_ota (IGNEUM_APP_NO_OTA=1 or --sweep skips the OTA tick and refuses Check now, logged at start), 6 lines in engine.rs, separable like the quit-source hunk; the playbooks and tools/ci/second-engine-check.sh (file not pipe, tree ended, NO_OTA set) ride with it. Without it any test engine on an old Cargo version can install over the fleet's app (section 3a) |
arrived after the merge |
ember-tune b671c8b (and the tune behind it) |
the C35 fix: Cmd::Quit(&'static str) so every "quit:" line names its sender (the window host's stdin, the host gone, POST /api/quit, the sweep's end), elevation_allowed() = Power control alone (the unattended sweep on PC 1 raised one UAC prompt at 22:30Z under the old rule), no power cap at start under --sweep; the quit-source hunk is separable (main.rs 2 lines, server.rs 1 line, engine.rs the Quit arm, elevation_allowed and its test) |
arrived after the tree closed at 23bc2b2 (the app, the DMG and PC 1's job carry it); not among the branches named for this cut |
fud-close (the ledger closer's main branch, a22ba27a60a6f1c64; its ready tip was due about 23:05Z) |
45 public-text fixes on the site and litepaper, spec 8.3 and 8.8, two CI checks, relay fixes; touches packfile.h and host.c, so taking it means the two Windows workers, the Mac worker and the DMG rebuilt from the merged tip (G5) |
offered by the Counter ASIC coordinator at 22:5xZ after the tree closed; not among the branches named for this cut; the fork-side ledger-fixes is not in 0.3.11 either |
explorer d7e797c (and 3e01212) |
/api/stats gains proving; tools/ci/public-api-check.mjs then FAILS when the live API lacks it, and ci.yml runs that check against the live site on every master push, which reads the OLD API until Vercel redeploys after the push (the reviewer's C36) |
not in 23bc2b2 (only on the explorer branch); its merge needs the check to retry for a few minutes or to require proving only when observer_updated_at is newer than the commit |
| the app's job queue | an update-now job pre-empts a running job instead of queuing behind it, or the queue reports its wait in the STATUS line (tonight PC 2's update waited 11 minutes behind a hung sweep's cap, section 7; the Counter ASIC coordinator's ask) | app change |
proving-v1 c36dfea |
docs only, after the code tip 22c2363 | its agent's choice: docs follow |
the 0.3.10 list (release-0.3.10.md section 11): fork pack-loop 05ef0fa3, job-console, the rest of opencl-rdna4, opencl-rdna4-telemetry |
unchanged |
11. Open after the cut
| Item | What | Owner |
|---|---|---|
| the miner's dead channel (section 8) | igneum-miner mine grpc://... keeps subscribed=true after the node restarts and retries template and submit calls on the closed channel for ever (11 min 20 s tonight, 5,317 errors, hash burned at 26 MH/s with 0 accepted); it must re-subscribe on the first Not connected to server and the app's card must read "the miner lost the node" instead of "mining, 26 MH/s". Until the fix: every hand restart of node 1 is followed by publish-jobs.sh add --kind restart --target d937c69d --what miners (the runbook's step), and the Mac's card is read by its accepted count, not its MH/s |
miner (next cut) |
| the HiveOS package carries no override | packaging/hive/h-run.sh starts the rig's node without --override-params-file; a rig on igneum-hive-0.3.11.tar.gz runs on genesis params and is refused by every peer; the file must come from the Flight Sheet's extra config as PEERS= does (section 5) |
packaging (next cut) |
| the prepared pack and a restart near the hour (section 8) | an engine restarted in the last minute before an epoch boundary has no prepared pair for the next hour (80 s of seed-mismatch faults on PC 2 at 23:52:22Z, recovered by the attempt-aware path); the prepare should run at engine start for the next epoch too, or the update-now path should wait out the boundary when it is under 2 min away | app |
| PC 2's first shard after an update | Failed to create the CUDA prover impl: ... PermissionDenied (a GPU-server socket ...) on block 74094 at 23:52:14Z, the 0.3.10 socket class (fixed 22:01Z for the running engine; the root socket outlives the engine restart); the later shards proved. The socket unlink belongs in the engine's prover start, not only in the playbooks (prover-socket-check.sh covers the playbooks) |
app / proving |
| the relay machine named PC1 (section 3a) | it is the 1ccfe586 box: rename it (node tools/relay.mjs name DESKTOP-KMCV30N PC2), give the ae432dc7 box its own agent, and no relay task to "PC1" without the Counter ASIC coordinator's word until then |
relay (the project lead in the morning for the PC 1 box) |
| PC 1 down since 22:31:06Z | relaunch in the morning (the installed app, double-clicked or through the Start menu, never elevated); it takes 0.3.11 and the nine-field object through the manifest, one node death inside the update possible (section 8); the intake's first lines of win-ae432dc7-... are the check; ember-tune b671c8b carries the C35 fix for the quit's sender |
the project lead, then the release engineer |
| the laptop (37ba0461) | silent since 23:28:28Z, off the console; its 0.3.10 install kept it silent 55 min once already; when it uploads again its first STATUS line must show 0.3.11, the nine-field object in the manifest log line and 0139ab9d... | watch |
| Sam's Mac (3a9bf309) | 0.3.9, quit since 20:47Z; min_supported is 0.3.0 so it updates straight to 0.3.11 at its relaunch |
watch |
| the app's job queue | an update-now queues behind a running job (PC 2's step-1 update waited 11 min behind a hung sweep's 30 min cap, section 7); pre-empt, or report the wait in the STATUS line | app (section 10) |
| the public index at the edge | dl/public/igneum-downloads.json alternates between the unsigned and the signed copy at the edge for minutes after a deploy, so the ship's verify refuses it once per cut (0.3.10 and 0.3.11 both); --from console after it settles is the workaround; the verify should accept either copy while both carry the cut's hashes, or the deploy should purge the edge |
ship-app |
| the pre-push site flip | the pre-push hook rebuilds site/ and leaves the tree modified (git checkout -- site/ after every push); the site build is not idempotent |
site |
| the explorer branch's strict check (C36) | public-api-check.mjs on explorer d7e797c fails against the live API until Vercel redeploys; merge it right after a master push, not before |
explorer (section 10) |
| C1 (the reviewer) | the 0.3.10 exportSegments lacks daaScore and feesV1ActivationDaa; proving-v1's rpc.rs:982 (eb32c645) carries them; the decision on which the aggregator reads is due 16:00Z 6 October |
the Counter ASIC coordinator |
| the Metal worker at resume (G4b) | "the pair was not prepared" at the Mac's 23:39:10Z resume (section 7), compiled inline; same class as the PC 2 boundary row above | app |
| the 0.3.10 list | release-0.3.10.md section 11, unchanged: fork pack-loop 05ef0fa3, job-console, the rest of opencl-rdna4, opencl-rdna4-telemetry, the per-job zip pin check |
12. The merge and the push
The plan's last branch commit 7ab72f3 (sections 8, 9, 11); git merge --no-ff release-0.3.11 onto the local master 063baf6 (the CLAUDE.md
standing rule of 22:52Z, unpushed until now) = master 30cd292 (parents 063baf6, 7ab72f3), no conflict (the site files merged clean this time);
pushed 00:13:04Z (b38f3de..30cd292); the pre-push hook's site flip (bench.html, downloads.json, index.html, journey.json, miner.html)
restored with git checkout -- site/. CI on 30cd292: ci run 37392831312, windows-ci run 37392831184 (both started 00:13:13Z). No
release tag: 0.3.9 and 0.3.10 carry none (the repo's tags are backups and the CA2 history marks), so none is cut here without the word.
The fleet at the push: the Mac 25.7 MH/s with 55 accepted since its miner restart, PC 2 105 to 116 MH/s, every live node on 0139ab9d...,
tip DAA 141,715 at 00:12:36Z.