igneum-network/igneum
1
0
Fork 0
Code Issues Pull requests Releases Activity
igneum/docs/plans/release-0.3.9.md
igneum-labs 7eed16a29a Pre-public scrub, the text pass (7 October 2026, 19:5x UK): no founder name, personal login, earlier business or personal address in any tracked text file, and a gate check that keeps it so 
The sweep (main's item 1): 199 tracked text files, 783 lines. The founder's full name, first name and possessive become "the founder" (sentence starts capitalised); the lowercase operating-system user name in WSL paths and commands becomes <user>; the second owner login becomes "the second owner login"; the three earlier businesses and the two other brands become "the other business", "the earlier entity", "the earlier business" and "another brand"; the Chrome profile rule names the igneum.network profile, not the profile's label. The standing commit login igneum-labs is not a founder term here: the fresh-repository step renames it in the history (docs/plans/history-rewrite.md, tools/repo/fresh-repo.sh).

The patterns never appear in plain text in the tree (a plaintext list would be the hit): tools/ci/founder-strings.b64 (perl regex, tab, a sample per row) is read by tools/ci/founder-strings-check.sh (every tracked text file, perl, known-failed first: the self-test plants each row's sample in a fixture and the hit must name the file), by tools/community/discord-hooks.mjs (the guard's founder and business rows; the test takes its fixtures from the samples) and by tools/repo/fresh-repo.sh (the business names of the rewrite rules). site/forbidden-strings.txt carries the same patterns as b64: lines, decoded case-insensitive by site/scrub.mjs and tools/ci/launch-gates-check.mjs (whose fixture now plants an encoded made-up name). The check runs in the gate's tree checks on every merge.

Not in this commit, by main's word: the 105 commit messages and 40 personal-identity commits that need the history rewrite (listed, not run), and the secrets found by gitleaks over the history (reported with owners).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 18:39:50 +00:00

408 lines
41 KiB
Markdown
Raw Blame History

# Igneum Miner 0.3.9: the fee switch to the devnet, 5 October 2026
Release engineer, from 15:34 UTC, the founder: "give it everything we have, ready for 7pm" (18:00 UTC). Worktree
`/Users/joshm/Projects/igneum-wt-ship039`, branch `release-0.3.9` from master 23d11d5 (the 0.3.8 merge; nothing else
on master). Every Mac build through `/Users/joshm/Projects/igneum/tools/lock/with-lock.sh build` at `nice -n 19` with
`-j 4`. Times are UTC. `release-0.3.8.md` is the template for the cut and the proving rollout; `release-0.3.6.md`
section 5 step 4 for the override publish.
## 1. What 0.3.9 carries
| Change | Where | State |
|---|---|---|
| The prover core mirrors both fee tables and the `fees_v1_activation_daa` switch (the shard input carries the schedule and the block's DAA score; one pinned guest meters before and after the switch as the node does) | `proving/igneum-prove/core/src/config.rs`, `pgas.rs`, `shard.rs`, `executor.rs`, the export and the host, branch `fee-switch` | (pending) |
| A new pinned guest: new shard program id, new aggregator id | `proving/igneum-prove/elf/` | (pending) |
| The runbook for the switch | `docs/plans/fee-switch-devnet.md` | (pending) |
Scope change at 16:16Z (coordinator, from the txgen agent's result: shards with content fail the native-execution veto
unless `igneum_exportSegments` names the block index and body position per entry): 0.3.9 also carries the NODE change
`txgen-export` a24ab01a (one file, `igneum/exec/src/rpc.rs`, 124 insertions, plus its unit test), from the separate
clone `vendor/igneum-node-txgen`, fast-forwarded onto the fork's `release-0.3.6` in `vendor/igneum-node-036`
(2b6d23ef -> a24ab01a, 16:23:58Z). The exporter side (`export blocks_of`, fixtures block-72803 and block-72854 with
node-plan sidecars) is on master (txgen merge 49796b0, taken into this branch). The RPC change touches no consensus
parameter, so the digest must stay f10a4eab... (checked with the 20-s scratch run below). The node is therefore
rebuilt: Mac arm64 here under the lock, Windows exes by the Mac cross-build (never a PC-built Windows node),
Linux by a PC 1 build job (for HiveOS; the seed keeps the glibc-2.36 cross-build of 2b6d23ef, the change is RPC only
and the hand nodes and the seed export nothing), and the node's `kaspa-rpc-service` and `igneum-exec` suites on PC 2.
## 2. The branch
| Commit | What |
|---|---|
| 9f1c555 | `origin/master` merged in (local master was 7 commits behind: the fud-b wording merge, the site rebuild, rotation-3 executed; no app or packaging file, two lines in `tools/ship-app.mjs`); the ship tool's preflight refuses a tree behind origin/master |
| 49796b0 | `origin/master` merged again at 16:23Z (txgen 49796b0, testnet-infra 83a0bcf, public-release 1eb94b0: the exporter's `blocks_of`, the node-plan fixtures, `build-job.mjs --node-tests`, `ship-app.mjs --public`, dl/public, the HiveOS package); no conflict, 0.3.9 still in all 6 |
| f7a41a2 | Merge `fee-switch` (15bb6cd, landed 16:24Z). Conflicts: `proving/igneum-prove/export/src/main.rs` (only the two `use` lines: master's `ensure` kept, fee-switch's `FeeParams`, `FeeSchedule` taken, `SHARD_PROVING_GAS_BUDGET` gone), `docs/bench-log.md` (both entries), `docs/testnet/README.md` (both sentences), `site/index.html` and `site/journey.json` (master's, then `node site/build.mjs`: 13 pages, `link-check` 518 links, 0 broken) |
| 2b6bf7f | `Igneum Miner 0.3.9: ...`, the six version files (`node tools/ship-app.mjs --check`: 0.3.9 in all 6) |
| dfc0cfc | `packaging/windows/node-source.pin` = a24ab01a (written by `push-inputs.sh`, section 5b) |
| 00c7b4f | `infra/devnet/restart-hand-nodes.sh` and `restart-seed.sh` take the fork tip's commit and the seed's cross-built binary by variable (section 5c); pushed 16:42Z, `windows.yml` dispatched again on it: run 37342911129, green 16:48:10Z. THE RELEASE COMMIT for the installer (every later commit is docs and site) |
| 97b1a74 | `origin/master` fb076de merged (12 more commits: fud-a round 6, the Igneum Labs LTD imprint, `tools/ci/no-conflict-markers.sh` with a pre-push hook that rebuilds the site; among the build inputs only `.github/workflows/ci.yml`). `docs/bench-log.md` both entries; the site taken from master and rebuilt (519 links, 0 broken, no markers) |
| (next) | `site: rebuilt on this tree`: the pre-push hook's build of `site/index.html` and `site/journey.json` differed from master's copy (two journey entries' wording), which left the tree dirty and the ship's first run (16:48:47Z) stopped in preflight; the generator's own output committed |
| 1269db8 | `origin/master` 318a2de merged (coordinator 16:32Z: housekeeping, the signed jobs envelope `igneum-jobs.signed.json` on both sides, `test-publish-jobs.sh`, `ship-app.mjs` carries the envelope in FOLDER_FILES and `--public`); it rewrites four app sources (`jobs.rs`, `jobrun.rs`, `jobbuild.rs`, `bin/ota-sign.rs`), so the chained build that had started at 16:31Z was killed at its app step and restarted from this tree, and the app tests rerun. The fork's own `housekeeping` branch is NOT in 0.3.9 (0.3.10) |
## 3. Tests and checks, with the command
| Tip | Command | Result |
|---|---|---|
| 23d11d5 (before the merge) | `tools/ci/identity-check.sh` | 0 hits over 206 files, 15:36Z |
| 23d11d5 | `tools/ci/copied-sources-check.sh` | every copying build script re-stamps its sources |
| 23d11d5 | `tools/ci/pinned-guests-check.sh` | elf/ matches its manifest, no script builds a guest outside pin-guests.sh |
| 23d11d5 | `node tools/ci/check-workflow-shell.mjs` | 12 run blocks in 2 workflows, 25 .ps1, 0 findings |
| 23d11d5 | `node --test relay/test/parse.test.mjs relay/test/auth.test.mjs relay/test/wake.test.mjs` | 15 pass |
| 23d11d5 | `node --test app/igneum-app/ui/notices.test.mjs app/igneum-app/ui/update-card.test.mjs` | 10 pass |
| 0.3.9 bump | `node tools/ship-app.mjs --check` | 0.3.9 in all 6 (bumped with the tool's own six patterns, 15:38Z) |
| 2b6bf7f | `tools/ci/identity-check.sh`, `copied-sources-check.sh`, `pinned-guests-check.sh` (the NEW `elf/` matches its manifest), `check-workflow-shell.mjs`, the relay tests (15), the UI tests (10) | all as at 23d11d5, 16:28Z |
| fork a24ab01a + app 2b6bf7f | PC 2 build job `build-20261005-162506` (`node tools/build-job.mjs run --target 1ccfe586 --targets linux --node-tests "kaspa-rpc-service igneum-exec" --app-tests "igneum-app"`) | started 16:25Z, done 16:27:55Z (139 s): Linux node built in 78 s (cache warm), `kaspa-rpc-service` and `igneum-exec` suites exit 0 in 36 s, `igneum-app` exit 0 in 5 s; its Linux igneumd 374fccad... (48,736,744, glibc 2.39: not for the seed) placed under `infra/cross/out/` of this worktree |
| 1269db8 | `cargo test --release -p igneum-app` (under the build lock, this Mac; the final app tree, with the housekeeping merge) | ok: 76 (lib) + 27 (ota-sign) + 8 (prove-verify), 0 failed, 16:33 to 16:41Z |
| 1269db8 | `cargo test --release` in `proving/igneum-prove` (the whole workspace, under the build lock, this Mac: 11 min wall, almost all of it compiling beside the chained build) | ok: core 8, export 3 + fixtures 2, host 9 (1 ignored), aggregator, pin and program 0, 0 failed, 16:32 to 16:43Z |
| 1269db8 | `gh workflow run windows.yml --ref release-0.3.9` -> run 37341810898 | green 16:39:45Z (parse checks; engine, window host, payload, installer, smoke run; the inputs step against the a24ab01a inputs of 5b) |
| 9f1c555 + bump | `cargo test --release -p igneum-app` (under the build lock, this Mac; the app crate is not touched by `fee-switch`, so this run stands for the final tree) | ok: 75 (lib) + 26 (ota-sign) + 8 (prove-verify), 0 failed, 15:49 to 15:54Z (target dir cloned by APFS from the 0.3.8 worktree) |
## 4. The state of the network before the cut (15:35Z)
| Node | Binary | Override file | Digest |
|---|---|---|---|
| Mac app node d937c69d, PC 1 ae432dc7, PC 2 1ccfe586 | app 0.3.8, node 2b6d23ef | the manifest's `{33000, 84100}` | (the live one) |
| The observer and node 1 on this Mac (hand nodes) | `vendor/igneum-node/target-release/release/igneumd`, the 0.3.5 build 20139145: NO `fees_v1_activation_daa` in the binary (0 string hits; 2b6d23ef has 3) | `/tmp/igneum-devnet/override-v3.json` `{33000, 84100}` | f10a4eab... |
| The seed 188.245.5.161 | `/opt/igneum/v4/bin/igneumd` c98a23da..., the 0.3.5 cross-build, glibc 2.34; Debian 12, glibc 2.36 | `/etc/igneum/override-v3.json` `{33000, 84100}` | f10a4eab... |
| PC 37ba0461 | 0.3.7, silent since 13:52Z | | |
| Sam's Mac 3a9bf309 | 0.3.5, node 20139145, silent since 09:43Z; reads the OLD downloads folder | | |
So the three hand-run nodes cannot read a `fees_v1_activation_daa` line at all: they must move to the 2b6d23ef node
before the switch. The 0.3.6 digest check (release-0.3.6.md 8g) showed 2b6d23ef with the same override file gives
the same digest f10a4eab..., so that swap changes nothing on the network. The Mac arm64 2b6d23ef binary is the DMG's
(`vendor/igneum-node-036/target-integration/release/igneumd`, 64138a17...). The PC 1 Linux build of 2b6d23ef
(c24fd2c5..., in the 0.3.6 worktree) needs glibc 2.38 and the seed refuses it (`GLIBC_2.38 not found`), so the seed's
binary is the Mac zig cross-build (`infra/cross/build-linux.sh`, glibc 2.36 target), started 15:41Z under the lock
(`NODE_SRC=vendor/igneum-node-036`, target dir `vendor/igneum-node/target-release-linux`, the 0.3.5 cross-build's cache).
The hand nodes moved first, 15:44Z, with the UNCHANGED override file (the scratchpad script extended to take the
binary and an optional H: `restart-hand-nodes-039.sh 84100`): observer restarted 15:44:04Z, node 1 15:44:16Z, both
`igneumd/2.1.0-2b6d23ef`, both `Fees on igneum-devnet: ... calibrated v1 from DAA score never`, both digest
f10a4eab... (unchanged), peers reconnected within 20 s and blocks accepted at DAA 112,496. The field itself was
checked on the same binary with a scratch node and `fees_v1_activation_daa: 200000` (15:42Z, 25 s, ports 60995/60996):
`Calibrated v1 fees from the override file: chain blocks metered with the adopted table, budgets and floors from DAA
score 200000`, digest d5b5028b... (the digest moves with H, so the H of the runbook has its own).
The cross build: try 1 (15:41 to 15:54Z) compiled every crate and was killed at the final link of `igneumd` by a
SIGTERM from outside this session (not a compile error; the sender is unknown); try 2 (15:55 to 15:59:11Z, 270 s,
the link only) gave `igneumd` c28756d08a61194d8ef945885ce22d8c48d1045843bbb01a41f0e2358b226e3e (47,451,112 bytes,
`GLIBC_2.34` at most, 2b6d23ef in its strings, the fee field 3 times) and `igneum-miner` a122885a... (9,631,272).
The seed moved at 16:00:21Z with the UNCHANGED override file (the scratchpad `restart-seed-039.sh 84100`, which
copies the file by scp and swaps the binary as the 0.3.5 hot-swap did; its first run aborted at `igneumd.new
--version` because the node's `--version` exits non-zero under `set -e`, nothing changed, `|| true` added, run
again): `/opt/igneum/v4/bin/igneumd` c28756d0..., previous kept as `igneumd.prev-035`, unit active 16:00:22Z,
`igneumd/2.1.0-2b6d23ef`, `calibrated v1 from DAA score never`, digest f10a4eab... (unchanged), first peer back at
16:00:29Z. So from 16:00Z every node on the devnet runs 2b6d23ef and can read the switch.
Downloads folders: the rotation was executed at 15:25Z (`rotation-phase-2.md` section 8): `dl-token.next` became
`dl-token`, the OLD folder was stripped to the 0.3.8 manifest, its DMG and its installer for Sam's 0.3.5 Mac. So
`--dl-both` (which needs `dl-token.next`) no longer applies; the ship publishes the NEW folder (every 0.3.6+ app) and
the OLD folder gets the same 0.3.9 manifest, DMG and installer by hand (`publish-manifest.sh --dest --base-url`).
## 5. The Mac binaries and the DMG
The Mac node of fork a24ab01a: `CARGO_TARGET_DIR=vendor/igneum-node/target-036 cargo build --release -j 4 -p kaspad -p igneum-miner --features kaspad/igneum-pow`
from `vendor/igneum-node-036`, under the lock, 16:24:19 to 16:28:53Z (incremental on the 0.3.6 target): `igneumd`
03b8959054d89448023e5b4cec21c8fc2425e53eac900e1813631769ec24386e (40,968,064 bytes, `a24ab01a` in its strings),
`igneum-miner` ef438be2... (8,514,864, byte-identical to 0.3.6's: the miner did not change). Copied into
`vendor/igneum-node-036/target-integration/release/` (the 2b6d23ef igneumd kept beside it as `igneumd.2b6d23ef`).
### 5a. The hand nodes on a24ab01a
The observer (16:30:25Z) and node 1 (16:30:37Z) restarted on the a24ab01a Mac binary with today's override (the
scratchpad script, `BIN=` the new build): `igneumd/2.1.0-a24ab01a`, digest f10a4eab... on both, peers back within 20 s,
new pids 16875 and 17017 (node 1 under caffeinate 17019). The digest of a24ab01a was read first with a scratch node
(ports 60995/60996, 22 s each): today's override `{33000, 84100}` -> f10a4eab... (16:29:19Z); the switch override
`{33000, 84100, fees_v1_activation_daa 210000}` -> `Calibrated v1 fees from the override file: ... from DAA score
210000`, digest ab8847da538dead1dc10e046dfaadab3c1c35928e3748810c4e050d4a886087a (16:29:45Z), equal to the runbook's
reading on 2b6d23ef: the RPC change moves no parameter.
### 5b. The Windows payload inputs (16:31 to 16:32Z)
The Windows exes by the Mac cross-build (`CARGO_TARGET_DIR=vendor/igneum-node/target-036-win proto-cuda/windows-node/cross-build.sh vendor/igneum-node-036 4`,
under the lock, 16:24:23 to 16:29:04Z, 4 min 39 s incremental on the 0.3.7 target): `igneumd.exe`
0e844ff7b5e82ed987f97c1cd7197a16ca609de7fa4688f0a2ca6e420007bb41 (50,779,136 bytes, `a24ab01a` in its strings),
`igneum-miner.exe` eb77bf93... (10,778,624, byte-identical to 0.3.7's). Installed into
`vendor/igneum-node-036/target-integration/x86_64-pc-windows-gnu/release/` (the 0.3.7 pair kept in `0.3.7-exes/`), then
`IGNEUM_WIN_RELEASE=<that> IGNEUM_NODE_SRC=vendor/igneum-node-036 packaging/windows/push-inputs.sh` from the worktree:
signed, deployed, `payload-inputs.json` HTTP 200 at 16:32:09Z, `node-source.pin` a24ab01a2e10cecf575bcea372310b871081ec64.
PC 1 build job `build-20261005-162451` (`--targets linux`, 16:24:51Z, done 16:29:15Z, 219 s): every stage ok, Linux
igneumd 445b25fa... (48,736,744, glibc 2.39, for HiveOS, not the seed), placed under `infra/cross/out/`.
`cargo check --release` of the merged prover workspace (core, export, host): ok, 16:26 to 16:30Z.
### 5c. The seed on a24ab01a
The Linux node of a24ab01a for the seed: `infra/cross/build-linux.sh` (zig, glibc 2.36 target) from
`vendor/igneum-node-036` under the lock, 16:30:10 to 16:41:04Z (654 s): `igneumd`
7e26e374586ad8ba539e371c6cbbd74f0f77aee1e2e97cd25f0182fcbda458a5 (47,453,736 bytes, `GLIBC_2.34` at most, `a24ab01a` in
its strings), kept in the main checkout's `infra/cross/out-039/`. Staged on the seed as `/root/v4/out/igneumd.039x`
(sha equal, `--version` runs on Debian 12), then the hot-swap with today's override (scratchpad `restart-seed-039.sh 84100`):
restart 16:41:37Z, unit active 16:41:38Z (MainPID 117943), `igneumd/2.1.0-a24ab01a`, `calibrated v1 from DAA score
never`, digest f10a4eab... (unchanged), a peer back within 10 s; the 2b6d23ef binary kept as `igneumd.prev-035`
(the name the 16:00Z swap gave the 0.3.5 one is now `igneumd.prev-035` too: the second swap overwrote that copy with
the 2b6d23ef binary, so the 0.3.5 binary survives only as `/root/v4/out/igneumd.035`).
The repository's `infra/devnet/restart-hand-nodes.sh` and `restart-seed.sh` (from `fee-switch`) are adjusted on this
branch for the new tip: the hand-node script checks for `IGNEUMD_COMMIT` (default a24ab01a) instead of 2b6d23ef; the
seed script takes `IGNEUMD_LINUX` (default `infra/cross/out-039/igneumd`) and `IGNEUMD_LINUX_SHA256` (default
7e26e374...) instead of the PC 1 build c24fd2c5..., which the seed's glibc 2.36 refuses (`GLIBC_2.38 not found`,
tried 15:36Z); both still take the whole override object and print the pids and the digest lines. They are the
scripts of section 10 steps (c) and (d).
### 5d. The app, the prover tools and the fixtures (the chained script, 16:33 to 16:43Z)
`build-chain-039.sh` (the 0.3.8 chain with the paths changed), under the lock: `cargo build --release` of the app
(16:33 to 16:42Z, the housekeeping merge rebuilt the engine and the signer), then
`cargo build --release -p igneum-prove-export -p igneum-prove-host` in `proving/igneum-prove` (16:42 to 16:43Z, the
host embeds the NEW `elf/`; no Succinct toolchain). `igneum-prove-host --mode id` on this tree:
```
pinned guests: shard program id 0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a (2832504 bytes, sha256 0x150f4c05a2951fc5) aggregator id 0x474678f35f7545db28055d5e5bbc308231d84a5a072202087a2a8d5b09123896
```
Both equal `elf/manifest.json` (pinned 2026-10-05T16:20:38Z, SP1 6.8.1, circuit v6.1.0). `--mode native` (the package
gate's native half) on every fixture in `proving/fixtures/`: the nine block fixtures (338, 341, 344, 56 x2, 58927,
72803, 72854, 78) and the three fee-switch fixtures (`fees-switch-prototype`, `fees-v1-shards2`, `fees-v1-shards3`)
all ok. The chain's loop had stopped at `block-72803-skipped-copies.json.node-plan.json`, the node-plan SIDECAR that
the `block-*.json` glob now matches (`missing field format`: not a fixture), so the last two block fixtures and the
three fee-switch ones were run by hand afterwards. The chain's DMG step therefore ran separately (below).
### 5e. The DMG (16:44:09 to 16:45:08Z, under the lock)
`NODE=<fork>/target-integration/release/igneumd MINER=.../igneum-miner packaging/mac/build-dmg.sh` from the worktree:
node igneumd 2.1.0 (a24ab01a, 03b89590...), engine 0.3.9, the prover host 6ede0b9e... (58,544,672 bytes before the
strip, 45,314,144 in the bundle) and export 347da361..., manifest folder fingerprint ed9c4d2e (`dl-token`, the
rotated one), `hdiutil verify` VALID.
| Artefact | sha256 | Size |
|---|---|---|
| `packaging/mac/dist/Igneum-Miner-0.3.9.dmg` (build 202610051644) | 5e57c5735dc99e44475c43fba8ecffadae9dbd3a150c32ef4858587a98256de3 | 41,333,873 |
86,454 bytes larger than 0.3.8's (41,247,419): the new ELFs and the host's fee mirror.
## 6. The prover package for PC 2
`SKIP_GATE=1 proving/windows-wsl2/make-package.sh <scratch>/igneum-prove-wsl2-039.zip` from the worktree at 1269db8 (16:34Z):
b5e5fa6ed4d62fee43b8fc737ce41c3665b5f4573cf9426e34012874031a282a, 1,530,932 bytes; `package/proving/igneum-prove/elf/`
carries the NEW ELFs (program 2,832,504 bytes, aggregator 319,744), the two keys and `manifest.json` (shard
`0x2b1a81cb...`, aggregator `0x474678f3...`). The gate skipped on instruction as in 0.3.8; its native half runs in the
chained build on the DMG's host (section 5).
The fetch job `fetch-prove-039` (`--dir prove --extract --extract-dir igneum-prove-wsl2-039 --fresh`) was added at 16:34:54Z
through the MAIN checkout's `publish-jobs.sh` (coordinator's rule, 16:31Z: that script writes the signed envelope
`igneum-jobs.signed.json`; a publish from an older script leaves it stale, which is exactly what `prove-off-pc2-039` had done at
16:02Z through this worktree's copy: live envelope 15:53:42Z against a 16:02Z pair). The add stopped at the signature:
the main checkout's `app/igneum-app/target/release/igneum-ota-sign` was built 07:38Z, before the `sign-jobs` subcommand
existed (the script printed the signer's usage). The signer built from this tree (the chained build's app step, which
carries `bin/ota-sign.rs` with `sign-jobs`) is copied over it, then `publish-jobs.sh sign --deploy` publishes the file
with the job and a fresh envelope. BUT: the failed `add` had not written the job (the file is replaced only after it
is signed), so the 16:42Z `sign --deploy` re-published the file WITHOUT `fetch-prove-039`, PC 2 had nothing to fetch, and
the 16:46:04Z live file (84 jobs, another agent's build job) confirmed the absence at 16:48Z. The job was added again
at 16:49:20Z through the main script with the new signer (`added: fetch-prove-039 ... until 2026-10-06T04:49:21Z`,
envelope and pair verified, apps woken, stamp f45d504a); PC 2 was then busy with another agent's build job
(`build-20261005-164604`, started 16:48Z), one job at a time, so the fetch queues behind it. The signer step: the signer from this tree (595,024 bytes, built 16:41Z, `sign-jobs` present) copied
over the main checkout's (the 07:38Z one kept as `igneum-ota-sign.pre-envelope`), `publish-jobs.sh sign --deploy`
16:42:13Z: `live jobs files verified at .../igneum-jobs.signed.json and the plain pair (try 1 of 12)`, apps woken (stamp
2026-10-05T16:42:14Z.9e4b22bb). Lesson for the class: a script that needs a newer signer than the checkout has built
fails with the signer's usage text and no exit code of its own; the next `publish-jobs.sh` should check for `sign-jobs`
in the signer's usage before it writes anything (a 0.3.10 item).
## 7. The proving rollout order (proving/README.md), with times
| Step | What | When |
|---|---|---|
| (1) provers off | Mac: already off (the app's `/api/state`: `enabled false, status off`, 15:46Z). PC 2: run job `prove-off-pc2-039` (the 0.3.8 PowerShell script: `POST <app.url>/api/prove {"on":false}`, the app state, the node's `igneum_getProvingStatus`), published 16:02:06Z (apps woken, stamp e933f990), started early so the drain overlaps the wait for the `fee-switch` branch | ran 16:02:27 to 16:02:38Z: `api/prove off: {"ok":true}`; PC 2's node pool `{entries 18, failed 0, pending 0, verified 178}`, paidShards 388, tip DAA 113,603 |
| (2) empty pool | A 30-s watch on the Mac app node's `igneum_getProvingStatus` that prints every change and every error line: 16:02:36Z `entries 18, pending 0, verified 178`, tip DAA 113,601; then one entry fewer every 30 to 60 s (16:03:07Z 16, 16:07:17Z 10, 16:11:48Z 5), no error line, no new record | OBSERVED empty 16:15:50Z: `entries 0, pending 0, failed 0, verified 178` (13 min 23 s after PC 2's prover went off; the 600-DAA record window at 0.965 blocks/s is 10.4 min) |
| (3) new host on every node | The Mac, PC 1, PC 2: by `update-now-039` (published 16:56:12Z through the main script, apps woken, stamp 029c1bb7), see 9. PC 2's WSL tools from the package: `fetch-prove-039` and `rebuild-prover-pc2-039` (7a) | |
| (4) `--mode id` | PC 2 (`/opt/igneum`, in the rebuild job, 16:54:5xZ): shard `0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a`, aggregator `0x474678f3...`, equal to the DMG's host (5d). Mac: the installed app after the update, see 9 | |
| (5) provers on | PC 2: run job `prove-on-pc2-039`; the Mac's prover stays off as before (it was off at 0.3.8 too) | (pending) |
### 7a. PC 2's WSL tools from the package
| Job | Published | Result |
|---|---|---|
| `fetch-prove-039` (fetch, `--dir prove --extract --extract-dir igneum-prove-wsl2-039 --fresh`; the first add at 16:34:54Z never wrote the job, section 6) | 16:49:21Z | ran 16:49:48Z: the zip (1,530,932 bytes, sha256 ok) extracted into `%LOCALAPPDATA%\igneum\prove\igneum-prove-wsl2-039` |
| `rebuild-prover-pc2-039` (run, PowerShell, the 0.3.8 script with the folder names changed) | 16:54:09Z | ran 16:54:42 to 16:54:53Z as root in Ubuntu-24.04: rsync of `package/` into `~/igneum-prove` (`--delete`, target kept), every source re-stamped, `elf/` listed (the package manifest's shard id `0x2b1a81cb...`), `cargo clean -p igneum-prove-core -p igneum-prove-export -p igneum-prove-host`, `cargo build --release ... --features igneum-prove-host/cuda` (the three crates only, every dependency cached from the 0.3.8 build), installed into `/opt/igneum` (host 73,036,832 bytes 29cc476898c3ab2666110ef65f5b432ac243d1e87e565b57b28e4d140fcbfeee, export 3,602,008 bytes f8b7c19d...; `ldd`: 0 libs missing) |
| `--mode id` on PC 2 (`/opt/igneum/igneum-prove-host --mode id`, in the same job) | | `pinned guests: shard program id 0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a (2832504 bytes, sha256 0x150f4c05a2951fc5) aggregator id 0x474678f35f7545db28055d5e5bbc308231d84a5a072202087a2a8d5b09123896`: equal to the Mac DMG's |
## 8. The ship (16:33 to 16:54Z)
`git push -u origin release-0.3.9` at 1269db8 (16:33:55Z), `gh workflow run windows.yml --ref release-0.3.9` -> run
37341810898 (green 16:39:45Z); the script commit 00c7b4f pushed 16:42Z and dispatched again -> run 37342911129
(started 16:42:42Z, green 16:48:10Z: parse checks; engine, window host, payload, installer, smoke run; the G13 inputs
step against the a24ab01a inputs of 5b). Two more merges of master followed (97b1a74, docs and site only; then the
site rebuilt as ea08798 and the plan as the next commit), so the ship state file got `sha` = 00c7b4f, what the
tool's own commit step would have recorded, and the run resumed with `--from ci`, as the 0.3.6 cut did.
```
node tools/ship-app.mjs 0.3.9 --node vendor/igneum-node-036 --branch release-0.3.9 --public \
--activation-height 84100 --deadline-note "proving v0" --notes "The prover mirrors the fee switch (new pinned guest, shard id 0x2b1a81cb...); node a24ab01a: igneum_exportSegments names the mergeset and every entry's block and body position; the devnet fee-switch runbook" \
--from ci
```
No `--dl-both`: the rotation of 15:25Z left one token folder (`dl-token.next` is gone; the tool would refuse it).
`--public` (coordinator, 16:32Z): the same version into `dl/public/` and the `/public/` aliases. Three false starts
in preflight (16:48:47Z, 16:50:25Z, 16:50:54Z): the pre-push hook's site build had left `site/index.html` and
`site/journey.json` modified after the push, then the plan file (tracked since the 97b1a74 merge) was modified; the
tool refuses any modified tracked file besides the six, which is right.
| Step | Result |
|---|---|
| preflight | ok: tree ea08798 + the plan commit, 0.3.9 in all 6, fork a24ab01a, gh igneum-labs, live inputs a24ab01a built 16:31:46Z |
| ci | already: run 37342911129 green |
| fetch | Igneum-Miner-Setup-0.3.9.exe 18.9 MB, igneum-windows-app.zip 26.4 MB |
| dmg | already (5e) |
| copy | the DMG into the downloads folder |
| manifest | 0.3.9 mac+windows, signed (key 8f186e37...), verified locally, and in `dl/public/` |
| deploy | one deploy, 16:51Z |
| verify | the three files and the manifest match the local copies; `dl/public/` and the `/public/` aliases serve the local bytes (try 8 of 12: the edge took about 70 s) |
| console | item #360 "Igneum Miner 0.3.9 shipped (mac+windows)"; exit 0 at 16:54:53Z, 211 s |
| File | sha256 | Size |
|---|---|---|
| Igneum-Miner-0.3.9.dmg | 5e57c5735dc99e44475c43fba8ecffadae9dbd3a150c32ef4858587a98256de3 | 41,333,873 |
| Igneum-Miner-Setup-0.3.9.exe | 1d62a4dbdcef768ebd6fac5e46b199fea4de8854f4cf4c905eee49a6ed9334ce | 19,836,912 |
| igneum-windows-app.zip | 22996a692128a97183c39b37b65275ad33eace4a7d89d83d8f097b315c062adc | 27,683,238 |
Manifest 0.3.9 published 2026-10-05T16:51:46Z.
The live manifests, read 16:55:13Z, compared field by field with the 0.3.8 manifest saved before the cut (section 4 of
this plan, read 15:38Z):
| Field | 0.3.8 (before) | 0.3.9 token folder | 0.3.9 `dl/public/` | 0.3.9 OLD folder |
|---|---|---|---|---|
| version, channel, min_supported_version | 0.3.8, devnet, 0.3.0 | 0.3.9, devnet, 0.3.0 | same | same |
| published_at | 2026-10-05T13:33:22Z | 2026-10-05T16:51:46Z | 2026-10-05T16:51:46Z | 2026-10-05T16:55:1xZ |
| consensus | override {difficulty_v2_activation_daa 33000, proving_v0_activation_daa 84100}, activation_height 84100, deadline_note "proving v0" | identical | identical | identical (carried over from the OLD folder's own 0.3.8 manifest) |
| fees_v1_activation_daa, tuning | absent | absent | absent | absent |
| mac | dmg 53c12b6e..., 41,247,419 | dmg 5e57c573..., 41,333,873, URL in its folder | same bytes, public URL | same bytes, URL in the OLD folder |
| windows | inno-setup 86cffd14..., 19,794,320 | inno-setup 1d62a4db..., 19,836,912, URL in its folder | same bytes, public URL | same bytes, URL in the OLD folder |
The OLD folder (what Sam's 0.3.5 Mac reads; `rotation-phase-2.md` 8b keeps it until 7 October): the scratchpad
`publish-old-folder.sh` copied the DMG and the installer from the token folder (byte-compared), ran
`publish-manifest.sh --dest <old> --base-url <old url> --no-deploy` with the same notes, activation height and deadline
note (the override carried over from the folder's own manifest), one deploy of the site by hand (16:55:18Z; `--deploy`
refuses `--dest`), then `--verify-only` for the OLD base URL: `version 0.3.9, byte-identical, signature OK` (16:55:44Z).
## 9. The machines after the publish (manifest live 16:51:46Z)
Baseline 15:35Z: Mac d937c69d app 0.3.8 node 2b6d23ef DAA 111,914 24.0 MH/s; PC 1 ae432dc7 0.3.8 DAA 111,876 110.2 MH/s;
PC 2 1ccfe586 0.3.8 DAA 111,895 122.3 MH/s; Sam's Mac 3a9bf309 silent 5 h (0.3.5); PC 37ba0461 not on the console's
card any more (silent since 13:52Z). A machine counts as updated only when its node reports a DAA score and the miner
a hash rate on 0.3.9.
| Machine | On 0.3.9 | Its log (`IGNEUM-APP version=0.3.9`) |
|---|---|---|
| PC 2 1ccfe586 (Windows) | `update-now-039` ran 16:56:42Z (30 s after the publish, woken); engine restart 16:56:51Z (unix 1791219412), 39 s after the job; node a24ab01a started 16:56:55Z, `node proof verifier reported: command` +6 s; console 17:30Z: node 2.1.0-a24ab01a, DAA 118,757, 120.6 MH/s | run `win-1ccfe586-20261005-165651` |
| Mac d937c69d | `update-now-039` ran 16:56:43Z; engine restart 16:57:10Z (unix 1791219431); node a24ab01a started (pid 58734), verifier `command` +10 s; console 17:30Z: node a24ab01a, DAA 118,710, 25.4 MH/s | run `mac-d937c69d-20261005-165710`; the node log `node-20261005-165710.log` |
| PC 1 ae432dc7 (Windows) | `update-now-039` ran 16:58:22Z (its wake came 100 s later than the others'); engine restart 16:58:34Z (unix 1791219515); node a24ab01a started (pid 13780), verifier `command` +6 s; a SECOND engine start at 17:26:31Z (run `win-ae432dc7-20261005-172631`, node pid 22280) whose cause is not this rollout (no job of this cut targets PC 1 after 16:56Z; another agent's job or a restart of its own; unverified); console 17:32Z: node a24ab01a, DAA 118,867, 116.8 MH/s | runs `win-ae432dc7-20261005-165834` and `-172631` |
| PC 37ba0461 (Windows, the US laptop) | not reached: silent since 13:52Z (on 0.3.7); the manifest and the job wait for it | |
| Sam's Mac 3a9bf309 | not reached: silent since 09:43Z (0.3.5); reads the OLD folder, which carries 0.3.9 since 16:55Z | |
The machines watch (`watch-machines.sh`, once a minute from 16:57Z) printed every machine on 0.3.9 with node
a24ab01a and a DAA score from its 17:00Z tick on, but its success test still named the old node commit (2b6d23ef), so
it reported TIMEOUT at 17:33Z instead of success at about 17:00Z: 25 minutes lost to a watcher that was not shown to
fire on a known-finished case first (the CLAUDE.md rule, again; the script is fixed to take the commit as an argument).
`--mode id` on the installed Mac app (`/Applications/Igneum Miner.app/Contents/Resources/bin/igneum-prove-host`, sha
6846d351..., the DMG's stripped copy): shard `0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a`,
aggregator `0x474678f3...`, equal to PC 2's `/opt/igneum` host (7a). The Mac app node's pool after its restart:
`{entries 0, failed 0, pending 0, verified 0}`, paidShards 388, tip DAA 119,097 (17:34Z).
Provers on: `prove-on-pc2-039` published 17:33:32Z, ran 17:35:55 to 17:36:16Z: `api/prove on: {"ok":true}`; the app's
state 20 s later: `backend cuda`, `enabled true`, `available true`, `current "block 77691 shard 0 (0 txs, 0 pgas, open)"`,
keys 10, "proving on the GPU".
## 10. The fee switch
The runbook (`docs/plans/fee-switch-devnet.md` on the `fee-switch` branch, read 15:44Z while still uncommitted): H = 210,000
(DAA 112,227 at 15:40:13Z, 0.965 blocks/s, so about 19:50Z on 6 October; the 24-hour rule holds for a publish before
about 19:50Z on 5 October), expected digest `ab8847da538dead1dc10e046dfaadab3c1c35928e3748810c4e050d4a886087a` with
`{"difficulty_v2_activation_daa":33000,"proving_v0_activation_daa":84100,"fees_v1_activation_daa":210000}`. Order:
(a) `publish-manifest.sh --version 0.3.9 --override '<that object>' --activation-height 210000 --deadline-note "fees v1" --deploy`
(the NEW folder; the OLD folder by hand with `--dest`/`--base-url`, one deploy, `--verify-only`), (b) `update-now` to
every app, wait for `Calibrated v1 fees from the override file: ... 210000` on every app node, (c) the hand nodes,
(d) the seed, then the digest sweep. The app side is confirmed in code: `ota.rs write_override` writes the manifest's
object verbatim whenever it differs from `override.json` (same version or not) and the engine restarts the node at a
safe moment.
### 10b. The publish (17:37 to 17:42Z)
Order kept: the proving rollout was complete (the first cross-verified shard under the new id, section 10c) before
anything consensus-bearing was published. The object:
`{"difficulty_v2_activation_daa":33000,"proving_v0_activation_daa":84100,"fees_v1_activation_daa":210000}`; H = 210,000;
tip DAA 119,097 at 17:34Z, so H - tip = 90,903 >= 86,400 at the publish (the 24-hour rule holds; H lands about 19:50Z
on 6 October at 0.965 blocks/s).
| Step | When | What |
|---|---|---|
| (a) the token folder and `dl/public/` | 17:37:35Z, died at once; again 17:38:34Z, written and deployed | `packaging/ota/publish-manifest.sh --version 0.3.9 --override '<object>' --activation-height 210000 --deadline-note "fees v1" --notes "Calibrated v1 fees from DAA score 210000 ..." --public --deploy` from the worktree. The first run printed only `failed printing to stdout: Broken pipe` and nothing was written (the signer's `embedded` output into `head -1` under `pipefail`, a race the ship tool's own run of the same script did not hit; the second run went through). The second run: mac and windows entries carried over from the folder's 0.3.9 manifest, signed (key 8f186e37...), `dl/public/` rewritten and signed, one deploy; exit 1 only because the public folder's `igneum-downloads.json` (1,317 bytes) was "NOT the local bytes yet" after 12 tries at the edge; the token manifest verified live (`--verify-only`, 17:41Z: byte-identical, signature OK) and the public manifest read live at 17:41:13Z carries the object |
| (b) the OLD folder | 17:37:35Z, verified 17:37:5xZ | `publish-old-folder.sh 0.3.9 "<notes>" --override '<object>' --activation-height 210000 --deadline-note "fees v1"`: written, deployed, `version 0.3.9, byte-identical, signature OK` (Sam's 0.3.5 Mac, if it wakes, takes 0.3.9 from this folder and then the object) |
| (c) `update-now` to every app | `update-now-fees-210000` 17:37:51Z (ran against the token manifest BEFORE (a) had landed: a no-op), then `update-now-fees-210000-b` 17:41:34Z (apps woken, stamp 3fa32fec) | the apps re-read the manifest, `ota.rs write_override` writes the object to `override.json` and the engine restarts the node at a safe moment |
The live manifests after (a), read 17:41:13Z: token folder and `dl/public/` both `0.3.9, published 2026-10-05T17:38:34Z,
consensus {activation_height 210000, deadline_note "fees v1", override {difficulty_v2_activation_daa 33000,
fees_v1_activation_daa 210000, proving_v0_activation_daa 84100}}`; the OLD folder the same object, published 17:37:35Z.
### 10c. The first cross-verified shard under the new id (before the publish)
PC 2's prover (key e809e396..., payout 0xCAfc6e74...) took block 77691 shard 0 at 17:35:5xZ; the Mac app node
(`node-20261005-165710.log`): `proof record for block 77691 shard 0 by e809e396... (peer): accepted, verify Command(".../igneum-prove-host")`
at 17:36:28.747Z, `proof of block 77691 shard 0 ...: verified in 0.2 s: RESULT verify: VERIFIED in 0.026 s` at
17:36:29.235Z, `chain block 77730: proof record for block 77691 shard 0 ... paid 1791503640000000000 wei to 0xCAfc6e74...`
at 17:36:31.460Z (1.7915 IGN). Proven on PC 2 under `0x2b1a81cb...`, verified on the Mac under the same id in 26 ms.
### 10d. The restarts, in the order they happened
| Node | Restarted with the object | Its lines |
|---|---|---|
| PC 2 1ccfe586 app node | 17:40:55Z (from the first update-now, 17:37:51Z: its app re-read the manifest after (a) had landed at 17:38:34Z) | `node-20261005-165651-r2.log` (digest job `digest-pcs-210000`, 17:43:08Z): `Refusing peer 192.168.68.64:26611 (node 1) and 188.245.5.161:26611 (the seed): consensus params digest mismatch, local ab8847da538dead1dc10e046dfaadab3c1c35928e3748810c4e050d4a886087a remote f10a4eab...`; `override.json` = the object |
| Mac d937c69d app node | 17:42:15Z (`update-now-fees-210000-b` ran 17:42:13Z) | `node-20261005-165710-r2.log`: `Calibrated v1 fees from the override file: ... from DAA score 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a`; `override.json` = the object; refusing node 1 and the seed until they moved |
| PC 1 ae432dc7 app node | between 17:46:06Z (its update-now-b) and 17:50Z: its card shows DAA 120,282 with a peer and 115 MH/s at 17:50Z, in step with the network, which a node on the old digest could not be (every peer would refuse it) | its digest line is OWED: `digest-pcs-210000` reported only from PC 2 and `digest-pc1-210000-b` (17:50:49Z) had not reported by 17:55Z (PC 1 took 100 s and 4 min to fetch the two update-now jobs: slow to wake today) |
| The seed | 17:48:27Z (`infra/devnet/restart-seed.sh '<object>'`, the a24ab01a cross-build 7e26e374... re-copied and checked, unit active, MainPID 119125) | journal: `Calibrated v1 fees ... 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a` |
| The observer | 17:48:53Z (pid 39882) | `observer-v4.out`: `Calibrated v1 fees ... 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a`; peers: the seed and 192.168.68.67 at once |
| Node 1 | 17:49:42Z (pid 40679, caffeinate 40681) | `node1.out`: the same three lines; peers at 17:49:42Z and 17:49:55Z |
Two script faults on the way, both fixed on this branch: (1) `infra/devnet/restart-hand-nodes.sh` refused the a24ab01a
binary (`strings | grep -q`: `-q` closes the pipe, `pipefail` turns `strings`' SIGPIPE into a failure); now `grep -c`
compared with 0, and `lines()` no longer ends the script when a node's lines are not in the log within 10 s; (2) the
scratchpad script used instead stopped after the observer for that second reason (`set -e` on an empty grep), so
node 1 was restarted by its pid by hand at 17:49:42Z with the same command line and the same file. Between 17:40:55Z and
17:49:42Z the network was split as the runbook predicts (app nodes on ab8847da, the three hand-run nodes on
f10a4eab), 8 min 47 s; node 1 was alone for the last 49 s.
### 10e. The digest sweep (17:55Z)
| Node | Digest | Read from |
|---|---|---|
| Mac app node d937c69d | ab8847da538dead1dc10e046dfaadab3c1c35928e3748810c4e050d4a886087a | its node log (above); app state: DAA 120,587, 6 peers, synced, `consensus_switch_daa` 210000 |
| PC 2 app node 1ccfe586 | ab8847da... | its node log through the digest job |
| PC 1 app node ae432dc7 | owed (the card's DAA and peer say the new digest; the line itself not yet read) | |
| The seed | ab8847da... | the journal through `restart-seed.sh` |
| The observer | ab8847da... | `/tmp/igneum-devnet/observer-v4.out` |
| Node 1 | ab8847da... | `/tmp/igneum-devnet/node1.out` |
Equal to the runbook's digest for the object (read on 2b6d23ef) and to this cut's reading on a24ab01a (5a).
## 10a. The fee-switch branch, as polled
| Time | State of `/Users/joshm/Projects/igneum-wt-feeswitch` (branch `fee-switch` at 23d11d5) |
|---|---|
| 15:34Z | 2 modified files (core Cargo.toml, config.rs), no runbook |
| 15:38Z | 12 files (core, export, host, gen.mjs, net.sh) |
| 15:43Z | 18 files; `docs/plans/fee-switch-devnet.md` present with `SHARD_ID_SECTION` unfilled; `elf/` still the 0.3.8 pin 0x0dfade07 (pinned 12:07:56Z) |
| 15:47 to 16:11Z | 23 then 24 files; the agent's simnet driver cut round 1 (15:47 to 15:53Z), round 2 failed (`only 0 of 11 receipts after 90000 ms`, 16:09Z), resumed 16:11:15Z (`provingGasLimit 0x1d4c0` at tip DAA 874, the v1 side); no commit, no re-pin, the placeholder still in the runbook |
## 11. Open after the cut
| Item | State |
|---|---|
| PC 1's digest line | owed: `digest-pc1-210000-b` (17:50:49Z) unreported at 17:55Z; read it with `node tools/jobs.mjs digest-pc1-210000-b` |
| The Mac's miner at 0.0 MH/s since the engine's own restart at 17:45:16Z (its hourly update slot, minute 57; `update check: 0.3.9 is current`) | the node is fine (DAA 120,587, 6 peers, synced); the miner's state after that restart not yet read; not caused by the switch (the node restart at 17:42:15Z was separate and mining was already 0.0 at the 17:47Z card) |
| PC 37ba0461 (0.3.7, silent since 13:52Z) and Sam's Mac (0.3.5, silent since 09:43Z) | take 0.3.9 and the object on their next check (the OLD folder carries both); until then each is refused by every peer from its next restart |
| `dl/public/igneum-downloads.json` | the publisher saw the previous bytes at the edge for its 12 tries at 17:39Z; at 17:42:38Z the live file was byte-identical to the local one (sha 3aa32e75...) |
| The 0.3.9 `publish-jobs.sh` through the worktree before the master merge | left the envelope stale once (16:02Z); every later publish went through the main checkout's script, which rewrote it |
| `make-package.sh` gate | skipped on instruction (`SKIP_GATE=1`); its native half ran on every fixture with the DMG's host (5d); PC 2's CUDA build proved and the Mac verified block 77691 within 40 s of provers-on, the stronger check |
| H = 210,000 at about 19:50Z on 6 October | step 8 of the runbook (budgets, base fees, the first v1 shard) is tomorrow's check |
| Carried from 0.3.8 section 11 | the PC 2 jobs-file signature refusal of 13:19Z (did not recur); the 0.3.6 items |
Reference in a new issue View git blame Copy permalink