igneum/docs/plans/difficulty-v2-rollout-devnet.md

Difficulty v2 on the live devnet: rollout plan, 4 October 2026

Prepared by the rollout engineer; the Hetzner rehearsal (section 7) passed. Nothing in this file has been run on the devnet. The main session runs it in the order of section 4. Background: docs/analysis/difficulty-2026-10-04-oscillation.md (the finding, rule v2, section 8 rollout), docs/bench-log.md "difficulty rule v2". the project lead approved the rollout on 4 October 2026 ("1-5 approved and anything else needed") with two constraints, both built in here: the PCs get igneumd v2 only through an OTA app version, and the activation height leaves at least three hours from the manifest publish.

1. What changes and what does not

Only igneumd changes. A v2 node rules v1 for every block whose DAA score is below difficulty_v2_activation_daa and v2 from the block whose DAA score reaches it, so the chain, the genesis and the databases stay. The miners follow templates and need nothing. A node that reaches the height WITHOUT the field in its override file keeps rule v1 and forks off at that block (seen on the 3-node test network, section 7 of the analysis), so every node of the devnet must carry the same N before the height arrives: Mac node 1, the observer node, the seed, PC 1 and PC 2.

The default on every network is u64::MAX (never). Only the override file sets it. The binaries built today carry no height; the height is operational data (section 3).

2. The binaries (built 4 October 2026, 13:31 to 13:5x UTC, on the loaded Mac at nice 19, 4 cargo jobs)

Source: vendor/igneum-node-v4 at 3bfe346f (devnet-v4), which is a21ff239 (difficulty v2) plus one commit that touches igneum/miner/src/main.rs only; kaspad does not depend on the miner crate, so igneumd is the a21ff239 node. Each binary was built into its own target directory; the live binaries (target-integration, target-linux, the seed's /opt/igneum/v4/bin, the Hetzner nodes' /opt/igneum/bin) were not touched.

Platform	Path	sha256 (igneumd)	Build	Verified
Mac arm64	vendor/igneum-node/target-v2/release/igneumd	8b0fbd27fa2286548256110c4e6655fe6620db8ab36bad584fcbdbe89179539a	144 s incremental (target dir seeded from the v4 worktree's own cache)	--version = igneumd 2.1.0; started on a private suffix with {"difficulty_v2_activation_daa": 123456} and printed Difficulty rule v2 from the override file: active from DAA score 123456
Linux x86-64 (glibc 2.36)	infra/cross/out-v2/igneumd (from vendor/igneum-node/target-v2-linux)	70751def7e2274f596d91fe29e9a5344d7ceca3065c1378f12ed5308f26e2233	173 s incremental (seeded from target-linux, cargo-zigbuild)	ELF x86-64 PIE; carries the field name and the v2 line; the Hetzner roll printed the line on all 12 nodes (section 7)
Windows x86-64	vendor/igneum-node/target-v2/x86_64-pc-windows-gnu/release/igneumd.exe (50,170,880 bytes)	27c2ce858a497e3d7197c3f1e868c18b588793cf8bee4ce6e0978dd473d4c5bc	8 min 50 s (cross-build.sh pattern, mingw, 4 jobs; target dir seeded from igneum-node-win)	PE32+ x86-64; strings carries the field name and the v2 line; same DLL imports as the shipped 0.3.1 exe; it cannot run here

infra/cross/out-v2/igneum-miner and the Windows igneum-miner.exe were built alongside (the scripts build both); they are the 3bfe346f miner (seed-mismatch and stall guards). The Hetzner roll shipped igneumd only.

3. The activation height N, and how every node learns it

Rule (the project lead, 4 October 2026): N = DAA score at the manifest publish + 10,800 (three hours at 1 block/s). Why three hours: the apps check the manifest on start and every 60 minutes plus up to 10 minutes of jitter, then download, then wait for a safe moment (synced node, no program boundary within 180 s, no worker starting; up to 6 hours), and the "urgent" path (install at once, red bar) opens only when the node's DAA score is within 1,800 blocks of N (app/igneum-app/src/manifest.rs, FORK_URGENT_BLOCKS). With 10,800 both PCs have seen the manifest within 70 minutes, have the download within minutes of that, and still have 30 minutes of urgent installation if no safe moment came.

The gap found while preparing this, and its fix (committed with this plan, to review before cutting the version): the app launched igneumd with no --override-params-file (app/igneum-app/src/engine.rs, node_args), so an OTA-delivered igneumd v2 would have run with the default (never) and forked at N. Now igneum-app.json (the file the packagers write next to the engine) may carry "node_override_params": {...}; the engine writes it to <app data>/app/override-params.json and adds --override-params-file=<that path> to the node's command line (config::Packaged::node_override_params, Engine::node_override_file; test packaged_carries_the_node_override_params). Both packagers read it from ONE line in packaging/mac/packaged-config.sh:

NODE_OVERRIDE_PARAMS='{"difficulty_v2_activation_daa": N}'

build-dmg.sh sources that file; the Windows make-payload.sh (run by CI from the pushed tree) reads the line with sed. So N is fixed when that line is committed, which is BEFORE the publish. To satisfy the rule at publish:

• choose N = (DAA score now) + 14,400 when editing the line (the extra 3,600 covers the CI build, the DMG and the publish; CI took about 20 minutes today);
• at publish, read the DAA score again and check N - DAA >= 10,800; if not, raise N, commit, rebuild, re-cut.

The same N goes verbatim into the override files of Mac node 1, the observer node and the seed (section 4, steps 4 to 6), which the main session restarts by hand; the PCs get it through the app. The DAA score: on the Mac, IGNEUM_RPC=ws://127.0.0.1:28640 python3 infra/cloud-devnet/node/wrpc.py call getBlockDagInfo | python3 -c 'import json,sys; print(json.load(sys.stdin)["virtualDaaScore"])' (the observer node's JSON listener; read-only).

4. The order, with the exact commands

All Mac commands from /Users/joshm/Projects/igneum. N below is the number from section 3. Steps 1 to 3 are the package; steps 4 to 6 the hand-run nodes (any time after N is fixed and before the height; the order observer, seed, node 1 is lowest risk first); step 7 the watch.

Step 1: fix N and cut 0.3.2

# N = current DAA + 14,400 (section 3)
sed -i '' "s/^NODE_OVERRIDE_PARAMS=.*/NODE_OVERRIDE_PARAMS='{\"difficulty_v2_activation_daa\": N}'/" packaging/mac/packaged-config.sh
# version 0.3.2: app/igneum-app/Cargo.toml, app/windows/version.h, packaging/windows/resources/igneum-app.rc (as CI demands)
# commit as igneum-labs, push: .github/workflows/windows.yml builds the installer from the payload inputs of step 2

Step 2: the Windows payload inputs (done today; redo only if the exe changes)

packaging/windows/push-inputs.sh was run with IGNEUM_WIN_RELEASE=vendor/igneum-node/target-v2/x86_64-pc-windows-gnu/release (section 7a has the sha256 of the igneumd.exe it carried and the deploy result). The workflow fetches payload-inputs.zip and refuses a mismatched sha256. If the inputs must be re-pushed:

IGNEUM_WIN_RELEASE=vendor/igneum-node/target-v2/x86_64-pc-windows-gnu/release packaging/windows/push-inputs.sh

Step 3: the Mac DMG and the manifest

NODE=vendor/igneum-node/target-v2/release/igneumd packaging/mac/build-dmg.sh
#   MINER stays the default (vendor/igneum-node/target-integration/release/igneum-miner, the 0.3.1 miner) unless the
#   miner is meant to move too; then MINER=vendor/igneum-node/target-v2/release/igneum-miner (3bfe346f guards)
# check the packaged config inside the bundle carries N:
python3 -c 'import json; print(json.load(open("packaging/mac/build/dmg/Igneum Miner.app/Contents/Resources/igneum-app.json"))["node_override_params"])'

# publish the Mac entry with the activation height (NOT run by the rollout engineer; this is the publish of section 3)
packaging/ota/publish-manifest.sh --version 0.3.2 --mac packaging/mac/dist/Igneum-Miner-0.3.2.dmg \
    --notes "difficulty v2 from DAA score N" --activation-height N --deadline-note "difficulty v2" --deploy
# Windows entry when CI has the installer (carries the Mac entry over, same version):
packaging/windows/fetch-ci-artifacts.sh --deploy
# record the DAA score at this moment and check N - DAA >= 10,800 (section 3)

What the PCs then do: within 70 minutes the 0.3.1 app sees 0.3.2, downloads and verifies the installer, installs at the next safe moment (urgent from N - 1,800), relaunches as 0.3.2, whose engine writes %LOCALAPPDATA%\igneum\app\override-params.json and starts igneumd.exe --override-params-file=.... The node's log (node-<stamp>.log in the app's log folder, uploaded by the log intake) starts with Difficulty rule v2 from the override file: active from DAA score N. The STATUS lines of the log intake show the run id and app version per PC; both must show 0.3.2 before N - 1,800.

Step 4: the observer node (26640/28640, a follower; observer.mjs keeps running under tools/observer/run.sh)

printf '{"difficulty_v2_activation_daa": N}\n' > /tmp/igneum-devnet/override-v2.json
kill -INT 4584; while kill -0 4584 2>/dev/null; do sleep 1; done          # pid of the observer igneumd on 4 Oct 2026 (ps -eo pid,command | grep observer-v4)
cd vendor/igneum-node && nohup ../igneum-node/target-v2/release/igneumd --devnet --nodnsseed --disable-upnp \
  --appdir=/tmp/igneum-devnet/observer-v4 --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 --listen=127.0.0.1:26641 \
  --addpeer=127.0.0.1:26611 --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 \
  --override-params-file=/tmp/igneum-devnet/override-v2.json --nologfiles --yes >> /tmp/igneum-devnet/observer-v4.out 2>&1 &
cd /Users/joshm/Projects/igneum
head -3 /tmp/igneum-devnet/observer-v4.out | grep "Difficulty rule v2"    # the line must be there
# observer.mjs loses the feed, exits 2, run.sh restarts it 3 s later (the 4 Oct restart loop); the live page is back within a minute

The command line is the one running on 4 October 2026 (ps), with the binary and the override flag changed. The existing database opens under v2 (same chain).

Step 5: the seed (igneum-seed-1, 188.245.5.161, unit igneumd-v4, binary /opt/igneum/v4/bin/igneumd)

stage-v4.sh is not the tool here: its install copies over the running binary (cp onto a busy text file fails) and it was written for a disabled unit. The hot swap by hand, from the Mac (the key is ~/.ssh/igneum_ed25519):

scp -i ~/.ssh/igneum_ed25519 infra/cross/out-v2/igneumd infra/cross/out-v2/version.txt root@188.245.5.161:/root/v4/out/
ssh -i ~/.ssh/igneum_ed25519 root@188.245.5.161 '
  set -e
  sha256sum /root/v4/out/igneumd | cut -c1-64        # must print 70751def7e2274f596d91fe29e9a5344d7ceca3065c1378f12ed5308f26e2233
  cp -p /opt/igneum/v4/bin/igneumd /opt/igneum/v4/bin/igneumd.prev
  cp /root/v4/out/igneumd /opt/igneum/v4/bin/igneumd.new && chmod +x /opt/igneum/v4/bin/igneumd.new
  printf "{\"difficulty_v2_activation_daa\": N}\n" > /etc/igneum/override-v2.json
  sed -i "s#^EXTRA_ARGS=.*#EXTRA_ARGS=--override-params-file=/etc/igneum/override-v2.json#" /etc/igneum/seed-v4.env
  systemctl stop igneumd-v4
  mv -f /opt/igneum/v4/bin/igneumd.new /opt/igneum/v4/bin/igneumd
  systemctl start igneumd-v4
  sleep 5; journalctl -u igneumd-v4 --no-pager -n 40 -o cat | grep -m1 "Difficulty rule v2"
  cat /etc/igneum/seed-v4.env'
infra/seed-nodes/health.sh            # OK ... unit=active-v4 synced=True peers>=1 within a minute

run-seed-v4.sh appends EXTRA_ARGS to the flag list, so the env line is all the launcher needs. Rollback of this step: EXTRA_ARGS= back in the env file, mv igneumd.prev igneumd, systemctl restart igneumd-v4.

Step 6: Mac node 1 (26610/26611; the Metal miner mines through it)

The nohup line of docs/plans/cutover-2026-10-04.md step (b)2 with the binary and the override flag changed. The database in /tmp/igneum-devnet/node1 opens under v2 (same chain), so no move.

kill -INT 33114; while kill -0 33114 2>/dev/null; do sleep 1; done        # pid of node 1's igneumd on 4 Oct 2026 (caffeinate 33116 exits with it)
cd vendor/igneum-node
nohup caffeinate -dims target-v2/release/igneumd --devnet --nodnsseed --disable-upnp --enable-unsynced-mining \
  --appdir=/tmp/igneum-devnet/node1 --rpclisten=0.0.0.0:26610 --listen=0.0.0.0:26611 \
  --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 \
  --override-params-file=/tmp/igneum-devnet/override-v2.json --nologfiles --yes >> /tmp/igneum-devnet/node1-v4.out 2>&1 &
cd /Users/joshm/Projects/igneum
tail -5 /tmp/igneum-devnet/node1-v4.out | grep "Difficulty rule v2"
# the Metal miner (pid 33633, igneum-miner mine grpc://127.0.0.1:26610 ... --worker packaging/mac/build/worker/igneum-bench):
# check it still submits after the restart (its log); if it exited, start it again with the same command line (ps -eo command | grep 'igneum-miner mine')
vendor/igneum-node/target-integration/release/igneum-miner watch 1 grpc://127.0.0.1:26610     # peers=2.., synced=true, blocks climbing

Step 7: the watch

When	Where	Expected
after each restart	the node's first lines	Difficulty rule v2 from the override file: active from DAA score N
before N - 1,800	log intake STATUS lines, the live page's miner list	both PCs on 0.3.2 (their node logs carry the v2 line)
N	observer difficulty events (/tmp/igneum-devnet/observer-mjs-v4.out), getBlockDagInfo on 26640 and 26610, health.sh on the seed	one sink across node 1, the observer and the seed; the PCs' blocks keep being accepted (the live page's per-miner counts keep moving); no "rejected block" in any log
N to N + 600	the same	difficulty keeps moving, no node stuck at N
the next epoch boundary and the next PC join or leave	observer difficulty events	no 1.3x bursts every 2 to 3 minutes (section 1 of the analysis); the std of log difficulty over 15 minutes under 0.05

If a node forks at N (its sink stops advancing, or its peer count drops and its blocks are rejected): its override file lacks N or carries another value. Fix the file, restart that node; its database re-syncs to the common chain (the v2 side) because a v2 node rejects the v1 side's blocks from N on.

5. Rollback

Before N: remove the field (or the flag) on every node and restart; nothing has happened on the chain. After N: there is no rollback to v1 without a chain split; the way back is to fix any node that lacks the field. The Hetzner nodes keep igneumd.prev and ./rollout-v2.sh back for their own chain.

6. Pieces this plan adds to the repository

Piece	What
app/igneum-app/src/config.rs, engine.rs, packaging/windows/make-payload.sh	node_override_params in the packaged config, written to the app data dir and passed as --override-params-file; the Windows payload's JSON field. These three files were swept into another agent's commit b8b349a (the app log intake change) before this plan was committed; cargo test config:: passes (packaged_carries_the_node_override_params)
packaging/mac/packaged-config.sh	the NODE_OVERRIDE_PARAMS line (empty until the main session sets N) and its JSON field in igneum-app.json
infra/cloud-devnet/rollout-v2.sh	the one-node-at-a-time roll, the common-sink check and the watch used for the rehearsal
infra/cross/out-v2/	the Linux v2 binaries and version.txt (not committed: binaries)

7. The Hetzner rehearsal (igneum-devnet-20, 12 nodes, own chain)

Run 4 October 2026, 14:14 to 15:20 UTC, after the morning's hop.sh schedule had ended (its end line at 14:02 UTC) and with the other agent's second partition.sh sin 10 (14:03:25 to 14:15:44 UTC) still cut while nodes 01 to 04 were rolled; the Singapore nodes were rolled after its heal. Raw logs: infra/cloud-devnet/results/2026-10-04/rollout-v2.log (the roll), rollout-v2-watch.log (31 checks, one a minute, from DAA 14,646 to 16,800).

Step	What happened
staging	rollout-v2.sh stage: the Mac to the four gateways, then gateway to private node over the zone networks (seconds). Lesson: the Mac to Hillsboro ran at about 32 KB/s (20 MB in 10 minutes); copying igneum-01 to igneum-04 server to server took 12 s
N	DAA 14,362 at 14:14:37 UTC plus 1,800, rounded: N = 16,170
the roll	12 nodes in nodes.tsv order, 14:14:39 to 14:18:16 UTC (15 to 30 s each). Every node printed Difficulty rule v2 from the override file: active from DAA score 16170 and rejoined within 5 s (igneum-04: 15 s) with 2 to 3 peers and a DAA score within 2 of the reference. The miners stopped with their node (Requires=igneumd) and were started again; so did the block logs, which the first version of the script did not restart (fixed; they were off from each node's restart until 14:56 UTC, so the 5-s sample series has a gap across N and the record across N is the watch's one-minute checks and the journals)
N and N + 600	the watch's 31 checks: at every check all 12 nodes held the reference's older sink as a chain block, the DAA scores were within 2 to 12 of each other and the difficulty within 1% across nodes. Difficulty by check (median): 77.4k at DAA 14,646, 81.9k at 15,160, 76.7k at 16,161 (just before N), 74.6k at 16,250, 74.5k at 16,344, 76.0k at 16,420, 77.8k at 16,511, 78.3k at 16,587, 81.9k at 16,665, 86.8k at 16,740, 86.6k at 16,800: it kept moving through N with no step at the switch
common chain, the definitive check	getVirtualChainFromBlock from the chain block that was igneum-01's sink at 14:17 UTC (DAA about 14,500) on all 12 nodes at 15:00 UTC: 1,623 to 1,628 chain blocks each (the spread is the newest 5), and the SAME block at index 1,100 (DAA 16,062, before N), 1,250 (DAA 16,304), 1,400 (DAA 16,565) and 1,550 (DAA 16,797, past N + 600) on every node. No node forked at N
hash-rate step under v2	hop.sh "half:4:600;all:1:600" from 14:57 UTC: section 7b

7b. Settle times, v1 (morning) against v2 (afternoon)

The morning's run (hop.sh default schedule, rule v1, 12:32 to 15:02 BST; results/2026-10-04/hop.md, committed in 105b5f8) and the afternoon's (hop.sh "half:4:600;all:1:600", rule v2 active since DAA 16,170, 14:57 to 15:17 UTC; results/2026-10-04/v2/hop.md, hop-series.tsv, compare.md), compared over the first 600 s of the same two steps. "half:4" is six of the twelve nodes at 4 miner threads (on 2 vCPU, so about 1.5x the network's hash rate, not 2.5x); "all:1" is the step back down.

run	phase	2-min rate: first back within 10% of 60/min (s)	stays within 10% for 3 min from (s)	overshoot	difficulty start	difficulty at 300 s	at 600 s	std log D 300 to 600 s	max D / min D 300 to 600 s
v1	2 step up, half 4 threads	161	not in 600 s	22%	8.01e+04	1.28e+05	8.95e+04	0.169	1.51
v1	3 step down, all 1	272	not in 600 s	32%	1.15e+05	8.88e+04	1e+05	0.031	1.18
v2	1 step up, half 4 threads	157	not in 600 s	34%	8.41e+04	1.02e+05	1.15e+05	0.053	1.15
v2	2 step down, all 1	172	not in 600 s	37%	1.14e+05	9.7e+04	8.19e+04	0.087	1.45

Per minute, 2-min block rate (blocks/min) and median difficulty, first 10 minutes of each step:

s	v1 up rate	v1 up D	v2 up rate	v2 up D	v1 down rate	v1 down D	v2 down rate	v2 down D
0	65.5	80.1k	1277.5	84.1k	62.0	115.2k	51.0	114.3k
60	75.5	103.5k	1326.5	94.3k	56.0	113.1k	41.0	108.8k
120	72.0	109.5k	80.5	124.4k	46.5	110.9k	40.0	72.4k
180	61.5	109.5k	57.0	94.5k	42.5	108.4k	60.5	89.2k
240	65.5	115.7k	67.5	99.7k	45.5	75.7k	65.0	99.1k
300	66.5	127.9k	75.5	102.4k	62.0	88.8k	58.0	97.0k
360	65.0	133.7k	61.0	102.2k	58.0	104.8k	53.5	93.9k
420	57.5	123.4k	61.5	107.5k	47.5	102.6k	47.5	91.7k
480	48.0	112.8k	70.0	116.3k	43.0	101.1k	49.5	87.8k
540	64.5	89.5k	63.0	117.4k	52.5	101.2k	54.5	79.6k
600	nan	nank	nan	nank	nan	nank	nan	nank

Reading, with the caveats stated once: one run of each rule, CPU trickle miners (12 one-thread nodes, the 2-min rate at 60/min carries a Poisson std of about 6.5% before any hash-rate noise), the first two v2 "up" rows (1,277 and 1,326 blocks/min) are the series builder bridging the sample gap of section 7 (the block logs were off until 14:56 UTC) and are not rates, and the v1 step-up phase ran 94 minutes instead of 15 because the Mac starved hop.sh (its first 600 s are unaffected). Both rules bring the 2-min rate back within 10% of 60/min after about 160 s on the step up (v1 161 s, v2 157 s) and v2 sooner on the step down (172 s against 272 s); neither holds it there for three minutes within 600 s on this network (hop.md's criterion, "never" for every 600-s phase on both runs; the morning's 15-min phases gave v1 751 s and 646 s). The difference is in the difficulty after the first five minutes: under v1 the step up ended with the difficulty swinging 128k to 134k to 89k between 300 and 540 s (max over min 1.51, std of log difficulty 0.169: the flip of section 2 of the analysis), under v2 it climbed 102k to 117k (1.15, 0.053) with the rate 61 to 70 per minute. On the step down v2 reached the one-thread level (82k; the morning's steady value was 80k) by 600 s where v1 was still at 100k after 600 s and 97k after 900 s. The morning's rule-v1 numbers for the full schedule are in hop.md (phase 2 settled 751 s, phase 4 241 s, phase 5 646 s on the 3-min criterion).

7a. The Windows payload inputs (pushed 4 October 2026, 13:46 UTC)

packaging/windows/push-inputs.sh with IGNEUM_WIN_RELEASE=vendor/igneum-node/target-v2/x86_64-pc-windows-gnu/release published payload-inputs.zip (sha256 66b4dc51c4dfc3f099e6c04cb5b15945ec3e255a3c9662064c7632e0b8f46eb8) to the downloads host; the live payload-inputs.json carries igneumd.exe 27c2ce858a497e3d7197c3f1e868c18b588793cf8bee4ce6e0978dd473d4c5bc (50,170,880 bytes) and igneum-miner.exe 19793fd09609adc1eaba11f8d3bea76ae93bb8b0f98d25154b2dcfd055f52401 (10,110,464 bytes, the 3bfe346f miner with the seed-mismatch and stall guards; the 0.3.1 payload of 13:31 UTC carried 108261fe..., so the next CI build ships the newer miner as well). The CUDA and OpenCL workers, the NVRTC DLLs and the mingw DLLs are the same files as before. No manifest was written or published.