igneum/docs/analysis/difficulty-2026-10-04-oscillation.md

Difficulty oscillation on the live devnet, 4 October 2026: record, replay, cause, rule v2

4 October 2026, consensus-engineer. Follows difficulty-2026-10-03.md (the rule) and sim/difficulty/attacks/README.md (the attacks). Everything measured here is reproducible from sim/difficulty/ (record, hash-rate truth, simulator with the DAG model, test-network script) and the devnet-v4 branch of vendor/igneum-node (worktree vendor/igneum-node-v4).

1. The finding

Live devnet v4 (genesis 10:05 BST, docs/bench-log.md "devnet v4 cut-over"), 09:05 to 10:58 UTC. One RTX 5090 (PC 1, 122 MH/s, 8 identities through its own node) plus the Mac's Metal worker (26.7 MH/s, on the Mac node) and an integrated Radeon (2.8 MH/s) mined from 09:11. A second RTX 5090 (PC 2, 124 MH/s, 8 identities, through the Mac node) joined at 10:12:17 UTC, left at 10:31:25 and came back at 10:35:19. The network went from about 152 to about 280 MH/s. The two PCs are clones with the same Windows computer name, so their identities carried the same vote keys (fixed the same day: identities now come from a per-install machine id); the log intake tells them apart by run id.

What the rule did, from the observer's difficulty events and the header record:

Period (UTC)	What was mining	Difficulty band (node convention, 2^255 / target)	Blocks per minute	Chain blocks merging 2+ blues
09:24 to 10:04	PC 1 + Mac + Radeon	56.7M to 67M, within 1% per minute from 09:24 (one excursion to 82M at 09:43)	49 to 79	38.7%
10:14 to 10:16	PC 2 joins	70M to 144M in 90 s (23 harden steps of 3%)
10:17 to 10:32	both PCs	101.8M to 164.3M, 5 peaks of 1.33x spaced 132 chain blocks, std of log difficulty 0.123	54 to 81	36.3%
10:33 to 10:36	PC 2 off, then back	116M down to 67M, back to 106M
10:37 to 10:54	both PCs	103M to 150M, 4 peaks of 1.35x spaced 150 chain blocks, std 0.072; the floor rises 110M to 127M	55 to 76	34.5%
11:02	epoch boundary (DAA 7,200 at 11:01:52) and both PCs restarted for the machine-id package	152M down to 77M in 3 minutes
11:07 to 11:17	one PC + Mac	69.0M to 72.7M, within 1.3% per minute, no flips	45 to 73

The true level for 280 MH/s at one block a second is 139M. The rule sat on a floor of 110M to 127M (the chain ran at 1.1 to 1.2 blocks a second) and burst to 146M to 164M every two to three minutes. The observer's per-poll events show the bursts as stacked clamp steps: "up 6.1%" is two 3% hardens inside one 2-second poll, "up 15.9%" five, "down 24.3%" three 10% eases.

Records: sim/difficulty/records/live-2026-10-04.csv (8,090 headers through 11:18 UTC from the observer node's read-only wRPC, pull_live.py: hash, DAA and blue score, stamp, bits, difficulty, parents, selected parent, blue work, mergeset sizes, vote key) and live-2026-10-04-hashrate.csv (587 STATUS lines of the four GPU workers from the log intake: the last-30-s hash rate every 30 s, with run id and node; the Mac Metal worker does not upload, its 26.7 MH/s is the bench-log value of the hourly swap).

2. The cause

Spec 2.3 as implemented on 3 October: the reference lane R is the epoch window, every block of the block's own epoch (the epoch lane for the first 600 blocks, Kaspa's sampled window restricted to the epoch after that); the short lane S is the newest 120 chain blocks; S rules while S and R disagree by more than 25%, else R rules; the output moves at most 3% harder or 10% easier per block.

A hash-rate step inside an epoch leaves R polluted by the pre-step blocks for the rest of the hour. PC 2 joined 7 minutes into epoch 1 (the boundary was 10:05:09). From then on R was the average of 7 minutes at 152 MH/s and m minutes at 280: 224 MH/s at m = 7, 252 at m = 36. S read the true 280 with its own noise (about 10% for 120 linearly weighted blocks, more on a DAG where a chain step carries one to three blocks' work). While S/R was above 1.25 (the first 7 minutes after the join) S ruled and the target was right. Once R had climbed to within 25% of S, R ruled and the target eased to R's level, 11 to 20% too easy. Every upward noise excursion of S past 1.25 R re-engaged S, the target hardened 3% a block toward S's momentary value (which at that moment was 25 to 45% above R), then S fell back under 1.25 R, R ruled again and the target eased 10% a block back to the floor. A relaxation oscillation with period set by how often S's noise crosses the trigger and amplitude set by the trigger band and S's noise: 1.3x to 1.5x peaks every two to three minutes, exactly the record.

The two other suspects in the brief are real but secondary. The DAG's bursts (38% of chain blocks merge two or more blues; a chain step's work is one to three blocks') widen S's noise, which sets how often the trigger is crossed. The per-block clamps stack inside a burst of chain blocks (three 10% eases in two seconds), which steepens each flip. Neither starts the oscillation: the chain-only simulator with a 1.85x step 10 minutes into an epoch oscillates the same way (section 4). The epoch boundary ends it: the new epoch's R starts clean, which is what the live chain did at 11:02 (section 1, last two rows; the hash rate also stepped down there, so the live boundary is consistent with the cause, not a clean test of it).

The 3 October simulator could not see this because its step profiles put the step at an epoch boundary (up50 and down50 step at 3 h = DAA 10,800) or ran steps so large (10x hops) that S/R stayed outside the band until R had washed out.

3. The DAG model and the replay

sim/difficulty/sim.py --live (new, section "DAG model"). Miners on nodes: PC 1 on the PC node, PC 2 and the Mac on the Mac node, the two nodes 95 ms apart. A miner fetches a template from its node every 150 ms (the Mac: 630 ms, one job), the template reaches the card 260 ms later (the Mac: 675 ms, the 2-deep job queue; igneum-miner lines 947 to 975 and 1018), the miner refetches 30 ms after its own block; the block carries the parents and the node's stamp of its template, as igneum-miner does (it never touches the stamp). GHOSTDAG with every merged block blue: selected parent by blue work, blue work of a chain step = the work of the selected parent's mergeset blues (checked on the record: 215 of 215 single-merge steps), DAA score advances by the mergeset size. The rule runs as SampledDifficultyManager::igneum_difficulty_bits: the chain walk from the virtual's selected parent with blue-work increments over sanitised-clock steps, and the sampled window (every 4th block of each mergeset, selected parent first then descending blue work, 661 samples) restricted to the epoch. The hash-rate schedule is the STATUS series of the four workers plus the Mac's constant. One scale factor (--dag-delay 0.75) was fitted to the merge fraction; nothing else was tuned to the record.

Three seeds against the record, the join window 10:20 to 10:31 UTC (three short windows after the join, both PCs on, the whole window polluted):

Source	Blocks/s	Chain blocks merging 2+ %	Difficulty mean M	Min M	Max M	Std log D	Peaks over 15%	Mean peak/trough	Blocks/min min	Blocks/min max	Lane flips	Short lane %
record	1.08	36.3	128.2	101.8	164.3	0.134	4	1.37	59	72
simulator, rule v1	1.10	37.1	128.8	110.7	167.2	0.115	4.3	1.31	56.7	73.7	22.3	38.1

The whole polluted window 10:17 to 10:32: record std 0.123, 5 peaks of 1.33x, 101.8M to 164.3M, 54 to 81 blocks a minute; simulator 0.118, 5.3 peaks of 1.33x, 104.1M to 169.2M, 53.7 to 77.0. The rejoin window 10:42 to 10:54: record 0.067, 3 peaks of 1.31x; simulator 0.045, 1.7 peaks of 1.25x. The solo window 09:25 to 10:04: record 0.053 with 1 peak; simulator 0.054 with 4.7 peaks (the simulator flips more in the first 20 minutes, where R still carried the Mac-only minutes before PC 1; the record flipped there too, 09:14 to 09:23, 52M to 78M). The simulator reproduces the amplitude, the period and the mean level. Raw outputs: sim/difficulty/results.md, section "Live replay".

4. Candidates

Measured on the live replay (3 seeds, mean), the join window 10:20 to 10:31 and the rejoin window 10:42 to 10:54. The standard from the brief: damped within three short windows (the join window starts three short windows after the step). True difficulty for the hash rate: 139M.

Rule	Join: std log D	Join: peaks	Join: lane flips	Join: mean M	Rejoin: std log D	Rejoin: peaks	Rejoin: mean M	Solo: std log D
v1 (spec 2.3 of 3 October)	0.115	4.3	22.3	128.8	0.045	1.7	125.0	0.054
(a) short lane 240	0.125	1.3	10.3	132.8	0.043	0.3	125.2	0.060
(a) short lane 360	0.107	1.0	11.3	132.9	0.041	0.0	125.6	0.068
(b) ease clamp 3%	0.113	3.0	20.3	127.1	0.055	1.3	125.1	0.062
(c) clamp once per DAA second	0.132	4.3	20.7	129.3	0.054	1.0	125.7	0.067
(d) hysteresis, leave at 10%	0.090	2.0	1.7	142.2	0.054	0.7	130.4	0.068
(e) median of three on S	0.124	4.7	15.0	129.0	0.056	2.3	126.6	0.056
soft trigger (blend 10% to 25%)	0.110	3.0	13.0	137.6	0.055	1.0	128.5	0.070
reference window 1,200	0.108	3.7	14.3	133.4	0.048	1.0	130.8	0.038
reference window 900	0.081	0.7	8.3	134.2	0.063	0.0	129.5	0.043
reference window 720	0.055	0.0	4.3	136.4	0.052	0.0	132.3	0.053
reference window 600	0.024	0.0	0.0	140.9	0.028	0.0	137.8	0.044
reference window 600, epoch lane only (v2)	0.026	0.0	0.0	142.6	0.031	0.0	136.1	0.050
reference window 480, epoch lane only	0.021	0.0	0.0	141.3	0.019	0.0	135.0	0.052
reference window 600 + hysteresis	0.035	0.3	1.0	141.9	0.044	0.3	135.8	0.046

The brief's five candidates each leave the join window between 0.09 and 0.13: they shape the oscillation (fewer, larger swings with a longer short lane; S ruling almost always with hysteresis, 0.09 from S's own DAG noise) without removing its cause, which is R. Capping the reference window at the newest 600 DAA score of the epoch removes it: R carries a step for 10 minutes instead of an hour, so by the join window S and R agree, R rules, and the target sits on the true level (140.9M against 139M) with the noise of a 600-block work-over-time estimate (std 0.024, no flips). The cap at 600 makes Kaspa's sampled long lane moot (600 DAA holds at most 150 samples, the long lane's minimum), so v2 uses the epoch lane alone over that window: one estimator, one clock (the sanitised clock), no flicker between the two estimators at the boundary. 480 is marginally better on the live case and 720 clearly worse; 600 (10 minutes at target, the long lane's old minimum) is kept.

Chain-only check (sim.py, a 1.85x step 10 minutes into epoch 1, seeds 7 to 9, 8 to 25 minutes after the step): v1 std of log difficulty 0.179 / 0.136 / 0.189 with 102 / 142 / 96 lane flips; v2 0.052 / 0.022 / 0.081 with the mean rate ratio 1.013 / 0.998 / 0.974.

5. Rule v2

One change to spec 2.3 rule 2: the reference lane R is the epoch lane over the newest min(k, 600) chain blocks of the block's epoch (DAA score at least max(epoch start, DAA - 600)), blended with the parent's implied rate as k_ref : 16 where k_ref is the DAA span of that window; Kaspa's sampled window is not consulted. The hold (8 blocks of the epoch), the short lane (newest 120 of the same walk), the trigger (25%), the clamps (3% harden, 10% ease), the sanitised clock and the floor are unchanged. Constant REF_WINDOW_V2 = 600 (kaspa_consensus_core::igneum::difficulty).

Activation: difficulty_v2_activation_daa, a consensus parameter (Params, OverrideParams, the --override-params-file parser, following pow_epoch_blocks). Every network's default is u64::MAX (never); the override file sets it. A node upgraded ahead of the height rules v1 for every block whose DAA score is below it and v2 from the block whose DAA score reaches it, so the devnet keeps its genesis and its chain. Implementation: reference_window(daa_score, epoch_blocks, activation) returns the epoch start, the reference start and the v2 flag; igneum_difficulty_bits bounds the chain walk at the reference start, skips the long lane in v2 and passes k_ref to the pure igneum_target, which blends the epoch lane on it. Cost: the walk is 600 chain blocks per header in v2 (2 compact-header, 2 blue-work and 2 clock reads each, all cached stores), the same as the first 600 blocks of every epoch today; the 3 October note to re-measure before the 4 BPS step stands.

Unit tests (cargo test --release -p kaspa-consensus --lib difficulty): 15 pass, the 12 of 3 and 4 October unchanged plus reference_window_switches_at_the_activation_height (never, one block before, at, inside a fresh epoch, activation 0), v2_reference_window_follows_a_step_inside_the_epoch_where_v1_eases_into_it (1,200 blocks at rate 1 then a step to 2 and a dip of the short lane to 1.4 with the parent at 1.4: v1's polluted reference reads 1.147, within 25% of 1.4, rules, and the target EASES by the 10% clamp; v2's reference over the newest 600 reads 1.55, rules, and the target HARDENS by the 3% clamp) and v1_and_v2_agree_in_a_steady_epoch. cargo test --release -p kaspa-consensus-core --lib params: 7 pass (override_params_carry_the_difficulty_v2_activation: every network's default is never, the file sets it, u64::MAX round-trips through JSON; the fast-time file test now checks the field too).

6. What v2 costs elsewhere

Synthetic set (sim.py, seed 7, honest stamps; the 3 October standard for the 50x step-up is under 90 s):

Case	v1	v2
Hash rate x50, settled s	61.7	65.5
Hash rate x50, overshoot	0.140	0.222
Hash rate /50, settled s	628.5	752.7
Hash rate /50, worst gap s	32.3	61.8
Epoch step +-30%, first within 10% s (mean of 6)	86.5	113.5
Epoch step +-30%, settled s (mean of 6)	143.7	143.4
3x pool hopping every 15 min, settled s	189.5	181.9
10x pool hopping every 15 min, settled s	211.3	199.6
Polluted window (devnet case), settled s	62.7	66.6
Polluted window, overshoot	0.180	0.089
Genesis 10x too hard, settled s	321.9	321.9
Genesis 10x too easy, settled s	58.7	58.7
Steady std of the expected block rate, constant hash rate	0.038	0.049
Steady blocks/min CV (Poisson alone 0.129)	0.135	0.130
Steady std, 10%/h random walk	0.062	0.060

The cost is the reference lane's noise: a 600-block work-over-time estimate carries about 4% of Poisson noise where Kaspa's 2,644-block window carried 2%, so the steady std of the expected rate rises from 0.038 to 0.049; the realised blocks per minute do not change (CV 0.130 against 0.135, Poisson alone 0.129). The 50x step-down is slower on this seed (753 s against 629 s, worst gap 62 s against 32 s): with blocks 50 s apart the reference window holds 12 blocks instead of the epoch's, and the epoch lane's prior blend holds it back; the step-up stays under the 90 s standard. Hops are slightly faster and the polluted window's overshoot halves, both because the reference forgets sooner. With +-500 ms stamp jitter the v1 numbers of 3 October stand (results.md); v2 was not re-run with jitter.

Attack suite (sim/difficulty/attacks/attacks.py, seeds 7 to 9, the forger under the 10 s rules; v1 rows are the 4 October rule's rows of attacks/results.md, "igneum-san" there, v2 rows results.md "Rule v2"):

Scenario	Measure	v1	v2
1. Pool hopping, 24 h, hopper at 10 / 30 / 50 / 100% of the base, greedy	hopper's blocks per hash over the base, minus 1	+1.5% / +1.0% / +0.9% / +0.7%	+2.5% / +1.6% / +1.2% / +1.0%
1. the same with a 60 s dwell		+1.3% / +0.1% / -1.7% / -4.0%	+2.3% / +0.8% / -0.3% / -1.5%
2. Pulsed rental, 50x for 10 min every hour	pulser's blocks per hash over the base, minus 1; weight per hash	-96.4%; 0.262	-96.3%; 0.262
2. the same, once		-97.1%; 0.130; worst gap 78 s	-97.2%; 0.130; worst gap 98 s
3. Timestamp stretching, 30 / 50% forger, latest / earliest / alternating	block-rate drift after 1 h	+0.8 / +0.8 / +0.4% and +0.9 / +0.7 / +1.1% (worst seed +2.7%)	+0.2 / -0.2 / +0.1% and +0.4 / -0.8 / +0.5% (worst seed -1.5%)
4. Short-lane oscillation, 25% miner on and off every 120 blocks	std of rate after / before	3.75 (worst 4.82)	3.29 (worst 3.71)
5. Epoch games: hold dodger / 10x hold flooder / 30% hold flooder	gamer's blocks per hash over the base, minus 1	0.0% / +0.7% / +0.3%	0.0% / +0.4% / 0.0%
6. Polluted window, 10x miner joins at 0 / 480 / 480 / 0 and leaves at 600 / 600 / 720 / 1,200	settled after the leave, s	297 / 287 / 312 / 329	297 / 288 / 312 / 331
7. Flood at 85 blocks a second	floor reached, no overflow	block 2,635 (unit test)	unchanged (the floor is outside the lanes)
Base profiles, 3-seed means: up50 / down50 / epoch30 / hop10 / polluted / warmup-hard	settled s	154 / 762 / 88 / 245 / 75 / 327	150 / 822 / 88 / 233 / 76 / 327
Base profiles	steady std	0.042	0.053

What v2 loses: the greedy hopper gains one more point (at most +2.5% against +1.5%), because a reference that forgets in 10 minutes follows the hopper's own effect on the difficulty sooner; the 50x step-down is 8% slower; the steady estimate is noisier by a quarter. What it gains: the live oscillation (sections 3 and 4), a smaller forger drift (within 1% on every cell, worst seed 1.5% against 2.7%: both lanes now run on the sanitised clock, so a forged step is paid back in both), a smaller oscillation gain in scenario 4 (3.3 against 3.8), and the pulsed renter's recovery gap 10% shorter. The pulse, the epoch games and the polluted window are unchanged to the first decimal.

7. Test network

sim/difficulty/testnet_v2.py, 3 igneumd nodes from devnet-v4 with the change (ports 29600 to 29622, appdir /tmp/igneum-diffv2/run1, igneum-devnet-960), infra/fast-time/override-60x.json plus genesis bits 0x1f010000 (2^16 hashes per block), the devnet's program epoch (3,600 blocks, lead 600: the 60x file's one-minute epoch would empty the reference window every minute and hide the pollution) and difficulty_v2_activation_daa 900 on nodes 1 and 2; node 3 started without it (rule v1 for ever) as a follower. CPU miners igneum-miner mine --engine igneum-pow, 4 threads each: A on node 1 for the run, B on node 2 from minute 4 (a 2x step inside epoch 0 under v1), off at minute 19 (a step down under v2), back at minute 23 (a step up under v2). Record: records/testnet-v2-2026-10-04.csv (1,471 blocks, 1,249 chain blocks, 18.7% with two or more parents), schedule beside it.

The switch: node 1's virtual DAA score reached 900 at 1,022 s (12:47:52 BST). Node 3 rejected the first v2 block, "difficulty of 520437997 is not the expected value of 520406991", struck and banned its peer and stayed at DAA 900 for the rest of the run (901 headers exported, a prefix of node 1's); nodes 1 and 2 exported the same 1,472 headers and the same sink through the end. The node logs print "Difficulty rule v2 from the override file: active from DAA score 900" at start.

The rule: the Mac carried four other agents' builds during the run (load average 184 at minute 10, 418 at minute 46), the miners ran at a fraction of a core each, and the python schedule itself was starved for 80 minutes after minute 26 (the chain ends at 1,560 s). The delivered hash rate swung by 2x on its own: the difficulty fell 40% after miner B joined (7,868 at 240 s to 4,308 at 480 s) and rose back to 7,400 by 660 s, so the v1 phase cannot be read as oscillation evidence either way. Under v2: the leave at 1,140 s eased the difficulty from 6,589 to 5,972 over 180 s with the std of log difficulty 0.036 and no peak over 15%; the rejoin at 1,380 s hardened 6,154 to 8,312 within 60 s and then held 8,333 to 8,826 (within 3%) until the chain stopped. Per phase (30-s bins in schedule.log, testnet_v2.py --analyse):

Phase	Chain blocks	Std log D	Peaks over 15%	Min	Max
A alone, warm-up from a 2x too-hard genesis (60 to 240 s)	153	0.207	1	7,953	14,650
B joins under v1 (240 to 480 s)	146	0.155	0	4,603	7,968
both under v1 (480 to 1,020 s)	402	0.208	1	3,910	7,451
the switch (1,020 to 1,140 s)	121	0.069	1	5,674	8,028
B leaves under v2 (1,140 to 1,380 s)	202	0.036	0	5,929	6,639
B rejoins under v2 (1,380 to 1,680 s)	168	0.149	1	6,066	9,157

What the test network establishes: the switch takes effect at the height and nowhere else, a node without the parameter forks off there, two nodes with it stay on one sink, and v2 follows a step down and a step up on a real DAG with no rejected block between the v2 nodes. The damping comparison between v1 and v2 rests on the live record and the two simulators (sections 3 and 4); the run should be repeated on a quiet machine, which takes 30 minutes.

8. Rollout

Binaries: devnet-v4 of vendor/igneum-node at the commit of this entry, built as today's devnet binaries are (Mac: cargo build --release -p kaspad -p igneum-miner --features kaspad/igneum-pow; Linux: infra/cross/build-linux.sh; Windows: the launcher package's igneumd.exe). Only igneumd changes; the miners follow templates and need nothing. A node without the parameter in its override file keeps rule v1 and forks off at the height (section 7), so every node of a chain must carry the same value before the height arrives.

1. The 12 Hetzner nodes (infra/cloud-devnet, their own chain) first: put "difficulty_v2_activation_daa" in their override file at their current DAA score plus 1,800 (30 minutes), restart them one by one, then have one miner join and leave 10 minutes into an epoch and read the difficulty events: no flips after the height.
2. The devnet: choose the activation DAA score about two hours ahead of the first restart (7,200 blocks, so every node can be restarted in turn with the chain running). Order: the observer node (26640/28640, a follower, lowest risk), the seed (/opt/igneum/v4/bin, infra/seed-nodes/stage-v4.sh), Mac node 1 (26610/26611), PC 1's node (its launcher's igneumd.exe; PC 2 mines through the Mac node and needs nothing). Each node gets the same "difficulty_v2_activation_daa": N in its override file. The founder restarts the live processes; this entry does not.
3. Watch the observer's difficulty events across the height and the next epoch boundary; with a miner joining inside an epoch the floor and the bursts of section 1 must not return.

A new network sets the parameter to 0 (v2 from genesis).

9. Not done

• The DAG model has one fitted scale (the miners' template staleness); its merge fraction is 35 to 37% against the record's 34 to 39%, the Mac's stale stamps are modelled as its job queue. Red blocks are not modelled (none seen on the record: every merged block was blue).
• The Mac Metal worker's hash rate is the bench-log constant, not a log series.
• The live boundary at 11:02 coincided with a hash-rate step (both PCs restarted), so the clean prediction "the oscillation stops at the epoch boundary with the hash rate unchanged" is checked in the simulator only.
• The 600-block reference window is noisier than Kaspa's 2,644-block sampled window in steady state (section 6); a longer window with the pollution washed out by a different mechanism (a window that restarts at the last clamped ramp) was not tried.