igneum/docs/plans/morning-2026-10-06.md

Morning summary, 6 October 2026

Written for the project lead at 08:40 UTC. Every number is in docs/bench-log.md or the named plan with the command that produced it. Failures are listed with the passes.

The headline

Everything on the overnight list landed. The one thing that could have gone wrong, switching the live chain to program class v3, held.

Piece	State	Number
0.3.11 (class v3 + proving v1)	On master 630da6b, three master CI runs green, rolled out to the Mac, PC 2, the seed, both hand nodes; PC 1 took it at 07:00	Live manifest 0.3.11, nine-field override, digest 0139ab9d
Counter ASIC 2.0 crossing	Crossed at 03:51:42 UTC, watcher verdict PASS	58.7 blocks a minute before, 59.2 after; three hourly swaps since, no pause, 0 refusals
12 GB proving	Floor broken on a patched SP1 server (prover-floor bcc6d68); real card lands today, test on PC 2	Proves alone: 10.3 GB, 5.7 s a shard. Mines AND proves as a core-only prover handing its proof to a big-card aggregator: 8.1 GB beside the miner, 19.8 s a shard (measured 09:15 on the 5090's allocation). 16 GB mines and proves compressed (12.9 GB, 17.4 s). 8 GB stays out
On-die recompute chip	Emulated on the 5090's own L2 (ledger-pc2 564acab)	0.256x honest, 5.1x worse per joule
Ledger	Rounds 2 and 3 closed (fud-close d16bc3b, ledger-rebase abb08a5)	47 consequence rows, 42 closed or taken, 14 decisions
Branches ready for later cuts	pool-v0, rig-install, repro-bench b776199, ember-tune 9a6469f, ota-k2, asic-history, proving-methods	measured where PC 2 allowed

Since you got up (07:00 to 08:40 UTC)

What	State
PC 1	Back on 0.3.11 at 07:00, 5090 and 9070 XT mining; integrated card off. The update needed your click because the app's hourly rollout slot had not come; fixed as a catch-up rule (update-catchup 2207cd7, 0.3.12)
Mac	Mining paused through the app (persists); its node runs
Chain	19 miner ids, 225 MH/s on the two PCs
Zero proven segments	Solved: the prover claims a whole segment and proves its 8 blocks in order (9 segments in 30 minutes on PC 2 beside the miner, 72 of 72 shard records paid, 11 percent hash cost). The segment record itself needs a consensus switch on the node fork (proving_v1_fresh_rule_daa), folded into 0.3.12
Ember Tune re-run on PC 1	Failed at 07:56 with no rows: the playbook wrote the test engine's settings with a byte-order mark, the engine parsed defaults, sat idle 35 minutes. Nothing was set on either card; the app restored its miners by itself. Fixed (8273494, watchdog 1e9550e, CI check). Re-run ember-tune-pc1-3 needs one more click when you are back
GPU list order	Cards ordered by performance, integrated last (card-order ffb2bfa, 0.3.12)
Counter ASIC 3.0	Running, all seven items plus a new item 8. See below
0.3.12	Being prepared: the app items plus the node fork with the segment switch; stops at the publish gate for your go

Counter ASIC 3.0 so far

The finding that matters: the chip that wins is not the clever recompute chip 2.0 priced. It is a stored-dataset chip, the whole dataset in DRAM, a 28 nm memory-controller die doing dependent reads.

Attacker	Per chip vs 5090	Per joule vs 5090	Source
On-die recompute chip (2.0's model)	0.92x with the 3x allowance	1.86x	chip-model-v3
Same, emulated on the 5090's L2	0.256x	0.2x	M16 inline bench
Stored-dataset chip, GDDR7	1.22x	5.1x	item 1
Stored-dataset chip, HBM3	about 1.2x	7.5x to 9.2x	item 1
Ethash precedent (E3, A10)		2.1x to 4.8x	history rows 3, 4
Stored-dataset chip with the latency shadow filled (item 8, N = 100,000, parity cores)		2.7x vs 5090, 1.4x vs M5 Max	item 8 Mac rows

Why: at the hash the 5090 spends about 55 W on memory and the rest keeping a GPU alive at 0.15 percent of its integer budget. The lever is RandomX's lever: make the hash use the rest of the chip. The 5090 can hide about 330,000 operations per hash behind its 128 reads; today it hides 512. Item 8 measures that fill: on the Mac it costs 1.5 percent of rate at 100,000 ops, the verifier barely notices, and the chip's edge drops from 5x to 2.7x. The 5090's rows are queued on PC 2. The deciding number is the chip core's energy per op against a GPU's ALU, which item 8 is pricing.

Closed at 08:50. One class v4 candidate, measured on the hash's own numbers and ready for its six-gate run on your word: mixer x8 plus 100,000 operations of program work per hash. The 5090 loses 0.2 percent of rate and the M5 Max 1.5 percent; the verifier adds 0.17 ms per warp; bit-exact on Metal, CUDA, Apple OpenCL and the CPU emulation. The chip must then carry a 14,000-lane ALU array, and its per-joule edge over the 5090 falls from 5.6x to 2.1x at a core as efficient as the GPU's, 1.5x at a realistic one. Your test as a number: the chip crosses 2x only if its datapath spends under half the energy per op that a GPU does. The cost per tier: a 5090 draws 431 W instead of 350 for the same blocks (a rig pays about 23 percent more electricity), the M5 Max 37 W instead of 21, a pool user sees nothing. The 9070 XT and 4060-class rows are owed, the AMD ones because PC 1 was left alone.

Other items: item 2 (per-day random derivation) works bit-exact at no hash cost and drops the recompute chip to 0.29x to 0.43x, but at full size its CPU verifier is over the 10 ms gate on an old core; it goes in as a reserve, the half-size draw passes, and the class v4 candidate is the pairing of derivation class and program length under one verifier gate. Item 3: cryptanalysis brief and budget line, USD 80,000 to 160,000, one firm plus one academic group, verdict GO to commission, nobody contacted. Items 4 and 5: share-pattern detector in the observer (fires on a fabricated fixed design, quiet on the devnet), issuance trigger at USD 20,000 a day, FPGA soft overlay 0.3x to 0.4x a 5090 per watt, layer 9 ranked above layer 7. Items 6 and 7 in flight.

Decisions for you

The full list with recommendations is in consequences-decisions.md (14) and the 3.0 status file. The ones that bite first:

0. Run the six gates on the class v4 candidate, or wait for the 9070 XT and 4060-class rows first (3.0 status, decision 2).
1. The public claim "under 2x". True of the recompute chip per chip, false of the stored-dataset chip per joule. Two re-wordings are drafted in the 3.0 status file; nothing on the site changed. Pick one before any public push.
2. The segment rule. Answered at 08:50: it is a consensus rule (as shipped a fresh segment record is valid for an 8-second window). The fix is on the node fork behind a new switch proving_v1_fresh_rule_daa, so 0.3.12 carries the node and goes out as a two-manifest publish with the switch at tip + 14,400. Measured on PC 2 beside the miner: 9 whole segments in 30 minutes, 72 of 72 shard records paid, 11 percent of hash rate.
3. 0.3.12 go. The state-reply fix, the update catch-up, the card order, Ember Tune and its guards, plus the node fork with the segment switch (two-manifest publish, switch at tip + 14,400). No prompt on any machine.
4. The 12 GB card. Into PC 2 when it lands (PC 1 has no prover toolchain); the playbooks are written. Core-only provers need a pool-protocol change (a core hand-off format only aggregators accept, the compressor's credit, the prover's signature over the proof hash); nothing on chain changes. Decide whether that goes into the pool spec now.
5. Cryptanalysis budget (D11 and item 3), growth mapping (b) (D4), a release-tag convention, the bounty only once escrowed.

What went wrong, plainly

• Epoch-34 pack outage, 18:23 UTC, 56 minutes of both PCs down: a pack-attempt bug in the worker loader. Hot fix shipped by job; class fix in 0.3.10.
• GitHub Actions outage forced a PC-built 0.3.10. 0.3.11's first CI run then failed in the census crate nobody updated for class v3; fixed (2a62735), rerun green.
• Credits ran out at 19:58 UTC and killed three agents; resumed.
• The 9070 XT dropped off the bus four times; the reading was wrong, the relay's "PC1" is PC 2.
• A 2.2x verifier regression on the mixer branch, caught before publish.
• Two Mac-only v3 outages caught by the Metal gate before publish.
• "12 GB proves" was false on the shipped prover; the floor was SP1's server code; patched.
• A job quit PC 1's app at 22:31 UTC (a second engine ran the urgent updater, whose installer sent quit). Rule written, CI gate added; PC 1 stayed down all night. Fixed in e600e63.
• Three dark proving windows on PC 2, all the same state-reply bug (paid_wei above u64::MAX empties the reply); 0.3.12's first item.
• The Mac miner lost its node subscription for 11 minutes after a node restart (C43, 0.3.12).
• The 0.3.11 update on PC 1 waited on the hourly slot and needed your click; catch-up rule in 0.3.12.
• Ember's first real run burned 35 minutes and your prompt on a byte-order mark; fixed with a watchdog and a CI check.
• The 07:45 summary task never fired on its own and its manual run stalled on a tool prompt; this document was written by hand.

Today's hands list

1. 0.3.12 go when the shipper reports it green.
2. One click for the Ember re-run on PC 1.
3. The 3060 into PC 2; the playbook runs the same fixture as the sweeps.
4. 16:00 UTC: the fee-switch check (every prover on 0.3.11 before H = 210,000 at about 19:15 UTC).
5. USB copies of the key backup (10:00 reminder).
6. The decisions above.

Since noon (13:15 UTC)

What	State
0.3.12	Shipped: merged to master 494c9c7, both master CI runs green, every node on the ten-field object, the fresh-record rule arms at about 15:55 UTC
The 12 GB card	Measured on the 4070 in PC 1: proves alone 7.6 GB in 7.7 s, mines and proves beside its own miner 9.0 GB in 24 s, every proof verified. The packaging row (our signed build of the patched server in the app) is built and verifying on PC 2; when it ships the public line moves to "12 GB mines and proves"
Rented fleet	36 boxes on Vast and RunPod, about USD 7.5 an hour; live page https://dl.igneum.network/fleet-22adafa34bc2. Real-card rows so far: 24 GB stock server proves; 12 GB mines and proves at 10.1 GB on Linux; 10 GB and 8 GB prove alone at 2^26 (the floor is now "8 GB proves alone, slowly"); the miner costs 4x to 8x on proof time
Apple proving	SP1 on the M5 Max CPU proves the v1 shard in 72 s. RISC Zero's current release has no Metal prover (575 s on the CPU), so the Apple route is SP1 on the CPU; RISC Zero keeps the version slot for the 8 GB CUDA tier and its 5.7x smaller proof
Counter ASIC 3.0	Closed: one class v4 candidate ready for its gate run on your word (see the 3.0 section above)
Ember on PC 1	Three runs failed on tooling (byte-order mark, a locked scratch folder); fixed each time; the real run needs you at PC 1 for one click, since Windows cancels an unanswered prompt after two minutes. 0.3.13 carries the helper that makes it the last prompt ever. Rented containers refuse power caps, so tuned priors come only from PC 1 and PC 2

Found by the fleet, both fixes in flight for 0.3.13

1. A fresh node never executes the EVM. A node syncing from the seed through the pruning proof has no blocks below the pruning point, and the execution follower, which walks from genesis, sits silent: zero wallet, empty explorer, no proving work, mining fine. Every 0.3.12 joiner today is in that state; only genesis-era nodes execute. Fix: a loud "not synced" status, and an execution state snapshot at the pruning point verified against the header's state root, fetched from a peer. Stop-gap for tonight's fleet: a full sync from genesis, or a copy of the observer's execution data.
2. The seed drops every fresh peer every 30 seconds. The per-checkpoint certificate burst fills the finality route and the inherited rule closes a full route's connection; the baseline shows 11,700 already-known certificates resent in five minutes. Fix: a sized route, no replay during sync, drop instead of disconnect, and a guard counter.

Afternoon (16:00 UTC)

What	State
0.3.13	Approved by the project lead at 15:20 UTC with the one-time devnet state reset: execution restarts empty at chain block 27,276 (11:40 UTC today) and re-derives forward; everything earned since is back, the first days' balances are gone; the chain, finality and the hash untouched. Carries the execution persistence and snapshot path, the fresh-joiner fix, the finality route fix (an echo of old certificates, 13,354 re-locks in 7 minutes at the seed) and Ember's helper. Two publishes, three new override fields, protocol 15 to 16, no prompt anywhere
Rented fleet, phase 1	Done on 11 real cards for USD 9: every NVIDIA card from 8 GB proves; 12 GB and up mine and prove at once; 8 GB proves alone or mines beside a core-only prover (docs/analysis/prover-tiers-real-cards.md). The 8x 4090 rig is measuring on RunPod; an 8x 5090 is refused by both providers so far; no provider rents consumer AMD
Ember	Root cause of every failed run found and measured: the engine's own folder lock stripped the permissions off files a job copied in. Dry run with no prompt passed (5090 127 MH/s at 316 W, 4070 28.7 at 103 W). The table run goes on 0.3.13 with the project lead's one click, which registers the helper and is the last prompt ever
PC 1	5090, 4070 and the 9070 XT all attached (two enclosures). Queue: 0.3.13, the Ember run, then the owed AMD rows for the class v4 candidate
Packaged prover server	Built and verified on PC 2 (prover-floor bf7b174): the patched server shipped signed in the app, fail-fast where it hung, a per-shard timeout with threshold step-down, tiers from the real-card rows. Ships in 0.3.14 with the public line
Apple	SP1 on the M5 Max CPU proves a shard in 72 s; RISC Zero's release has no Metal prover (575 s on the CPU). The Mac prover path uses the CPU; RISC Zero keeps the version slot for the 8 GB CUDA tier