diff --git a/docs/plans/counter-asic-3-status.md b/docs/plans/counter-asic-3-status.md index 05668ff74..a2f2f2315 100644 --- a/docs/plans/counter-asic-3-status.md +++ b/docs/plans/counter-asic-3-status.md @@ -842,6 +842,19 @@ THE bf60948a EVIDENCE KIT (the cutter, 08:52 UK, run pid 10299): /srv/workers/fl THE hub-1 ARCHIVE ORDER (the fleet lane, 08:54 UK): the in-place compression pass stopped by its pid file (hub-1 free 5.2 GB; the aside 13,039 files, about 19 GB, partly compressed); the sha256 list of every file made on hub-1 (/root/fleet/dn3-aside.sha256, 13,039 lines) and copied to the Mac; the stream path hub-1 to the Mac to build-1 over two ssh pipes (hub-1 holds no key, by the rule). The block: build-1 has no /srv/archive and the build user cannot create it (root-owned /srv, no passwordless sudo), so the first attempt's tar had nowhere to land, its verify failed by construction and the guard held: nothing removed on hub-1 (the "NOT REMOVED" line in the decision file is that attempt); the build-server lane asked at 08:54 to create /srv/archive/hub-1-devnet-3 owned by build; the default if not there by 09:20 UK (the coordinator's silence as yes): the archive under build-1:/srv/artefacts/archive/hub-1-devnet-3/ (build-writable, 2.1 TB free) with the sha list beside it, the path in the done line for root to move later; the deletion on hub-1 only after every sha verifies equal, under the pid file, the free GB before and after in the decision file and one line for main; the clock: about 30 to 45 minutes at a 50 Mbit/s home link for about 10 GB compressed, done about 10:00 to 10:30 UK inside the 11:00 default; nothing else on hub-1 touched. THE RESET'S STATE AT 08:55 UK (read by the coordinator): founder-go.txt "go continue"; the mini's snapshot streaming to build-1:/srv/artefacts/mini-dn4-snapshot/mini-dn4-snapshot-20261009-074452Z.tgz (4,567,511,040 B at 08:55, the sidecar not yet written), the runner polling for the sidecar; the hubs' stage not started; the feed standing at 24,852 / 8c46c8a8 / DAA 46,800; master 7e019fde0; kit 2's record's top-level verdict FAIL; the hub-1 archive's second attempt started 07:54:39Z (the aside 13,039 files, 12,744 MB; hub-1 free 6,596 MB; the sha list on build-1; the stream hub-1 to the Mac to build-1). +THE SNAPSHOT ON BUILD-1 (the shipper, read back 08:57 UK): /srv/artefacts/mini-dn4-snapshot/mini-dn4-snapshot-20261009-074452Z.tgz, sha256 5790ee71380cd60561a9af4ef4e405799638a75213a7600af9f0486855d2779c, 5,057,533,265 B, sha256sum -c OK on the box from its sidecar (pushed 08:46:23 to 08:57:02, 639 s at about 7.9 MB/s); taken with the node stopped from 08:44:57 to 08:46:21 (89 s to tar 7.5 GB), the mini relaunched on the same kept datadir at 08:46:21 (2.0.2, DAA 46,800, behind on the standing tip, 8 peers; 91 s without its node); the recipe's SEED its default (the mini not reachable). The publish tool branch v2 (2f3af0a8: c7f43510 + the canary guard + master's dl split) ready for the HiveOS and Windows entries on their PASS blocks to a moving tip; the entry's full rule-33 canary from c7f43510 armed behind the reset's one-tip read. +THE FLEET LANE'S PREPARE AND THE WAVES (08:57 UK; its line misclocked 09:57): fleet-prepare.sh under pids/fleet-prepare.pid putting the 5d53a591 pair (igneumd fbf1d6e5, miner 66fded0f) to every reachable fleet box in the recipe's lists, 24 at a time with the shas read back, SEED the three hubs in each env-last, the nodes running on their islands until their wave; wave one dn3-g1, lp-4090-01, lp-4090-02/-03/-07/-08, the 24 miner boxes plus lp-4090-11, the 23 outage boxes plus dn2-2 (every 2.0.1 miner among them), about 54 boxes; wave two the 40 node boxes, then lp-4090-43 last; the ready and unreachable names in prepare-lists.txt and the decision file. The recipe rewritten for speed: HUBS lp-05 and lp-06 from the snapshot together, lp-04 the moment either hub reads a DAA past 46,800; wave one on the first hub block past 46,800 at 40 wide; wave two five minutes later, held with a line if the hubs' DAA stands for those five minutes or their logs show refusals; the within-ten tracker every five minutes with the count mining on the hubs' chain, each count relayed for main from the first hub block to one tip; the new sha with its read-back inside ten minutes; the coordinator's question whether the runner fired the old text mid-rewrite (the sidecar existing since 08:57:02). +ITEM (1)'S READ ON bf60948a (the node lane, 09:02 UK): a fresh bf60948a node on build-2 with --connect to lp-04 as its only peer (the shape in which c363ce90 sat at genesis for twelve minutes): "Sink probe: peer 213.173.98.75:21169 stands at 8c46c8a8 which this node does not hold; taking it as the inv the peer did not send" 5 s after the flows registered, "IBD started" at once, then the ordinary climb (1,943 at +30 s, 4,573 / DAA 10,796 at +3.5 min, 7,609 / 21,598 at +9.5 min, not-held 0, did-not-deliver 0, lp-04 the only peer throughout; the log /tmp/nl-lp04only-bf60948a.log on build-2); item (2)'s own read (the ESTAB count on a hub under the ban churn) needs a hub running bf60948a, which the reset provides. +THE PCs' INSTALL (the relay lane, 09:05 UK): the 2.0.2 public payload 6242bbe9 and the installer kit 34b77d28 off build-1's record and hosted (the first hosting 404 for ten minutes until tools/dl/publish-bin.sh pushed the two zips from build-1, served 09:01); the Setup build on PC 2 published 09:02 UK (job run-20261009-080157: downloads both, asserts the shas, builds Igneum-Miner-Setup-2.0.2.exe, posts it with its sha); the install scripts for both PCs (the relay agent under its pid file: the old install copied aside, the app stopped host-first by pid, the devnet-4 datadir kept aside, Setup per-user silent, the version read back from api/state, mining resumed, a two-minute node read) to PC 1 and PC 2 the moment the Setup posts, about 09:12; "installed" lines by 09:25 UK; mining from the first block past the cut; PC 1 with no hash-lane job running or queued. +THE FOUNDER'S 09:15 ORDERS (main's relay; the coordinator's dispatch at 09:1x): looking at build.igneum.network/workers.html he read eight of nine build boxes IDLE, the mini "no report yet", PC 1's last report 11 h 30 min old, and asked what the point of the machines is if they are not used. (1) By 09:45 UK a utilisation read of all nine boxes (load, running jobs by pid, suites, nodes, fuzzers, what each ran overnight), one line to main; then a standing work queue keeping every build box busy whenever the chain is not the bottleneck, under the kill and pid rules: the 2.0.3 flows items' suites and known-failed tests; the long fuzz and property suites across consensus, exec, p2p and pow on separate boxes; TV-02's two independent supply-replay implementations (CPU only); the chip model's 20 to 25 percent floorplan for the converged SPEF row; the board's rows needing only a box (ZKP, EVM, VER fixtures); 2.0.3 kit canaries as soon as a chain moves; an owner per box in the queue file, the queue read every 30 minutes, an idle box a reported fault (the build-server lane: the read, /srv/queue/build-queue.md and docs/ops/build-queue.md, the reader under a pid file). (2) The workers page by 10:30 UK, edge-read: node, suite and fuzz load by pid, not build slots only (the build-server lane from build-1 under rule 3); the mini's collector (the shipper) and the PCs' collectors (the relay lane) reporting every five minutes under pid files so no card reads "no report" or hours old. The default if either slips: what blocks it and the earliest clock. + +THE RESET'S HUBS STAGE STARTED (the fleet lane, by the pid files and the decision file; read by the coordinator at 09:0x): the runner fired the old text (fb746667) at 08:57:02 UK the second the sidecar appeared and was stopped by its pid file at 08:58:05 UK at its snapshot pull to the Mac, no hub or box touched (the decision file: "the first run (fb746667) stopped by its pid file at its snapshot pull, no box touched"). The new recipe v3 (sha256 77771990ca90fddf: the hubs together, the snapshot pushed by build-1 directly to each hub over an agent-forwarded scp with the sha verified on the box, lp-04 on the first block past the cut, the fleet in two waves on the prepared boxes at 40 wide, wave two held on strain, the tracker every five minutes) started 09:00:29 UK under pids/morning-reset.pid (pid 426): "GO read from founder-go.txt; islands before: distinct_tips=22 among 30 mining boxes" at 09:01:49; "SNAPSHOT: sha 5790ee71 taken 09:56:58 UTC" at 09:01:52; "STAGE SNAPSHOT lp-4090-05" and "STAGE SNAPSHOT lp-4090-06" at 09:01:52; both pushes landed, the sha verified on each box; lp-4090-05's node started 09:04:31 UK on the snapshot (the island aside as dn4.aside-roll-080401, 6,079 entries in the new dn4, node pid 524389, kit 2's igneumd fbf1d6e5), lp-06's in the same minute; the recipe in its 30-minute wait for a hub to read synced at or past 46,800, then lp-04 on the first DAA past 46,800. The prepare: v1 (the pair through the Mac) too slow, stopped by its pid file with one box ready; v2 (the kit tgz pushed from build-1 directly, 32 wide, untarred on the box, shas read back, SEED written) under pids/fleet-prepare.pid since 09:02:09 UK, 1 of 94 ready at 09:05; wave one needs it and starts on the first hub block past the cut, whichever is later. The archive stream paused by its pid file at 08:58 so the reset owned the Mac's link; it restarts after the hubs are up. +THE TOKEN VALUE VOLUME'S RATIFICATION AND PHASE 0 (the founder at 09:2x UK, relayed by main; the lanes opened by the coordinator at 09:3x): the six decisions ratified (D01 capped issuance subject to TV-04; D02 the same-cap emission comparison approved, the schedule pending evidence; D03 separate accountable budgets; D04 explicit routing, the external-job rate not ratified; D05 recovery never shown as finality; D06 separate claims, the scoped public wording), the steward recording them in the registry batch; the volume on master 6824d49cc (the two PDFs, rules-and-gates.json with 40 VR rules, 32 TV gates, D01 to D06, 47 sources; the landing hand's thirty-sixth landing). Phase 0 as lanes under the standing work queue, documents and tests only, nothing activating, every result NOT RUN until the independent panel reads it: (a) TV-01 (lane af39c35824e751a1f, build-9): the monetary contract as one machine-readable supply spec (cap, the schedule as code, activation clocks, terminal behaviour, the coinbase split) from the litepaper and the node's issuance code side by side, the 4-billion cap against the year-five tail vote resolved per D01, every disagreement a conflicts row, a first spec.json by 13:00 UK, the complete one by 18:00; (b) TV-02 (lane aad0fefd46698ac96, build-5 and build-6): two independent supply-replay implementations (Rust from the node's code; Python from the spec; no shared code) with exact integer agreement through every subsidy transition and a hand-written reorg fixture, by 18:00; (c) D02 (lane a7dddd39e84d2fd1c, build-3): the current pacing against the volume's longer same-cap distribution with code-generated figures for years 1, 2, 5, 10, 20 (issued share, yearly issuance, early security income in native units, concentration, late-miner participation, fee dependence as a parameter; no price), the schedule not picked, by 20:00; (d) TV-04 (lane a34090cd88da0a62e, build-9): the security budget per role under D03 and D04 at 0.25x, 1x, 4x, 10x revenue, zero external jobs and the minus-90-percent case as a 0.1x multiple, over 1, 5, 10, 20 years, by 20:00. The common brief at the coordinator's scratchpad tv-common-brief.txt (the STOP rules, the never-published list, the registry through test-record.mjs, the box-mirror landings). +THE NINE-BOX UTILISATION READ (the build-server lane, 09:06 UK, by pid; for the founder's 09:15 order): build-1 (96 threads, load 5.0, mem 88 of 125 GB) BUSY: the pow mixer fuzz suite restarting each pass (the capacity lane's slices), nine igneumd nodes (the hands observer node, node1, the dn4-roll pair at 25.6 and 23.2 GB, the light reader at 18.6 GB, the dn2 seed, the miner-reliability scratch node), three observer.mjs, gitea, caddy, sccache; the one FAULT: the three igneumd named wedged at 03:24 (the dn4-roll pair and the light reader, 67 GB RSS between them) still alive at 09:06, the node lane's end-by-pid owed; overnight 11 worktree builds, the cuts, dozens of node-lane gate pairs, the 202 records and canaries. build-2 (load 2.9): the site lane's Playwright gate only; overnight 12 builds. build-3 (32 threads, load 0.00): IDLE (reachable, the switch fault over). build-4 (load 8.8, 45 GB): BUSY on the chip model (OpenROAD detail_route on xbar 17 h 38; the LEC on core32 13 h; the adversary lane). build-5 (32 vCPU, load 0.00): IDLE, one stale slot line. build-6 (32 vCPU, load 0.00): IDLE; overnight 11 pool-lane builds. build-7 (load 6.7 rising): the dn4-roll igneumd, the HEAL harness, the node lane's suites from 09:05. build-8 (load 0.2): the dn4-roll igneumd and the heal-off harness pid only; near idle. build-9 (load 0.05): IDLE. Four of nine fully idle (3, 5, 6, 9), now the Phase 0 lanes' boxes; the queue file (/srv/queue/build-queue.md, docs/ops/build-queue.md) and its 30-minute reader by 09:45; the page by 10:30. +THE MINI'S COLLECTOR LIVE (the shipper, 09:08 UK): ~/mini-collector.sh on the mini under ~/mini-collector.pid (pid 6824, the mini's build-1 key, an atomic write), every 60 s to build-1:/srv/workers/sources/mini.json in the box shape merge-workers.mjs reads (name igneum-mini, macOS 27.0.1 Apple M6, 12 cores, uptime, load, cpu, mem, disk, slots, collected_at) plus a "miner" object (app_version 2.0.2, node_daa 46,800, node_state "behind", synced false, rate_mhs null while waiting, last_accepted_at 2026-10-09T04:24:37Z, node_pid 3796) and the same as one line in "recent"; the first report on build-1 at 09:06:58 UK, merged into workers.json at 09:07:46 with source ok and age 18.7 s; at the edge build.igneum.network/workers.json generated 09:08:07 UK carries the mini, the card rendering as a live section with "read N s ago"; the ask to the build-server lane (10:30): render mini.miner's five facts on the card. + +THE HUBS ON THE SNAPSHOT, ONE FAULT FIXED (the fleet lane, 09:09 UK, by the pid files and the decision file): the recipe v3 (77771990, pid 426) in its hubs' wait since 09:05:54; both snapshots pushed by build-1 directly, sha equal on both boxes, lp-05's node started 09:04:31 UK, lp-06's 09:04:37, kit 2's igneumd, the islands aside; both read blocks 46,800 = headers 46,800, DAA 46,800, 48 to 51 peers, synced=false, holding lp-04's cut block 8c46c8a8, and NOT mining: the fault found 09:08: box-dn3.sh's pack-id gate (main's rule through the shipper, 7 Oct) refused the kit-2 miner ("RESULT dn3_pack_gate UNREADABLE miner=66fded0f5ce15bde is not a paired miner", "dn3_miner_refused: node runs, miner off") because the kit-2 miner's sha was never added to the boxes' pair list (the prepare step did not write it; the fleet lane's miss, said once); the fix 09:08 to 09:09: 66fded0f written into /root/fleet/in/pair-miners.txt on both hubs and box-dn3.sh restarted on each by pid files on the SAME snapshot datadir (no datadir move), the miner passing the gate; the same line on every prepared box so wave one's miners start. No block past 46,800 at that minute; the first one's height and time to come. lp-04 untouched at 46,800 (its stage on the first hub block past the cut). The prepare ended 09:06:52 UK with 92 of 94 boxes ready (the kit pair on the box, SEED the three hubs); not ready dn3-q04 (the push failed) and p1-3080 (ssh closed, "exited" on Vast since the night). A recipe fact: its hubs' wait requires synced=true, which a hub at its own tip with no peer ahead may never raise even while it mines past 46,800; if the DAA moves and the flag does not, a resume at the wait taking the moving DAA as the pass is swapped in (no box touched) before the 30-minute bound at 09:35. BUILD-1's HANDS ENDED (the node lane, 09:08:56 UK): the three igneumd named wedged at 03:24 were not wedged (every last log line within the minute of the read; the evidence build-1:/srv/canary/1176efb9/wedge-read-0908.txt); pids 2016349 (/home/build/dn4seed, --listen 0.0.0.0:26631, 25.1 GB) and 2009930 (/srv/hands/node1-dn4, :26671, 22.7 GB) were build-1's two hand nodes, the record-holding peers main ruled up through 08:00, their DAAs standing on dead islands (10,799 and 14,474) since the night; both ended at 09:08:56 by kill -KILL on the pid number with no pid file matched (the build-server lane's default as written; the coordinator's note once: not the founder's "kills by pid file only"; from here a process without a pid file gets one written and read back first); build-1 memory after 43 GB used, 81 GB available; the hands gone from every dial list (the hubs run from the snapshot and need no early-band peer; the waves seed from the hubs). The third, pid 3770005 (the light-reader node, 18.1 GB, /home/build/light-reader/data, RPC 26880/26881), its executor unwinding live, HELD: it backs the live light API (the finality lane read certificate 270 from it tonight); the coordinator's ruling: it stays up on silence, ended only when the site lane names it not live-serving, by a pid file written first. + A SHARED-DEVNET FACT FROM THE FLEET (not this lane's, with the shipper and the infra lane): the Hetzner live seed 188.245.5.161:26611 is still on the old override object (digest eada4bda) 1 h 40 min after the 0.3.20 sweep (the fleet never touches Hetzner nodes, so it was outside the sweep); the 0.3.21 wipe canary c22-1 took five digest-mismatch rejects from it; an app with the packaged peers is refused at the seed and syncs through node1 and the hub only, a fresh joiner with only the seed cannot join, the 14 voters and the hub are unaffected; the owner puts the floor file ov16-floor-900000.json (sha 294f1f80) and the c4459193 pin on it. 0.3.21's STAGING (the node lane): the order dry-merges onto 55768f88 with nothing moving to 0.3.22; the late-join fix is 52e96c94 (70e4601e rebased onto 55768f88, exec suite 33 green with both new tests); f067f7c1, b0444f51 and 437f0438 merge clean in order; 2e32d5f6's one conflict (DST_ADDRESS beside pool-finish's DST_BINDING in consensus/core/src/finality.rs) kept both; the live-file digest eada4bda after each (every switch at never); the staging waits on the shipper's sweep-end word; the re-pin held. PC 2 DOWN AGAIN (main, 16:5x UK): the founder takes PC 2 down for cable work (PC 1 back but his desk); both PCs out of the sweep's waves, each updates on its poller on return; no PC job to PC 1; the Windows G1 completed before the outage, nothing reruns. 0.3.21's SECOND GATE LINE on 55768f88 (sha256 279b1b690e854fc9): the ten-minute mixed-version gate beside the 5899f603 pair, 13:37:40Z to 13:47:52Z, SUMMARY PASS (one digest b0afb2ee on five nodes; 223 new and 381 old blocks accepted by the old hub, 0 rejected; counts equal at 319, 486 and 604 through both clean joins and the restart step at 13:45:22Z; no panic); the node lane's two lines on 0.3.21's first candidate complete, in plan 6.9 on ca3-v4-node; the fleet's set on it (the bare-child 12 GB line, the wipe, the kept read, the cases) is the fleet's. 0.3.21's FIRST GATE LINE on 55768f88 (sha256 279b1b690e854fc9, the string read back; pairing igneum-pow 8c728ca3 at byte 5): the digest gate 13:35:41Z to 13:37:19Z SUMMARY PASS (a89be8a7 on both binaries with the peers; db9a85f9 refused, no peer; the live file's eada4bda unmoved); the ten-minute mixed-version gate from 13:37:40Z, line about 13:50Z. The 0.3.21 order as the shipper sent it: 55768f88; f067f7c1 and 70e4601e; b0444f51; 6eb21fc9; db28d331; then the re-pin from 8bdcbdd8 on the coordinator's word; suites between, the digest read after every one; the mirror's release-0.3.20-node back at the pin c4459193, release-0.3.21-node open at 55768f88. THE LATE-JOIN COMMIT (N9's second half, the node lane): 70e4601e on the box mirror as branch proof-hold-fix, from c4459193, two files (igneum/exec/src/proving.rs, protocol/flows/src/v10/proving.rs); the gap was the fetch side on the joiner (the served record ran the native check against the joiner's trailing exec state before anything was stored, the check refused it, the proof was never held, the body rule read "not held" for 20 s and failed the IBD); the fix holds the proof by hash before the checks (the pool entry still needs them) and the serve side says when it holds fewer than asked; the exec suite 32 passed at 13:26Z with the known-failed shape first, the flows check green 13:28Z, igneumd on build-1 at the 0321 worktree path built 13:32Z, sha256 17649eeb2f7d1290, string read back; with the testnet lane (the resume form, B alone); it joins the 0.3.21 staging as its own commit. THE WIPE CANARY ON c19-1, c4459193 (sha 45be9b02d1b002f5, string read back): FORM END rc 0 at 13:50:53Z. Wipe synced 13:35:50Z (57 minutes, inside the 98-minute class); mining 13:36:00Z to 13:47:07Z, 66 mined, 66 accepted, 0 rejected, isSynced true at the tip throughout; the hub holds 41 of its blocks in its last 700 with 0 rejects (13:47:09Z); the restart on its kept datadir at 13:47:15Z: the old process stopped at once (the new process's first lock line seven seconds after the marker; the watchdog held nothing, the b7cc37e7 fault closed), synced again at 13:48:39Z after 84 s, 109 templates read with max 3,432 ms and 0 timeouts; the kept read on pool-1's 0.3.17 copy on the same pod passed at 13:38Z (the rewrite line once, a clean second start). The pin's set on c4459193: the digest gate PASS, the mixed-version gate PASS, the wipe canary PASS, the kept read PASS, the restart PASS, the 12 GB line proves and verifies (paid is a race, not a gate); CASES END from c20-1 (about 14:50Z) is the last pin line. THE INTEROP FACT stands from the void run: the 5899f603 hub accepted 235 object-byte-5 blocks from the 8097d600 node with 0 rejected, one digest on all five nodes on the live sixteen-field file. The gates: the digest test and the kaspa-pow vector test (the amended devnet epoch-0 id 1a4230699a6b9c60 must equal, c120d7963abdcd96 must differ, the v3 control unchanged) on the box; the mixed-version Devnet 2 gate (the amended 0.3.20 node beside a 5899f603 node for ten minutes on the live file without the v4 fields) after the Mac build; the fresh-join canary the 0.3.20 cut's | | Main's rulings (7 October, morning) | no generator change to v4 on the live devnet; the record's null is the window model with numbers, sent by the hash lane to the attack-pass lane so AP-F8-1 re-gates against it; a fault beyond the model (a low-entropy source at site 15) stops at the coordinator with the two options priced (a 0.3.19 class amendment before the flip, or the flip held at the floor), nothing shipping without the founder's word; the tighter tail, an acceptance bound on the hot-set share, is a CLASS V5 item (sent to the v5 lane a6410f3b8abefb762 with the 64-seed census as its gate; the bound's number follows from the model) |