From 562c33e8e5310d1bd0f08808cce3273dcc2fd4c6 Mon Sep 17 00:00:00 2001 From: igneum-labs <337424239+igneum-labs@users.noreply.github.com> Date: Tue, 6 Oct 2026 20:33:27 +0000 Subject: [PATCH] Finality pause of 6 October 2026: incident entry, ledger F27 and the F19 correction, spec 03 W7, the leave plan and harness case - docs/bench-log.md: "6 October 2026, 18:40Z to 20:41Z, the live devnet's finality pause": the timeline from the hub's own log and the hands' logs (local time +01:00: both hands stopped 18:30:28Z, returned 18:41:54Z and 18:42:07Z, relaunched under launchd 18:45:57Z and 18:46:39Z), the cause in the code (rule v3's frozen table at lock 6842, 42.7 percent of it held by 20 keys that stopped at 17:20 to 18:30Z, expiring at DAA 216,402), what was ruled out with the line that rules it out (the hands, the aggregators, the "level is 0" rejects = the Igneum Wallet app's bundled 5 October igneumd on this Mac), and what "paused" meant per tier - docs/fud-ledger.md: F27 (the incident as a critic will post it, with the fix and the operational rule) and the F19 correction: a silent 40 percent under rule v3 stalls the full window, 30 days on mainnet, not "day 19 to 20" (the v2 arithmetic) - docs/spec/03-finality.md: W7, the departure announcement, and the Q5 note - docs/plans/finality-leave.md: the 0.3.16 plan (what changes, why this candidate, the Devnet 2 gate, what it does not do, per tier) - tools/finality-attacks/leave.mjs: the harness case on the v3 runner's shape (45 percent of weight stops with leaves; --silent is the known-failed case), designed and not yet run (it needs W7 binaries on the harness host); lib/net.mjs gains the noLeave option - infra/fast-time/override-60x.json: leave_delay 10 and the leave switch at never, explicit Fork: branch finality-pause-node on 4c6b129d (vendor/igneum-node-finpause). Co-Authored-By: Claude Fable 5.1 --- docs/bench-log.md | 25 +++++++ docs/fud-ledger.md | 13 +++- docs/plans/finality-leave.md | 67 +++++++++++++++++ docs/spec/03-finality.md | 3 +- infra/fast-time/override-60x.json | 4 +- tools/finality-attacks/leave.mjs | 112 +++++++++++++++++++++++++++++ tools/finality-attacks/lib/net.mjs | 1 + 7 files changed, 222 insertions(+), 3 deletions(-) create mode 100644 docs/plans/finality-leave.md create mode 100644 tools/finality-attacks/leave.mjs diff --git a/docs/bench-log.md b/docs/bench-log.md index c37889aef..fd0df14a7 100644 --- a/docs/bench-log.md +++ b/docs/bench-log.md @@ -2597,3 +2597,28 @@ USD 20 an hour on community pods, against a devnet of 1.16 GH/s. Consequence: the devnet's hash is rentable for the price of a dinner, so nothing on it is a security result; the counter-ASIC and finality work is tested there for correctness, not for cost. The cost argument only starts at the TH/s scale, where the rental market's supply (not its price) is the limit, and that number belongs in the litepaper with this caveat. + +## 6 October 2026, 18:40Z to 20:41Z, the live devnet's finality pause (finality owner, worktree `finality-pause`, fork `finality-pause-node` on 4c6b129d) + +What happened, from the hub's own log (hub-1, RunPod 213.173.107.74, node bb43e9a8 up since 17:32:04Z, 54 peers; pulled read-only by the fleet agent), the two Mac hands' logs (local time +01:00) and the observer's rows (Horizon lane 3, `docs/analysis/horizon/finality-and-weight.md` 3.1): + +| Time (UTC) | Event | Source | +|---|---|---| +| 17:20 to 18:30 | 20 keys stop mining the live chain: 7 earlier leavers, then the 13 fleet keys the class v4 rehearsal job took at 18:27 to 18:30; together 3,026 of 7,083 blue blocks (42.7 percent) of the table frozen at the last lock | observer `live_checkpoints`, Horizon 3.1 | +| 18:30:28 | Both Mac hands stop (`igneumd has stopped...` at 19:30:28+01:00, SIGTERM); the wallet app's bundled node restarts 3 s later | `/tmp/igneum-devnet/node1.out`, `observer-v4.out`, `ps` | +| 18:30 to 18:39 | Locks 6824 to 6842 form WITHOUT the hands (83 of 93 signers, 78.5 to 79.7 percent of total); the hub builds 6836 itself under the any-node fallback (`certificate built for checkpoint 6836 ... by 73 of 88 voters ... aggregator 6e80f3ef`) | hub-1 log | +| 18:39:36.925 | Last lock: `certificate at index 6842 received: 71 of 91 voters, weight 4924 of 7078`, `LOCKED ... 78.8% of total, 78.7% of the table frozen at lock 6841`; folded 18:39:38.702 to 79 of 91 | hub-1, p1-4090 (18:39:37.115) | +| about 18:40 | 6843 determined (DAA 209,233); 75 of 93 voters' votes, 53.1 percent of total: under two thirds, no certificate anywhere | observer rows, every node's log from here shows only `determined` | +| 18:41:54 / 18:42:07 | The hands return, IBD from 192.168.68.67, receive and fold the certificates up to 6842, determine 6843 onward, lock nothing | the hands' logs | +| 18:45:57 / 18:46:39 | The hands are SIGTERMed again and relaunched under launchd (the observer's node panics once at `igneum/exec/src/rpc.rs:591` index out of bounds and is relaunched) | `~/Library/Logs/Igneum/*.out` | +| 19:14:53 | Checkpoint 6912: the stayers hold 74.9 percent of the sliding table and still no lock: they hold 57.3 percent of the table frozen at 6842 | observer rows | +| 19:52 to 20:1x | API `finality.active=false`, `latest_locked_index=6842`; the hub and p1-4090 log `determined` every 30 s and nothing else | `/api/stats`, fleet pull | +| 20:4x (expected) | The frozen table expires when a checkpoint block's DAA reaches 209,202 + 7,200 = 216,402 (sink rate 0.9905 DAA/s measured 18:30 to 20:01Z); first lock about index 7082 | arithmetic; the watcher's result is appended below | + +Cause (confirmed in the code, `consensus/src/processes/finality.rs` `evaluate` and `frozen_table`): rule v3 (Q5) needs the signers at two thirds of the table frozen at the last lock on the chain, and `frozen_table()` returns `None` only when the checkpoint block's DAA is a full window past the lock's. The 20 departed keys hold 42.7 percent of that table, so no signer set can reach two thirds of it until it expires, whatever the sliding table says. Nothing on the aggregation path is involved: the fallback (any node, 15 DAA after determination) exists and fired, and every node computes the same frozen table from the chain. + +Ruled out, with the line that rules it out: the hands as a star (locks 6824 to 6842 formed while both were down); vote routing through the VRF aggregators (the hub's own `certificate built` at 18:36:38Z under the fallback); votes dropped by the `level is 0 when it's expected to be at least 38` rejects (those are the Igneum Wallet app's bundled igneumd on this Mac, 2.121.2.18 public and 192.168.68.64 on the LAN, a 5 October build with no commit string at protocol version 12 and no params digest, dialling node 1 and the hub every 30 s, asking for block cd6666c9, failing to compute a class v3 block's level and dropping: 172 rejects on the hub since 18:30Z, all from that address, a block relay reply and not on any vote path; the fix is a 0.3.14 node in the wallet bundle); the 0.3.15 canary's version-1026 blocks (from 19:14Z, after the pause began). + +What "paused" meant per tier (and means on mainnet for the same event, a 43 percent departure in three minutes): a holder saw `finality_active` false for 2 hours (30 days on mainnet under v3, 7.7 days under v2) and lost nothing: blocks, execution, payouts and the hourly program continued and every lock before 6842 stood (the F20 guarantees, measured); a home miner or rig mined and was paid, its blocks counted; a prover's segment records kept being paid through the pause (`segment record 143654..143661 ... accepted` at 20:53 local); a pool that moves servers without a leave holds the whole network's finality for a window, and with one (0.3.16) only its own hour; a rollup customer's proofs continued but anything that waits for a certified checkpoint waited the whole pause, which on mainnet is a month and is why the departure announcement is a launch requirement. + +What was done: (1) the timeline and cause reported to main at 20:0xZ; (2) no operator action proposed: the frozen table is chain state, identical on every node, so no restart or re-dial can end the pause, only the departed keys returning (9.4 points of the frozen table would do) or the expiry; (3) the W7 departure announcement implemented on the fork (`docs/plans/finality-leave.md`): a signed leave item carried in blocks removes its key from the sliding and the frozen denominators `leave_delay` (3,600 DAA) after its first carrier until the window has passed, behind `finality_leave_activation_daa` and protocol version 17, with the miner sending it on a clean stop and `igneum-miner leave` by hand; unit test with the known-failed case (silent 40 percent pauses for the window) and the fixed case (lock within the delay); (4) ledger F27, the F19 correction (a silent 40 percent under v3 stalls the full window, not "day 19 to 20"), spec 03 W7. diff --git a/docs/fud-ledger.md b/docs/fud-ledger.md index 60fae6e91..5bc550722 100644 --- a/docs/fud-ledger.md +++ b/docs/fud-ledger.md @@ -1505,7 +1505,7 @@ Evidence: M1, O-1.17, spec 1.13 and 1.16, M16's arithmetic. Experiment: the scor ### F19. Old vote keys can be bought; fresh hashrate cannot buy weight "Weight is 30 days of blocks per key, and keys are free to make and free to sell. I do not rent hashrate for 20 days. I buy, borrow or steal the vote keys of pools that already mined those 20 days. Your simulation models the renter and never the buyer." -Status: Answered with evidence (`sim/results_v2.md` scenario K, 4 October 2026, at the 0.85 and the 2/3 floor, seeds 7 to 19): a bought key is worth the blocks it holds and nothing more. Keys worth 20% of the window plus 30% of hashrate never reach a third; keys worth 40% hold the veto from purchase until day 19 to 20 and are worth 30% on day 30, the same as fresh hashrate; 0 conflicting locks in every row. The cost at the 2/3 floor: a silent 40% buyer stalls 63,307 to 68,716 of 86,400 checkpoints in 30 days (305 to 1,085 at the old floor). The 40/40/20 row is scenario I (0 conflicts). O-3.15 was decided on 4 October 2026 (the 2/3 floor). Not modelled: a seller who keeps a copy of the key and equivocates; the price of a pool's key against F5's hashrate cost is not a simulator question. Was: Open, experiment scheduled (O-3.15). +Status: Answered with evidence, CORRECTED for rule v3 (6 October 2026, 21:1xZ, the finality owner after the live devnet's pause): the "day 19 to 20" figure below is the rule v2 arithmetic (the sliding table alone, `30 (1 - 1/(3x))` days for a silent share x) and is no longer what the live rule does. Under rule v3 (Q5, active since N3 = 135,200 on 5 October 2026) a silent 40 percent that holds the veto stalls the FULL weight window, 30 days on mainnet, because the table frozen at the last lock keeps the silent keys at their weight until that lock is one window old; measured on the devnet tonight: 20 keys holding 42.7 percent of the frozen table stopped at 18:27 to 18:30Z, the last lock was 6842 at 18:39:36Z, the stayers held 74.9 percent of the sliding table from 19:14Z and still no lock; the first lock came at the frozen table's expiry, DAA 216,402 (lock 6842's DAA 209,202 plus the 7,200-DAA window), about 20:41Z (`docs/analysis/horizon/finality-and-weight.md` 3.1 and 4.1; bench-log "6 October 2026, 18:40Z to 20:41Z, the live devnet's finality pause"). What ends it sooner: the silent keys returning and signing (0 minutes after, J and L1), or, once the W7 leave rule of 0.3.16 is active, a leave by the departing keys (`leave_delay`, one hour, after its first carrier); a key that goes silent WITHOUT leaving still holds the veto for the window, under v3 as under v2, which is the rule's price and is stated in 3.3.1. Was: Answered with evidence (`sim/results_v2.md` scenario K, 4 October 2026, at the 0.85 and the 2/3 floor, seeds 7 to 19): a bought key is worth the blocks it holds and nothing more. Keys worth 20% of the window plus 30% of hashrate never reach a third; keys worth 40% hold the veto from purchase until day 19 to 20 and are worth 30% on day 30, the same as fresh hashrate; 0 conflicting locks in every row. The cost at the 2/3 floor: a silent 40% buyer stalls 63,307 to 68,716 of 86,400 checkpoints in 30 days (305 to 1,085 at the old floor). The 40/40/20 row is scenario I (0 conflicts). O-3.15 was decided on 4 October 2026 (the 2/3 floor). Not modelled: a seller who keeps a copy of the key and equivocates; the price of a pool's key against F5's hashrate cost is not a simulator question. Was: Open, experiment scheduled (O-3.15). Answer: Correct, and the ledger had no entry for it. Spec 3.1 W6 says weight is the only Sybil-resistant quantity because it takes public mining to earn; it does not say what happens when an earned key changes hands. A compromised or sold key carries its whole 30-day history, so the 10-day and 20-day figures of F5 apply only to an attacker who mines; a buyer's day count is zero. What limits it today: W5 moves weight to a successor only by a message the old key signs (O-3.11), equivocation strips a key for 30 days (3.6), and pool keys are few and public (F10). What is missing: the acquisition cost of the top pools' keys against the hashrate cost of F5, whether weight should decay faster than 30 days when a key's blocks stop matching its earlier profile, and whether W5 should make the successor re-earn. Gate 3, in `finality_v2.py`: k% of weight changes hands at day 0 against the same k% arriving as fresh hashrate, for k in 20, 34 and 40, with the 40/40/20 partition row the reviewer asked for under the active-set rules and the floor, reporting time to a conflicting lock under each. @@ -1969,6 +1969,17 @@ Answer: Correct, measured. The red-team run's first scenario errored on it (`doc Evidence: the first run's `/tmp/igneum-redteam-fin/n0/node.log` parse line (kept in the session scratchpad `rt/logs/fa_s8/n0/node.log`), `rt/logs/ord_s2.log`. +### F27. Twenty keys stopped without a word and finality paused for two hours; on mainnet that is a month +"Your fleet operator moved 13 boxes to another chain at 18:30Z and your own devnet's finality stopped at 18:40Z and did not come back for two hours, with 93 voters online and the hub up. Under your frozen table a third of weight that goes quiet, a data centre fire, a hosting ban, holds every lock for 30 days. You called that a feature." + +Status: Fix built, pending rollout (6 October 2026, 21:1xZ, the finality owner; fork branch `finality-pause-node` on 4c6b129d, sized for 0.3.16: the W7 departure announcement of spec 03, `finality_leave_activation_daa`, a digest move). Incident: ledger-grade timeline in bench-log "6 October 2026, 18:40Z to 20:41Z, the live devnet's finality pause"; cause and candidates in `docs/analysis/horizon/finality-and-weight.md` (Horizon lane 3). Operational rule in force since 20:0x UK (CLAUDE.md, the standing fleet): a standing box never leaves the live chain for an experiment; any orchestrated departure over 10 percent of weight is staged in slices under 10 percent an hour. + +Answer: Correct on the facts and on the mainnet arithmetic; the pause was the rule doing what 3.7 item 2 says, and the frozen table of Q5 is why it lasted a window rather than the 35 minutes rule v2 would have taken (the stayers held 74.9 percent of the sliding table from 19:14Z). The topology hypotheses (the hands as a star, the aggregators, votes dropped by the old wallet node's rejects) were all refuted from the hub's own log: locks 6824 to 6842 formed while both Mac hands were down, the hub built a certificate itself at 18:36:38Z under the any-node fallback, and every node holds the same frozen table because it is computed from the chain. What distinguishes a departure from a partition is a message the departing key can sign and a partitioned key cannot: the leave (W7). With it, an orchestrated or clean departure of any size ends its pause one hour after the first block that carries the leave, with 0 conflicting locks in every partition, eclipse and equivocator row of the simulator (3 seeds, mainnet scale: `sim/horizon/finality-and-weight/results/`), and an attacker who buys keys to leave them gains nothing over signing with them (4.2). What it does not cover, stated: a crash, a power cut, a region going dark still age out over the window; the app's Stop and the fleet library send the leave, a node that returns after an unplanned stop sends it then, and staging under 10 percent an hour keeps finality on with no protocol change at all (every lock re-freezes the table). The known-failed case is the first assertion of the node's unit test `a_signed_leave_ends_the_pause_after_the_delay_and_a_silent_departure_holds_it_for_the_window`: a silent 40 percent pauses until the frozen table expires; with the leave, the first lock comes `leave_delay` after the carrier. + +Per tier, what "paused" meant tonight and means on mainnet: a holder saw `finality_active` false for 2 hours (30 days on mainnet for the same event) and lost nothing: blocks, execution, payouts and the hourly program continued, every lock before 6842 stood, exchanges fall back to the 12-hour finality depth of 3.9; a miner (home, rig) mined and was paid as before and its blocks counted toward weight; a prover's records were paid (segment records kept landing through the pause); a pool: one leave per server on maintenance ends its own share's pause in an hour, and a pool that moves servers without one holds the network for a window; a rollup customer: proofs continued, but a bridge that waits for a certified checkpoint waited the whole pause (two hours tonight, a month on mainnet without the leave), which is the number that makes W7 a launch requirement and not a nicety. + +Evidence: hub-1's log 18:30Z to 19:56Z (fleet pull, `scratchpad/finality-pause/hub-1.log`), `/tmp/igneum-devnet/node1.out` and `observer-v4.out` (local time +01:00), the observer's `live_checkpoints` rows, `consensus/src/processes/finality.rs` `frozen_table`. Experiment: the W7 harness case on the Devnet 2 gate of 0.3.16 (45 percent of weight stops with leaves, first lock within `leave_delay` plus one checkpoint, 0 conflicts in the 50/50 and 60/40 splits and the 34 percent eclipse). + ### X19. Operational knobs and silences in the shipped node "A slow-clock node disconnects every peer on every relayed block and never says why; the handshake's `time_offset` is computed and unused; `IGNEUM_ATTACK_TS_OFFSET_MS` and `IGNEUM_POW_STRIKES` are compiled into the live binary; `timestamp_deviation_tolerance` is dead and still accepted." diff --git a/docs/plans/finality-leave.md b/docs/plans/finality-leave.md new file mode 100644 index 000000000..4931932b7 --- /dev/null +++ b/docs/plans/finality-leave.md @@ -0,0 +1,67 @@ +# The departure announcement (spec 03 W7): plan for 0.3.16 + +Written 6 October 2026, 21:2xZ, by the finality owner, during the live devnet's finality pause (18:40Z to about 20:41Z). +Cause and candidates: `docs/analysis/horizon/finality-and-weight.md` (Horizon lane 3). Incident: `docs/bench-log.md` +"6 October 2026, 18:40Z to 20:41Z, the live devnet's finality pause"; ledger F27 and the F19 correction. Code: fork +branch `finality-pause-node` on 4c6b129d (release 0.3.14's node), worktree +`/Users/joshm/Projects/igneum-wt-finality-pause/vendor/igneum-node-finpause`. + +## 1. What changes + +| Piece | Where | What | +|---|---|---| +| The leave item | `consensus/core/src/finality.rs` | `Leave { daa, pubkey, signature }` (152 bytes) signed under `DST_LEAVE` over `"igneum-leave-v1/" || chain_id || 0 || daa_le`; `FinalityItem::Leave` tag 4; `FinalityGossip::Leave`; `FinalityParams::leave_delay` (serde default 3,600; mainnet and devnet 3,600) | +| The switch | `consensus/core/src/config/params.rs` | `finality_leave_activation_daa` on `Params` and `OverrideParams` (never on mainnet, devnet, simnet; 0 on the testnet, which is born on every switch), in the digest with `leave_delay` | +| The rule | `consensus/src/processes/finality.rs` | `leaves_at` (like `bans_at`: a key has left at C when a block in C's past carries its leave and `daa(e) + leave_delay <= daa(C) < daa(e) + window`); `voters_at` applies leaves with the bans (the sliding table); `frozen_floor` subtracts the keys that left at C from the frozen table's denominator and their signatures from its numerator (Q5); `ingest_leave` (one record per key, carriers dated), `submit_leave`, carriage in `template_section` LAST and only once the rule is active; persisted state layout 3 (`leaves`), layouts 2 and 1 read and upgraded | +| p2p | `protocol/flows/src/v10/finality.rs`, `flow_context.rs`, `v17/`, `service.rs` | `KIND_LEAVE = 3`; protocol version 17; `broadcast_finality_leave` to peers at 17 or later only (an older relay flow returns a protocol error on an unknown kind and drops the peer); every peer still reads leaves from blocks | +| RPC | `rpc/*` | `submitFinalityLeave` (hex of the wire bytes; `accepted`, `reason`), op 157, proto ids 1129 and 1130 | +| The miner | `igneum/miner/src/main.rs` | `mine` sends the leave for every identity when `secs` run out unless `--no-leave`; `igneum-miner leave