# Igneum devnet observer

Watches one Igneum node over wRPC JSON and writes what it sees to Neon, so `/api/live` and `/live` on the site can show the devnet as it runs. Node 22 or newer, no dependencies.

## Run

```
node tools/observer/observer.mjs
```

Environment, every value optional:

| Variable | Default | Meaning |
|---|---|---|
| `IGNEUM_RPC` | `ws://127.0.0.1:28610` | The node's wRPC JSON url. 28610 is the igneum-devnet wRPC JSON port (`consensus/core/src/network.rs`). Start the node with `--rpclisten-json=127.0.0.1:28610` or the port of your choice. |
| `DATABASE_URL` | read from `~/.config/igneum/env` | Neon connection string. Never commit it. |
| `LIVE_RETAIN_HOURS` | `24` | Hours of blocks kept in `live_blocks`. Older rows are deleted once a minute. |
| `LIVE_TABLE_PREFIX` | empty | Prefix for every table name, so a test observer against a test network can write `fintest_live_*` without touching the site. |
| `IGNEUM_EVM_RPC` | `http://127.0.0.1:26800` | The execution layer's JSON-RPC (http) of a node on the proving build, for `igneum_getShardPlan`, `igneum_getProofRecords` and `igneum_getProvingStatus`. On the Mac that port is the Igneum Miner app's node, so an app restart (an OTA at its slot minute, a quit) blips the proving feed with `fetch failed` lines until it is back; the observer's own node (`observer-v4`) has no `--evm-rpclisten`. The default is the Mac app's node; the observer node itself has no EVM listener yet (start it with `--evm-rpclisten=127.0.0.1:<port>` once it runs the proving build and point this at it). A node without the RPCs (method not found) gives `proving = {supported: false}`, rechecked every 5 minutes; an unreachable endpoint is retried every 20 s. |

A node started by another tool may listen on gRPC only. Then run your own non-mining peer with a JSON listener, on ports that do not clash with the devnet's (gRPC 26610, P2P 26611):

```
vendor/igneum-node/target/release/igneumd --devnet --nodnsseed --disable-upnp \
  --appdir=/tmp/igneum-obsnode --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 \
  --listen=127.0.0.1:26641 --connect=127.0.0.1:26611 --nologfiles
IGNEUM_RPC=ws://127.0.0.1:28640 node tools/observer/observer.mjs
```

## What it does

- Subscribes to `blockAdded` and `virtualChainChanged`; polls `getBlockDagInfo`, `getInfo`, `getConnectedPeerInfo`, `getSinkBlueScore` and `estimateNetworkHashesPerSecond` every 2 s.
- Decodes the miner address from the coinbase payload script (same bech32 variant as `crypto/addresses`). The fork's `vote_key_hash` header field is stored per block; the first 8 hex characters are the miner's short id on the site.
- `engine` is the miner's tag in the coinbase extra data after the node's version prefix. The node exposes no engine name over RPC, so this is null on devnet v0.
- When the node refuses the hash-rate estimate (it needs a 1,000-block window) the observer reports blue work added per second over the last 10 minutes instead.
- Proving v0 (4 Oct 2026, spec 7.7): every chain block (the `isChainBlock` flag of a new block, or a `virtualChainChanged` addition) has its shard plan read over `IGNEUM_EVM_RPC` as `{blockHash}` (the chain block hash is the same hash on both layers), one `live_proofs` row per shard in state `planned`; a plan the EVM node has not executed yet is retried with a growing delay (up to 30 tries). The proof records of the chain blocks of the last 10 minutes are polled in rotation (80 blocks per 2 s tick while active, 10 before activation, four calls in flight); a shard moves to `proving` (a record in the node's pool), `verified` (the SP1 proof verified by the node's verifier, or the record carried by a block and checked by consensus, which is what happens before activation) or `paid` (a carrying segment paid it). `lag_daa` is the carrier's DAA score minus the block's; `prover` is the first 8 hex characters of the record's vote key hash. A block whose every shard is paid, or older than 10 minutes, leaves the rotation. Events: `proving` (activation reached, first paid shard seen) and `prover_seen` (one per prover per run). A reorg drops the removed chain blocks from the rotation.
- Explorer (5 Oct 2026, `docs/plans/explorer.md`): every block row also carries what `/explorer`, `/block/<hash>` and `/address/<addr>` show, all read from the `blockAdded` notification itself (no extra RPC per block, measured: 282 against 283 wRPC calls per minute, 785 against 776 EVM calls in the same minute, before and after): `tx_count` (EVM transactions in the block), `evm_miner` (the coinbase's `IGNA` payout address, else the vote key hash's low 20 bytes as `consensus/core/src/evm.rs` falls back to), `proof_records` (records in the `IGNP` section, 274 bytes each), `subsidy_sompi` (the `E(daa)` the payload declares), `paid_sompi` (the coinbase outputs' sum), `selected_parent`, `number` (the chain block number, filled in when the shard plan arrives) and `detail` (header fields, mergeset, coinbase outputs, EVM transaction hashes, certificate indices). `live_state.rpc_load` counts this process's RPC calls per minute; `live_state.supply_check` is the hourly comparison of the newest 500 blocks with the emission rule (`site/lib/emission.mjs`, spec 2.5): the declared subsidy against `blockSubsidy(daa, 1)`, and each block's outputs against the declared subsidies of the blocks it merges. The observer now imports `site/lib/emission.mjs` and `site/lib/eth.mjs` (keccak for transaction hashes); autosync restarts only on `observer.mjs` and `run.sh` changes, so a change to those two libraries needs a restart by hand.
- Finality v2 (3 Oct 2026): subscribes to `FinalityLock` (the node's lock event) and polls `getFinalityCheckpoints` every 2 s and `getFinalityWeights` every 10 s. Every checkpoint the node reports is upserted into `live_checkpoints`; a checkpoint turning `locked` writes the event `checkpoint N locked (xx% of weight, yy% of active, v votes of n voters) at block h`. The weights snapshot (total, active, per key) goes into `live_state.finality`. A node from before the finality layer answers the RPC with an error; the observer then logs once and skips finality.

## Tables

Created on start if missing.

| Table | Rows | Columns |
|---|---|---|
| `live_blocks` | one per block, kept `LIVE_RETAIN_HOURS` | `hash`, `blue_score`, `daa_score`, `timestamp_ms`, `parents` (count), `parent_hashes`, `is_chain_block`, `vote_key_hash`, `miner_address`, `engine`, `received_at`, `color`; explorer: `tx_count`, `evm_miner`, `proof_records`, `subsidy_sompi`, `paid_sompi`, `selected_parent`, `number`, `detail` (jsonb). Indexes on `received_at`, `timestamp_ms`, `(vote_key_hash, received_at)`, `(evm_miner, received_at)`, `(miner_address, received_at)`, `number`. |
| `live_state` | one row, updated every 2 s | `block_count`, `header_count`, `blue_score`, `difficulty`, `hashes_per_second_estimate`, `peers`, `mempool`, `node_version`, `network`, `blocks_60s`, `blocks_per_minute` (60 pairs of minute epoch ms and count), `observer_started_at`, `updated_at` |
| `live_events` | one per event, kept 7 days | `ts`, `kind`, `text`. Kinds: `observer`, `miner_seen`, `miner_quiet`, `miner_back`, `peer_joined`, `peer_left`, `difficulty` (step over 5%), `checkpoint_locked`. |
| `live_checkpoints` | one per checkpoint index, kept 7 days | `index`, `hash`, `blue_score`, `daa_score`, `state` (proposed, certified, locked), `signed_weight`, `active_weight`, `total_weight`, `fraction_active`, `fraction_total`, `votes_seen`, `voters`, `aggregators` (key hashes whose sortition proof made them aggregators), `locked_at`, `first_seen_at`, `updated_at`. |
| `live_proofs` | one per planned shard of a chain block, kept `LIVE_RETAIN_HOURS` | `block_hash`, `shard`, `shards` (in the plan), `block_number`, `block_daa`, `block_ts`, `pgas`, `state` (planned, proving, verified, paid), `prover` (id8), `verified`, `carried_by`, `carrier_number`, `carrier_daa`, `lag_daa`, `payout_wei`, `received_at`, `updated_at`. Primary key `(block_hash, shard)`, index on `received_at`. |
| `live_state.proving` | jsonb, updated every 2 s | `supported` (false with `reason` when the node has no proving RPCs or the endpoint is unreachable), `active`, `activation_daa`, `tip_daa`, `verifier`, `pool` (entries, pending, verified, failed), `paid_shards_total`, `shard_budget_pgas`, `blocks_10m`, `blocks_fully_proven_10m`, `shards_proven_10m`, `shards_paid_10m`, `median_proof_lag_s` (median `lag_daa` of the last 10 minutes; the devnet targets one DAA step per second), `provers_10m`, `open_blocks`, `pending_plans`, `evm_rpc`. |
| `live_state.vendor_share` | jsonb, updated every 60 s (`tools/observer/vendor-share.mjs`, Counter ASIC 3.0 item 7) | `at`, `window_s` (600), `network_hps`, `fleet_reported` (`by_vendor` {nvidia, amd, apple, intel, unknown} with `mhs`, `workers`, `share`; `total_mhs`, `workers`, `machines`, `rows` per worker with label, machine id8, card, mhs, ages), `chain_attributed` (`by_vendor` with `blue_blocks`, `miners`, `share`, `mhs`; `blue_blocks_total`, `mapped_keys`, `unknown_miner_ids`), `by_vendor` (the two readings side by side), `coverage` (1 minus the unknown share). Fleet-reported = the sum of `now=MH/s wall` from each intake-reporting worker's newest STATUS line in the window (our fleet only); chain-attributed = each vote key's share of the window's blue blocks times the network estimate, vendor from the worker that logged the key, else unknown. `docs/benchmarks/repro.md` section 8. |
| `live_state.finality` | jsonb, updated every 2 s | `params`, `chain_id`, `next_index`, `finality_active`, `latest_locked_index`, `latest_locked_hash`, `latest_locked_blue_score`, `weights` (`total_weight`, `active_weight`, `voters`, `keys[]` with `id`, `blocks`, `voter`, `participation`, `stripped_until_daa`, `revealed`). |

## Keeping it current (autosync)

`tools/observer/autosync.sh` (loop: `nohup bash tools/observer/autosync.sh &` from the shared checkout; log `/tmp/igneum-devnet/autosync.out`) fetches origin/master every 5 min and fast-forwards the shared checkout when it is clean under `tools/observer` and `site/api`; a failed fast-forward logs git's reason and the dirty files the incoming commits also touch. Whoever moved HEAD (this loop or a pull by hand), the observer is restarted (a `kill`; `run.sh` starts it again in 3 s) whenever the checked-out `observer.mjs` or `run.sh` differs from what the running one started from: the key is the two blob ids hashed, kept in `/tmp/igneum-devnet/observer.tree` and rewritten at each restart. `tools/observer/autosync.sh check` prints the key, the marker and whether a restart is due (exit 3 when it is). A change to `autosync.sh` itself does not restart the observer, but the running loop keeps its old code: replace it by hand (kill it, start it again) after such a change; write the marker first (`git rev-parse HEAD:tools/observer/observer.mjs HEAD:tools/observer/run.sh | tr -d '\n' | shasum | cut -c1-40 > /tmp/igneum-devnet/observer.tree`) when the running observer is already current, or the new loop restarts it once for nothing. Each restart logs `seeded N checkpoint states` and, if a lock is recorded over a seeded state, names that state.

## Reading it

`site/api/stats.mjs`, `site/api/supply.mjs` and `site/api/explorer.mjs` serve `/api/stats`, `/api/supply` and `/api/explorer` (docs/api/public-stats.md). `site/api/live.mjs` serves `/api/live` from these tables in five indexed queries (`proving` from `live_state.proving`; every block carries `shards: [{i, n, state, prover, lag, payout, pgas}]` and `proven`). `LIVE_TABLE_PREFIX` on the API reads a test observer's tables. `site/live.html` polls it every 2 s. The site shows OFFLINE when `live_state.updated_at` is older than 30 s.

## Detector (Counter ASIC 3.0 item 4a, 6 October 2026)

`tools/observer/detector.mjs`, hooked into `observer.mjs` with one `setInterval` (every 60 s) and one column, `live_state.detector` (jsonb, the state below). Events go to `live_events` as kind `detector`. The question it answers: does a group of miner ids behave like one fixed design (MoneroCrusher's method, `docs/analysis/asic-resistance-history.md` section 4.3 addition 4)? The chain does nothing with the answer; the answer is the trigger for the epoch-length signal (`docs/plans/epoch-length.md` section 11). Tests: `node --test tools/observer/detector.test.mjs` (a fabricated honest population stays quiet, a fabricated fixed design under three ids alerts after the hold). Dry run against the live tables, read-only: `node tools/observer/detector.mjs --dry` (`--json` for the state).

### What it reads

| Input | Where | Note |
|---|---|---|
| Miner id | `live_blocks.vote_key_hash`, first 8 hex | A card mines under 1, 2 or 8 vote keys (`app/igneum-app/src/detect.rs`: 8 on a card with 8 GB or more, 2 on a smaller one, 1 on an iGPU or a Mac), so one machine is several ids; the devnet's 4 machines are 20 to 27 ids. The detector never assumes an id is a machine |
| Program | the epoch, `floor(daa_score / 3600)` (spec 01 section 1.12; `DETECTOR_EPOCH_LEN` follows a signalled change) | one program per epoch; the program id itself is not in the header |
| Work per block | `detail.bits` through `calc_work` (`vendor/igneum-node/consensus/src/processes/difficulty.rs`), as a double | a miner's implied rate in an epoch = its blue blocks' summed work / the epoch's wall seconds, the chain's own `estimate_network_hashes_per_second` rule restricted to one id |
| Nonce | `detail.nonce` (u64, kept as a string) | low and high 4 bits, and whether consecutive nonces of an id increase |
| Card models and rates | `miner_logs`: the app's `GPUs:` line (any upload of the last 7 days) and the workers' `STATUS ... now=<x> MH/s wall` lines (last 6 hours), extracted server-side | the app's own machines only; a stranger's card is not in the intake (owed: the coinbase tag could carry the model from 0.3.12) |

An epoch is closed when the tip is past its end and settled when the tip is 1,200 DAA past it (colours final); settled epochs are read once and cached, so a run reads at most two epochs (under 7,500 rows) plus one `UPDATE`. Start-up reads the window once (about 25,000 rows in 5,000-row pages).

### The statistics, per id over the window (`DETECTOR_WINDOW_EPOCHS`, default 6 closed epochs)

| Statistic | Rule | Flag | What it catches, what it misses |
|---|---|---|---|
| Per-program spread | sd of log implied rate over the epochs the id is present in (30 or more blue blocks), minus the Poisson part (`sqrt(mean 1/blue)`) in quadrature = the excess spread; only for a steady id (no step over 1.65x between consecutive epochs: a step is the machine doing something else, not the program) | `spread` when the excess is over 10% over 4 or more epochs | a design whose cost follows the program (a hard-datapath FPGA, a compute-bound sequencer). Misses the on-die recompute chip: its cost is the item derivation, the same for every program (`chip-model-v3.md`), so its spread is a GPU's |
| Epoch-start share | the id's blue blocks in the first tenth of each epoch (by DAA) over its blocks in the window | `late_start` under 2% with 200 or more blocks (binomial p about 1e-6 at the honest 10%) | a design compiled per program (epoch-length.md section 5.1: 42 to 160 min per bitstream, so it mines nothing in the first minutes). Misses every design that executes the program |
| Nonce pattern | chi-square of the low 4 bits and of the high 4 bits against uniform (15 degrees of freedom), and the fraction of increasing consecutive nonces | `nonce` when either chi-square exceeds 37.70 (p = 0.001) or the increasing fraction leaves [0.35, 0.65] with 200 or more pairs; 32 nonces minimum | a counter from 0, a per-core stride, a design that fixes the high word. Misses a design that draws random starts, as the app does |
| Card band | the window-median implied rate against every band / d for d in {1, 2, 8} (the identity counts), within 30% | `band_high` above every band by 30% (one id faster than any known card: a pool key or a design); `band` when steady and outside every band / d | a pool key trips `band_high` and is honest; the flag is evidence, never the alert |
| Correlation | two-way residuals (log rate minus the id's mean minus the epoch's common factor over the ids present, clipped at +-30%), Pearson over the epochs two ids share (5 or more); an edge at r over 0.8; maximal cliques of 3 or more ids | a clique whose members carry a design flag = `design_candidate`; a clique without = `machine_group` (one card's identities, one operator's machines: honest) | k or more ids moving as one and leaking a signature. Misses a design that is latency-bound like a GPU, draws random nonces and mines from the first second of each epoch: that design is invisible here, which is why the plan calls the detector a response-time tool and not a layer |

The alert: a `design_candidate` clique that holds for `DETECTOR_HOLD` net windows (default 6: one count up per newly closed epoch with a candidate, one down without), so at the base epoch about 11 hours, at the 600-s floor about 110 minutes. Event texts: `Detector: miner <id> flagged <flag> (<evidence>)`, `Detector: <n> ids move as one machine with design flags ...; held h of M windows`, `Detector ALERT: ...`, `Detector: the alert cleared`. Thresholds live in `DEFAULTS` (`DETECTOR_R`, `DETECTOR_K`, `DETECTOR_HOLD`, `DETECTOR_WINDOW_EPOCHS`, `DETECTOR_EPOCH_LEN` override them).

### The devnet's honest baseline (read-only dry run, 6 October 2026, 07:5x UTC, window epochs 40 to 45, tip DAA 168,422; the Mac's load average at the run was 3.6 to 4.9 (`uptime`, 08:46 UTC), which does not touch these chain-side numbers in any case)

Epochs 39 to 45 are the PC 1 outage (the Ember Tune quit, 22:31 UTC on 5 October), so the steady window holds 4 ids: PC 2's RTX 5090 under 2 keys, the M5 Max under 1, the Intel UHD laptop under 1. Network implied rate 125.4 to 130.6 MH/s per epoch against the cards' own STATUS sum of 143.9 (0.89x: reds, pending blocks and template latency are not in the chain figure).

| id | card | blue blocks | implied MH/s (median) | spread sd | Poisson sd | excess | first-tenth share | nonce chi2 low / high (crit 37.70) | increasing | band match |
|---|---|---|---|---|---|---|---|---|---|---|
| 9915d263 | 5090 (PC 2), 1 of 2 keys | 8,247 | 49.99 | 2.3% | 2.7% | 0% | 10.7% | 23.4 / 22.0 | 0.501 | 5090/2 |
| 00cec3ae | 5090 (PC 2), 2 of 2 | 8,245 | 50.11 | 3.9% | 2.7% | 2.8% | 9.3% | 13.7 / 9.4 | 0.501 | 5090/2 |
| 8fafda27 | M5 Max | 4,133 | 25.35 | 6.5% | 3.8% | 5.2% | 9.8% | 16.9 / 10.8 | 0.496 | M5 Max/1 |
| 4c022439 | Intel UHD | 265 | 1.59 | 13.4% | 15.2% | 0% | 9.1% | 20.2 / 26.0 | 0.512 | Intel UHD/1 |

Correlation: 6 pairs, max r 0.53 with the mean factor of the first commit (0.10 to 0.53 with the median core factor of the second), no edge at 0.8, no group, no flag, no alert. Over all 28 ids with 32 or more nonces in the 24-hour table (40,550 blocks with `detail`): chi-square maximum 26.8 (n = 38) and 26.0 (n = 793), KS maximum 0.232 at n = 42 (p = 0.01 critical 0.251), increasing fraction 0.438 to 0.520; `nonce mod 32` over all blocks chi-square 22.1 at 31 degrees of freedom. So the honest nonce is uniform over the full 64 bits: the serve protocol hands each job a 32-aligned 64-bit `nonce_start` (`proto-cuda/nvrtc/worker.cpp` lines 17 and 1175) and the kernel adds the lane index; the start is drawn at random per job. The 3-epoch window 36 to 38 (22 ids, PC 1 ramping down) had 4 of 28 pairs over r = 0.9: with 3 points a correlation is noise, which is why 5 shared epochs are the minimum. Card bands from the intake (6 hours of STATUS lines, p5 / p50 / p95): 5090 112.6 / 114.8 / 121.7 MH/s (n 781; the bench's 136 to 137 is the card to itself, the app's live rate shares it with the node and the prover); M5 Max 26.5 / 27.3 / 28.5 (674); Intel UHD 1.7 / 1.8 / 2.0 (754); from the 30-hour sample while PC 1 ran: RX 9070 XT 16.8 / 17.0 / 19.1 (355), gfx1036 2.6 / 2.8 / 3.4, M4 Max laptop 8.4 / 19.8 / 21.9 (478).

The honest population, in one line: excess per-program spread 0 to 5.2% on three card models (the census's 0.8 to 3.2% six-era spread plus the Mac's own load), first-tenth share 9.1 to 10.7%, nonces uniform, pairwise residual correlation under 0.55.

### Calibration on the merged tree, 09:2x UTC (window epochs 41 to 46, PC 1 back from 07:2x UTC, 20 ids, 4 correlated)

Three readings the coordinator raised, reproduced read-only and answered:

| Reading | Cause | Rule |
|---|---|---|
| The two PC 2 keys 00cec3ae and 9915d263 at r 0.94 (one edge, no group, held 0) | not the rate rule: an id's rate is its blue blocks' summed `calc_work(bits)` over the epoch's wall seconds, never a share of a network estimate, so nothing divides two ids by the same moving number. It was the epoch common factor: the mean over every id present let the Mac (paused and resumed, a 178% step) push every other residual the same way in the epochs it was off, and the two 5090 keys, each at Poisson sd 2.7%, moved together with it. The factor is now the median over the core ids (steady, present in every window epoch; the median over all present when the core is under 3): the same window reads max r 0.10, no edge. A residue of real co-movement stays between two keys of one card (PC 2's 5090 is shared with the prover, so both keys dip together), which is the `machine_group` case and needs k = 3 ids; two ids never form a group | changed in `analyse` (commit 2 on ca3-detector); the false-positive statement below stands, and the devnet's max pairwise r is now 0.10 to 0.53 across the two windows run |
| Each 5090 key at about 50 MH/s against the fleet band 99.4 / 115.6 / 123.1 | PC 2's worker runs the card under 2 vote keys (`STATUS ... identities=2 accepted_by_identity=8044/8009`), so 2 x 50.4 = 100.8 MH/s on chain against the card's own 115.6 median: 0.87x, the same ratio the network shows (125 to 131 MH/s on chain against 143.9 of STATUS: reds, pending blocks and template latency are not chain work). Not the hours off (every epoch of the window has PC 2 on) and not a share against total blocks (the rate is work, not a share) | the `/2` is the app's identity count (`detect.rs`: 1, 2 or 8); the band test is calibrated to the app and an unknown miner under another key count, or a pool, reads `band` or `band_high`, which are evidence only and never the alert by themselves. A testnet miner's key count belongs in the coinbase tag beside its card model (owed, 0.3.12) |
| The Mac 8fafda27 unsteady (spread 70%, step 178%) | the Mac was paused and resumed inside the window | by design `unsteady` only withholds the `spread` flag (a pause is not a program); the id stays in the correlation set, because a pause shared by several ids is exactly what identifies one machine's keys (`machine_group`), and it cannot become a `design_candidate` without a design flag. What changed: an unsteady id no longer enters the epoch common factor (above), so its pause cannot correlate other ids with each other |

### False positives, and what the public testnet's first week must add (history check 2)

Under the null (independent residuals, n = 6 epochs) one pair reads r over 0.8 with p about 0.028 (t = 2.67 on 4 degrees of freedom) and over 0.9 with p about 0.007; chance 3-cliques per window are about C(m, 3) p^3: 0.09 at m = 30 ids, about 100 at m = 300. So the clique alone is never the alert; the candidate also needs a design flag on a member (chance per id: `nonce` about 0.002, `late_start` about 1e-6, `spread` unknown until the per-model baseline exists; the devnet's maximum excess is 5.2% against the 10% line) and 6 net windows. At n = 12 (`DETECTOR_WINDOW_EPOCHS=12`) p(r over 0.8) is about 0.001 and chance 3-cliques at m = 300 are about 0.005 per window: set 12 once the testnet has over 100 ids. The first week must add: (1) the excess spread per card model over 12 or more epochs (the devnet has three models over 6); (2) the nonce layout of third-party miners and pools (a stratum extranonce in the high word is honest and non-uniform, so until each software is baselined a `nonce` flag is evidence, not an alert); (3) bands for cards the project does not own, which needs the card model in the coinbase tag; (4) the `machine_group` count, to see what an 8-key card looks like at scale.

Consequences per tier: the detector costs a miner nothing (it runs on the observer; one `UPDATE` a minute, two epochs re-read a minute, two regexp scans of the intake every 30 minutes). A home card under 8 keys appears as a `machine_group`, never an alert by itself. A pool's key trips `band_high`, which is informational; a pool that publishes its card mix clears it. The alert's action is the signal of epoch-length.md section 11, whose cost per tier is that document's section 7.