igneum/tools/observer

Igneum devnet observer

Watches one Igneum node over wRPC JSON and writes what it sees to Neon, so /api/live and /live on the site can show the devnet as it runs. Node 22 or newer, no dependencies.

Run

node tools/observer/observer.mjs

Environment, every value optional:

Variable	Default	Meaning
IGNEUM_RPC	ws://127.0.0.1:28610	The node's wRPC JSON url. 28610 is the igneum-devnet wRPC JSON port (consensus/core/src/network.rs). Start the node with --rpclisten-json=127.0.0.1:28610 or the port of your choice.
DATABASE_URL	read from ~/.config/igneum/env	Neon connection string. Never commit it.
LIVE_RETAIN_HOURS	24	Hours of blocks kept in live_blocks. Older rows are deleted once a minute.
LIVE_TABLE_PREFIX	empty	Prefix for every table name, so a test observer against a test network can write fintest_live_* without touching the site.
IGNEUM_EVM_RPC	http://127.0.0.1:26800	The execution layer's JSON-RPC (http) of a node on the proving build, for igneum_getShardPlan, igneum_getProofRecords and igneum_getProvingStatus. On the Mac that port is the Igneum Miner app's node, so an app restart (an OTA at its slot minute, a quit) blips the proving feed with fetch failed lines until it is back; the observer's own node (observer-v4) has no --evm-rpclisten. The default is the Mac app's node; the observer node itself has no EVM listener yet (start it with --evm-rpclisten=127.0.0.1:<port> once it runs the proving build and point this at it). A node without the RPCs (method not found) gives proving = {supported: false}, rechecked every 5 minutes; an unreachable endpoint is retried every 20 s.

A node started by another tool may listen on gRPC only. Then run your own non-mining peer with a JSON listener, on ports that do not clash with the devnet's (gRPC 26610, P2P 26611):

vendor/igneum-node/target/release/igneumd --devnet --nodnsseed --disable-upnp \
  --appdir=/tmp/igneum-obsnode --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 \
  --listen=127.0.0.1:26641 --connect=127.0.0.1:26611 --nologfiles
IGNEUM_RPC=ws://127.0.0.1:28640 node tools/observer/observer.mjs

What it does

• Subscribes to blockAdded and virtualChainChanged; polls getBlockDagInfo, getInfo, getConnectedPeerInfo, getSinkBlueScore and estimateNetworkHashesPerSecond every 2 s.
• Decodes the miner address from the coinbase payload script (same bech32 variant as crypto/addresses). The fork's vote_key_hash header field is stored per block; the first 8 hex characters are the miner's short id on the site.
• engine is the miner's tag in the coinbase extra data after the node's version prefix. The node exposes no engine name over RPC, so this is null on devnet v0.
• When the node refuses the hash-rate estimate (it needs a 1,000-block window) the observer reports blue work added per second over the last 10 minutes instead.
• Proving v0 (4 Oct 2026, spec 7.7): every chain block (the isChainBlock flag of a new block, or a virtualChainChanged addition) has its shard plan read over IGNEUM_EVM_RPC as {blockHash} (the chain block hash is the same hash on both layers), one live_proofs row per shard in state planned; a plan the EVM node has not executed yet is retried with a growing delay (up to 30 tries). The proof records of the chain blocks of the last 10 minutes are polled in rotation (80 blocks per 2 s tick while active, 10 before activation, four calls in flight); a shard moves to proving (a record in the node's pool), verified (the SP1 proof verified by the node's verifier, or the record carried by a block and checked by consensus, which is what happens before activation) or paid (a carrying segment paid it). lag_daa is the carrier's DAA score minus the block's; prover is the first 8 hex characters of the record's vote key hash. A block whose every shard is paid, or older than 10 minutes, leaves the rotation. Events: proving (activation reached, first paid shard seen) and prover_seen (one per prover per run). A reorg drops the removed chain blocks from the rotation.
• Explorer (5 Oct 2026, docs/plans/explorer.md): every block row also carries what /explorer, /block/<hash> and /address/<addr> show, all read from the blockAdded notification itself (no extra RPC per block, measured: 282 against 283 wRPC calls per minute, 785 against 776 EVM calls in the same minute, before and after): tx_count (EVM transactions in the block), evm_miner (the coinbase's IGNA payout address, else the vote key hash's low 20 bytes as consensus/core/src/evm.rs falls back to), proof_records (records in the IGNP section, 274 bytes each), subsidy_sompi (the E(daa) the payload declares), paid_sompi (the coinbase outputs' sum), selected_parent, number (the chain block number, filled in when the shard plan arrives) and detail (header fields, mergeset, coinbase outputs, EVM transaction hashes, certificate indices). live_state.rpc_load counts this process's RPC calls per minute; live_state.supply_check is the hourly comparison of the newest 500 blocks with the emission rule (site/lib/emission.mjs, spec 2.5): the declared subsidy against blockSubsidy(daa, 1), and each block's outputs against the declared subsidies of the blocks it merges. The observer now imports site/lib/emission.mjs and site/lib/eth.mjs (keccak for transaction hashes); autosync restarts only on observer.mjs and run.sh changes, so a change to those two libraries needs a restart by hand.
• Finality v2 (3 Oct 2026): subscribes to FinalityLock (the node's lock event) and polls getFinalityCheckpoints every 2 s and getFinalityWeights every 10 s. Every checkpoint the node reports is upserted into live_checkpoints; a checkpoint turning locked writes the event checkpoint N locked (xx% of weight, yy% of active, v votes of n voters) at block h. The weights snapshot (total, active, per key) goes into live_state.finality. A node from before the finality layer answers the RPC with an error; the observer then logs once and skips finality.

Tables

Created on start if missing.

Table	Rows	Columns
live_blocks	one per block, kept LIVE_RETAIN_HOURS	hash, blue_score, daa_score, timestamp_ms, parents (count), parent_hashes, is_chain_block, vote_key_hash, miner_address, engine, received_at, color; explorer: tx_count, evm_miner, proof_records, subsidy_sompi, paid_sompi, selected_parent, number, detail (jsonb). Indexes on received_at, timestamp_ms, (vote_key_hash, received_at), (evm_miner, received_at), (miner_address, received_at), number.
live_state	one row, updated every 2 s	block_count, header_count, blue_score, difficulty, hashes_per_second_estimate, peers, mempool, node_version, network, blocks_60s, blocks_per_minute (60 pairs of minute epoch ms and count), observer_started_at, updated_at
live_events	one per event, kept 7 days	ts, kind, text. Kinds: observer, miner_seen, miner_quiet, miner_back, peer_joined, peer_left, difficulty (step over 5%), checkpoint_locked.
live_checkpoints	one per checkpoint index, kept 7 days	index, hash, blue_score, daa_score, state (proposed, certified, locked), signed_weight, active_weight, total_weight, fraction_active, fraction_total, votes_seen, voters, aggregators (key hashes whose sortition proof made them aggregators), locked_at, first_seen_at, updated_at.
live_proofs	one per planned shard of a chain block, kept LIVE_RETAIN_HOURS	block_hash, shard, shards (in the plan), block_number, block_daa, block_ts, pgas, state (planned, proving, verified, paid), prover (id8), verified, carried_by, carrier_number, carrier_daa, lag_daa, payout_wei, received_at, updated_at. Primary key (block_hash, shard), index on received_at.
live_state.proving	jsonb, updated every 2 s	supported (false with reason when the node has no proving RPCs or the endpoint is unreachable), active, activation_daa, tip_daa, verifier, pool (entries, pending, verified, failed), paid_shards_total, shard_budget_pgas, blocks_10m, blocks_fully_proven_10m, shards_proven_10m, shards_paid_10m, median_proof_lag_s (median lag_daa of the last 10 minutes; the devnet targets one DAA step per second), provers_10m, open_blocks, pending_plans, evm_rpc.
live_state.vendor_share	jsonb, updated every 60 s (tools/observer/vendor-share.mjs, Counter ASIC 3.0 item 7)	at, window_s (600), network_hps, fleet_reported (by_vendor {nvidia, amd, apple, intel, unknown} with mhs, workers, share; total_mhs, workers, machines, rows per worker with label, machine id8, card, mhs, ages), chain_attributed (by_vendor with blue_blocks, miners, share, mhs; blue_blocks_total, mapped_keys, unknown_miner_ids), by_vendor (the two readings side by side), coverage (1 minus the unknown share). Fleet-reported = the sum of now=MH/s wall from each intake-reporting worker's newest STATUS line in the window (our fleet only); chain-attributed = each vote key's share of the window's blue blocks times the network estimate, vendor from the worker that logged the key, else unknown. docs/benchmarks/repro.md section 8.
live_state.finality	jsonb, updated every 2 s	params, chain_id, next_index, finality_active, latest_locked_index, latest_locked_hash, latest_locked_blue_score, weights (total_weight, active_weight, voters, keys[] with id, blocks, voter, participation, stripped_until_daa, revealed).

Keeping it current (autosync)

tools/observer/autosync.sh (loop: nohup bash tools/observer/autosync.sh & from the shared checkout; log /tmp/igneum-devnet/autosync.out) fetches origin/master every 5 min and fast-forwards the shared checkout when it is clean under tools/observer and site/api; a failed fast-forward logs git's reason and the dirty files the incoming commits also touch. Whoever moved HEAD (this loop or a pull by hand), the observer is restarted (a kill; run.sh starts it again in 3 s) whenever the checked-out observer.mjs or run.sh differs from what the running one started from: the key is the two blob ids hashed, kept in /tmp/igneum-devnet/observer.tree and rewritten at each restart. tools/observer/autosync.sh check prints the key, the marker and whether a restart is due (exit 3 when it is). A change to autosync.sh itself does not restart the observer, but the running loop keeps its old code: replace it by hand (kill it, start it again) after such a change; write the marker first (git rev-parse HEAD:tools/observer/observer.mjs HEAD:tools/observer/run.sh | tr -d '\n' | shasum | cut -c1-40 > /tmp/igneum-devnet/observer.tree) when the running observer is already current, or the new loop restarts it once for nothing. Each restart logs seeded N checkpoint states and, if a lock is recorded over a seeded state, names that state.

Reading it

site/api/stats.mjs, site/api/supply.mjs and site/api/explorer.mjs serve /api/stats, /api/supply and /api/explorer (docs/api/public-stats.md). site/api/live.mjs serves /api/live from these tables in five indexed queries (proving from live_state.proving; every block carries shards: [{i, n, state, prover, lag, payout, pgas}] and proven). LIVE_TABLE_PREFIX on the API reads a test observer's tables. site/live.html polls it every 2 s. The site shows OFFLINE when live_state.updated_at is older than 30 s.

Detector (Counter ASIC 3.0 item 4a, 6 October 2026)

tools/observer/detector.mjs, hooked into observer.mjs with one setInterval (every 60 s) and one column, live_state.detector (jsonb, the state below). Events go to live_events as kind detector. The question it answers: does a group of miner ids behave like one fixed design (MoneroCrusher's method, docs/analysis/asic-resistance-history.md section 4.3 addition 4)? The chain does nothing with the answer; the answer is the trigger for the epoch-length signal (docs/plans/epoch-length.md section 11). Tests: node --test tools/observer/detector.test.mjs (a fabricated honest population stays quiet, a fabricated fixed design under three ids alerts after the hold). Dry run against the live tables, read-only: node tools/observer/detector.mjs --dry (--json for the state).

What it reads

Input	Where	Note
Miner id	live_blocks.vote_key_hash, first 8 hex	A card mines under 1, 2 or 8 vote keys (app/igneum-app/src/detect.rs: 8 on a card with 8 GB or more, 2 on a smaller one, 1 on an iGPU or a Mac), so one machine is several ids; the devnet's 4 machines are 20 to 27 ids. The detector never assumes an id is a machine
Program	the epoch, floor(daa_score / 3600) (spec 01 section 1.12; DETECTOR_EPOCH_LEN follows a signalled change)	one program per epoch; the program id itself is not in the header
Work per block	detail.bits through calc_work (vendor/igneum-node/consensus/src/processes/difficulty.rs), as a double	a miner's implied rate in an epoch = its blue blocks' summed work / the epoch's wall seconds, the chain's own estimate_network_hashes_per_second rule restricted to one id
Nonce	detail.nonce (u64, kept as a string)	low and high 4 bits, and whether consecutive nonces of an id increase
Card models and rates	miner_logs: the app's GPUs: line (any upload of the last 7 days) and the workers' STATUS ... now=<x> MH/s wall lines (last 6 hours), extracted server-side	the app's own machines only; a stranger's card is not in the intake (owed: the coinbase tag could carry the model from 0.3.12)

An epoch is closed when the tip is past its end and settled when the tip is 1,200 DAA past it (colours final); settled epochs are read once and cached, so a run reads at most two epochs (under 7,500 rows) plus one UPDATE. Start-up reads the window once (about 25,000 rows in 5,000-row pages).

The statistics, per id over the window (DETECTOR_WINDOW_EPOCHS, default 6 closed epochs)

Statistic	Rule	Flag	What it catches, what it misses
Per-program spread	sd of log implied rate over the epochs the id is present in (30 or more blue blocks), minus the Poisson part (sqrt(mean 1/blue)) in quadrature = the excess spread; only for a steady id (no step over 1.65x between consecutive epochs: a step is the machine doing something else, not the program)	spread when the excess is over 10% over 4 or more epochs	a design whose cost follows the program (a hard-datapath FPGA, a compute-bound sequencer). Misses the on-die recompute chip: its cost is the item derivation, the same for every program (chip-model-v3.md), so its spread is a GPU's
Epoch-start share	the id's blue blocks in the first tenth of each epoch (by DAA) over its blocks in the window	late_start under 2% with 200 or more blocks (binomial p about 1e-6 at the honest 10%)	a design compiled per program (epoch-length.md section 5.1: 42 to 160 min per bitstream, so it mines nothing in the first minutes). Misses every design that executes the program
Nonce pattern	chi-square of the low 4 bits and of the high 4 bits against uniform (15 degrees of freedom), and the fraction of increasing consecutive nonces	nonce when either chi-square exceeds 37.70 (p = 0.001) or the increasing fraction leaves [0.35, 0.65] with 200 or more pairs; 32 nonces minimum	a counter from 0, a per-core stride, a design that fixes the high word. Misses a design that draws random starts, as the app does
Card band	the window-median implied rate against every band / d for d in {1, 2, 8} (the identity counts), within 30%	band_high above every band by 30% (one id faster than any known card: a pool key or a design); band when steady and outside every band / d	a pool key trips band_high and is honest; the flag is evidence, never the alert
Correlation	two-way residuals (log rate minus the id's mean minus the epoch's common factor over the ids present, clipped at +-30%), Pearson over the epochs two ids share (5 or more); an edge at r over 0.8; maximal cliques of 3 or more ids	a clique whose members carry a design flag = design_candidate; a clique without = machine_group (one card's identities, one operator's machines: honest)	k or more ids moving as one and leaking a signature. Misses a design that is latency-bound like a GPU, draws random nonces and mines from the first second of each epoch: that design is invisible here, which is why the plan calls the detector a response-time tool and not a layer

The alert: a design_candidate clique that holds for DETECTOR_HOLD net windows (default 6: one count up per newly closed epoch with a candidate, one down without), so at the base epoch about 11 hours, at the 600-s floor about 110 minutes. Event texts: Detector: miner <id> flagged <flag> (<evidence>), Detector: <n> ids move as one machine with design flags ...; held h of M windows, Detector ALERT: ..., Detector: the alert cleared. Thresholds live in DEFAULTS (DETECTOR_R, DETECTOR_K, DETECTOR_HOLD, DETECTOR_WINDOW_EPOCHS, DETECTOR_EPOCH_LEN override them).

The devnet's honest baseline (read-only dry run, 6 October 2026, 07:5x UTC, window epochs 40 to 45, tip DAA 168,422; the Mac's load average at the run was 3.6 to 4.9 (uptime, 08:46 UTC), which does not touch these chain-side numbers in any case)

Epochs 39 to 45 are the PC 1 outage (the Ember Tune quit, 22:31 UTC on 5 October), so the steady window holds 4 ids: PC 2's RTX 5090 under 2 keys, the M5 Max under 1, the Intel UHD laptop under 1. Network implied rate 125.4 to 130.6 MH/s per epoch against the cards' own STATUS sum of 143.9 (0.89x: reds, pending blocks and template latency are not in the chain figure).

id	card	blue blocks	implied MH/s (median)	spread sd	Poisson sd	excess	first-tenth share	nonce chi2 low / high (crit 37.70)	increasing	band match
9915d263	5090 (PC 2), 1 of 2 keys	8,247	49.99	2.3%	2.7%	0%	10.7%	23.4 / 22.0	0.501	5090/2
00cec3ae	5090 (PC 2), 2 of 2	8,245	50.11	3.9%	2.7%	2.8%	9.3%	13.7 / 9.4	0.501	5090/2
8fafda27	M5 Max	4,133	25.35	6.5%	3.8%	5.2%	9.8%	16.9 / 10.8	0.496	M5 Max/1
4c022439	Intel UHD	265	1.59	13.4%	15.2%	0%	9.1%	20.2 / 26.0	0.512	Intel UHD/1

Correlation: 6 pairs, max r 0.53 with the mean factor of the first commit (0.10 to 0.53 with the median core factor of the second), no edge at 0.8, no group, no flag, no alert. Over all 28 ids with 32 or more nonces in the 24-hour table (40,550 blocks with detail): chi-square maximum 26.8 (n = 38) and 26.0 (n = 793), KS maximum 0.232 at n = 42 (p = 0.01 critical 0.251), increasing fraction 0.438 to 0.520; nonce mod 32 over all blocks chi-square 22.1 at 31 degrees of freedom. So the honest nonce is uniform over the full 64 bits: the serve protocol hands each job a 32-aligned 64-bit nonce_start (proto-cuda/nvrtc/worker.cpp lines 17 and 1175) and the kernel adds the lane index; the start is drawn at random per job. The 3-epoch window 36 to 38 (22 ids, PC 1 ramping down) had 4 of 28 pairs over r = 0.9: with 3 points a correlation is noise, which is why 5 shared epochs are the minimum. Card bands from the intake (6 hours of STATUS lines, p5 / p50 / p95): 5090 112.6 / 114.8 / 121.7 MH/s (n 781; the bench's 136 to 137 is the card to itself, the app's live rate shares it with the node and the prover); M5 Max 26.5 / 27.3 / 28.5 (674); Intel UHD 1.7 / 1.8 / 2.0 (754); from the 30-hour sample while PC 1 ran: RX 9070 XT 16.8 / 17.0 / 19.1 (355), gfx1036 2.6 / 2.8 / 3.4, M4 Max laptop 8.4 / 19.8 / 21.9 (478).

The honest population, in one line: excess per-program spread 0 to 5.2% on three card models (the census's 0.8 to 3.2% six-era spread plus the Mac's own load), first-tenth share 9.1 to 10.7%, nonces uniform, pairwise residual correlation under 0.55.

Calibration on the merged tree, 09:2x UTC (window epochs 41 to 46, PC 1 back from 07:2x UTC, 20 ids, 4 correlated)

Three readings the coordinator raised, reproduced read-only and answered:

Reading	Cause	Rule
The two PC 2 keys 00cec3ae and 9915d263 at r 0.94 (one edge, no group, held 0)	not the rate rule: an id's rate is its blue blocks' summed calc_work(bits) over the epoch's wall seconds, never a share of a network estimate, so nothing divides two ids by the same moving number. It was the epoch common factor: the mean over every id present let the Mac (paused and resumed, a 178% step) push every other residual the same way in the epochs it was off, and the two 5090 keys, each at Poisson sd 2.7%, moved together with it. The factor is now the median over the core ids (steady, present in every window epoch; the median over all present when the core is under 3): the same window reads max r 0.10, no edge. A residue of real co-movement stays between two keys of one card (PC 2's 5090 is shared with the prover, so both keys dip together), which is the machine_group case and needs k = 3 ids; two ids never form a group	changed in analyse (commit 2 on ca3-detector); the false-positive statement below stands, and the devnet's max pairwise r is now 0.10 to 0.53 across the two windows run
Each 5090 key at about 50 MH/s against the fleet band 99.4 / 115.6 / 123.1	PC 2's worker runs the card under 2 vote keys (STATUS ... identities=2 accepted_by_identity=8044/8009), so 2 x 50.4 = 100.8 MH/s on chain against the card's own 115.6 median: 0.87x, the same ratio the network shows (125 to 131 MH/s on chain against 143.9 of STATUS: reds, pending blocks and template latency are not chain work). Not the hours off (every epoch of the window has PC 2 on) and not a share against total blocks (the rate is work, not a share)	the /2 is the app's identity count (detect.rs: 1, 2 or 8); the band test is calibrated to the app and an unknown miner under another key count, or a pool, reads band or band_high, which are evidence only and never the alert by themselves. A testnet miner's key count belongs in the coinbase tag beside its card model (owed, 0.3.12)
The Mac 8fafda27 unsteady (spread 70%, step 178%)	the Mac was paused and resumed inside the window	by design unsteady only withholds the spread flag (a pause is not a program); the id stays in the correlation set, because a pause shared by several ids is exactly what identifies one machine's keys (machine_group), and it cannot become a design_candidate without a design flag. What changed: an unsteady id no longer enters the epoch common factor (above), so its pause cannot correlate other ids with each other

False positives, and what the public testnet's first week must add (history check 2)

Under the null (independent residuals, n = 6 epochs) one pair reads r over 0.8 with p about 0.028 (t = 2.67 on 4 degrees of freedom) and over 0.9 with p about 0.007; chance 3-cliques per window are about C(m, 3) p^3: 0.09 at m = 30 ids, about 100 at m = 300. So the clique alone is never the alert; the candidate also needs a design flag on a member (chance per id: nonce about 0.002, late_start about 1e-6, spread unknown until the per-model baseline exists; the devnet's maximum excess is 5.2% against the 10% line) and 6 net windows. At n = 12 (DETECTOR_WINDOW_EPOCHS=12) p(r over 0.8) is about 0.001 and chance 3-cliques at m = 300 are about 0.005 per window: set 12 once the testnet has over 100 ids. The first week must add: (1) the excess spread per card model over 12 or more epochs (the devnet has three models over 6); (2) the nonce layout of third-party miners and pools (a stratum extranonce in the high word is honest and non-uniform, so until each software is baselined a nonce flag is evidence, not an alert); (3) bands for cards the project does not own, which needs the card model in the coinbase tag; (4) the machine_group count, to see what an 8-key card looks like at scale.

Consequences per tier: the detector costs a miner nothing (it runs on the observer; one UPDATE a minute, two epochs re-read a minute, two regexp scans of the intake every 30 minutes). A home card under 8 keys appears as a machine_group, never an alert by itself. A pool's key trips band_high, which is informational; a pool that publishes its card mix clears it. The alert's action is the signal of epoch-length.md section 11, whose cost per tier is that document's section 7.