igneum/tools/observer

Igneum devnet observer

Watches one Igneum node over wRPC JSON and writes what it sees to Neon, so /api/live and /live on the site can show the devnet as it runs. Node 22 or newer, no dependencies.

Run

node tools/observer/observer.mjs

Environment, every value optional:

Variable	Default	Meaning
IGNEUM_RPC	ws://127.0.0.1:28610	The node's wRPC JSON url. 28610 is the igneum-devnet wRPC JSON port (consensus/core/src/network.rs). Start the node with --rpclisten-json=127.0.0.1:28610 or the port of your choice.
DATABASE_URL	read from ~/.config/igneum/env	Neon connection string. Never commit it.
LIVE_RETAIN_HOURS	24	Hours of blocks kept in live_blocks. Older rows are deleted once a minute.
LIVE_TABLE_PREFIX	empty	Prefix for every table name, so a test observer against a test network can write fintest_live_* without touching the site.
IGNEUM_EVM_RPC	http://127.0.0.1:26800	The execution layer's JSON-RPC (http) of a node on the proving build, for igneum_getShardPlan, igneum_getProofRecords and igneum_getProvingStatus. On the Mac that port is the Igneum Miner app's node, so an app restart (an OTA at its slot minute, a quit) blips the proving feed with fetch failed lines until it is back; the observer's own node (observer-v4) has no --evm-rpclisten. The default is the Mac app's node; the observer node itself has no EVM listener yet (start it with --evm-rpclisten=127.0.0.1:<port> once it runs the proving build and point this at it). A node without the RPCs (method not found) gives proving = {supported: false}, rechecked every 5 minutes; an unreachable endpoint is retried every 20 s.

A node started by another tool may listen on gRPC only. Then run your own non-mining peer with a JSON listener, on ports that do not clash with the devnet's (gRPC 26610, P2P 26611):

vendor/igneum-node/target/release/igneumd --devnet --nodnsseed --disable-upnp \
  --appdir=/tmp/igneum-obsnode --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 \
  --listen=127.0.0.1:26641 --connect=127.0.0.1:26611 --nologfiles
IGNEUM_RPC=ws://127.0.0.1:28640 node tools/observer/observer.mjs

What it does

• Subscribes to blockAdded and virtualChainChanged; polls getBlockDagInfo, getInfo, getConnectedPeerInfo, getSinkBlueScore and estimateNetworkHashesPerSecond every 2 s.
• Decodes the miner address from the coinbase payload script (same bech32 variant as crypto/addresses). The fork's vote_key_hash header field is stored per block; the first 8 hex characters are the miner's short id on the site.
• engine is the miner's tag in the coinbase extra data after the node's version prefix. The node exposes no engine name over RPC, so this is null on devnet v0.
• When the node refuses the hash-rate estimate (it needs a 1,000-block window) the observer reports blue work added per second over the last 10 minutes instead.
• Proving v0 (4 Oct 2026, spec 7.7): every chain block (the isChainBlock flag of a new block, or a virtualChainChanged addition) has its shard plan read over IGNEUM_EVM_RPC as {blockHash} (the chain block hash is the same hash on both layers), one live_proofs row per shard in state planned; a plan the EVM node has not executed yet is retried with a growing delay (up to 30 tries). The proof records of the chain blocks of the last 10 minutes are polled in rotation (80 blocks per 2 s tick while active, 10 before activation, four calls in flight); a shard moves to proving (a record in the node's pool), verified (the SP1 proof verified by the node's verifier, or the record carried by a block and checked by consensus, which is what happens before activation) or paid (a carrying segment paid it). lag_daa is the carrier's DAA score minus the block's; prover is the first 8 hex characters of the record's vote key hash. A block whose every shard is paid, or older than 10 minutes, leaves the rotation. Events: proving (activation reached, first paid shard seen) and prover_seen (one per prover per run). A reorg drops the removed chain blocks from the rotation.
• Finality v2 (3 Oct 2026): subscribes to FinalityLock (the node's lock event) and polls getFinalityCheckpoints every 2 s and getFinalityWeights every 10 s. Every checkpoint the node reports is upserted into live_checkpoints; a checkpoint turning locked writes the event checkpoint N locked (xx% of weight, yy% of active, v votes of n voters) at block h. The weights snapshot (total, active, per key) goes into live_state.finality. A node from before the finality layer answers the RPC with an error; the observer then logs once and skips finality.

Tables

Created on start if missing.

Table	Rows	Columns
live_blocks	one per block, kept LIVE_RETAIN_HOURS	hash, blue_score, daa_score, timestamp_ms, parents (count), parent_hashes, is_chain_block, vote_key_hash, miner_address, engine, received_at. Indexes on received_at, timestamp_ms, (vote_key_hash, received_at).
live_state	one row, updated every 2 s	block_count, header_count, blue_score, difficulty, hashes_per_second_estimate, peers, mempool, node_version, network, blocks_60s, blocks_per_minute (60 pairs of minute epoch ms and count), observer_started_at, updated_at
live_events	one per event, kept 7 days	ts, kind, text. Kinds: observer, miner_seen, miner_quiet, miner_back, peer_joined, peer_left, difficulty (step over 5%), checkpoint_locked.
live_checkpoints	one per checkpoint index, kept 7 days	index, hash, blue_score, daa_score, state (proposed, certified, locked), signed_weight, active_weight, total_weight, fraction_active, fraction_total, votes_seen, voters, aggregators (key hashes whose sortition proof made them aggregators), locked_at, first_seen_at, updated_at.
live_proofs	one per planned shard of a chain block, kept LIVE_RETAIN_HOURS	block_hash, shard, shards (in the plan), block_number, block_daa, block_ts, pgas, state (planned, proving, verified, paid), prover (id8), verified, carried_by, carrier_number, carrier_daa, lag_daa, payout_wei, received_at, updated_at. Primary key (block_hash, shard), index on received_at.
live_state.proving	jsonb, updated every 2 s	supported (false with reason when the node has no proving RPCs or the endpoint is unreachable), active, activation_daa, tip_daa, verifier, pool (entries, pending, verified, failed), paid_shards_total, shard_budget_pgas, blocks_10m, blocks_fully_proven_10m, shards_proven_10m, shards_paid_10m, median_proof_lag_s (median lag_daa of the last 10 minutes; the devnet targets one DAA step per second), provers_10m, open_blocks, pending_plans, evm_rpc.
live_state.finality	jsonb, updated every 2 s	params, chain_id, next_index, finality_active, latest_locked_index, latest_locked_hash, latest_locked_blue_score, weights (total_weight, active_weight, voters, keys[] with id, blocks, voter, participation, stripped_until_daa, revealed).

Keeping it current (autosync)

tools/observer/autosync.sh (loop: nohup bash tools/observer/autosync.sh & from the shared checkout; log /tmp/igneum-devnet/autosync.out) fetches origin/master every 5 min and fast-forwards the shared checkout when it is clean under tools/observer and site/api; a failed fast-forward logs git's reason and the dirty files the incoming commits also touch. Whoever moved HEAD (this loop or a pull by hand), the observer is restarted (a kill; run.sh starts it again in 3 s) whenever the checked-out observer.mjs or run.sh differs from what the running one started from: the key is the two blob ids hashed, kept in /tmp/igneum-devnet/observer.tree and rewritten at each restart. tools/observer/autosync.sh check prints the key, the marker and whether a restart is due (exit 3 when it is). A change to autosync.sh itself does not restart the observer, but the running loop keeps its old code: replace it by hand (kill it, start it again) after such a change; write the marker first (git rev-parse HEAD:tools/observer/observer.mjs HEAD:tools/observer/run.sh | tr -d '\n' | shasum | cut -c1-40 > /tmp/igneum-devnet/observer.tree) when the running observer is already current, or the new loop restarts it once for nothing. Each restart logs seeded N checkpoint states and, if a lock is recorded over a seeded state, names that state.

Reading it

site/api/live.mjs serves /api/live from these tables in five indexed queries (proving from live_state.proving; every block carries shards: [{i, n, state, prover, lag, payout, pgas}] and proven). LIVE_TABLE_PREFIX on the API reads a test observer's tables. site/live.html polls it every 2 s. The site shows OFFLINE when live_state.updated_at is older than 30 s.

Independence columns and the nightly table (O-X.1, ledger X5 and X14; branch ledger-observer, 5 October 2026 night, NOT deployed)

The running observer is untouched. On this branch observer.mjs adds four tables and three timers so the phase 5 gate ("N_ind >= 1,000 over 30 days with top-10 share of window weight under 50%", site/journey.json) can be read from stored data. The independence definition is the project lead's decision (docs/plans/ledger-decisions.md item 3); the table labels its column proposed.

Table	Rows	Filled by	Columns
live_peer_asn	one per announcing peer address	peerAsnTick every 10 min: getConnectedPeerInfo against the offline prefix table tools/observer/asn-table.txt (IGNEUM_ASN_TABLE overrides). No third-party lookup is made at run time; the table is filled by hand from a dated BGP dump (RIPEstat, Team Cymru bulk whois, or a pyasn dump of RouteViews). Private ranges read local; a miss stays source = 'none'	address, asn, asn_name, source, first_seen_at, updated_at
live_key_machines	one per (vote key, machine)	keyMachineTick every 10 min: the log intake (miner_logs, same database as tools/logs.mjs), every upload of the last day with vote_key_hash=<64 hex> (the miner's identity line, igneum/miner/src/main.rs), keyed by the upload's machine (the STATUS line's run id carries the same id8)	key_hash, machine, label, first_seen_at, last_seen_at
live_pool_statements	one per pool	poolStatementTick every 10 min: tools/observer/pool-statements/<pool>.json (IGNEUM_POOL_STATEMENTS overrides) parsed and verified against registry.json; a failing statement is stored with verified = false and the reason	pool, pubkey, keys, signed_at, verified, error, statement, received_at
live_concentration	one per day	nightlyTick at 00:05 UTC from what the observer holds: getFinalityWeights (hashing), the window's checkpoints (aggregation by certificateAggregator), live_certificates with their stored voter tables (signing: signed weight per key over the bitmap), live_proofs in state paid (proving), and the three tables above (N_ind)	day, daa, checkpoint_index, node_version, hashing, signing, proving, aggregation (each {keys, total, top1, top3, top10, unit, ...}), n_ind, n_ind_definition (proposed), n_ind_eligible, n_ind_unattributed, computed_at

What is open: a block names its producer by vote key and a peer announces an address, and nothing ties the two, so the ASN per key is null until the app reports its public address with its uploads (an app-owner item); n_ind_unattributed counts the keys with no attribute at all. The pure functions live in tools/observer/lib/concentration.mjs (payload decoder, voter table rebuild, shares, N_ind, pool statement, ASN table) and lib/nightly.mjs (the row); tools/finality-attacks/x14-concentration.mjs uses the same decoder and table rebuild against a live node. Tests: node --test 'tools/observer/test/*.test.mjs' (9 tests).