igneum/tools/observer
igneum-labs 344893ee11 Observer: decoupled ingest, lag metric, per-minute by header time, feed self-check, restart loop
After the 4 Oct 2026 stall (no block stored from 11:57 to 13:15 UTC, then 7,022
blocks in two minutes). Notifications only enqueue; a drain loop handles them
in bounded batches. Block flush, colour marking and certificate work each run
on their own timer and never wait on one another. Mergesets come from the
notification's verbose data (bounded cache); getBlock only on a miss, four at
a time. live_state gains observer_lag_s and queue_depth; the API serves them;
the page shows "observer N s behind" past 30 s instead of waiting for the
first block. blocks_per_minute and blocks_60s are bucketed by the block's own
timestamp and reseeded from the table on start, so a catch-up fills past
minutes instead of painting a spike. If no blockAdded arrives for 60 s while
the node's block_count advances, the observer resubscribes; after two failed
attempts it exits 2 and tools/observer/run.sh restarts it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-04 13:26:46 +00:00
..
observer.mjs Observer: decoupled ingest, lag metric, per-minute by header time, feed self-check, restart loop 2026-10-04 13:26:46 +00:00
README.md Finality v2: fork reading guide, spec implementation notes, bench entry, observer checkpoints, live page locks 2026-10-03 21:54:24 +00:00
run.sh Observer: decoupled ingest, lag metric, per-minute by header time, feed self-check, restart loop 2026-10-04 13:26:46 +00:00

Igneum devnet observer

Watches one Igneum node over wRPC JSON and writes what it sees to Neon, so /api/live and /live on the site can show the devnet as it runs. Node 22 or newer, no dependencies.

Run

node tools/observer/observer.mjs

Environment, every value optional:

Variable Default Meaning
IGNEUM_RPC ws://127.0.0.1:28610 The node's wRPC JSON url. 28610 is the igneum-devnet wRPC JSON port (consensus/core/src/network.rs). Start the node with --rpclisten-json=127.0.0.1:28610 or the port of your choice.
DATABASE_URL read from ~/.config/igneum/env Neon connection string. Never commit it.
LIVE_RETAIN_HOURS 24 Hours of blocks kept in live_blocks. Older rows are deleted once a minute.
LIVE_TABLE_PREFIX empty Prefix for every table name, so a test observer against a test network can write fintest_live_* without touching the site.

A node started by another tool may listen on gRPC only. Then run your own non-mining peer with a JSON listener, on ports that do not clash with the devnet's (gRPC 26610, P2P 26611):

vendor/igneum-node/target/release/igneumd --devnet --nodnsseed --disable-upnp \
  --appdir=/tmp/igneum-obsnode --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 \
  --listen=127.0.0.1:26641 --connect=127.0.0.1:26611 --nologfiles
IGNEUM_RPC=ws://127.0.0.1:28640 node tools/observer/observer.mjs

What it does

  • Subscribes to blockAdded and virtualChainChanged; polls getBlockDagInfo, getInfo, getConnectedPeerInfo, getSinkBlueScore and estimateNetworkHashesPerSecond every 2 s.
  • Decodes the miner address from the coinbase payload script (same bech32 variant as crypto/addresses). The fork's vote_key_hash header field is stored per block; the first 8 hex characters are the miner's short id on the site.
  • engine is the miner's tag in the coinbase extra data after the node's version prefix. The node exposes no engine name over RPC, so this is null on devnet v0.
  • When the node refuses the hash-rate estimate (it needs a 1,000-block window) the observer reports blue work added per second over the last 10 minutes instead.
  • Finality v2 (3 Oct 2026): subscribes to FinalityLock (the node's lock event) and polls getFinalityCheckpoints every 2 s and getFinalityWeights every 10 s. Every checkpoint the node reports is upserted into live_checkpoints; a checkpoint turning locked writes the event checkpoint N locked (xx% of weight, yy% of active, v votes of n voters) at block h. The weights snapshot (total, active, per key) goes into live_state.finality. A node from before the finality layer answers the RPC with an error; the observer then logs once and skips finality.

Tables

Created on start if missing.

Table Rows Columns
live_blocks one per block, kept LIVE_RETAIN_HOURS hash, blue_score, daa_score, timestamp_ms, parents (count), parent_hashes, is_chain_block, vote_key_hash, miner_address, engine, received_at. Indexes on received_at, timestamp_ms, (vote_key_hash, received_at).
live_state one row, updated every 2 s block_count, header_count, blue_score, difficulty, hashes_per_second_estimate, peers, mempool, node_version, network, blocks_60s, blocks_per_minute (60 pairs of minute epoch ms and count), observer_started_at, updated_at
live_events one per event, kept 7 days ts, kind, text. Kinds: observer, miner_seen, miner_quiet, miner_back, peer_joined, peer_left, difficulty (step over 5%), checkpoint_locked.
live_checkpoints one per checkpoint index, kept 7 days index, hash, blue_score, daa_score, state (proposed, certified, locked), signed_weight, active_weight, total_weight, fraction_active, fraction_total, votes_seen, voters, aggregators (key hashes whose sortition proof made them aggregators), locked_at, first_seen_at, updated_at.
live_state.finality jsonb, updated every 2 s params, chain_id, next_index, finality_active, latest_locked_index, latest_locked_hash, latest_locked_blue_score, weights (total_weight, active_weight, voters, keys[] with id, blocks, voter, participation, stripped_until_daa, revealed).

Reading it

site/api/live.mjs serves /api/live from these tables in three indexed queries. site/live.html polls it every 2 s. The site shows OFFLINE when live_state.updated_at is older than 30 s.