12 KiB
The devnet hands move to igneum-build-1 (node 1 and the observer)
the project lead's decision, 6 October 2026: the Mac runs nothing the network depends on. After the 0.3.15 cut tonight, node 1 and the observer leave the Mac's launchd agents and run on igneum-build-1 (188.40.146.49, Falkenstein, the build server of docs/plans/build-server.md) as systemd units. The shipper (owner of infra/devnet/restart-hand-nodes.sh and the exec recovery recipe) and the build-server agent run it together on the shipper's "0.3.15 live" line.
1. What runs on the Mac today (read 6 Oct 2026, 19:5x UK)
| Hand | How it runs | Flags that matter | Data |
|---|---|---|---|
| node 1 | launchd network.igneum.devnet.node1, KeepAlive, under caffeinate -dims, binary igneum-wt-ship0314/vendor/igneum-node-0314/target-integration/release/igneumd (0.3.14), log ~/Library/Logs/Igneum/node1.out |
--devnet --enable-unsynced-mining --appdir=/tmp/igneum-devnet/node1 --rpclisten=0.0.0.0:26610 --listen=0.0.0.0:26611 --evm-rpclisten=127.0.0.1:26791 --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 --override-params-file=/tmp/igneum-devnet/override-v3.json --igneum-exec-snapshot=/tmp/igneum-devnet/node1-copy-snapshot.bin,ac101f13... --nodnsseed --disable-upnp --nologfiles --yes |
856 MB (consensus, evm with exec-snapshot.bin 127,564,588 B and .prev, meta) |
| observer node | launchd network.igneum.devnet.observer, KeepAlive, same binary, log observer.out |
--appdir=/tmp/igneum-devnet/observer-v4 --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 --listen=127.0.0.1:26641 --addpeer=127.0.0.1:26611 --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 + the same override and snapshot flags; no EVM listener |
822 MB |
| observer process | tools/observer/run.sh loop (nohup, since 5 Oct 20:39) running node tools/observer/observer.mjs from the shared checkout, IGNEUM_RPC=ws://127.0.0.1:28640, IGNEUM_EVM_RPC default http://127.0.0.1:26800 (the Miner app's node), DATABASE_URL from ~/.config/igneum/env; tools/observer/autosync.sh (since 4 Oct) fast-forwards the shared checkout and restarts it on observer changes |
writes Neon (live_* tables); /api/live and /live read Neon only |
Also found: the Mac's own miner (igneum-miner, pid 7721) holds a gRPC connection to node 1 on 26610. The override file today:
{"difficulty_v2_activation_daa":33000,"proving_v0_activation_daa":84100,"fees_v1_activation_daa":210000,"finality_v3_activation_daa":135200,"program_class_v3_activation_daa":154800,"proving_v1_activation_daa":154800,"proving_v1_segment_blocks":8,"proving_v1_unproven_daa":600,"proving_v1_aggregator_share_bps":1000,"proving_v1_fresh_rule_daa":198000,"exec_restart_number":27276,"exec_restart_hash":"bb45cf0d...","exec_restart_trust_daa":200000}.
Node 1's last exec lines: exec state loaded from a snapshot: tip 141700 ... and exec sync: resumed from .../node1/igneum-devnet/datadir/evm/exec-snapshot.bin (tip 141700, 127564588 bytes, sha256 0x67637c55...).
2. What runs on the box afterwards
| Unit | Runs | Ports | Files |
|---|---|---|---|
igneum-node1.service |
/srv/hands/bin/run-node1.sh -> /srv/hands/bin/igneumd (0.3.15 Linux, the box's own build), --appdir=/srv/hands/node1, --externalip=188.40.146.49 |
p2p 0.0.0.0:26611 (ufw open); gRPC 127.0.0.1:26610, wRPC JSON 127.0.0.1:28610, EVM 127.0.0.1:26791 on loopback |
/srv/hands/hands.env (IGNEUMD, OVERRIDE, SNAPSHOT, EXTERNAL_IP, PEERS), /srv/hands/override.json, /srv/hands/node1-copy-snapshot.bin |
igneum-observer-node.service |
run-observer-node.sh, --appdir=/srv/hands/observer-node, peers node 1 on the box and the seed |
p2p 127.0.0.1:26641, gRPC 127.0.0.1:26640, wRPC JSON 127.0.0.1:28640, EVM 127.0.0.1:26840 (new: the observer's proving feed came from the Miner app's node on the Mac) |
same env and override |
igneum-observer.service |
/usr/local/bin/node tools/observer/observer.mjs in /srv/observer/igneum (a clone of the mirror /srv/igneum.git, master), EnvironmentFile=/srv/observer/env (mode 600, owner build: DATABASE_URL copied from the Mac's ~/.config/igneum/env, IGNEUM_RPC=ws://127.0.0.1:28640, IGNEUM_EVM_RPC=http://127.0.0.1:26840; never in the repo), Restart=always 3 s (run.sh's loop) |
outbound to Neon only | /srv/observer/env |
igneum-observer-sync.timer |
every 5 min observer-sync.sh: fast-forward the clone from the mirror, restart the observer when tools/observer or site/lib changed (autosync.sh's job; the mirror is fed by every build-remote.sh and run-from-mac.sh push) |
All units: Restart=always, journald (journalctl -u igneum-node1 -f), MemoryMax=24G on the nodes, enabled for reboot, run as user
build (one uid on a single-purpose box; the units are the separation). Installer: infra/build-server/hands/install-hands.sh
(idempotent, inert: enables, never starts). Mover: infra/build-server/hands/move-hand.sh (dry run by default, --go executes).
3. Ports, names, firewall
| Port | On the Mac today | On the box | ufw |
|---|---|---|---|
| 26611 TCP | node 1 p2p, open | node 1 p2p, open, --externalip so peers learn it |
open already |
| 26610 TCP | node 1 gRPC on 0.0.0.0 (the Mac's miner connects to it) |
gRPC on loopback; a Mac tool reaches it through ssh -L 26610:127.0.0.1:26610 build@188.40.146.49 |
not opened (decision for main: open it to a fixed IP list if a miner must feed node 1 from outside) |
| 26791 TCP | node 1 EVM RPC, loopback | loopback | closed |
| 26640, 28640, 26641 TCP | observer node, loopback | loopback (+ EVM 26840) | closed |
| 26811 TCP | (the TESTNET seed p2p port, not an RPC port: infra/seed-nodes/config.sh) | unused by the hands | open from provision; harmless |
DNS (deSEC, main adds): node1.devnet.igneum.network A 188.40.146.49, observer.devnet.igneum.network A 188.40.146.49. The
observer node listens on loopback only, so its name is for the future public endpoint and for the runbooks' wording. The public
API (/api/live, /live) reads Neon and is unchanged.
4. The move, one hand at a time (run on the shipper's "0.3.15 live" line)
| Step | Command (Mac) | What happens | Proof |
|---|---|---|---|
| 0 | ssh root@188.40.146.49 'bash -s' < infra/build-server/hands/install-hands.sh |
units, run scripts, dirs, the observer clone; nothing started (done 6 Oct, see section 6) | systemd-analyze verify clean, units enabled and inactive |
| 1 | move-hand.sh binary --node /Users/joshm/Projects/igneum-wt-ship0315/vendor/igneum-node-0315 |
the 0.3.15 Linux igneumd built ON the box (tools/build-remote.sh), installed as /srv/hands/bin/igneumd-0.3.15-<sha>, commit string checked; the Mac's override file and the exec snapshot copied; hands.env pointed at them |
igneumd --version, commit in strings, sha256 lines |
| 2 | move-hand.sh observer-node --go |
hot rsync of /tmp/igneum-devnet/observer-v4 (822 MB) while the Mac node runs; launchctl bootout of the Mac's observer agent (KeepAlive would restart a killed pid); final rsync (the delta, seconds, node stopped so RocksDB is consistent); systemctl start igneum-observer-node |
the unit's first [igneum-exec] exec sync: resumed from ... line and its first Accepted N blocks lines, printed by the script |
| 3 | move-hand.sh observer --go |
/srv/observer/env written (scp, mode 600, owner build); the Mac's autosync.sh, run.sh and observer.mjs stopped FIRST (two writers would duplicate Neon rows), then systemctl start igneum-observer |
the observer's first journal lines (rpc load, events); /api/live fresh within a minute |
| 4 | move-hand.sh node1 --go |
the same as step 2 for node 1 (856 MB); node 1 is the last node the Mac serves, the observer node on the box already peers with the seed, so the network never loses both hands | node 1's first exec line and PoW accepted lines on the box |
| 5 | move-hand.sh unload --go |
the two plists moved aside so a login never brings the Mac hands back; the Mac's miner loses node 1's RPC (section 5) | pgrep igneumd on the Mac shows only the wallet's node |
| 6 | move-hand.sh status, 10 minutes later |
both nodes at the network tip, observer writing, /live current |
the status output in this plan's section 6 |
Exec recovery on the box: each node resumes from its data dir's evm/exec-snapshot.bin (copied with the data dir); if that file
is bad or missing, --igneum-exec-snapshot=/srv/hands/node1-copy-snapshot.bin,<sha256> (the Mac's recovery snapshot, copied in
step 1) is taken, as the launchd agents do today; the override's exec_restart_number/exec_restart_hash/exec_restart_trust_daa
travel unchanged. A snapshot from the seed is the fallback the shipper's recipe names; the p2p snapshot path refuses one below the
node's tip (CLAUDE.md, Devnet 2 rules).
Rollback at any step: the Mac's plist is still in ~/Library/LaunchAgents until step 5; launchctl bootstrap gui/$(id -u) <plist>
brings a hand back on the Mac within 10 s, and the box unit is stopped with systemctl stop. Data dirs are copies; nothing is deleted
on the Mac.
5. Decisions and consequences for main (DECIDED by main, 6 October 2026, 19:1x UTC)
| Question | Decision |
|---|---|
| The Mac's miner on node 1's gRPC | stops (paused by the project lead's order anyway; the Mac mines nothing) |
| The LAN peer 192.168.68.67 (a PC) | the shipper adds --addpeer=188.40.146.49:26611 to the PCs' and the Mac's app node args in the 0.3.15 update; the seeds carry the public peers already |
| node 1's gRPC 26610 | loopback only on the box; Mac tools tunnel |
| DNS | node1.devnet.igneum.network and observer.devnet.igneum.network -> 188.40.146.49 exist in deSEC (checked: ns1.desec.io answers both) |
The table below is the reasoning that led to them.
| Finding | Means | Proposed |
|---|---|---|
The Mac's own miner (igneum-miner pid 7721) mines through node 1's gRPC 26610 |
after step 4 it loses its node; the project lead's rule says the Mac runs nothing the network depends on, and a miner is hashrate, not a dependency | either it stops with node 1, or it follows through an ssh tunnel to the box (ssh -L 26610:...); main decides, default: it stops |
192.168.68.67:26611 is a LAN peer of both hands |
unreachable from the box; the box peers with the seed 188.245.5.161 and the two hands peer with each other | hands.env PEERS=188.245.5.161:26611; whoever runs 192.168.68.67 adds --addpeer=188.40.146.49:26611 if it relied on node 1 |
| CLAUDE.md "Running agents on this Mac" says the box never hosts a live-devnet node (my R6, 6 Oct 18:xx) | contradicted by the project lead's decision the same evening | rewritten in this commit: the box hosts the two hands as units; it still holds no secret beyond /srv/observer/env (DATABASE_URL, mode 600) |
| The observer's proving feed on the Mac read the Miner app's node (26800) | on the box there is no app node | the observer node gets --evm-rpclisten=127.0.0.1:26840 (0.3.15 runs the proving build) and the observer reads it; if its shard plans lag node 1's, point IGNEUM_EVM_RPC at node 1's 26791 |
| autosync.sh followed origin/master from GitHub; the box has no GitHub credential | the box's clone follows the MIRROR, which moves only when a Mac agent pushes (every build-remote.sh and run-from-mac.sh run pushes the branch it builds; run-from-mac.sh pushes every branch) | enough today; a read-only deploy key on the box (generated there, added by main to the repository) would make it follow GitHub directly, open |
| Two hands stop for one to three minutes each during the move (the final rsync and the start) | the other hand serves throughout; the observer feed pauses once for about a minute (step 3) | accepted by the order above |
| Data dirs are rsynced hot then with the node stopped | the hot pass moves 99 percent of 1.7 GB with the hands up; the stopped pass is the delta | the Mac's upload rate decides the hot pass (minutes); measured in section 6 |
6. Run log (filled as it happens)
RUNLOG