igneum/docs/plans/hands-on-build-1.md

The devnet hands move to igneum-build-1 (node 1 and the observer)

The founder's decision, 6 October 2026: the Mac runs nothing the network depends on. After the 0.3.15 cut tonight, node 1 and the observer leave the Mac's launchd agents and run on igneum-build-1 (188.40.146.49, Falkenstein, the build server of docs/plans/build-server.md) as systemd units. The shipper (owner of infra/devnet/restart-hand-nodes.sh and the exec recovery recipe) and the build-server agent run it together on the shipper's "0.3.15 live" line.

1. What runs on the Mac today (read 6 Oct 2026, 19:5x UK)

Hand	How it runs	Flags that matter	Data
node 1	launchd network.igneum.devnet.node1, KeepAlive, under caffeinate -dims, binary igneum-wt-ship0314/vendor/igneum-node-0314/target-integration/release/igneumd (0.3.14), log ~/Library/Logs/Igneum/node1.out	--devnet --enable-unsynced-mining --appdir=/tmp/igneum-devnet/node1 --rpclisten=0.0.0.0:26610 --listen=0.0.0.0:26611 --evm-rpclisten=127.0.0.1:26791 --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 --override-params-file=/tmp/igneum-devnet/override-v3.json --igneum-exec-snapshot=/tmp/igneum-devnet/node1-copy-snapshot.bin,ac101f13... --nodnsseed --disable-upnp --nologfiles --yes	856 MB (consensus, evm with exec-snapshot.bin 127,564,588 B and .prev, meta)
observer node	launchd network.igneum.devnet.observer, KeepAlive, same binary, log observer.out	--appdir=/tmp/igneum-devnet/observer-v4 --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 --listen=127.0.0.1:26641 --addpeer=127.0.0.1:26611 --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 + the same override and snapshot flags; no EVM listener	822 MB
observer process	tools/observer/run.sh loop (nohup, since 5 Oct 20:39) running node tools/observer/observer.mjs from the shared checkout, IGNEUM_RPC=ws://127.0.0.1:28640, IGNEUM_EVM_RPC default http://127.0.0.1:26800 (the Miner app's node), DATABASE_URL from ~/.config/igneum/env; tools/observer/autosync.sh (since 4 Oct) fast-forwards the shared checkout and restarts it on observer changes	writes Neon (live_* tables); /api/live and /live read Neon only

Also found: the Mac's own miner (igneum-miner, pid 7721) holds a gRPC connection to node 1 on 26610. The override file today: {"difficulty_v2_activation_daa":33000,"proving_v0_activation_daa":84100,"fees_v1_activation_daa":210000,"finality_v3_activation_daa":135200,"program_class_v3_activation_daa":154800,"proving_v1_activation_daa":154800,"proving_v1_segment_blocks":8,"proving_v1_unproven_daa":600,"proving_v1_aggregator_share_bps":1000,"proving_v1_fresh_rule_daa":198000,"exec_restart_number":27276,"exec_restart_hash":"bb45cf0d...","exec_restart_trust_daa":200000}. Node 1's last exec lines: exec state loaded from a snapshot: tip 141700 ... and exec sync: resumed from .../node1/igneum-devnet/datadir/evm/exec-snapshot.bin (tip 141700, 127564588 bytes, sha256 0x67637c55...).

2. What runs on the box afterwards

Unit	Runs	Ports	Files
igneum-node1.service	/srv/hands/bin/run-node1.sh -> /srv/hands/bin/igneumd (0.3.15 Linux, the box's own build), --appdir=/srv/hands/node1, --externalip=188.40.146.49	p2p 0.0.0.0:26611 (ufw open); gRPC 127.0.0.1:26610, wRPC JSON 127.0.0.1:28610, EVM 127.0.0.1:26791 on loopback	/srv/hands/hands.env (IGNEUMD, OVERRIDE, SNAPSHOT, EXTERNAL_IP, PEERS), /srv/hands/override.json, /srv/hands/node1-copy-snapshot.bin
igneum-observer-node.service	run-observer-node.sh, --appdir=/srv/hands/observer-node, peers node 1 on the box and the seed	p2p 127.0.0.1:26641, gRPC 127.0.0.1:26640, wRPC JSON 127.0.0.1:28640, EVM 127.0.0.1:26840 (new: the observer's proving feed came from the Miner app's node on the Mac)	same env and override
igneum-observer.service	/usr/local/bin/node tools/observer/observer.mjs in /srv/observer/igneum (a clone of the mirror /srv/igneum.git, master), EnvironmentFile=/srv/observer/env (mode 600, owner build: DATABASE_URL copied from the Mac's ~/.config/igneum/env, IGNEUM_RPC=ws://127.0.0.1:28640, IGNEUM_EVM_RPC=http://127.0.0.1:26840; never in the repo), Restart=always 3 s (run.sh's loop)	outbound to Neon only	/srv/observer/env
igneum-observer-sync.timer	every 5 min observer-sync.sh: fast-forward the clone from the mirror, restart the observer when tools/observer or site/lib changed (autosync.sh's job; the mirror is fed by every build-remote.sh and run-from-mac.sh push)

All units: Restart=always, journald (journalctl -u igneum-node1 -f), MemoryMax=24G on the nodes, enabled for reboot, run as user build (one uid on a single-purpose box; the units are the separation). Installer: infra/build-server/hands/install-hands.sh (idempotent, inert: enables, never starts). Mover: infra/build-server/hands/move-hand.sh (dry run by default, --go executes).

3. Ports, names, firewall

Port	On the Mac today	On the box	ufw
26611 TCP	node 1 p2p, open	node 1 p2p, open, --externalip so peers learn it	open already
26610 TCP	node 1 gRPC on 0.0.0.0 (the Mac's miner connects to it)	gRPC on loopback; a Mac tool reaches it through ssh -L 26610:127.0.0.1:26610 build@188.40.146.49	not opened (decision for main: open it to a fixed IP list if a miner must feed node 1 from outside)
26791 TCP	node 1 EVM RPC, loopback	loopback	closed
26640, 28640, 26641 TCP	observer node, loopback	loopback (+ EVM 26840)	closed
26811 TCP	(the TESTNET seed p2p port, not an RPC port: infra/seed-nodes/config.sh)	unused by the hands	open from provision; harmless

DNS (deSEC, main adds): node1.devnet.igneum.network A 188.40.146.49, observer.devnet.igneum.network A 188.40.146.49. The observer node listens on loopback only, so its name is for the future public endpoint and for the runbooks' wording. The public API (/api/live, /live) reads Neon and is unchanged.

4. The move, one hand at a time (run on the shipper's "0.3.15 live" line)

Step	Command (Mac)	What happens	Proof
0	ssh root@188.40.146.49 'bash -s' < infra/build-server/hands/install-hands.sh	units, run scripts, dirs, the observer clone; nothing started (done 6 Oct, see section 6)	systemd-analyze verify clean, units enabled and inactive
1	move-hand.sh binary --node /Users/joshm/Projects/igneum-wt-ship0315/vendor/igneum-node-0315	the 0.3.15 Linux igneumd built ON the box (tools/build-remote.sh), installed as /srv/hands/bin/igneumd-0.3.15-<sha>, commit string checked; the Mac's override file and the exec snapshot copied; hands.env pointed at them	igneumd --version, commit in strings, sha256 lines
2	move-hand.sh observer-node --go	hot rsync of /tmp/igneum-devnet/observer-v4 (822 MB) while the Mac node runs; launchctl bootout of the Mac's observer agent (KeepAlive would restart a killed pid); final rsync (the delta, seconds, node stopped so RocksDB is consistent); systemctl start igneum-observer-node	the unit's first [igneum-exec] exec sync: resumed from ... line and its first Accepted N blocks lines, printed by the script
3	move-hand.sh observer --go	/srv/observer/env written (scp, mode 600, owner build); the Mac's autosync.sh, run.sh and observer.mjs stopped FIRST (two writers would duplicate Neon rows), then systemctl start igneum-observer	the observer's first journal lines (rpc load, events); /api/live fresh within a minute
4	move-hand.sh node1 --go	the same as step 2 for node 1 (856 MB); node 1 is the last node the Mac serves, the observer node on the box already peers with the seed, so the network never loses both hands	node 1's first exec line and PoW accepted lines on the box
5	move-hand.sh unload --go	the two plists moved aside so a login never brings the Mac hands back; the Mac's miner loses node 1's RPC (section 5)	pgrep igneumd on the Mac shows only the wallet's node
6	move-hand.sh status, 10 minutes later	both nodes at the network tip, observer writing, /live current	the status output in this plan's section 6

Exec recovery on the box: each node resumes from its data dir's evm/exec-snapshot.bin (copied with the data dir); if that file is bad or missing, --igneum-exec-snapshot=/srv/hands/node1-copy-snapshot.bin,<sha256> (the Mac's recovery snapshot, copied in step 1) is taken, as the launchd agents do today; the override's exec_restart_number/exec_restart_hash/exec_restart_trust_daa travel unchanged. A snapshot from the seed is the fallback the shipper's recipe names; the p2p snapshot path refuses one below the node's tip (CLAUDE.md, Devnet 2 rules).

Rollback at any step: the Mac's plist is still in ~/Library/LaunchAgents until step 5; launchctl bootstrap gui/$(id -u) <plist> brings a hand back on the Mac within 10 s, and the box unit is stopped with systemctl stop. Data dirs are copies; nothing is deleted on the Mac.

5. Decisions and consequences for main (DECIDED by main, 6 October 2026, 19:1x UTC)

Question	Decision
The Mac's miner on node 1's gRPC	stops (paused by the founder's order anyway; the Mac mines nothing)
The LAN peer 192.168.68.67 (a PC)	the shipper adds --addpeer=188.40.146.49:26611 to the PCs' and the Mac's app node args in the 0.3.15 update; the seeds carry the public peers already
node 1's gRPC 26610	loopback only on the box; Mac tools tunnel
DNS	node1.devnet.igneum.network and observer.devnet.igneum.network -> 188.40.146.49 exist in deSEC (checked: ns1.desec.io answers both)

The table below is the reasoning that led to them.

Finding	Means	Proposed
The Mac's own miner (igneum-miner pid 7721) mines through node 1's gRPC 26610	after step 4 it loses its node; the founder's rule says the Mac runs nothing the network depends on, and a miner is hashrate, not a dependency	either it stops with node 1, or it follows through an ssh tunnel to the box (ssh -L 26610:...); main decides, default: it stops
192.168.68.67:26611 is a LAN peer of both hands	unreachable from the box; the box peers with the seed 188.245.5.161 and the two hands peer with each other	hands.env PEERS=188.245.5.161:26611; whoever runs 192.168.68.67 adds --addpeer=188.40.146.49:26611 if it relied on node 1
CLAUDE.md "Running agents on this Mac" says the box never hosts a live-devnet node (my R6, 6 Oct 18:xx)	contradicted by the founder's decision the same evening	rewritten in this commit: the box hosts the two hands as units; it still holds no secret beyond /srv/observer/env (DATABASE_URL, mode 600)
The observer's proving feed on the Mac read the Miner app's node (26800)	on the box there is no app node	the observer node gets --evm-rpclisten=127.0.0.1:26840 (0.3.15 runs the proving build) and the observer reads it; if its shard plans lag node 1's, point IGNEUM_EVM_RPC at node 1's 26791
autosync.sh followed origin/master from GitHub; the box has no GitHub credential	the box's clone follows the MIRROR, which moves only when a Mac agent pushes (every build-remote.sh and run-from-mac.sh run pushes the branch it builds; run-from-mac.sh pushes every branch)	enough today; a read-only deploy key on the box (generated there, added by main to the repository) would make it follow GitHub directly, open
Two hands stop for one to three minutes each during the move (the final rsync and the start)	the other hand serves throughout; the observer feed pauses once for about a minute (step 3)	accepted by the order above
Data dirs are rsynced hot then with the node stopped	the hot pass moves 99 percent of 1.7 GB with the hands up; the stopped pass is the delta	the Mac's upload rate decides the hot pass (minutes); measured in section 6

6. Run log (filled as it happens)

Run on 6 Oct 2026 (UTC; box clock is UTC+2). Inputs: the shipper's "0.3.15 live" became 0.3.16 (same tree, f1ea7a38); publish 2 live with the sixteen-field object, digest eada4bda8aa8368c2b2c3d17744bc7a70a0ff0e996dad681884d3ac5de1207eb; main's gate: port 26611 on the box was held by the fleet agent's Devnet 2 seed, moved to 26621 at 23:06:57 ("26611 free on the box").

Time	Step	Result
23:06 to 23:14	binary (tree from the Mac's node1 agent: igneum-wt-ship0315/vendor/igneum-node-0315, f1ea7a38)	/srv/hands/bin/igneumd-2.1.0-f1ea7a38, sha256 7f0bde70cf2e72a3a9479ba6421f2b37162c3dd4391bf413717c99e6f168668d, commit string f1ea7a3 present; the sixteen-field override from the live manifest's consensus.override; recovery snapshot ac101f13... as the fallback flag. First attempt died silently after the version print: igneumd --version exits 1 (fixed)
23:06 to 23:10	hot pre-sync	observer-v4 920 MB in 97 s, node1 937 MB in 98 s, hands running
23:15:11 to 23:15:32	observer-node --go	first executing line 01:15:28 box time: "[igneum-exec] exec state loaded from a snapshot: tip 149781 e98b33f2..., state root 0x55a5892f..."; digest eada4bda... MATCH; "Program class v4 from the override file: active from epoch 231"; listeners 26640, 28640, 26641, 26840 loopback; the hand was down about 10 s
23:15:52 to 23:16:00	observer --go	/srv/observer/env written (600, build); the Mac's autosync, run.sh, observer.mjs stopped first; started 23:15:56, "subscribed on igneum-devnet, node 2.1.0", 2871 checkpoint states seeded. Two minutes of getBlockDagInfo timeouts while its node settled (/api/live stale 41 s at 23:16:12), none after; rpc load 150 wRPC/min, 1366 EVM/min
23:18:36 to 23:18:58	node1 --go	first executing line 01:18:53: "exec state loaded from a snapshot: tip 149806 f54ef53d..., state root 0xe1cd365e..." and "exec sync: resumed from /srv/hands/node1/.../exec-snapshot.bin (tip 149806, 142713733 bytes)"; digest MATCH; p2p 0.0.0.0:26611, RPC loopback; IBD of 419 headers from 213.173.107.74 01:19:12 to 01:21:42, then PoW blocks accepted from DAA 227716 (420 in the next 2 min, 6 established peers)
23:22:15	unload --go	both plists moved aside (.moved-to-build-1-20261006); no hand igneumd on the Mac
23:22	steady state	igneum-node1, igneum-observer-node, igneum-observer active; /api/live age 1.4 s, height 149878; box load 27 to 34 (capacity layer and CI runner share it)

6a. The 0.3.17 hotfix on the box and the seed (7 Oct 2026, 05:30 to 05:32 UTC)

The cut moved twice in the night (12153428 failed its canary; b3c228fa became 5899f603 with the test-only fixes and the toolchain pin), so the pairs were built three times; the shipped one is fork 5899f603 paired with igneum release-0.3.17 at 250fd371 (the tool's first line reads "pairs with igneum 250fd371 (detached): igneum-pow 0.2.0"). Read-back per node: first executing line, commit string 5899f603 in the running binary, "Consensus params digest" eada4bda... from the journal, and the engine from the binary (6 igneum-pow/src/ paths; this tree has no igneum_getNodeInfo, the RPC answers -32601).

Time	Step	Result
05:30:41	move-hand.sh binary --node <5899f603 tree> --override-json <live consensus.override>	igneumd-2.1.0-5899f603, sha256 401bfd54d3cb58736c18ada6c1796b62839a5bc02addcc6bb4ca01d0879810f8 (49,720,416 B); the 16-field object (byte-equal to the 23:14 one; the manifest reads 0.3.17); recovery snapshot as the fallback flag
05:30:46 to 05:30:57	move-hand.sh restart observer-node --digest eada4bda... --go	first executing line 07:30:50 box time "exec state loaded from a snapshot: tip 157682 f2e6041c..."; PoW accepted at once; commit string present; digest MATCH; engine igneum-pow from the binary; the observer process reconnected by itself (1 tick failure in the first minute, none after); down about 6 s
05:31:12 to 05:31:22	move-hand.sh restart node1 ... --go	first executing line 07:31:16 "tip 157689 f6cfb8bf..."; PoW accepted from daa 253285; commit string present; digest MATCH; engine igneum-pow; after: 11 peers on 26611, 126 blocks in the next minute, /api/live age 0.6 s
05:31:36 to 05:32:10	IGNEUMD_LINUX=<class-seed igneumd 39165c1f...> IGNEUMD_LINUX_SHA256=... infra/devnet/restart-seed.sh '<object>'	the script's own check "binary needs GLIBC 2.34, the seed has 2.36"; unit active 05:32:00; /opt/igneum/v4/bin/igneumd sha256 39165c1f..., 2 commit strings 5899f60, 6 igneum-pow paths; digest eada4bda...; program class v4 from epoch 231; exec resumed from its own snapshot at tip 157696; 265 blocks in the two minutes after; down about 24 s. The seed's igneum-miner is not part of restart-seed.sh

Held for 0.3.18: the 12153428 pairs (hands igneumd f0f2db86..., seed igneumd d8d431e5...). Not byte-for-byte with the shipper's own native build of the same inputs (d712b498): the two trees sit at different paths on the box and prost's generated code embeds OUT_DIR; a "reproduced" row stays a same-tree comparison unless both sides pass --remap-path-prefix.

Open after the move: the Mac's Igneum Miner app (0.3.16, running since 23:12) did NOT start its own igneumd once 26610/26611 were free (nothing bound them five minutes later; the only Mac igneumd is the Wallet's on 26620/26621/26800). A job never restarts the installed app, so the app lane or main decides how its node starts. The observer's proving feed now comes from the observer node's own EVM listener 26840.