igneum/infra/cloud-devnet/README.md

Igneum cloud devnet: 20 nodes across regions

A private Igneum devnet (igneum-devnet-20: own handshake magic, own genesis) on 20 small Linux VMs in five locations, each running igneumd with the real lottery hash and one CPU trickle miner with its own BLS vote key. Built for two measurements that the single-site devnet cannot give: block propagation and reorg behaviour under real inter-region latency (gate 3, spec 03 C1 and the floor rule), and the difficulty controller under hash-rate steps (spec 02 section 2.3). Everything here is scripts; nothing runs or spends until create.sh is answered "yes".

Hetzner Cloud through hcloud is the primary path. DigitalOcean through doctl is the variant for create.sh (and destroy.sh); the rest is provider-neutral once nodes.tsv exists.

The command sequence (the project lead approves the cost, then this, start to finish)

cd infra/cloud-devnet
brew install hcloud                                  # once; doctl for the DigitalOcean variant
# the API token is read from ~/.config/igneum/hetzner-token (the seed-node scripts use the same file)
./create.sh                                          # 1. shows the plan and the live prices, asks "yes", creates 20 VMs, writes nodes.tsv
./provision.sh                                       # 2. source tarball -> builder VM -> igneumd + igneum-miner -> every node (about 20 min)
./start.sh                                           # 3. nodes, block logs, then the 20 trickle miners
./status.sh                                          # one line per node (blocks, blue, DAA, tips, peers, difficulty, sink, miner)
./experiments/latency.sh 10                          # 4. RTT matrix, then 10 min of block propagation samples
./experiments/partition.sh sin 10                    # 5. cut Singapore off for 10 min, heal, reorg depth and heal time
./experiments/hop.sh                                 # 6. hash-rate steps for the controller (about 75 min, schedule in the script)
./experiments/collect.sh                             # 7. pull logs and RPC samples -> results/<date>/, summary.md, bench-log entry
./stop.sh && ./destroy.sh                            # 8. stop, then delete every VM (hourly billing ends)

Time from "yes" to the first block: VM creation 1 min, build on the builder 10 to 25 min (approximate; 500 crates plus rocksdb), install 2 min, miners' first 256 MiB cache 10 to 30 s. Steps 4 to 6 are independent; 4 and 6 can run at the same time, 5 should run alone.

The observer and the live page (tools/observer): the RPC never leaves a node's loopback, so either ./experiments/observer.sh tunnel (an ssh tunnel from the Mac; then run the observer here) or ./experiments/observer.sh remote on (installs Node 22 on node 1 and runs the observer there, keeping the Mac out of it). With LIVE_TABLE_PREFIX=cloud_ the record goes to cloud_live_* tables; without the prefix the cloud network takes over /live on the site (stop the Mac's observer first). site/api/live.mjs reads the unprefixed tables only.

Cost

Hetzner API prices for this project on 3 Oct 2026, net USD per month, hourly billing (the API's price_hourly):

Location	Type	vCPU	RAM	Per month	Per hour	Note
fsn1, nbg1, hel1	cx23	2 Intel	4 GB	6.49	0.0104	the cheapest x86 type the project can create
sin	cpx22	2 AMD	4 GB	30.99	0.0497	cx not sold outside the EU
ash, hil	cpx21	3 AMD	4 GB	37.49	0.0601	the old cpx line, only still sold in the US; cpx11 (2 GB) is 20.49
builder, any EU	cx43	8 Intel	16 GB	18.49	0.0296	deleted after the build

Plus USD 0.60 per month net per primary IPv4 (pricing API). The account bills in USD (the pricing API reports currency USD, VAT 20%); VAT is added for a private customer; the API's gross column is net x 1.2.

Mix	Nodes	Per month net	Per hour	An evening (6 h)
4 per location (default REGIONS)	8 EU, 4 sin, 4 ash, 4 hil	USD 476	USD 0.76	about USD 5
EU-heavy (REGIONS=hel1,fsn1,nbg1,hel1,fsn1,nbg1,hel1,ash,hil,sin)	14 EU, 2 sin, 2 ash, 2 hil	USD 303	USD 0.47	about USD 3
DigitalOcean s-2vcpu-4gb in any region (DO pricing page, USD 24/mo, 0.036/h)	20	USD 480	USD 0.71	about USD 4.50

The run is meant to last one evening, so the hourly column is the one that matters: well under USD 10 all in, with the builder. Leaving the network up costs the monthly column.

What each script does

Script	What
config.sh	every setting (provider, N, regions, types per location, suffix, genesis bits, mesh degree, miner threads, source tree)
create.sh	ssh key, firewall (22, 26611, icmp), N servers round-robin over REGIONS, writes nodes.tsv (name, index, region, ip)
make-source.sh	git archive HEAD of NODE_SRC and igneum-pow plus the uncommitted files (SRC_MODE=head+dirty, the default: on 3 Oct 2026 every worktree's branch work is uncommitted) into build/src.tar.gz, same layout as the Windows package's src.zip
provision.sh	build: builder VM, builder/build-on-builder.sh (apt deps, rustup, cargo build --release -p kaspad -p igneum-miner --features igneum-pow), binaries to build/bin/, builder deleted. install: binaries, node/wrpc.py, units and /etc/igneum/node.env on every node
node/run-igneumd.sh	the flag list: --devnet --devnet-suffix=20 --override-params-file (genesis_bits) --rpclisten=127.0.0.1:26610 --rpclisten-json=127.0.0.1:28610 --listen=0.0.0.0:26611 --externalip --addpeer x2 --outpeers=0 --maxinpeers=32 --nodnsseed --disable-upnp --nologfiles --enable-unsynced-mining
node/igneum-miner.service	igneum-miner mine grpc://127.0.0.1:26610 1 1000000000 <node> --engine igneum-pow --payout-label <node>: one CPU thread, its own vote key, votes on every checkpoint
node/wrpc.py	standard-library wRPC JSON client (the node's WebSocket RPC): call, sample, watch-blocks (blocks.tsv, chain.tsv, samples.tsv under /var/log/igneum)
start.sh, stop.sh, status.sh, destroy.sh	as named; destroy.sh deletes everything labelled igneum=devnet and the firewall
experiments/latency.sh	ping matrix between all nodes, then block propagation from the per-node arrival logs joined on hash (p50, p90, p99 per node and per region)
experiments/partition.sh	iptables cut of one region's p2p traffic to the rest, heal after N min, reorg depth (getVirtualChainFromBlock from each node's sink at heal, plus the largest virtualChainChanged removal), heal time (sinks converge), conflicting locks (getFinalityCheckpoints on both sides)
experiments/hop.sh	a schedule of miner-thread changes per node set (step up, step down, a region off), logged for the controller analysis
experiments/collect.sh	journals, block/chain/sample logs, RPC snapshots into results/<date>/nodes/<name>/; analyze.py summary and a bench-log entry template; APPEND_BENCH_LOG=1 appends it to docs/bench-log.md
experiments/analyze.py	the offline analysis (RTT table, propagation join, lock conflicts, hop phases, reorg-depth histogram)
experiments/observer.sh	the observer hookup (tunnel or remote)

Network design choices

• Own network id. --devnet-suffix=20 gives igneum-devnet-20 (own handshake magic, own data directory) and the override file sets genesis_bits (0x1e400000, 2^18 hashes per block: three 6-thread CPU miners on the M5 Max held about 1 block/s at this value, fork-divergence genesis row), which recomputes the genesis hash. A node of this network can never complete a handshake with the live devnet or the seed.
• Sparse mesh. Each node has two permanent outbound links (--addpeer to its ring neighbour and to the node half way round) and --outpeers=0, so the connection manager does not fill its default 8 outbound slots from exchanged addresses. Inbound links double the count: about 4 peers per node, 40 links in total. Blocks therefore cross several hops between regions, which is what the propagation measurement needs.
• --addpeer, never --connect: --connect sets the inbound limit to 0 (kaspad/src/daemon.rs), the Windows node lesson.
• RPC on loopback. Every measurement goes through ssh to wrpc.py on the node. The firewall opens 22, 26611 and icmp.
• One vote key per node. The miner's identity label is the node name, so weight accrues to 20 keys and the finality rule has 20 voters; the devnet finality parameters (interval 30, dust 5, presence 20) apply.
• Which tree runs. NODE_SRC defaults to vendor/igneum-node (master worktree: finality v2 work, uncommitted). For the controller experiment point it at vendor/igneum-node-diff (the difficulty worktree) or at a merged tree; the binary is cached in build/bin and REBUILD=1 rebuilds. build/src.stamp records what went in.
• Node 22 is not needed on the nodes: wrpc.py is standard-library Python. The observer (remote mode) is the one thing that installs Node, on one node, only when asked.

What the results answer

Measurement	Where it lands	Decides
RTT between regions, block propagation p50/p90/p99	results/<date>/latency/	the latency assumption behind GHOSTDAG k (5 s network delay bound at 1 BPS) and spec 03 C1's lock latency (simulated 2.5 s median at a 2 s inter-region delay)
Reorg depth distribution over the run	summary.md	checkpoint determination depth d (spec 03 C1, ledger F7: placeholder 60, devnet 20)
Partition: reorg depth, heal time, conflicting locks	results/<date>/partition-*/partition.md	the floor rule (lock needs 56.7% of total weight) under a real partition; ledger F2, F3, F11; O-3.6 (two certificates at one index)
Hop phases: settle time and overshoot per step	hop.md in the summary	spec 02 section 2.3's simulator claims (x50 settled 62 s, /50 657 s) on a real network with real timestamps

Caveats, stated once: CPU hash rate only (no GPU in this network), one evening of data, clocks by chrony, 20 nodes not 1,000. The numbers are the first real ones; they do not close gate 3 by themselves.

Failure notes

• create.sh refuses to run without an active hcloud context or doctl auth. It never creates a cloud account.
• A node whose database predates a binary change must be wiped: ssh root@<ip> 'systemctl stop igneumd; rm -rf /var/lib/igneum/*; systemctl start igneumd'.
• If provision.sh build dies on the builder, the log is build/build.log; KEEP_BUILDER=1 keeps the VM for a retry.
• partition.sh heal removes the iptables rules if a run was interrupted.
• destroy.sh is the only thing that stops the bill. Check the provider console afterwards.