diff --git a/docs/plans/cloud-devnet.md b/docs/plans/cloud-devnet.md new file mode 100644 index 000000000..2287e9203 --- /dev/null +++ b/docs/plans/cloud-devnet.md @@ -0,0 +1,70 @@ +# Cloud devnet and rented-GPU benchmarks: plan + +3 October 2026. Scripts in `infra/cloud-devnet/` and `infra/gpu-bench/`. Nothing in either directory spends until +the project lead answers "yes" to one command; the seed node (`docs/plans/seed-nodes.md`) is the only thing live tonight. + +## Experiment 1: a 20-node Igneum devnet across regions + +What: 20 small VMs (Hetzner Cloud, 4 each in Falkenstein, Helsinki, Ashburn, Hillsboro, Singapore) running `igneumd` +on a private network (`igneum-devnet-20`, own genesis, 2^18 hashes per block) with one CPU trickle miner and one BLS +vote key per node, wired as a sparse mesh of about 4 peers each. Three measurements: inter-region RTT and block +propagation delay per node; a region cut off with iptables for 10 minutes and healed, recording the reorg depth, +the heal time and whether any checkpoint locked with two hashes; and a schedule of hash-rate steps (4x on half the +nodes, a region off) for the difficulty controller. + +What it proves: + +| Measurement | Closes or informs | +|---|---| +| Reorg depth distribution under real latency, and the partition's reorg depth and heal time | gate 3: the checkpoint determination depth d (spec 03 C1, placeholder 60, devnet 20, ledger F7) is set from exactly this distribution; the floor rule (lock needs 56.7% of total weight) is tested against a real partition instead of the checkpoint-level simulator (ledger F2, F3, F11; O-3.6 two certificates at one index) | +| Block propagation p50/p90/p99 between regions | the 5 s network delay bound behind GHOSTDAG k at 1 BPS; spec 03's lock latency (simulated 2.5 s median at a 2 s inter-region delay) | +| Settle time and overshoot per hash-rate step | spec 02 section 2.3: the dual-lane controller's simulator numbers (x50 settled in 62 s, /50 in 657 s) on a real network with real timestamps; on the master tree it measures Kaspa's sampled DAA instead, so build from the `difficulty` worktree for this one | + +What it does not prove: nothing about GPU hash rates (CPU miners), nothing at 1,000 miners, one evening of data. + +Cost (Hetzner API prices of 3 Oct 2026, net): EUR 0.76 per hour for the 20 nodes at 4 per location (EUR 476 per +month if left running), or EUR 0.47 per hour EU-heavy; the builder VM about EUR 0.03 per hour for one hour. An evening +of six hours: about EUR 5. DigitalOcean variant: USD 0.71 per hour. Time to first block after "yes": about 25 minutes +(VM creation, a 10 to 25 minute build, install). + +Command sequence: `infra/cloud-devnet/README.md`, section "The command sequence". In short: `create.sh`, +`provision.sh`, `start.sh`, then `experiments/latency.sh 10`, `experiments/partition.sh sin 10`, `experiments/hop.sh`, +`experiments/collect.sh`, then `stop.sh` and `destroy.sh`. + +## Experiment 2: rented GPUs + +What: one RunPod pod per card (RTX 3060 if a host offers one, RTX 3090, RTX 4090, RTX 5090; an AMD RX 7900 XTX is not +offered by RunPod and Vast.ai lists no AMD consumer cards in general, approximate, so the ROCm recipe is shipped and +waits for an AMD box). Each pod gets the bundle (`proto-cuda/host.cu`, the three packs, `proto-opencl/host.c`, the +bench scripts), builds the worker with `--serve` support, and runs: the vectors gate, a 10-minute raw bench at 1 GiB +on the memory-hard pack, the dataset sweep 64 MiB to 1 GiB (the L2 cliff), the inline-dataset shortcut kernel against +the honest one (the ratio), and the hourly-recompile timing (nvcc to cubin as the worker's `prepare` path does it, +NVRTC in process, and the OpenCL compiler where an ICD is present). It prints one results row, appends it to +`results.md`, and uploads the row and the log to the intake (`/api/log`, the Windows package's key). + +What it proves: + +| Measurement | Closes or informs | +|---|---| +| Mhash/s at 1 GiB per card, 10 minutes sustained | the per-card hash rate table the litepaper and the miner community need; the first numbers for 3060, 3090 and 4090 (only the 5090 and the M5 Max are measured) | +| The sweep's cliff between the card's L2 and 1 GiB | ledger M1 ("the program space is tiny"): the defence is random reads over a dataset larger than any on-chip cache; the cliff per card is the measurement behind the "under 2x for a chip" target | +| Inline shortcut over honest rate | ledger M16 (a 256 MiB cache on a die): the recompute attacker's rate on three more cards than the M5 Max's 0.21; the 64 MiB-cache variant inside the 5090's L2 needs a pack exported with a smaller cache and is a follow-up | +| nvcc, NVRTC and OpenCL compile times per card | ledger M11 (hourly JIT on real rigs) and M17: whether the hourly program change costs milliseconds or seconds on real NVIDIA drivers, in process and out of process | + +Cost (RunPod pricing page, 3 Oct 2026, USD per hour): RTX 3090 0.22 community / 0.50 secure, RTX 4090 0.34 / 0.74, +RTX 5090 0.69 / 0.99; the 3060 is not on RunPod's table (Vast.ai about 0.05 to 0.10, approximate). One run is about +45 minutes, so the four NVIDIA cards cost about USD 2 on community cloud or USD 3.50 on secure cloud, plus a few +cents of storage. Per-minute billing. + +Command sequence: `infra/gpu-bench/README.md`. In short: `make-bundle.sh` on the Mac, start a pod from the image +recipe with SSH, `scp` the bundle, `./run.sh`, read `results.md`, paste the row into `docs/bench-log.md` with the +template. + +## The go/no-go question for the project lead + +Spend about EUR 5 and USD 2 to 4 tonight-to-tomorrow for (a) the first reorg-depth, propagation and partition numbers +from a real multi-region network, which gate 3 needs and the simulator cannot give, and (b) the per-card hash rates +and the three ASIC-resistance measurements (L2 cliff, inline ratio, recompile time) on 3060, 3090 and 4090, closing +the measurement half of ledger items M1, M11 and M16 for those cards? If yes: `cd infra/cloud-devnet && ./create.sh` +and `cd infra/gpu-bench && ./make-bundle.sh` are the two starting commands. If no: the scripts keep, nothing bills, +and the seed node stays at EUR 6.49 per month. diff --git a/docs/plans/seed-nodes.md b/docs/plans/seed-nodes.md new file mode 100644 index 000000000..98dae6212 --- /dev/null +++ b/docs/plans/seed-nodes.md @@ -0,0 +1,84 @@ +# Seed nodes: plan + +3 October 2026. Scripts in `infra/seed-nodes/`. The first seed is live. + +## The first seed + +| Item | Value | +|---|---| +| Name | igneum-seed-1 | +| Address | `188.245.5.161:26611` (Hetzner primary IPv4, auto-delete off, so a rebuilt server keeps it) | +| Provider, location, type | Hetzner Cloud, Falkenstein (fsn1), cx23 (2 Intel vCPU, 4 GB, 40 GB), Debian 12 | +| Cost | EUR 6.49 per month net, EUR 7.79 gross, EUR 0.0104 per hour, 20 TB traffic included (Hetzner API, 3 Oct 2026); plus the primary IPv4, about EUR 0.50 per month (approximate) | +| Firewall | inbound tcp 26611 from anywhere, tcp 22 from anywhere (the project lead's instruction; `SSH_SOURCE=me` narrows it to this Mac's IP), icmp; RPC bound to 127.0.0.1 only | +| Network | the shared devnet (`--devnet`, no suffix), built from `vendor/igneum-node` HEAD d62708a8 plus the uncommitted finality v2 work and `igneum-pow` HEAD, with `--features igneum-pow` | +| Role | p2p open, no mining, address manager on (serves `RequestAddresses`), `--nodnsseed`, UPnP off, `--externalip` set | +| How the Mac reaches it | the Mac's live node sits behind NAT (192.168.68.64), so the seed cannot dial in; the Mac dials out: `infra/seed-nodes/addpeer-from-mac.sh 188.245.5.161` (the `addPeer` RPC through grpcurl, permanent, no restart). If the live node is ever restarted by hand: `--addpeer=188.245.5.161:26611` | + +`infra/seed-nodes/seeds.txt` holds exactly `188.245.5.161:26611`. + +## How the seed list reaches clients + +1. Baked into the node. `vendor/igneum-node/consensus/core/src/config/params.rs` holds the per-network seed list as + `dns_seeders: &'static [&'static str]` on `Params`: `MAINNET_PARAMS` (line 617 on 3 Oct 2026), `TESTNET_PARAMS` + (670), `SIMNET_PARAMS` (721), `DEVNET_PARAMS` (785), all `&[]` since the rename commit emptied Kaspa's nine mainnet + and three testnet hostnames. The connection manager resolves each entry with `(seeder, default_p2p_port).to_socket_addrs()` + (`components/connectionmanager/src/lib.rs`, `dns_seed_single`), so a plain IPv4 literal works as an entry with no + DNS at all: `dns_seeders: &["188.245.5.161"]` on `DEVNET_PARAMS` is the whole change for the devnet, and the + testnet list is the same shape with the testnet seeds. The port is the network's default p2p port (devnet 26611; + the testnet port is still Kaspa's and must be set with the testnet genesis). The consensus engineer owns this edit. + Two consequences for packages: a client that passes `--nodnsseed` ignores the baked list (`kaspad/src/daemon.rs` + line 573: `dns_seeders` is emptied when `--nodnsseed` or `--connect` is given), so the Windows node package + (`proto-cuda/windows-node/start-node.ps1`) and the cloud scripts must drop `--nodnsseed` once the list is baked; + and the list is consulted only when the node is short of outbound peers, so a node with enough `--addpeer` + entries never asks a seed. +2. `SEED_PEERS` override in every package. Each launcher (Windows node, cloud devnet, seed nodes, the observer's + helper node) reads `SEED_PEERS` (comma-separated `ip:port`) and turns every entry into `--addpeer=`; the + baked list is the default when the variable is empty. This is what an operator uses when the baked list is stale + between releases. +3. DNS names only as a convenience. `seed1.igneum.network` and so on can point at the same addresses (the domains + are on Vercel nameservers, so a record each), and `dns_seeders` accepts a hostname too; but the IPs are the source + of truth because a DNS failure or a registrar problem must not stop bootstrapping, and because the public key of + nothing is involved: a seed only hands out addresses, it cannot forge blocks. + +## Public testnet seed set + +Three to five seeds across two providers and three regions. Proposed: + +| Seed | Provider | Location | Type | Per month net | +|---|---|---|---|---| +| igneum-seed-1 | Hetzner | Falkenstein (EU) | cx23 | EUR 6.49 (live) | +| igneum-seed-2 | Hetzner | Ashburn (US east) | cpx11 (2 GB) or cpx21 (4 GB) | EUR 20.49 or 37.49 | +| igneum-seed-3 | DigitalOcean | Singapore (sgp1) | s-2vcpu-4gb | USD 24 (DO pricing page) | +| igneum-seed-4 (optional) | DigitalOcean | New York or Frankfurt | s-2vcpu-4gb | USD 24 | +| igneum-seed-5 (optional) | Hetzner | Helsinki | cx23 | EUR 6.49 | + +Three seeds: about EUR 50 per month; five: about EUR 80 (approximate, mixed currencies). The US seed is the expensive +one because Hetzner's current cx line is EU-only. Every seed is created with `create-seed.sh` (the DigitalOcean +variant reserves an IP in the same way) and provisioned with `provision-seed.sh`, which adds the seeds already in +`seeds.txt` as `--addpeer` entries so the seeds form a full mesh among themselves. `health.sh` checks them all. + +## Rotation + +1. Add before removing: create and provision the replacement, run `health.sh` until it is synced and has peers. +2. Bake the new list (`params.rs`) and release packages with it; keep the old address in the list for one release so + clients on the previous build still bootstrap. +3. Keep the old IP alive until the release after that (a Hetzner primary IP or a DO reserved IP costs under EUR 1 per + month unattached, approximate), then delete the server and the IP, and remove the entry from `seeds.txt` and + `seeds.tsv`. +4. A compromised seed is the one case to remove first: delete the server, release the IP, bake and release the same + day. The damage a bad seed can do is bounded (it hands out addresses; the node's handshake and PoW checks are + unchanged), which is why the list may sit in a release rather than behind a signature. + +## Health and operations + +`health.sh` (one line per seed: p2p port reachable from the Mac, unit active, RPC answering, synced, blocks and +headers, connected peers, known and banned addresses, disk, memory, version); `--watch` repeats every minute. The +seed's journal: `ssh -i ~/.ssh/igneum_ed25519 root@188.245.5.161 journalctl -u igneumd -f`. Updating the binary: +`provision-seed.sh` again (it rebuilds on the VM) or `BUILD_WHERE=bin` to push a binary built by the cloud-devnet +builder. The database format changes with some fork commits; a seed that refuses to start after an update is wiped +(`rm -rf /var/lib/igneum/*`) and resyncs from its peers. + +## No-spend rule + +Only the first seed spends tonight (approved). The remaining seeds and the 20-node network wait for the morning. diff --git a/infra/cloud-devnet/.gitignore b/infra/cloud-devnet/.gitignore new file mode 100644 index 000000000..15fbe349b --- /dev/null +++ b/infra/cloud-devnet/.gitignore @@ -0,0 +1,4 @@ +# build artefacts and runtime state; results/ is committed on purpose +build/ +nodes.tsv +__pycache__/ diff --git a/infra/cloud-devnet/README.md b/infra/cloud-devnet/README.md new file mode 100644 index 000000000..67ac93a35 --- /dev/null +++ b/infra/cloud-devnet/README.md @@ -0,0 +1,120 @@ +# Igneum cloud devnet: 20 nodes across regions + +A private Igneum devnet (`igneum-devnet-20`: own handshake magic, own genesis) on 20 small Linux VMs in five +locations, each running `igneumd` with the real lottery hash and one CPU trickle miner with its own BLS vote key. +Built for two measurements that the single-site devnet cannot give: block propagation and reorg behaviour under +real inter-region latency (gate 3, spec 03 C1 and the floor rule), and the difficulty controller under hash-rate +steps (spec 02 section 2.3). Everything here is scripts; nothing runs or spends until `create.sh` is answered "yes". + +Hetzner Cloud through `hcloud` is the primary path. DigitalOcean through `doctl` is the variant for `create.sh` +(and `destroy.sh`); the rest is provider-neutral once `nodes.tsv` exists. + +## The command sequence (the project lead approves the cost, then this, start to finish) + +``` +cd infra/cloud-devnet +brew install hcloud # once; doctl for the DigitalOcean variant +# the API token is read from ~/.config/igneum/hetzner-token (the seed-node scripts use the same file) +./create.sh # 1. shows the plan and the live prices, asks "yes", creates 20 VMs, writes nodes.tsv +./provision.sh # 2. source tarball -> builder VM -> igneumd + igneum-miner -> every node (about 20 min) +./start.sh # 3. nodes, block logs, then the 20 trickle miners +./status.sh # one line per node (blocks, blue, DAA, tips, peers, difficulty, sink, miner) +./experiments/latency.sh 10 # 4. RTT matrix, then 10 min of block propagation samples +./experiments/partition.sh sin 10 # 5. cut Singapore off for 10 min, heal, reorg depth and heal time +./experiments/hop.sh # 6. hash-rate steps for the controller (about 75 min, schedule in the script) +./experiments/collect.sh # 7. pull logs and RPC samples -> results//, summary.md, bench-log entry +./stop.sh && ./destroy.sh # 8. stop, then delete every VM (hourly billing ends) +``` + +Time from "yes" to the first block: VM creation 1 min, build on the builder 10 to 25 min (approximate; 500 crates +plus rocksdb), install 2 min, miners' first 256 MiB cache 10 to 30 s. Steps 4 to 6 are independent; 4 and 6 can run +at the same time, 5 should run alone. + +The observer and the live page (`tools/observer`): the RPC never leaves a node's loopback, so either +`./experiments/observer.sh tunnel` (an ssh tunnel from the Mac; then run the observer here) or +`./experiments/observer.sh remote on` (installs Node 22 on node 1 and runs the observer there, keeping the Mac out of +it). With `LIVE_TABLE_PREFIX=cloud_` the record goes to `cloud_live_*` tables; without the prefix the cloud network takes +over `/live` on the site (stop the Mac's observer first). `site/api/live.mjs` reads the unprefixed tables only. + +## Cost + +Hetzner API prices for this project on 3 Oct 2026, net EUR per month, hourly billing (the API's `price_hourly`): + +| Location | Type | vCPU | RAM | Per month | Per hour | Note | +|---|---|---|---|---|---|---| +| fsn1, nbg1, hel1 | cx23 | 2 Intel | 4 GB | 6.49 | 0.0104 | the cheapest x86 type the project can create | +| sin | cpx22 | 2 AMD | 4 GB | 30.99 | 0.0497 | cx not sold outside the EU | +| ash, hil | cpx21 | 3 AMD | 4 GB | 37.49 | 0.0601 | the old cpx line, only still sold in the US; cpx11 (2 GB) is 20.49 | +| builder, any EU | cx43 | 8 Intel | 16 GB | about 20 | about 0.03 | deleted after the build (approximate: cx33 is 9.99, cx43 not priced in the check) | + +Plus about EUR 0.50 per month per primary IPv4 (approximate, Hetzner list price). VAT is added for a private +customer; the API's gross column is net x 1.2. + +| Mix | Nodes | Per month net | Per hour | An evening (6 h) | +|---|---|---|---|---| +| 4 per location (default `REGIONS`) | 8 EU, 4 sin, 4 ash, 4 hil | EUR 476 | EUR 0.76 | about EUR 5 | +| EU-heavy (`REGIONS=hel1,fsn1,nbg1,hel1,fsn1,nbg1,hel1,ash,hil,sin`) | 14 EU, 2 sin, 2 ash, 2 hil | EUR 303 | EUR 0.47 | about EUR 3 | +| DigitalOcean `s-2vcpu-4gb` in any region (DO pricing page, USD 24/mo, 0.036/h) | 20 | USD 480 | USD 0.71 | about USD 4.50 | + +The run is meant to last one evening, so the hourly column is the one that matters: well under EUR 10 all in, with +the builder. Leaving the network up costs the monthly column. + +## What each script does + +| Script | What | +|---|---| +| `config.sh` | every setting (provider, N, regions, types per location, suffix, genesis bits, mesh degree, miner threads, source tree) | +| `create.sh` | ssh key, firewall (22, 26611, icmp), N servers round-robin over `REGIONS`, writes `nodes.tsv` (name, index, region, ip) | +| `make-source.sh` | `git archive HEAD` of `NODE_SRC` and `igneum-pow` plus the uncommitted files (`SRC_MODE=head+dirty`, the default: on 3 Oct 2026 every worktree's branch work is uncommitted) into `build/src.tar.gz`, same layout as the Windows package's `src.zip` | +| `provision.sh` | `build`: builder VM, `builder/build-on-builder.sh` (apt deps, rustup, `cargo build --release -p kaspad -p igneum-miner --features igneum-pow`), binaries to `build/bin/`, builder deleted. `install`: binaries, `node/wrpc.py`, units and `/etc/igneum/node.env` on every node | +| `node/run-igneumd.sh` | the flag list: `--devnet --devnet-suffix=20 --override-params-file (genesis_bits) --rpclisten=127.0.0.1:26610 --rpclisten-json=127.0.0.1:28610 --listen=0.0.0.0:26611 --externalip --addpeer x2 --outpeers=0 --maxinpeers=32 --nodnsseed --disable-upnp --nologfiles --enable-unsynced-mining` | +| `node/igneum-miner.service` | `igneum-miner mine grpc://127.0.0.1:26610 1 1000000000 --engine igneum-pow --payout-label `: one CPU thread, its own vote key, votes on every checkpoint | +| `node/wrpc.py` | standard-library wRPC JSON client (the node's WebSocket RPC): `call`, `sample`, `watch-blocks` (blocks.tsv, chain.tsv, samples.tsv under /var/log/igneum) | +| `start.sh`, `stop.sh`, `status.sh`, `destroy.sh` | as named; `destroy.sh` deletes everything labelled `igneum=devnet` and the firewall | +| `experiments/latency.sh` | ping matrix between all nodes, then block propagation from the per-node arrival logs joined on hash (p50, p90, p99 per node and per region) | +| `experiments/partition.sh` | iptables cut of one region's p2p traffic to the rest, heal after N min, reorg depth (`getVirtualChainFromBlock` from each node's sink at heal, plus the largest `virtualChainChanged` removal), heal time (sinks converge), conflicting locks (`getFinalityCheckpoints` on both sides) | +| `experiments/hop.sh` | a schedule of miner-thread changes per node set (step up, step down, a region off), logged for the controller analysis | +| `experiments/collect.sh` | journals, block/chain/sample logs, RPC snapshots into `results//nodes//`; `analyze.py summary` and a bench-log entry template; `APPEND_BENCH_LOG=1` appends it to `docs/bench-log.md` | +| `experiments/analyze.py` | the offline analysis (RTT table, propagation join, lock conflicts, hop phases, reorg-depth histogram) | +| `experiments/observer.sh` | the observer hookup (tunnel or remote) | + +## Network design choices + +- Own network id. `--devnet-suffix=20` gives `igneum-devnet-20` (own handshake magic, own data directory) and the + override file sets `genesis_bits` (0x1e400000, 2^18 hashes per block: three 6-thread CPU miners on the M5 Max held + about 1 block/s at this value, fork-divergence genesis row), which recomputes the genesis hash. A node of this + network can never complete a handshake with the live devnet or the seed. +- Sparse mesh. Each node has two permanent outbound links (`--addpeer` to its ring neighbour and to the node half + way round) and `--outpeers=0`, so the connection manager does not fill its default 8 outbound slots from + exchanged addresses. Inbound links double the count: about 4 peers per node, 40 links in total. Blocks therefore + cross several hops between regions, which is what the propagation measurement needs. +- `--addpeer`, never `--connect`: `--connect` sets the inbound limit to 0 (kaspad/src/daemon.rs), the Windows node + lesson. +- RPC on loopback. Every measurement goes through ssh to `wrpc.py` on the node. The firewall opens 22, 26611 and icmp. +- One vote key per node. The miner's identity label is the node name, so weight accrues to 20 keys and the + finality rule has 20 voters; the devnet finality parameters (interval 30, dust 5, presence 20) apply. +- Which tree runs. `NODE_SRC` defaults to `vendor/igneum-node` (master worktree: finality v2 work, uncommitted). For + the controller experiment point it at `vendor/igneum-node-diff` (the `difficulty` worktree) or at a merged tree; the + binary is cached in `build/bin` and `REBUILD=1` rebuilds. `build/src.stamp` records what went in. +- Node 22 is not needed on the nodes: `wrpc.py` is standard-library Python. The observer (remote mode) is the one + thing that installs Node, on one node, only when asked. + +## What the results answer + +| Measurement | Where it lands | Decides | +|---|---|---| +| RTT between regions, block propagation p50/p90/p99 | `results//latency/` | the latency assumption behind GHOSTDAG k (5 s network delay bound at 1 BPS) and spec 03 C1's lock latency (simulated 2.5 s median at a 2 s inter-region delay) | +| Reorg depth distribution over the run | `summary.md` | checkpoint determination depth d (spec 03 C1, ledger F7: placeholder 60, devnet 20) | +| Partition: reorg depth, heal time, conflicting locks | `results//partition-*/partition.md` | the floor rule (lock needs 56.7% of total weight) under a real partition; ledger F2, F3, F11; O-3.6 (two certificates at one index) | +| Hop phases: settle time and overshoot per step | `hop.md` in the summary | spec 02 section 2.3's simulator claims (x50 settled 62 s, /50 657 s) on a real network with real timestamps | + +Caveats, stated once: CPU hash rate only (no GPU in this network), one evening of data, clocks by chrony, 20 nodes +not 1,000. The numbers are the first real ones; they do not close gate 3 by themselves. + +## Failure notes + +- `create.sh` refuses to run without an active `hcloud` context or `doctl` auth. It never creates a cloud account. +- A node whose database predates a binary change must be wiped: `ssh root@ 'systemctl stop igneumd; rm -rf /var/lib/igneum/*; systemctl start igneumd'`. +- If `provision.sh build` dies on the builder, the log is `build/build.log`; `KEEP_BUILDER=1` keeps the VM for a retry. +- `partition.sh heal` removes the iptables rules if a run was interrupted. +- `destroy.sh` is the only thing that stops the bill. Check the provider console afterwards. diff --git a/infra/cloud-devnet/builder/build-on-builder.sh b/infra/cloud-devnet/builder/build-on-builder.sh new file mode 100755 index 000000000..3c8e2d219 --- /dev/null +++ b/infra/cloud-devnet/builder/build-on-builder.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Runs ON the builder VM (Debian 12 or Ubuntu 22.04 and newer, x86_64) as root. Installs the toolchain, builds +# igneumd and igneum-miner with the real lottery hash (--features igneum-pow), leaves the binaries in /root/out/. +# Input: /root/src.tar.gz from make-source.sh. Idempotent; a second run only rebuilds what changed. +set -euo pipefail + +export DEBIAN_FRONTEND=noninteractive +log() { printf '%s %s\n' "$(date -u +%H:%M:%S)" "$*"; } + +if ! command -v cargo >/dev/null 2>&1 && [ ! -x "$HOME/.cargo/bin/cargo" ]; then + log "installing build dependencies" + apt-get update -qq + # rocksdb (librocksdb-sys) needs clang + libclang for bindgen and a C++ compiler; the gRPC and p2p crates run protoc. + apt-get install -y -qq build-essential clang libclang-dev llvm-dev pkg-config libssl-dev protobuf-compiler libprotobuf-dev curl git >/dev/null + log "installing rust (stable) with rustup" + curl -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable >/dev/null +fi +# shellcheck disable=SC1091 +. "$HOME/.cargo/env" + +cd /root +if [ -f src.tar.gz ]; then + rm -rf src; tar -xzf src.tar.gz; log "unpacked src.tar.gz" +fi +[ -d src/vendor/igneum-node ] || { echo "no src/vendor/igneum-node"; exit 1; } +cd src/vendor/igneum-node + +log "building igneumd and igneum-miner (release, --features igneum-pow); about 500 crates, 10 to 25 minutes on 8 vCPU (approximate)" +start=$(date +%s) +( while sleep 60; do printf '%s build running, %s s\n' "$(date -u +%H:%M:%S)" "$(( $(date +%s) - start ))"; done ) & ticker=$! +trap 'kill $ticker 2>/dev/null || true' EXIT +cargo build --release -p kaspad -p igneum-miner --features igneum-pow 2>&1 | tail -n 30 +kill $ticker 2>/dev/null || true +log "build finished in $(( $(date +%s) - start )) s" + +mkdir -p /root/out +cp target/release/igneumd target/release/igneum-miner /root/out/ +strip /root/out/igneumd /root/out/igneum-miner 2>/dev/null || true +/root/out/igneumd --version | head -1 | tee /root/out/version.txt +git -C /root/src/vendor/igneum-node rev-parse --short HEAD >> /root/out/version.txt 2>/dev/null || true +ls -la /root/out diff --git a/infra/cloud-devnet/config.sh b/infra/cloud-devnet/config.sh new file mode 100755 index 000000000..34d6fb08c --- /dev/null +++ b/infra/cloud-devnet/config.sh @@ -0,0 +1,55 @@ +# Igneum cloud devnet: settings. Sourced by every script in this directory through lib/common.sh. +# Override any value in the environment, for example `N=10 PROVIDER=digitalocean ./create.sh`. +# Nothing here spends money by itself; create.sh asks before it creates anything. + +PROVIDER="${PROVIDER:-hetzner}" # hetzner (hcloud CLI) or digitalocean (doctl CLI) +N="${N:-20}" # node count +PREFIX="${PREFIX:-igneum}" # server names igneum-01 .. igneum-20, the builder is igneum-builder + +# Regions, round-robin over the node index. Hetzner locations: hel1 Helsinki, fsn1 Falkenstein, ash Ashburn (US east), +# hil Hillsboro (US west), sin Singapore. With 20 nodes that is 4 per location: 8 in the EU, 4 US east, 4 US west, 4 Asia. +REGIONS="${REGIONS:-hel1,fsn1,ash,hil,sin}" +DO_REGIONS="${DO_REGIONS:-lon1,fra1,nyc3,sfo3,sgp1}" + +# Instance size per location. The node holds up to four 256 MiB lottery caches (M15 cap), rocksdb and the CPU +# miner's own 256 MiB cache, so 2 GB plans are too small. What this Hetzner project can create (API, 3 Oct 2026, net +# EUR per month): cx23 (2 Intel vCPU, 4 GB) 6.49 in fsn1/nbg1/hel1 only; cpx22 (2 AMD vCPU, 4 GB) 22.99 EU, 30.99 sin; +# cpx21 (3 vCPU, 4 GB) 37.49 in ash/hil only (the old cpx line is deprecated in the EU and Singapore since Dec 2025); +# cpx11 (2 GB) 20.49 in ash/hil. So: cx23 in Europe, cpx22 in Singapore, cpx21 in the US. SERVER_TYPE overrides all. +TYPE_BY_LOCATION="${TYPE_BY_LOCATION:-fsn1=cx23,nbg1=cx23,hel1=cx23,sin=cpx22,ash=cpx21,hil=cpx21}" +SERVER_TYPE="${SERVER_TYPE:-}" # empty = pick from TYPE_BY_LOCATION +DO_SIZE="${DO_SIZE:-s-2vcpu-4gb}" +BUILDER_TYPE="${BUILDER_TYPE:-cx43}" # 8 Intel vCPU, 16 GB, EUR 0.03/h approximate, for the one-off node build (deleted afterwards) +DO_BUILDER_SIZE="${DO_BUILDER_SIZE:-c-8}" +IMAGE="${IMAGE:-debian-12}" +DO_IMAGE="${DO_IMAGE:-debian-12-x64}" + +# SSH. The operations key ~/.ssh/igneum_ed25519 (3 Oct 2026, uploaded to Hetzner as "igneum-ops" by the seed-node +# scripts). If SSH_KEY_FILE is missing, create.sh generates an ed25519 pair there and uploads the public half. +SSH_KEY_NAME="${SSH_KEY_NAME:-igneum-ops}" +SSH_KEY_FILE="${SSH_KEY_FILE:-$HOME/.ssh/igneum_ed25519}" +SSH_USER="${SSH_USER:-root}" + +# The network. A suffixed devnet (igneum-devnet-20) has its own handshake magic and its own genesis (the bits +# override recomputes the genesis hash), so it can never peer with or pollute the live devnet on the Mac. +DEVNET_SUFFIX="${DEVNET_SUFFIX:-20}" +# 0x1e400000 = 2^18 expected hashes per block. Three 6-thread CPU miners on the M5 Max held about 1 block/s at this +# value (fork-divergence, genesis row); 20 one-thread cloud vCPUs are in the same range. The DAA takes over after that. +GENESIS_BITS="${GENESIS_BITS:-0x1e400000}" +MESH_OUT="${MESH_OUT:-2}" # outbound --addpeer links per node (ring plus a chord); inbound doubles it, about 4 peers each +MINER_THREADS="${MINER_THREADS:-1}" # the trickle: one CPU thread per node +MINER_STATUS_SECS="${MINER_STATUS_SECS:-60}" + +# Source to ship. NODE_SRC is the fork worktree whose code the network runs. master (vendor/igneum-node) carries the +# finality v2 work as uncommitted changes; the dual-lane difficulty controller is on the `difficulty` worktree +# (vendor/igneum-node-diff). Point NODE_SRC at the tree you want measured. +NODE_SRC="${NODE_SRC:-$REPO/vendor/igneum-node}" +POW_SRC="${POW_SRC:-$REPO/igneum-pow}" +# head = `git archive HEAD` only; head+dirty = HEAD plus every modified and untracked file of the worktree (target*/ excluded). +# All seven worktrees sit at d62708a8 with their branch work uncommitted (3 Oct 2026), so head+dirty is the default. +SRC_MODE="${SRC_MODE:-head+dirty}" + +# Ports (consensus/core/src/network.rs, devnet). RPC stays on loopback; only p2p is open to the world. +P2P_PORT=26611 +RPC_PORT=26610 +RPC_JSON_PORT=28610 diff --git a/infra/cloud-devnet/create.sh b/infra/cloud-devnet/create.sh new file mode 100755 index 000000000..e48d0db5a --- /dev/null +++ b/infra/cloud-devnet/create.sh @@ -0,0 +1,147 @@ +#!/usr/bin/env bash +# Create the cloud devnet VMs: N nodes, regions round-robin, one firewall, one SSH key. Writes nodes.tsv. +# Hetzner Cloud through `hcloud` (primary). DigitalOcean through `doctl` with PROVIDER=digitalocean. +# Spends money from the moment the servers exist (hourly billing on both providers). It prints the plan and +# the provider's live price and asks for "yes" first (YES=1 skips the question). +# +# Before the first run: `hcloud context create igneum` (paste the project's API token; the project is created by +# the project lead in the Hetzner console, not by this script) or `doctl auth init`. +# usage: ./create.sh create N nodes +# ./create.sh builder create only the builder VM (provision.sh does this itself when it needs one) + +. "$(dirname "$0")/lib/common.sh" + +what="${1:-nodes}" +mkdir -p "$BUILD_DIR" + +# ---- SSH key ------------------------------------------------------------------------------------------------- +if [ ! -f "$SSH_KEY_FILE" ]; then + log "no key at $SSH_KEY_FILE, generating an ed25519 pair (no passphrase; it only opens these test VMs)" + ssh-keygen -q -t ed25519 -N "" -C "igneum-devnet" -f "$SSH_KEY_FILE" +fi +PUBKEY_FILE="$SSH_KEY_FILE.pub" +[ -f "$PUBKEY_FILE" ] || die "missing $PUBKEY_FILE" + +# ---- Hetzner ------------------------------------------------------------------------------------------------------ +hetzner_prepare() { + need hcloud "brew install hcloud" + hcloud context active >/dev/null 2>&1 || die "no active hcloud context: hcloud context create igneum" + if ! hcloud ssh-key describe "$SSH_KEY_NAME" >/dev/null 2>&1; then + hcloud ssh-key create --name "$SSH_KEY_NAME" --public-key-from-file "$PUBKEY_FILE" >/dev/null + log "uploaded ssh key $SSH_KEY_NAME" + fi + if ! hcloud firewall describe "$PREFIX-devnet" >/dev/null 2>&1; then + hcloud firewall create --name "$PREFIX-devnet" --label igneum=devnet >/dev/null + hcloud firewall add-rule "$PREFIX-devnet" --direction in --protocol tcp --port 22 --source-ips 0.0.0.0/0 --source-ips ::/0 --description ssh >/dev/null + hcloud firewall add-rule "$PREFIX-devnet" --direction in --protocol tcp --port "$P2P_PORT" --source-ips 0.0.0.0/0 --source-ips ::/0 --description igneum-p2p >/dev/null + hcloud firewall add-rule "$PREFIX-devnet" --direction in --protocol icmp --source-ips 0.0.0.0/0 --source-ips ::/0 --description ping >/dev/null + log "created firewall $PREFIX-devnet (in: 22, $P2P_PORT, icmp; RPC never leaves loopback)" + fi +} + +hetzner_price() { + log "live price list for $1 (per location, EUR, excl. VAT):" + hcloud server-type describe "$1" 2>/dev/null | sed -n '/Pricings/,$p' | head -40 || true +} + +hetzner_create_one() { # name type location + local name="$1" type="$2" loc="$3" + if hcloud server describe "$name" >/dev/null 2>&1; then log "$name exists, keeping it"; return; fi + hcloud server create --name "$name" --type "$type" --image "$IMAGE" --location "$loc" \ + --ssh-key "$SSH_KEY_NAME" --firewall "$PREFIX-devnet" --label igneum=devnet --label role="${4:-node}" >/dev/null + log "created $name ($type, $loc)" +} + +hetzner_ip() { hcloud server ip "$1"; } + +# ---- DigitalOcean -------------------------------------------------------------------------------------------------- +do_prepare() { + need doctl "brew install doctl" + doctl account get >/dev/null 2>&1 || die "doctl is not authenticated: doctl auth init" + DO_KEY_ID=$(doctl compute ssh-key list --format ID,Name --no-header | awk -v n="$SSH_KEY_NAME" '$2 == n { print $1 }') + if [ -z "$DO_KEY_ID" ]; then + DO_KEY_ID=$(doctl compute ssh-key import "$SSH_KEY_NAME" --public-key-file "$PUBKEY_FILE" --format ID --no-header) + log "imported ssh key $SSH_KEY_NAME ($DO_KEY_ID)" + fi + DO_FW_ID=$(doctl compute firewall list --format ID,Name --no-header | awk -v n="$PREFIX-devnet" '$2 == n { print $1 }') + if [ -z "$DO_FW_ID" ]; then + DO_FW_ID=$(doctl compute firewall create --name "$PREFIX-devnet" --tag-names "$PREFIX-devnet" \ + --inbound-rules "protocol:tcp,ports:22,address:0.0.0.0/0,address:::/0 protocol:tcp,ports:$P2P_PORT,address:0.0.0.0/0,address:::/0 protocol:icmp,address:0.0.0.0/0,address:::/0" \ + --outbound-rules "protocol:tcp,ports:all,address:0.0.0.0/0,address:::/0 protocol:udp,ports:all,address:0.0.0.0/0,address:::/0 protocol:icmp,address:0.0.0.0/0,address:::/0" \ + --format ID --no-header) + log "created firewall $PREFIX-devnet ($DO_FW_ID), applied by tag" + fi +} + +do_price() { log "live price list for $1 (USD):"; doctl compute size list --format Slug,Memory,VCPUs,Disk,PriceMonthly,PriceHourly | grep -E "^Slug|^$1 " || true; } + +do_create_one() { # name size region + local name="$1" size="$2" reg="$3" + if doctl compute droplet get "$name" >/dev/null 2>&1; then log "$name exists, keeping it"; return; fi + doctl compute droplet create "$name" --size "$size" --image "$DO_IMAGE" --region "$reg" --ssh-keys "$DO_KEY_ID" \ + --tag-names "$PREFIX-devnet,role:${4:-node}" --wait >/dev/null + log "created $name ($size, $reg)" +} + +do_ip() { doctl compute droplet get "$1" --format PublicIPv4 --no-header; } + +# ---- plan and confirm -------------------------------------------------------------------------------------------- +if [ "$PROVIDER" = digitalocean ]; then + do_prepare; TYPE="$DO_SIZE"; BTYPE="$DO_BUILDER_SIZE" +else + hetzner_prepare; TYPE="${SERVER_TYPE:-by-location}"; BTYPE="$BUILDER_TYPE" +fi + +if [ "$what" = builder ]; then + log "plan: 1 builder VM $BTYPE in $(region_of 1) on $PROVIDER (deleted by provision.sh after the build unless KEEP_BUILDER=1)" + if [ "$PROVIDER" = digitalocean ]; then do_price "$BTYPE"; else hetzner_price "$BTYPE"; fi + confirm "create the builder now (hourly billing starts)?" + if [ "$PROVIDER" = digitalocean ]; then do_create_one "$PREFIX-builder" "$BTYPE" "$(region_of 1)" builder; ip=$(do_ip "$PREFIX-builder") + else hetzner_create_one "$PREFIX-builder" "$BTYPE" "$(region_of 1)" builder; ip=$(hetzner_ip "$PREFIX-builder"); fi + printf '%s\n' "$ip" > "$BUILD_DIR/builder.ip" + log "builder $ip (saved to build/builder.ip)" + exit 0 +fi + +log "plan: $N nodes ($IMAGE) on $PROVIDER, regions round-robin:" +for i in $(seq 1 "$N"); do + reg=$(region_of "$i"); t="$TYPE"; [ "$PROVIDER" = digitalocean ] || t=$(type_for_location "$reg") + printf ' %s %s %s\n' "$(node_name "$i")" "$reg" "$t" +done +if [ "$PROVIDER" = digitalocean ]; then do_price "$TYPE"; else + for t in $(for i in $(seq 1 "$N"); do type_for_location "$(region_of "$i")"; done | sort -u); do hetzner_price "$t"; done +fi +log "cost at the Hetzner API prices of 3 Oct 2026 (net EUR per month: cx23 6.49 EU, cpx22 30.99 sin, cpx21 37.49 ash/hil):" +log " 20 nodes as 4 per location = 8 x 6.49 + 4 x 30.99 + 8 x 37.49 = EUR 476/mo, EUR 0.76/h; an evening of 6 h about EUR 5 plus the builder (about EUR 0.03/h)" +log " EU-heavy alternative REGIONS=hel1,fsn1,nbg1,hel1,fsn1,nbg1,hel1,ash,hil,sin (14 EU, 2 each elsewhere): about EUR 303/mo, EUR 0.47/h" +log " DigitalOcean s-2vcpu-4gb: USD 24 per node per month (USD 0.036/h, DO pricing page): 20 nodes USD 480/mo, USD 0.71/h" +confirm "create $N servers now (hourly billing starts)?" + +: > "$NODES_FILE.tmp" +for i in $(seq 1 "$N"); do + name=$(node_name "$i"); reg=$(region_of "$i") + if [ "$PROVIDER" = digitalocean ]; then do_create_one "$name" "$TYPE" "$reg"; else hetzner_create_one "$name" "$(type_for_location "$reg")" "$reg"; fi +done +log "waiting 20 s for the servers to boot, then collecting addresses" +sleep 20 +for i in $(seq 1 "$N"); do + name=$(node_name "$i"); reg=$(region_of "$i") + if [ "$PROVIDER" = digitalocean ]; then ip=$(do_ip "$name"); else ip=$(hetzner_ip "$name"); fi + printf '%s\t%s\t%s\t%s\n' "$name" "$i" "$reg" "$ip" >> "$NODES_FILE.tmp" +done +mv "$NODES_FILE.tmp" "$NODES_FILE" +cp "$NODES_FILE" "$BUILD_DIR/nodes-$(date -u +%Y%m%d-%H%M%S).tsv" +log "wrote $NODES_FILE:" +cat "$NODES_FILE" + +log "checking ssh on every node (cloud-init can take a minute)" +for try in 1 2 3 4 5 6; do + bad=0 + while IFS=$'\t' read -r name idx reg ip; do + nssh "$ip" true >/dev/null 2>&1 || { bad=$((bad + 1)); } + done < "$NODES_FILE" + [ "$bad" = 0 ] && break + log "$bad nodes not reachable yet (try $try), waiting 15 s"; sleep 15 +done +[ "$bad" = 0 ] || log "WARNING: $bad nodes still unreachable over ssh; provision.sh will retry them" +log "done. Next: ./provision.sh" diff --git a/infra/cloud-devnet/destroy.sh b/infra/cloud-devnet/destroy.sh new file mode 100755 index 000000000..1716718bd --- /dev/null +++ b/infra/cloud-devnet/destroy.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Delete every VM of the devnet (nodes and builder) and the firewall. The SSH key stays on the provider (free). +# Run experiments/collect.sh first: the logs on the VMs go with them. +. "$(dirname "$0")/lib/common.sh" + +if [ "$PROVIDER" = digitalocean ]; then + need doctl "brew install doctl" + ids=$(doctl compute droplet list --tag-name "$PREFIX-devnet" --format ID,Name --no-header) + [ -n "$ids" ] || log "no droplets tagged $PREFIX-devnet" + printf '%s\n' "$ids" + confirm "delete these droplets and the firewall $PREFIX-devnet?" + [ -n "$ids" ] && doctl compute droplet delete -f --tag-name "$PREFIX-devnet" + doctl compute droplet delete -f "$PREFIX-builder" 2>/dev/null || true + fw=$(doctl compute firewall list --format ID,Name --no-header | awk -v n="$PREFIX-devnet" '$2 == n { print $1 }') + [ -n "$fw" ] && doctl compute firewall delete -f "$fw" +else + need hcloud "brew install hcloud" + hcloud server list -l igneum=devnet + confirm "delete every server labelled igneum=devnet and the firewall $PREFIX-devnet?" + names=$(hcloud server list -l igneum=devnet -o columns=name -o noheader) + for s in $names; do hcloud server delete "$s" >/dev/null && log "deleted $s"; done + hcloud firewall delete "$PREFIX-devnet" >/dev/null 2>&1 && log "deleted firewall $PREFIX-devnet" || true +fi +if [ -f "$NODES_FILE" ]; then + d=$(results_dir_for); cp "$NODES_FILE" "$d/nodes-destroyed-$(date -u +%H%M%S).tsv"; rm -f "$NODES_FILE" "$BUILD_DIR/builder.ip" +fi +printf '%s\tdestroy\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$(results_dir_for)/events.log" +log "done. Check the provider console once: nothing labelled igneum should remain." diff --git a/infra/cloud-devnet/experiments/analyze.py b/infra/cloud-devnet/experiments/analyze.py new file mode 100644 index 000000000..c1211738e --- /dev/null +++ b/infra/cloud-devnet/experiments/analyze.py @@ -0,0 +1,308 @@ +#!/usr/bin/env python3 +"""Offline analysis of a results// directory (runs on the Mac, standard library only). + + analyze.py rtt region-pair RTT table from /rtt.tsv + analyze.py propagation per-node and per-region block propagation delay from + /nodes//blocks.tsv (window t0..t1) + analyze.py locks conflicting locks between the checkpoint dumps at heal + analyze.py hop block rate and difficulty per hop.log phase (node 1 samples) + analyze.py summary summary.md of everything present under + analyze.py bench-entry a docs/bench-log.md entry (template with the numbers filled) +""" +import glob, json, os, statistics, sys, time +from collections import defaultdict + + +def read_nodes(path): + nodes = [] + for line in open(path): + f = line.rstrip("\n").split("\t") + if len(f) >= 4: + nodes.append({"name": f[0], "index": int(f[1]), "region": f[2], "ip": f[3]}) + return nodes + + +def pct(xs, p): + if not xs: + return float("nan") + xs = sorted(xs) + k = min(len(xs) - 1, max(0, int(round((len(xs) - 1) * p)))) + return xs[k] + + +def fmt(x, nd=1): + return "n/a" if x != x else f"{x:.{nd}f}" + + +# --------------------------------------------------------------------------------------------------------------- +def rtt(d, nodes_file): + nodes = read_nodes(nodes_file) + by_ip = {n["ip"]: n for n in nodes} + by_name = {n["name"]: n for n in nodes} + pairs = defaultdict(list) + for line in open(os.path.join(d, "rtt.tsv")): + f = line.rstrip("\n").split("\t") + if len(f) < 5 or f[3] == "NA": + continue + src, dst = by_name.get(f[0]), by_ip.get(f[1]) + if not src or not dst: + continue + key = tuple(sorted([src["region"], dst["region"]])) + pairs[key].append(float(f[3])) + regions = sorted({n["region"] for n in nodes}) + out = ["| region | " + " | ".join(regions) + " |", "|---|" + "---|" * len(regions)] + for a in regions: + row = [a] + for b in regions: + xs = pairs.get(tuple(sorted([a, b])), []) + row.append(fmt(statistics.median(xs)) if xs else "n/a") + out.append("| " + " | ".join(row) + " |") + out.append("") + out.append("Median of the per-pair average RTT in ms (ping, 5 packets). Same-region cells are the intra-location RTT.") + return "\n".join(out) + + +def propagation(d, nodes_file, t0, t1): + nodes = read_nodes(nodes_file) + t0, t1 = int(t0), int(t1) + seen = defaultdict(dict) # hash -> node -> recv_ms + header_ts = {} + for n in nodes: + p = os.path.join(d, "nodes", n["name"], "blocks.tsv") + if not os.path.exists(p): + continue + for line in open(p): + f = line.rstrip("\n").split("\t") + if len(f) < 3: + continue + try: + recv = int(f[0]) + except ValueError: + continue + if recv < t0 or recv > t1: + continue + seen[f[1]][n["name"]] = recv + try: + header_ts[f[1]] = int(f[2]) + except ValueError: + pass + total = len(seen) + full = {h: m for h, m in seen.items() if len(m) >= max(2, len(nodes) * 0.8)} + delays = defaultdict(list) + region_delays = defaultdict(list) + ts_vs_first = [] + reg = {n["name"]: n["region"] for n in nodes} + for h, m in full.items(): + first = min(m.values()) + for name, r in m.items(): + delays[name].append(r - first) + region_delays[reg[name]].append(r - first) + if h in header_ts: + ts_vs_first.append(first - header_ts[h]) + out = [f"Blocks in the window: {total}; blocks seen by at least 80% of nodes: {len(full)} (the join base).", ""] + out.append("| node | region | blocks | p50 ms | p90 ms | max ms |") + out.append("|---|---|---|---|---|---|") + for n in nodes: + xs = delays.get(n["name"], []) + out.append(f"| {n['name']} | {n['region']} | {len(xs)} | {fmt(pct(xs, 0.5), 0)} | {fmt(pct(xs, 0.9), 0)} | {fmt(max(xs) if xs else float('nan'), 0)} |") + out.append("") + out.append("| region | samples | p50 ms | p90 ms | p99 ms |") + out.append("|---|---|---|---|---|") + for r in sorted(region_delays): + xs = region_delays[r] + out.append(f"| {r} | {len(xs)} | {fmt(pct(xs, 0.5), 0)} | {fmt(pct(xs, 0.9), 0)} | {fmt(pct(xs, 0.99), 0)} |") + allx = [x for xs in delays.values() for x in xs] + out.append("") + out.append(f"All nodes: p50 {fmt(pct(allx, 0.5), 0)} ms, p90 {fmt(pct(allx, 0.9), 0)} ms, p99 {fmt(pct(allx, 0.99), 0)} ms, max {fmt(max(allx) if allx else float('nan'), 0)} ms " + f"(arrival at a node minus the first arrival anywhere; 0 for the node that produced or first received the block).") + if ts_vs_first: + out.append(f"First arrival minus the header timestamp: median {fmt(statistics.median(ts_vs_first), 0)} ms (miner clock and template age; negative = miner clock ahead).") + out.append("Clock caveat: chrony on every VM; per-node offsets in nodes//chrony.txt.") + return "\n".join(out) + + +def locks(pdir): + dumps = {} + for p in glob.glob(os.path.join(pdir, "checkpoints-*-at-heal.json")): + try: + dumps[os.path.basename(p)[12:-13]] = json.load(open(p)) + except Exception: + continue + locked = defaultdict(dict) # index -> hash -> [nodes] + for node, j in dumps.items(): + for cp in (j.get("checkpoints") or []): + if str(cp.get("state", "")).lower() == "locked": + locked[int(cp["index"])].setdefault(cp.get("hash"), []).append(node) + conflicts = {i: h for i, h in locked.items() if len(h) > 1} + if not dumps: + return "no checkpoint dumps (node without the finality layer, or the RPC was refused)" + lines = [f"nodes dumped: {', '.join(sorted(dumps))}; locked indices seen: {len(locked)}; conflicting: {len(conflicts)}"] + for i in sorted(conflicts): + lines.append(f" index {i}: " + "; ".join(f"{h[:12]} by {','.join(ns)}" for h, ns in conflicts[i].items())) + if not conflicts: + lines.append(" none: no index locked with two different hashes on the two sides") + return "\n".join(lines) + + +def read_samples(path): + rows = [] + for line in open(path): + f = line.rstrip("\n").split("\t") + if len(f) >= 9: + try: + rows.append({"t": int(f[0]), "sink": f[1], "blue": int(f[2] or 0), "daa": int(f[3] or 0), "tips": int(f[4]), + "peers": int(f[5]), "difficulty": float(f[6] or 0), "headers": int(f[7] or 0), "blocks": int(f[8] or 0)}) + except ValueError: + continue + return rows + + +def hop(d): + logp = os.path.join(d, "hop.log") + if not os.path.exists(logp): + return "no hop.log in this directory" + phases = [] + for line in open(logp): + f = line.rstrip("\n").split("\t") + if len(f) >= 3: + phases.append({"t": int(f[0]), "sel": f[1], "threads": f[2]}) + cands = sorted(glob.glob(os.path.join(d, "nodes", "*", "samples.tsv"))) + if not cands: + return "no samples (run collect.sh first)" + rows = read_samples(cands[0]) + out = [f"Samples from {cands[0].split('/')[-2]}; phases from hop.log.", "", + "| phase | selector | threads | length s | blocks/min mean | blocks/min last 3 min | difficulty start | difficulty end | settled s (3 min within 10% of 60/min) |", + "|---|---|---|---|---|---|---|---|---|"] + for i, ph in enumerate(phases): + if ph["sel"] == "end": + break + t_end = phases[i + 1]["t"] if i + 1 < len(phases) else rows[-1]["t"] + win = [r for r in rows if ph["t"] <= r["t"] <= t_end] + if len(win) < 2: + out.append(f"| {i + 1} | {ph['sel']} | {ph['threads']} | {(t_end - ph['t']) // 1000} | n/a | n/a | n/a | n/a | n/a |") + continue + mins = defaultdict(list) + for r in win: + mins[(r["t"] - ph["t"]) // 60000].append(r["blocks"]) + per_min = [(m, max(v) - min(v)) for m, v in sorted(mins.items()) if len(v) >= 2] + rates = [x for _, x in per_min] + settled = "never" + for k in range(len(rates) - 2): + if all(abs(rates[k + j] - 60) <= 6 for j in range(3)): + settled = str(per_min[k][0] * 60); break + mean_rate = (win[-1]["blocks"] - win[0]["blocks"]) / max(1, (win[-1]["t"] - win[0]["t"]) / 60000) + last3 = statistics.mean(rates[-3:]) if len(rates) >= 3 else float("nan") + out.append(f"| {i + 1} | {ph['sel']} | {ph['threads']} | {(t_end - ph['t']) // 1000} | {mean_rate:.1f} | {fmt(last3)} | {win[0]['difficulty']:.3g} | {win[-1]['difficulty']:.3g} | {settled} |") + out.append("") + out.append("blocks/min from the node's blockCount (every block, blue or red). Target 60. 'settled' = first minute of three in a row within 10% of target after the step (spec 02 uses a 100-block criterion on the 121-block mean rate; recompute from samples.tsv for the spec's definition).") + return "\n".join(out) + + +def summary(d, nodes_file): + nodes = read_nodes(nodes_file) + out = [f"# Cloud devnet results, {os.path.basename(d)}", ""] + ver = os.path.join(d, "version.txt") + if os.path.exists(ver): + out.append("Node build: " + " ".join(open(ver).read().split())) + stamp = os.path.join(d, "src.stamp") + if os.path.exists(stamp): + out.append("Source: " + "; ".join(l.strip() for l in open(stamp) if l.strip())) + region_counts = ", ".join("%s x %d" % (r, sum(1 for n in nodes if n["region"] == r)) for r in sorted({n["region"] for n in nodes})) + out.append(f"Nodes: {len(nodes)} ({region_counts})") + out.append("") + out.append("## Final state per node") + out.append("") + out.append("| node | region | blocks | headers | blue | daa | tips | peers | difficulty | synced | miner blocks found |") + out.append("|---|---|---|---|---|---|---|---|---|---|---|") + for n in nodes: + nd = os.path.join(d, "nodes", n["name"]) + s = read_samples(os.path.join(nd, "samples.tsv")) if os.path.exists(os.path.join(nd, "samples.tsv")) else [] + last = s[-1] if s else {} + synced = "?" + try: + synced = str(json.load(open(os.path.join(nd, "getInfo.json"))).get("isSynced")) + except Exception: + pass + found = "?" + ml = os.path.join(nd, "miner.log") + if os.path.exists(ml): + found = str(sum(1 for l in open(ml, errors="replace") if "accepted" in l.lower() and "block" in l.lower())) + out.append(f"| {n['name']} | {n['region']} | {last.get('blocks', '?')} | {last.get('headers', '?')} | {last.get('blue', '?')} | {last.get('daa', '?')} | {last.get('tips', '?')} | {last.get('peers', '?')} | {last.get('difficulty', 0):.3g} | {synced} | {found} |") + out.append("") + ev = os.path.join(d, "events.log") + if os.path.exists(ev): + out.append("## Events"); out.append(""); out.append("```"); out.append(open(ev).read().rstrip()); out.append("```"); out.append("") + lat = os.path.join(d, "latency") + if os.path.isdir(lat): + for f in ("rtt-by-region.md", "propagation.md"): + p = os.path.join(lat, f) + if os.path.exists(p): + out.append(f"## Latency: {f}"); out.append(""); out.append(open(p).read().rstrip()); out.append("") + for p in sorted(glob.glob(os.path.join(d, "partition-*", "partition.md"))): + out.append("## " + os.path.basename(os.path.dirname(p))); out.append(""); out.append(open(p).read().rstrip()); out.append("") + if os.path.exists(os.path.join(d, "hop.log")): + out.append("## Hash-rate steps (hop.sh)"); out.append(""); out.append(hop(d)); out.append("") + # reorg distribution over the whole run from chain.tsv (every virtualChainChanged removal) + removals = [] + for n in nodes: + p = os.path.join(d, "nodes", n["name"], "chain.tsv") + if os.path.exists(p): + for line in open(p): + f = line.rstrip("\n").split("\t") + if len(f) >= 3 and f[2].isdigit() and int(f[2]) > 0: + removals.append(int(f[2])) + out.append("## Reorg depth distribution (all nodes, whole run, virtualChainChanged removals > 0)") + out.append("") + if removals: + hist = defaultdict(int) + for r in removals: + hist[r] += 1 + out.append("| depth | count |"); out.append("|---|---|") + for k in sorted(hist): + out.append(f"| {k} | {hist[k]} |") + out.append("") + out.append(f"{len(removals)} reorgs; p50 {pct(removals, 0.5)}, p99 {pct(removals, 0.99)}, max {max(removals)} (spec 03 C1: d is set from this distribution).") + else: + out.append("no reorgs recorded (or no chain.tsv)") + return "\n".join(out) + + +def bench_entry(d, nodes_file): + nodes = read_nodes(nodes_file) + day = os.path.basename(d) + try: + date_txt = time.strftime("%-d %B %Y", time.strptime(day, "%Y-%m-%d")) + except ValueError: + date_txt = day + ver = " ".join(open(os.path.join(d, "version.txt")).read().split()) if os.path.exists(os.path.join(d, "version.txt")) else "" + regions = ", ".join("%s x %d" % (r, sum(1 for n in nodes if n["region"] == r)) for r in sorted({n["region"] for n in nodes})) + lines = [f"## {date_txt}, cloud devnet: {len(nodes)} igneumd nodes across regions, CPU trickle miners, latency, partition and hash-rate steps (consensus-engineer)", "", + f"Machines: {len(nodes)} Hetzner Cloud VMs ({regions}; type cpx21 unless noted), Debian 12, chrony. Node: {ver}, built on a builder VM from the source tarball in results/{day}/src.stamp. Network: igneum-devnet-, genesis bits , {len(nodes)} one-thread CPU miners (igneum-miner --engine igneum-pow), one BLS vote key per node, sparse --addpeer mesh (about 4 peers each).", "", + "Inter-region RTT (ms, median of pair averages): ", "", + "Block propagation (arrival minus first arrival anywhere): p50 ms, p90 ms, p99 ms, max ms; per region .", "", + "Partition , min: minority reorg depth max (getVirtualChainFromBlock from the minority sink at heal), majority ; converged s after heal; conflicting locks: .", "", + "Hash-rate steps (hop.sh): : settle times per step against spec 02 section 2.3 (simulator: x50 settled 62 s, /50 657 s).", "", + "Reorg depth distribution over the run: p50 , p99 , max over reorgs (summary.md).", "", + "Reading: . Caveats: CPU hash rate only (no GPU), one evening, clocks by chrony.", ""] + return "\n".join(lines) + + +def main(a): + if len(a) >= 4 and a[1] == "rtt": + print(rtt(a[2], a[3])) + elif len(a) >= 6 and a[1] == "propagation": + print(propagation(a[2], a[3], a[4], a[5])) + elif len(a) >= 3 and a[1] == "locks": + print(locks(a[2])) + elif len(a) >= 3 and a[1] == "hop": + print(hop(a[2])) + elif len(a) >= 4 and a[1] == "summary": + print(summary(a[2], a[3])) + elif len(a) >= 4 and a[1] == "bench-entry": + print(bench_entry(a[2], a[3])) + else: + sys.stderr.write(__doc__); sys.exit(2) + + +if __name__ == "__main__": + main(sys.argv) diff --git a/infra/cloud-devnet/experiments/collect.sh b/infra/cloud-devnet/experiments/collect.sh new file mode 100755 index 000000000..b3ca618c5 --- /dev/null +++ b/infra/cloud-devnet/experiments/collect.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Pull logs and RPC samples from every node into results//nodes// and write summary.md plus a +# bench-log entry template (results//bench-log-entry.md). APPEND_BENCH_LOG=1 also appends that entry to +# docs/bench-log.md (the log is append-only; edit the numbers in the entry file first if a run looked odd). +# ./experiments/collect.sh [date] default today (UTC) +. "$(dirname "$0")/../lib/common.sh" +require_nodes +out="$(results_dir_for "${1:-}")"; mkdir -p "$out/nodes" +log "collecting into $out" +cp "$NODES_FILE" "$out/nodes.tsv" +[ -f "$BIN_DIR/version.txt" ] && cp "$BIN_DIR/version.txt" "$BIN_DIR/src.stamp" "$out/" 2>/dev/null + +while IFS=$'\t' read -r name idx reg ip; do + ( + d="$out/nodes/$name"; mkdir -p "$d" + nssh "$ip" "journalctl -u igneumd --no-pager -o short-iso --since -24h | tail -c 30000000" > "$d/igneumd.log" 2>/dev/null + nssh "$ip" "journalctl -u igneum-miner --no-pager -o short-iso --since -24h | tail -c 5000000" > "$d/miner.log" 2>/dev/null + for f in blocks chain samples; do nscp "$SSH_USER@$ip:/var/log/igneum/$f.tsv" "$d/$f.tsv" 2>/dev/null || true; done + nssh "$ip" "chronyc tracking 2>/dev/null; echo; free -m; echo; df -h / | tail -1; echo; cat /etc/igneum/node.env" > "$d/host.txt" 2>/dev/null + for m in getInfo getBlockDagInfo getConnectedPeerInfo getFinalityWeights; do + nssh "$ip" "python3 /opt/igneum/bin/wrpc.py call $m" > "$d/$m.json" 2>/dev/null || true + done + nssh "$ip" "python3 /opt/igneum/bin/wrpc.py call getFinalityCheckpoints '{\"last\": 50}'" > "$d/getFinalityCheckpoints.json" 2>/dev/null || true + printf '[%s] %s blocks lines, %s chain lines, %s samples\n' "$name" "$(grep -c . "$d/blocks.tsv" 2>/dev/null || echo 0)" "$(grep -c . "$d/chain.tsv" 2>/dev/null || echo 0)" "$(grep -c . "$d/samples.tsv" 2>/dev/null || echo 0)" + ) & + while [ "$(jobs -r | wc -l)" -ge 10 ]; do sleep 1; done +done < "$NODES_FILE" +wait + +python3 "$HERE/experiments/analyze.py" summary "$out" "$NODES_FILE" | tee "$out/summary.md" +python3 "$HERE/experiments/analyze.py" bench-entry "$out" "$NODES_FILE" > "$out/bench-log-entry.md" +log "wrote $out/summary.md and $out/bench-log-entry.md" +if [ "${APPEND_BENCH_LOG:-0}" = 1 ]; then + { printf '\n'; cat "$out/bench-log-entry.md"; } >> "$REPO/docs/bench-log.md" + log "appended the entry to docs/bench-log.md (review it before committing)" +else + log "APPEND_BENCH_LOG=1 appends the entry to docs/bench-log.md" +fi diff --git a/infra/cloud-devnet/experiments/hop.sh b/infra/cloud-devnet/experiments/hop.sh new file mode 100755 index 000000000..4bd599d73 --- /dev/null +++ b/infra/cloud-devnet/experiments/hop.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# Experiment 1c: hash-rate steps for the difficulty controller (spec 02 section 2.3; the dual-lane rule is on the +# `difficulty` worktree, NODE_SRC=vendor/igneum-node-diff when building; on master it exercises Kaspa's sampled DAA). +# ./experiments/hop.sh "" +# schedule = phases separated by ";", phase = :: +# selector: all | half (odd indices) | odd | even | region: | - (indices) +# threads: miner threads on the selected nodes (0 stops the miner); unselected nodes keep their setting +# Default schedule (about 75 min): steady, 4x step up on half the nodes, back down, one region off, back: +# all:1:900;half:4:900;all:1:900;region:hel1:0:600;all:1:900 +# Every phase change is logged to results//hop.log (ts_ms, selector, threads); collect.sh and analyze.py hop +# line the difficulty samples up against it. +. "$(dirname "$0")/../lib/common.sh" +require_nodes +schedule="${1:-all:1:900;half:4:900;all:1:900;region:hel1:0:600;all:1:900}" +out="$(results_dir_for)"; logf="$out/hop.log" + +select_nodes() { + case "$1" in + all) node_names ;; + half|odd) awk -F'\t' '$2 % 2 == 1 { print $1 }' "$NODES_FILE" ;; + even) awk -F'\t' '$2 % 2 == 0 { print $1 }' "$NODES_FILE" ;; + region:*) nodes_in_region "${1#region:}" ;; + *-*) awk -F'\t' -v a="${1%-*}" -v b="${1#*-}" '$2 >= a && $2 <= b { print $1 }' "$NODES_FILE" ;; + *) die "bad selector $1" ;; + esac +} + +set_threads() { # node threads + local ip; ip=$(node_ip "$1") + if [ "$2" = 0 ]; then + nssh "$ip" "systemctl stop igneum-miner" + else + nssh "$ip" "sed -i 's/^MINER_THREADS=.*/MINER_THREADS=$2/' /etc/igneum/node.env && systemctl restart igneum-miner" + fi +} + +log "schedule: $schedule" +IFS=';' read -r -a phases <<< "$schedule" +for ph in "${phases[@]}"; do + sel="${ph%%:*}"; rest="${ph#*:}" + if [ "$sel" = region ]; then sel="region:${rest%%:*}"; rest="${rest#*:}"; fi + threads="${rest%%:*}"; secs="${rest#*:}" + nodes=$(select_nodes "$sel") + log "phase: $sel -> $threads thread(s) for $secs s ($(printf '%s\n' "$nodes" | wc -l | tr -d ' ') nodes)" + for n in $nodes; do set_threads "$n" "$threads" & done; wait + printf '%s\t%s\t%s\t%s\n' "$(date +%s)000" "$sel" "$threads" "$secs" >> "$logf" + sleep "$secs" +done +printf '%s\tend\t-\t-\n' "$(date +%s)000" >> "$logf" +log "done; phases in $logf. Pull the samples with ./experiments/collect.sh and read results//hop.md" diff --git a/infra/cloud-devnet/experiments/latency.sh b/infra/cloud-devnet/experiments/latency.sh new file mode 100755 index 000000000..241d80482 --- /dev/null +++ b/infra/cloud-devnet/experiments/latency.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Experiment 1a: inter-region RTT and block propagation delay. +# ./experiments/latency.sh [minutes] default 10 +# 1. RTT: every node pings every other node (5 pings, 0.2 s apart) in parallel; results//latency/rtt.tsv and a +# region-pair table. +# 2. Propagation: the block logs run anyway (igneum-blocklog); this waits `minutes`, pulls blocks.tsv from every node +# and joins on the block hash: first arrival anywhere = t0 of that block, per node delay = arrival - t0. +# Clocks: chrony on every VM; the per-node offset is recorded (chronyc tracking) and the join is only as good as it. +. "$(dirname "$0")/../lib/common.sh" +require_nodes +minutes="${1:-10}" +out="$(results_dir_for)/latency"; mkdir -p "$out/nodes" +all_ips=$(cut -f4 "$NODES_FILE" | tr '\n' ' ') +t_start=$(date +%s)000 + +log "RTT matrix: $(node_count) nodes x $(( $(node_count) - 1 )) targets, 5 pings each" +: > "$out/rtt.tsv" +while IFS=$'\t' read -r name idx reg ip; do + ( + nssh "$ip" "for t in $all_ips; do [ \"\$t\" = \"$ip\" ] && continue; r=\$(ping -c 5 -i 0.2 -W 2 \$t 2>/dev/null | awk -F'/' '/rtt|round-trip/ { print \$4\"\t\"\$5\"\t\"\$6 }'); printf '%s\t%s\t%s\n' '$name' \"\$t\" \"\${r:-NA\tNA\tNA}\"; done" 2>/dev/null + ) >> "$out/rtt.tsv" & +done < "$NODES_FILE" +wait +log "rtt.tsv: $(grep -c . "$out/rtt.tsv") pairs. Region-pair medians (avg RTT ms):" +python3 "$HERE/experiments/analyze.py" rtt "$out" "$NODES_FILE" | tee "$out/rtt-by-region.md" + +log "propagation window: $minutes min from $(date -u +%H:%M:%S) (the miners must be running: ./start.sh miners)" +sleep $(( minutes * 60 )) +t_end=$(date +%s)000 +while IFS=$'\t' read -r name idx reg ip; do + mkdir -p "$out/nodes/$name" + nscp "$SSH_USER@$ip:/var/log/igneum/blocks.tsv" "$out/nodes/$name/blocks.tsv" 2>/dev/null || log "no blocks.tsv on $name" + nssh "$ip" "chronyc tracking 2>/dev/null | grep -E 'System time|RMS offset'" > "$out/nodes/$name/chrony.txt" 2>/dev/null || true +done < "$NODES_FILE" +python3 "$HERE/experiments/analyze.py" propagation "$out" "$NODES_FILE" "$t_start" "$t_end" | tee "$out/propagation.md" +printf '%s\tlatency %s min, window %s..%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$minutes" "$t_start" "$t_end" >> "$(results_dir_for)/events.log" +log "written: $out/rtt.tsv, rtt-by-region.md, propagation.md" diff --git a/infra/cloud-devnet/experiments/observer.sh b/infra/cloud-devnet/experiments/observer.sh new file mode 100755 index 000000000..d2497afea --- /dev/null +++ b/infra/cloud-devnet/experiments/observer.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# Observer hookup: let tools/observer/observer.mjs watch one cloud node so /live on the site shows the 20-node network. +# The RPC never leaves the node's loopback, so the observer reaches it through an ssh tunnel (from the Mac) or runs +# on the node itself (remote mode, keeps the overloaded Mac out of it). +# +# ./experiments/observer.sh tunnel [node] ssh -L 28610 -> the node's wRPC JSON; prints the observer command +# ./experiments/observer.sh remote on [node] install Node 22 on the node, copy observer.mjs and DATABASE_URL, start a unit +# ./experiments/observer.sh remote off [node] stop and remove that unit (the DATABASE_URL file is deleted too) +# +# Tables: with LIVE_TABLE_PREFIX=cloud_ the observer writes cloud_live_* and the public page keeps showing the Mac's +# devnet (site/api/live.mjs reads the unprefixed tables; it has no prefix switch, so a side-by-side page needs that +# one-line change in site/). Without the prefix the cloud network REPLACES the Mac's devnet on /live: stop the +# Mac's observer first, or two writers fight over live_state. +. "$(dirname "$0")/../lib/common.sh" +require_nodes +mode="${1:-}"; node="${3:-${2:-}}" +case "$mode" in + tunnel) + node="${2:-$(node_name 1)}"; ip=$(node_ip "$node") + log "tunnel to $node ($ip): local ws://127.0.0.1:${LOCAL_PORT:-28620} -> node 127.0.0.1:$RPC_JSON_PORT (Ctrl+C ends it)" + log "in another shell, from the repo root:" + log " IGNEUM_RPC=ws://127.0.0.1:${LOCAL_PORT:-28620} LIVE_TABLE_PREFIX=cloud_ node tools/observer/observer.mjs # parallel record" + log " IGNEUM_RPC=ws://127.0.0.1:${LOCAL_PORT:-28620} node tools/observer/observer.mjs # takes over /live (stop the Mac's observer first)" + exec ssh "${SSH_OPTS[@]}" -N -L "${LOCAL_PORT:-28620}:127.0.0.1:$RPC_JSON_PORT" "$SSH_USER@$ip" + ;; + remote) + sub="${2:-}"; node="${3:-$(node_name 1)}"; ip=$(node_ip "$node") + if [ "$sub" = on ]; then + envf="$HOME/.config/igneum/env"; [ -f "$envf" ] || die "no $envf (DATABASE_URL)" + dburl=$(grep -E '^DATABASE_URL=' "$envf" | head -1) + [ -n "$dburl" ] || die "DATABASE_URL not in $envf" + log "installing Node 22 (NodeSource apt repository) and the observer on $node" + nssh "$ip" 'command -v node >/dev/null && node -v | grep -q "^v2[2-9]" || { curl -fsSL https://deb.nodesource.com/setup_22.x | bash - >/dev/null && apt-get install -y -qq nodejs >/dev/null; }; node -v' + nscp "$REPO/tools/observer/observer.mjs" "$SSH_USER@$ip:/opt/igneum/observer.mjs" + printf '%s\nIGNEUM_RPC=ws://127.0.0.1:%s\nLIVE_TABLE_PREFIX=%s\n' "$dburl" "$RPC_JSON_PORT" "${LIVE_TABLE_PREFIX:-cloud_}" | nssh "$ip" 'umask 077; cat > /etc/igneum/observer.env' + nssh "$ip" 'cat > /etc/systemd/system/igneum-observer.service </dev/null; rm -f /etc/systemd/system/igneum-observer.service /etc/igneum/observer.env; systemctl daemon-reload; echo removed' + else die "usage: observer.sh remote on|off [node]"; fi + ;; + *) die "usage: observer.sh tunnel [node] | observer.sh remote on|off [node]" ;; +esac diff --git a/infra/cloud-devnet/experiments/partition.sh b/infra/cloud-devnet/experiments/partition.sh new file mode 100755 index 000000000..d22c24aa8 --- /dev/null +++ b/infra/cloud-devnet/experiments/partition.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# Experiment 1b: cut one region off, heal it, measure the reorg depth and the heal time. +# ./experiments/partition.sh e.g. ./experiments/partition.sh sin 10 +# ./experiments/partition.sh heal remove every partition rule (if a run was interrupted) +# +# Cut: on every node of the region, iptables DROP of p2p traffic (port 26611, both directions, both roles) to and +# from every node outside the region, tagged with a comment so heal finds exactly these rules. The minority keeps +# mining on its own tips, the majority on theirs (both sides have --enable-unsynced-mining). +# Heal: the rules are deleted; the nodes reconnect through their --addpeer retries (backoff up to 8 minutes in +# components/connectionmanager, so a reconnect kick restarts igneumd on the minority side when RECONNECT_KICK=1, +# default on: a restart reconnects at once and the chain state is on disk). +# Measured, through the loopback RPC: +# reorg depth = removedChainBlockHashes of getVirtualChainFromBlock(start = the node's own sink at heal time), +# per node, after convergence; plus the largest single virtualChainChanged removal in chain.tsv +# heal time = seconds from heal until all nodes report at most 2 distinct sinks for 3 consecutive 5 s polls +# locks = getFinalityCheckpoints on both sides at heal: any index locked with two different hashes is a +# conflicting lock (the gate 3 question; the rule's floor predicts none) +. "$(dirname "$0")/../lib/common.sh" +require_nodes + +heal_all() { + log "removing partition rules everywhere" + on_all "iptables -S 2>/dev/null | grep -- '--comment igneum-partition' | sed 's/^-A/-D/' | while read -r r; do iptables \$r; done; true" >/dev/null +} + +region="${1:-}"; minutes="${2:-10}" +[ -n "$region" ] || die "usage: partition.sh | partition.sh heal" +if [ "$region" = heal ]; then heal_all; exit 0; fi +minority=$(nodes_in_region "$region"); [ -n "$minority" ] || die "no nodes in region $region (present: $(regions_present | tr '\n' ' '))" +majority=$(node_names | grep -vxF -f <(printf '%s\n' "$minority")) +outside_ips=$(for n in $majority; do node_ip "$n"; done | tr '\n' ' ') +stamp=$(date -u +%Y%m%d-%H%M%S) +out="$(results_dir_for)/partition-$region-$stamp"; mkdir -p "$out" +log "partition: region $region ($(printf '%s\n' "$minority" | wc -l | tr -d ' ') nodes) cut off for $minutes min; majority $(printf '%s\n' "$majority" | wc -l | tr -d ' ') nodes" + +snapshot() { # file: one sample line per node, prefixed with the node name + : > "$1" + while IFS=$'\t' read -r name idx reg ip; do + ( printf '%s\t%s\n' "$name" "$(nssh "$ip" 'python3 /opt/igneum/bin/wrpc.py sample' 2>/dev/null)" >> "$1" ) & + done < "$NODES_FILE"; wait +} +checkpoints() { # node file + nssh "$(node_ip "$1")" "python3 /opt/igneum/bin/wrpc.py call getFinalityCheckpoints '{\"last\": 100}'" > "$2" 2>/dev/null || echo '{}' > "$2" +} + +snapshot "$out/before.tsv" +t0=$(date +%s) +for n in $minority; do + rules="" + for ip in $outside_ips; do + rules="$rules iptables -I INPUT -s $ip -p tcp --dport $P2P_PORT -m comment --comment igneum-partition -j DROP;" + rules="$rules iptables -I INPUT -s $ip -p tcp --sport $P2P_PORT -m comment --comment igneum-partition -j DROP;" + rules="$rules iptables -I OUTPUT -d $ip -p tcp --dport $P2P_PORT -m comment --comment igneum-partition -j DROP;" + rules="$rules iptables -I OUTPUT -d $ip -p tcp --sport $P2P_PORT -m comment --comment igneum-partition -j DROP;" + done + nssh "$(node_ip "$n")" "$rules true" && log "cut $n" +done +printf '%s\tpartition %s cut\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$region" >> "$(results_dir_for)/events.log" +trap 'log "interrupted: healing"; heal_all' INT TERM + +: > "$out/during.tsv" +for i in $(seq 1 $(( minutes * 2 ))); do + sleep 30 + snapshot "$out/during-$i.tsv"; cat "$out/during-$i.tsv" >> "$out/during.tsv"; rm -f "$out/during-$i.tsv" + m_sinks=$(for n in $minority; do awk -F'\t' -v n="$n" '$1 == n { print $3 }' "$out/during.tsv" | tail -1; done | sort -u | wc -l | tr -d ' ') + M_sinks=$(for n in $majority; do awk -F'\t' -v n="$n" '$1 == n { print $3 }' "$out/during.tsv" | tail -1; done | sort -u | wc -l | tr -d ' ') + log "t+$(( i * 30 )) s: minority distinct sinks $m_sinks, majority distinct sinks $M_sinks" +done + +log "heal: snapshots and checkpoints on both sides, then rules off" +snapshot "$out/at-heal.tsv" +for n in $minority; do checkpoints "$n" "$out/checkpoints-$n-at-heal.json"; done +first_major=$(printf '%s\n' "$majority" | head -1); checkpoints "$first_major" "$out/checkpoints-$first_major-at-heal.json" +heal_all +t_heal=$(date +%s) +trap - INT TERM +if [ "${RECONNECT_KICK:-1}" = 1 ]; then + for n in $minority; do nssh "$(node_ip "$n")" 'systemctl restart igneumd igneum-blocklog' && log "kicked $n (restart, reconnects at once)"; done +fi +printf '%s\tpartition %s healed\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$region" >> "$(results_dir_for)/events.log" + +log "waiting for convergence (at most 2 distinct sinks for 3 polls, 5 s apart; up to 20 min)" +ok=0; converged_at="" +for i in $(seq 1 240); do + sleep 5 + snapshot "$out/poll.tsv" + d=$(cut -f3 "$out/poll.tsv" | grep -c . ); u=$(cut -f3 "$out/poll.tsv" | sort -u | grep -c .) + if [ "$u" -le 2 ] && [ "$d" = "$(node_count)" ]; then ok=$((ok + 1)); else ok=0; fi + [ $(( i % 6 )) = 0 ] && log "t+$(( i * 5 )) s after heal: $u distinct sinks" + if [ "$ok" -ge 3 ]; then converged_at=$(( $(date +%s) - t_heal - 10 )); break; fi +done +[ -n "$converged_at" ] && log "converged $converged_at s after heal" || log "no convergence within 20 min (see poll.tsv)" + +log "reorg depth from each node's own sink at heal (getVirtualChainFromBlock)" +: > "$out/reorg.tsv" +while IFS=$'\t' read -r name idx reg ip; do + sink=$(awk -F'\t' -v n="$name" '$1 == n { print $3 }' "$out/at-heal.tsv") + side=majority; printf '%s\n' "$minority" | grep -qx "$name" && side=minority + r=$(nssh "$ip" "python3 /opt/igneum/bin/wrpc.py call getVirtualChainFromBlock '{\"startHash\": \"$sink\", \"includeAcceptedTransactionIds\": false}'" 2>/dev/null \ + | python3 -c 'import json,sys; j=json.load(sys.stdin); print(len(j.get("removedChainBlockHashes",[])), len(j.get("addedChainBlockHashes",[])))' 2>/dev/null || echo "NA NA") + chainmax=$(nssh "$ip" "awk -F'\t' -v t=$(( t_heal * 1000 )) '\$1 >= t && \$3 > m { m = \$3 } END { print m + 0 }' /var/log/igneum/chain.tsv" 2>/dev/null || echo NA) + printf '%s\t%s\t%s\t%s\t%s\n' "$name" "$side" "$sink" "$r" "$chainmax" | tr ' ' '\t' >> "$out/reorg.tsv" +done < "$NODES_FILE" +{ + printf '# Partition %s, %s min, %s\n\n' "$region" "$minutes" "$stamp" + printf 'cut at %s, healed at %s, converged %s s after heal (criterion: at most 2 distinct sinks for 3 polls)\n\n' "$(date -u -r "$t0" +%H:%M:%S 2>/dev/null || date -u -d @"$t0" +%H:%M:%S)" "$(date -u -r "$t_heal" +%H:%M:%S 2>/dev/null || date -u -d @"$t_heal" +%H:%M:%S)" "${converged_at:-none}" + printf '| node | side | sink at heal | removed (reorg depth) | added | max single removal after heal |\n|---|---|---|---|---|---|\n' + awk -F'\t' '{ printf "| %s | %s | %s | %s | %s | %s |\n", $1, $2, substr($3, 1, 12), $4, $5, $6 }' "$out/reorg.tsv" + printf '\nMinority reorg depth, max: %s. Majority, max: %s.\n' "$(awk -F'\t' '$2 == "minority" && $4 != "NA" && $4 > m { m = $4 } END { print m + 0 }' "$out/reorg.tsv")" "$(awk -F'\t' '$2 == "majority" && $4 != "NA" && $4 > m { m = $4 } END { print m + 0 }' "$out/reorg.tsv")" + printf '\nConflicting locks at heal (same checkpoint index, different hash, both locked):\n' + python3 "$HERE/experiments/analyze.py" locks "$out" +} | tee "$out/partition.md" +log "written: $out/partition.md (plus before/during/at-heal/poll/reorg tsv and the checkpoint dumps)" diff --git a/infra/cloud-devnet/lib/common.sh b/infra/cloud-devnet/lib/common.sh new file mode 100644 index 000000000..2220ee04a --- /dev/null +++ b/infra/cloud-devnet/lib/common.sh @@ -0,0 +1,91 @@ +# Shared helpers for infra/cloud-devnet. Source this; it sources config.sh. +# Needs bash 3.2 or newer (the Mac's /bin/bash is fine), ssh, scp, python3. + +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +export REPO +# shellcheck source=../config.sh +. "$HERE/config.sh" + +NODES_FILE="${NODES_FILE:-$HERE/nodes.tsv}" # nameindexregionip, one line per node, written by create.sh +BUILD_DIR="$HERE/build" +BIN_DIR="$BUILD_DIR/bin" +RESULTS_DIR="$HERE/results" + +# Hetzner API token: the only line of ~/.config/igneum/hetzner-token (never printed, never in the repo). +if [ -z "${HCLOUD_TOKEN:-}" ] && [ -s "${HETZNER_TOKEN_FILE:-$HOME/.config/igneum/hetzner-token}" ]; then + HCLOUD_TOKEN="$(head -1 "${HETZNER_TOKEN_FILE:-$HOME/.config/igneum/hetzner-token}" | tr -d '[:space:]')"; export HCLOUD_TOKEN +fi + +log() { printf '%s %s\n' "$(date -u +%H:%M:%S)" "$*"; } +die() { log "ERROR: $*" >&2; exit 1; } +need() { command -v "$1" >/dev/null 2>&1 || die "$1 is not installed ($2)"; } + +# Hetzner location name for node index i (1-based) and the DigitalOcean one. +region_of() { + local i="$1" list + if [ "$PROVIDER" = digitalocean ]; then list="$DO_REGIONS"; else list="$REGIONS"; fi + local n; n=$(printf '%s' "$list" | tr ',' '\n' | grep -c .) + printf '%s' "$list" | tr ',' '\n' | sed -n "$(( (i - 1) % n + 1 ))p" +} + +node_name() { printf '%s-%02d' "$PREFIX" "$1"; } + +# Server type for a Hetzner location: SERVER_TYPE if set, else the TYPE_BY_LOCATION map, else cx23. +type_for_location() { + if [ -n "$SERVER_TYPE" ]; then printf '%s' "$SERVER_TYPE"; return; fi + local t; t=$(printf '%s' "$TYPE_BY_LOCATION" | tr ',' '\n' | awk -F= -v l="$1" '$1 == l { print $2 }') + printf '%s' "${t:-cx23}" +} + +# nodes.tsv accessors +require_nodes() { [ -s "$NODES_FILE" ] || die "no $NODES_FILE: run create.sh first"; } +node_count() { grep -c . "$NODES_FILE"; } +node_ip() { awk -F'\t' -v n="$1" '$1 == n || $2 == n { print $4; exit }' "$NODES_FILE"; } +node_region() { awk -F'\t' -v n="$1" '$1 == n || $2 == n { print $3; exit }' "$NODES_FILE"; } +node_names() { cut -f1 "$NODES_FILE"; } +nodes_in_region() { awk -F'\t' -v r="$1" '$3 == r { print $1 }' "$NODES_FILE"; } +regions_present() { cut -f3 "$NODES_FILE" | sort -u; } + +# Outbound peers of node index i: the ring neighbour i+1 and chords at i + k*N/MESH_OUT (1-based, wrapping). +# With MESH_OUT=2 that is i+1 and i+N/2; inbound links double it, so each node sees about 4 peers. +peers_of() { + local i="$1" n; n=$(node_count) + local k step idx out="" + for k in $(seq 1 "$MESH_OUT"); do + if [ "$k" = 1 ]; then step=1; else step=$(( (k - 1) * n / MESH_OUT )); fi + idx=$(( (i - 1 + step) % n + 1 )) + [ "$idx" = "$i" ] && continue + out="$out $(node_ip "$idx"):$P2P_PORT" + done + printf '%s' "$out" | sed 's/^ //' | tr ' ' ',' +} + +SSH_OPTS=(-i "$SSH_KEY_FILE" -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile="$HERE/build/known_hosts" -o ConnectTimeout=15 -o BatchMode=yes -o ServerAliveInterval=30) +nssh() { local ip="$1"; shift; ssh "${SSH_OPTS[@]}" "$SSH_USER@$ip" "$@"; } +nscp() { scp -q "${SSH_OPTS[@]}" "$@"; } + +# Run a command on every node in parallel (PAR at a time), prefixing output with the node name. +# usage: on_all '' (the command runs on the node through ssh) +on_all() { + local cmd="$1" par="${PAR:-10}" + mkdir -p "$BUILD_DIR" + node_names | xargs -P "$par" -I{} sh -c ' + name="$1"; cmd="$2"; ip=$(awk -F"\t" -v n="$name" "\$1 == n { print \$4; exit }" "'"$NODES_FILE"'") + ssh -i "'"$SSH_KEY_FILE"'" -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile="'"$HERE/build/known_hosts"'" -o ConnectTimeout=15 -o BatchMode=yes "'"$SSH_USER"'@$ip" "$cmd" 2>&1 | sed "s/^/[$name] /" + ' _ {} "$cmd" +} + +# One RPC call on a node through the loopback wRPC JSON endpoint (node/wrpc.py is installed by provision.sh). +rpc() { local node="$1" method="$2" params="${3:-{\}}"; nssh "$(node_ip "$node")" "python3 /opt/igneum/bin/wrpc.py call $method '$params'"; } + +results_dir_for() { local d="$RESULTS_DIR/${1:-$(date -u +%Y-%m-%d)}"; mkdir -p "$d"; printf '%s' "$d"; } + +genesis_bits_decimal() { printf '%d' "$GENESIS_BITS"; } + +confirm() { + if [ "${YES:-0}" = 1 ]; then return 0; fi + printf '%s [type yes]: ' "$1"; read -r a; [ "$a" = yes ] || die "not confirmed" +} diff --git a/infra/cloud-devnet/make-source.sh b/infra/cloud-devnet/make-source.sh new file mode 100755 index 000000000..663f6ee5e --- /dev/null +++ b/infra/cloud-devnet/make-source.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Export the node source as one tarball the builder VM can compile without any GitHub access: +# src/vendor/igneum-node/ `git archive HEAD` of NODE_SRC, plus (SRC_MODE=head+dirty) every modified and untracked file +# src/igneum-pow/ the same for POW_SRC +# The layout keeps the fork's relative path dependency `igneum-pow = { path = "../../../../igneum-pow" }` +# (vendor/igneum-node/consensus/pow/Cargo.toml) valid. Output: build/src.tar.gz and build/src.stamp (what went in). +# This is the Windows package's src.zip recipe (proto-cuda/windows-node/make-package.sh) in Linux form. + +. "$(dirname "$0")/lib/common.sh" + +mkdir -p "$BUILD_DIR" +stage="$BUILD_DIR/src" +rm -rf "$stage"; mkdir -p "$stage/vendor/igneum-node" "$stage/igneum-pow" + +export_tree() { # repo-dir dest label + local src="$1" dst="$2" label="$3" head dirty=0 + [ -d "$src" ] || die "$label: $src does not exist" + head=$(git -C "$src" rev-parse --short HEAD) + git -C "$src" archive --format=tar HEAD | tar -x -C "$dst" + if [ "$SRC_MODE" = "head+dirty" ]; then + # modified, deleted and untracked files of the worktree (never target directories or the .git dir) + local list; list=$(git -C "$src" ls-files -m -o --exclude-standard | grep -vE '^target' || true) + if [ -n "$list" ]; then + dirty=$(printf '%s\n' "$list" | grep -c .) + printf '%s\n' "$list" | while IFS= read -r f; do + if [ -e "$src/$f" ]; then mkdir -p "$dst/$(dirname "$f")"; cp -p "$src/$f" "$dst/$f"; else rm -f "$dst/$f"; fi + done + fi + fi + printf '%s: HEAD %s (%s), %s uncommitted files overlaid, mode %s\n' "$label" "$head" "$(git -C "$src" rev-parse --abbrev-ref HEAD)" "$dirty" "$SRC_MODE" +} + +{ + printf 'exported %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + export_tree "$NODE_SRC" "$stage/vendor/igneum-node" "igneum-node ($NODE_SRC)" + export_tree "$POW_SRC" "$stage/igneum-pow" "igneum-pow ($POW_SRC)" +} | tee "$BUILD_DIR/src.stamp" + +# the fork's Cargo.lock is tracked, so the builder resolves the same crate versions; drop any stale target dirs +rm -rf "$stage/vendor/igneum-node/target" "$stage/igneum-pow/target" +tar -C "$BUILD_DIR" -czf "$BUILD_DIR/src.tar.gz" src +rm -rf "$stage" +log "wrote $BUILD_DIR/src.tar.gz ($(du -h "$BUILD_DIR/src.tar.gz" | cut -f1))" +grep -q 'mode head$' "$BUILD_DIR/src.stamp" && log "NOTE: SRC_MODE=head ships HEAD only; on 3 Oct 2026 every worktree's branch work (finality v2, difficulty, hotswap) is uncommitted and would be left out" +exit 0 diff --git a/infra/cloud-devnet/node/blocklog.sh b/infra/cloud-devnet/node/blocklog.sh new file mode 100755 index 000000000..e326cc242 --- /dev/null +++ b/infra/cloud-devnet/node/blocklog.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# Block and chain log on a node VM: one line per block the node accepts (arrival time against the header timestamp), +# one line per virtual chain change (the reorg record), one sample line every 5 s. Written by wrpc.py into +# /var/log/igneum/{blocks,chain,samples}.tsv, collected by experiments/collect.sh. Restarts when the node restarts. +set -euo pipefail +mkdir -p /var/log/igneum +exec python3 /opt/igneum/bin/wrpc.py watch-blocks /var/log/igneum diff --git a/infra/cloud-devnet/node/igneum-blocklog.service b/infra/cloud-devnet/node/igneum-blocklog.service new file mode 100644 index 000000000..f68151854 --- /dev/null +++ b/infra/cloud-devnet/node/igneum-blocklog.service @@ -0,0 +1,18 @@ +[Unit] +Description=Igneum block arrival, chain change and sample log (wrpc.py watch-blocks) +After=igneumd.service +Requires=igneumd.service + +[Service] +Type=simple +User=igneum +Group=igneum +ExecStart=/opt/igneum/bin/blocklog.sh +Restart=always +RestartSec=5 +StandardOutput=journal +StandardError=journal +SyslogIdentifier=igneum-blocklog + +[Install] +WantedBy=multi-user.target diff --git a/infra/cloud-devnet/node/igneum-miner.service b/infra/cloud-devnet/node/igneum-miner.service new file mode 100644 index 000000000..b426a20b6 --- /dev/null +++ b/infra/cloud-devnet/node/igneum-miner.service @@ -0,0 +1,24 @@ +[Unit] +Description=Igneum CPU trickle miner (igneum-miner, real lottery hash, votes with this node's key) +After=igneumd.service +Requires=igneumd.service + +[Service] +Type=simple +User=igneum +Group=igneum +EnvironmentFile=/etc/igneum/node.env +# positional: . The label is the node name, so each node has its own +# BLS vote key and its own payout address (--payout-label), deterministic across restarts. +# 1e9 seconds = never stop on its own. MINER_THREADS=0 is handled by hop.sh (it stops the unit instead). +ExecStart=/bin/sh -c 'exec /opt/igneum/bin/igneum-miner mine grpc://127.0.0.1:26610 "$MINER_THREADS" 1000000000 "$NODE_NAME" --engine igneum-pow --payout-label "$NODE_NAME" --label "$NODE_NAME" --status-secs "$MINER_STATUS_SECS"' +Restart=always +RestartSec=15 +Nice=10 +MemoryMax=1200M +StandardOutput=journal +StandardError=journal +SyslogIdentifier=igneum-miner + +[Install] +WantedBy=multi-user.target diff --git a/infra/cloud-devnet/node/igneumd.service b/infra/cloud-devnet/node/igneumd.service new file mode 100644 index 000000000..e166af7bd --- /dev/null +++ b/infra/cloud-devnet/node/igneumd.service @@ -0,0 +1,22 @@ +[Unit] +Description=Igneum devnet node (igneumd) +After=network-online.target chrony.service +Wants=network-online.target + +[Service] +Type=simple +User=igneum +Group=igneum +EnvironmentFile=/etc/igneum/node.env +ExecStart=/opt/igneum/bin/run-igneumd.sh +Restart=always +RestartSec=10 +LimitNOFILE=65536 +# the node keeps up to four 256 MiB lottery caches plus rocksdb; stop a runaway before the VM swaps to death +MemoryMax=3200M +StandardOutput=journal +StandardError=journal +SyslogIdentifier=igneumd + +[Install] +WantedBy=multi-user.target diff --git a/infra/cloud-devnet/node/install-node.sh b/infra/cloud-devnet/node/install-node.sh new file mode 100755 index 000000000..66fe98183 --- /dev/null +++ b/infra/cloud-devnet/node/install-node.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Runs ON a node VM as root (Debian 12 or Ubuntu). Called by provision.sh with: +# install-node.sh +# Expects /opt/igneum/bin/{igneumd,igneum-miner,wrpc.py,run-igneumd.sh,blocklog.sh} and the three unit files in /root. +set -euo pipefail +name="$1"; idx="$2"; suffix="$3"; bits="$4"; peers="$5"; extip="$6"; threads="$7"; status_secs="${8:-60}" + +export DEBIAN_FRONTEND=noninteractive +if ! command -v chronyd >/dev/null 2>&1 || ! command -v iptables >/dev/null 2>&1; then + apt-get update -qq + apt-get install -y -qq chrony iptables python3 jq >/dev/null +fi +systemctl enable --now chrony >/dev/null 2>&1 || true + +id igneum >/dev/null 2>&1 || useradd --system --home /var/lib/igneum --shell /usr/sbin/nologin igneum +mkdir -p /var/lib/igneum /var/log/igneum /etc/igneum /opt/igneum/bin +chmod +x /opt/igneum/bin/* +chown -R igneum:igneum /var/lib/igneum /var/log/igneum + +# A private devnet: own handshake magic (suffix) and own genesis (bits override recomputes the genesis hash). +# OverrideParams is deny_unknown_fields, so only known keys go in here (consensus/core/src/config/params.rs). +printf '{"genesis_bits": %s}\n' "$bits" > /etc/igneum/override-params.json + +cat > /etc/igneum/node.env </dev/null 2>&1 +echo "installed $name (index $idx, suffix $suffix, bits $bits, peers $peers, $threads miner thread(s)); $(/opt/igneum/bin/igneumd --version | head -1)" diff --git a/infra/cloud-devnet/node/run-igneumd.sh b/infra/cloud-devnet/node/run-igneumd.sh new file mode 100755 index 000000000..445a051fe --- /dev/null +++ b/infra/cloud-devnet/node/run-igneumd.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# igneumd launcher on a node VM. Reads /etc/igneum/node.env and builds the flag list. +# The flags are the ones that work on the Windows node (proto-cuda/windows-node/start-node.ps1): --addpeer, never +# --connect (--connect sets the inbound limit to 0, kaspad/src/daemon.rs). --outpeers=0 stops the connection manager +# filling 8 outbound slots from exchanged addresses, which would turn the sparse mesh into a near-complete graph; +# the --addpeer links are permanent connections outside that target. RPC on loopback only; p2p on every interface. +# --enable-unsynced-mining: every node starts at genesis (timestamp 2026-10-03T00:00Z), so none is "synced" at first +# and the node would refuse templates (rpc/service/src/service.rs). +set -euo pipefail +. /etc/igneum/node.env + +args=(--devnet --devnet-suffix="$DEVNET_SUFFIX" --override-params-file=/etc/igneum/override-params.json + --appdir=/var/lib/igneum --rpclisten=127.0.0.1:26610 --rpclisten-json=127.0.0.1:28610 + --listen=0.0.0.0:26611 --externalip="$EXTERNAL_IP" + --nodnsseed --disable-upnp --nologfiles --enable-unsynced-mining --outpeers=0 --maxinpeers=32 --loglevel=info) +IFS=',' read -r -a peers <<< "${PEERS:-}" +for p in "${peers[@]}"; do [ -n "$p" ] && args+=(--addpeer="$p"); done +# shellcheck disable=SC2206 +[ -n "${EXTRA_ARGS:-}" ] && args+=($EXTRA_ARGS) +exec /opt/igneum/bin/igneumd "${args[@]}" diff --git a/infra/cloud-devnet/node/wrpc.py b/infra/cloud-devnet/node/wrpc.py new file mode 100755 index 000000000..1f50283cf --- /dev/null +++ b/infra/cloud-devnet/node/wrpc.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python3 +"""Minimal wRPC JSON client for igneumd, standard library only (any Debian python3, no websocket package). + +The node's wRPC JSON endpoint (--rpclisten-json=127.0.0.1:28610) is a plain WebSocket. Requests are +{"id": n, "method": "getBlockDagInfo", "params": {...}}; the answer carries the same id and either "params" (the +result) or "error"; notifications arrive as {"method": "blockAddedNotification", "params": {...}}. Field names are +camelCase (serde rename_all on the RPC model). This mirrors the Rpc class in tools/observer/observer.mjs. + + wrpc.py call [json-params] one call, prints the result as JSON + wrpc.py watch-blocks subscribe and append, forever: + /blocks.tsv recv_ms hash header_timestamp_ms blue_score daa_score parents + /chain.tsv recv_ms added removed first_removed_hashes(up to 3, comma separated) + /samples.tsv ts_ms sink blue_score daa_score tips peers difficulty headers blocks (every 5 s) + wrpc.py sample one samples.tsv line to stdout +Environment: IGNEUM_RPC (default ws://127.0.0.1:28610). +""" +import base64, json, os, socket, struct, sys, time, threading +from urllib.parse import urlparse + +RPC = os.environ.get("IGNEUM_RPC", "ws://127.0.0.1:28610") + + +class WebSocket: + """RFC 6455 client: text frames, masking, ping/pong, continuation frames.""" + + def __init__(self, url, timeout=10.0): + u = urlparse(url) + self.sock = socket.create_connection((u.hostname, u.port or 80), timeout=timeout) + key = base64.b64encode(os.urandom(16)).decode() + path = u.path or "/" + req = (f"GET {path} HTTP/1.1\r\nHost: {u.hostname}:{u.port}\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n" + f"Sec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n") + self.sock.sendall(req.encode()) + head = b"" + while b"\r\n\r\n" not in head: + chunk = self.sock.recv(4096) + if not chunk: + raise ConnectionError("handshake: connection closed") + head += chunk + status, _, rest = head.partition(b"\r\n\r\n") + if b" 101 " not in status.split(b"\r\n")[0]: + raise ConnectionError("handshake refused: " + status.split(b"\r\n")[0].decode(errors="replace")) + self.buf = rest + self.lock = threading.Lock() + + def _read(self, n): + while len(self.buf) < n: + chunk = self.sock.recv(65536) + if not chunk: + raise ConnectionError("connection closed") + self.buf += chunk + out, self.buf = self.buf[:n], self.buf[n:] + return out + + def send_text(self, text): + data = text.encode() + mask = os.urandom(4) + n = len(data) + if n < 126: + hdr = struct.pack("!BB", 0x81, 0x80 | n) + elif n < 65536: + hdr = struct.pack("!BBH", 0x81, 0x80 | 126, n) + else: + hdr = struct.pack("!BBQ", 0x81, 0x80 | 127, n) + masked = bytes(b ^ mask[i % 4] for i, b in enumerate(data)) + with self.lock: + self.sock.sendall(hdr + mask + masked) + + def _send_control(self, opcode, payload=b""): + mask = os.urandom(4) + with self.lock: + self.sock.sendall(struct.pack("!BB", 0x80 | opcode, 0x80 | len(payload)) + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload))) + + def recv_text(self): + """Returns the next complete text message (handles ping and continuation).""" + message = b"" + while True: + b0, b1 = self._read(2) + fin, opcode = b0 & 0x80, b0 & 0x0F + n = b1 & 0x7F + if n == 126: + n = struct.unpack("!H", self._read(2))[0] + elif n == 127: + n = struct.unpack("!Q", self._read(8))[0] + if b1 & 0x80: + mask = self._read(4) + payload = bytes(b ^ mask[i % 4] for i, b in enumerate(self._read(n))) + else: + payload = self._read(n) + if opcode == 0x9: + self._send_control(0xA, payload); continue + if opcode == 0xA: + continue + if opcode == 0x8: + raise ConnectionError("server closed the websocket") + message += payload + if fin: + return message.decode() + + def close(self): + try: + self._send_control(0x8) + except Exception: + pass + self.sock.close() + + +class Rpc: + def __init__(self, url=RPC): + self.ws = WebSocket(url) + self.next_id = 0 + self.notifications = [] + + def call(self, method, params=None, timeout=10.0): + self.next_id += 1 + rid = self.next_id + self.ws.send_text(json.dumps({"id": rid, "method": method, "params": params or {}})) + deadline = time.time() + timeout + while time.time() < deadline: + m = json.loads(self.ws.recv_text()) + if m.get("id") == rid: + if "error" in m and m["error"]: + raise RuntimeError(f"{method}: {m['error']}") + return m.get("params") + if m.get("method"): + self.notifications.append(m) + raise TimeoutError(method) + + def next_notification(self): + if self.notifications: + return self.notifications.pop(0) + while True: + m = json.loads(self.ws.recv_text()) + if m.get("method"): + return m + + +def sample_line(rpc): + dag = rpc.call("getBlockDagInfo") + peers = rpc.call("getConnectedPeerInfo").get("peerInfo", []) + try: + blue = rpc.call("getSinkBlueScore").get("blueScore") + except Exception: + blue = "" + return "\t".join(str(x) for x in [int(time.time() * 1000), dag.get("sink", ""), blue, dag.get("virtualDaaScore", ""), + len(dag.get("tipHashes", []) or []), len(peers), dag.get("difficulty", ""), + dag.get("headerCount", ""), dag.get("blockCount", "")]) + + +def watch_blocks(directory): + os.makedirs(directory, exist_ok=True) + while True: + try: + rpc = Rpc() + rpc.call("subscribe", {"BlockAdded": {}}) + rpc.call("subscribe", {"VirtualChainChanged": {"include_accepted_transaction_ids": False}}) + sys.stdout.write(f"{time.strftime('%H:%M:%S')} subscribed on {RPC}\n"); sys.stdout.flush() + last_sample = 0.0 + with open(os.path.join(directory, "blocks.tsv"), "a") as fb, open(os.path.join(directory, "chain.tsv"), "a") as fc, \ + open(os.path.join(directory, "samples.tsv"), "a") as fs: + while True: + now = time.time() + if now - last_sample >= 5.0: + try: + fs.write(sample_line(rpc) + "\n"); fs.flush() + except Exception as e: + sys.stdout.write(f"sample failed: {e}\n"); sys.stdout.flush() + last_sample = now + rpc.ws.sock.settimeout(5.0) + try: + m = rpc.next_notification() + except socket.timeout: + continue + recv_ms = int(time.time() * 1000) + p = m.get("params") or {} + inner = p.get("BlockAdded") or p.get("VirtualChainChanged") or p + if m["method"] == "blockAddedNotification" and inner.get("block"): + h = inner["block"].get("header", {}) + levels = h.get("parentsByLevel") or [] # array of arrays of hashes; level 0 = direct parents + direct = levels[0] if levels and isinstance(levels[0], list) else [] + fb.write("\t".join(str(x) for x in [recv_ms, h.get("hash", ""), h.get("timestamp", ""), h.get("blueScore", ""), + h.get("daaScore", ""), len(direct)]) + "\n") + fb.flush() + elif m["method"] == "virtualChainChangedNotification": + added = inner.get("addedChainBlockHashes", []) or [] + removed = inner.get("removedChainBlockHashes", []) or [] + fc.write("\t".join([str(recv_ms), str(len(added)), str(len(removed)), ",".join(removed[:3])]) + "\n") + fc.flush() + except Exception as e: + sys.stdout.write(f"{time.strftime('%H:%M:%S')} rpc lost ({e}); retrying in 5 s\n"); sys.stdout.flush() + time.sleep(5) + + +def main(argv): + if len(argv) >= 2 and argv[1] == "call": + params = json.loads(argv[3]) if len(argv) > 3 else {} + print(json.dumps(Rpc().call(argv[2], params), indent=None)) + elif len(argv) >= 3 and argv[1] == "watch-blocks": + watch_blocks(argv[2]) + elif len(argv) >= 2 and argv[1] == "sample": + print(sample_line(Rpc())) + else: + sys.stderr.write(__doc__); sys.exit(2) + + +if __name__ == "__main__": + main(sys.argv) diff --git a/infra/cloud-devnet/provision.sh b/infra/cloud-devnet/provision.sh new file mode 100755 index 000000000..31082b2ce --- /dev/null +++ b/infra/cloud-devnet/provision.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# Build the node once and install it on every node. +# +# ./provision.sh source tarball -> builder VM -> build/bin/{igneumd,igneum-miner} -> every node (systemd units) +# ./provision.sh build only the build (creates the builder VM if build/builder.ip is missing, deletes it afterwards) +# ./provision.sh install only the install on the nodes (needs build/bin/igneumd) +# +# Why a builder VM and not a cross-compile on the Mac: rustup here has only aarch64-apple-darwin and +# x86_64-pc-windows-gnu installed and there is no x86_64 Linux linker (no musl-cross, zig, cross or docker; checked +# 3 Oct 2026). A Linux cross-compile would need `rustup target add x86_64-unknown-linux-gnu` plus a linker +# (`brew install filosottile/musl-cross/musl-cross` and a musl target), and the rocksdb bindgen trick of the Windows +# cross-build again. The builder costs about EUR 0.05 for the hour (cpx41, approximate) and the Mac stays free. +# The binaries are cached in build/bin; REBUILD=1 forces a new build. + +. "$(dirname "$0")/lib/common.sh" + +what="${1:-all}" +mkdir -p "$BUILD_DIR" "$BIN_DIR" + +# ---- build --------------------------------------------------------------------------------------------------------- +do_build() { + if [ -x "$BIN_DIR/igneumd" ] && [ "${REBUILD:-0}" != 1 ]; then + log "build/bin/igneumd exists ($(cat "$BIN_DIR/version.txt" 2>/dev/null | tr '\n' ' ')); REBUILD=1 to rebuild" + return + fi + "$HERE/make-source.sh" + if [ ! -s "$BUILD_DIR/builder.ip" ]; then + YES="${YES:-0}" "$HERE/create.sh" builder + fi + bip=$(cat "$BUILD_DIR/builder.ip") + log "builder at $bip; waiting for ssh" + for try in $(seq 1 12); do nssh "$bip" true >/dev/null 2>&1 && break; sleep 10; done + nssh "$bip" true || die "builder $bip not reachable over ssh" + log "uploading source ($(du -h "$BUILD_DIR/src.tar.gz" | cut -f1)) and the build script" + nscp "$BUILD_DIR/src.tar.gz" "$HERE/builder/build-on-builder.sh" "$SSH_USER@$bip:/root/" + nssh "$bip" "bash /root/build-on-builder.sh" | tee "$BUILD_DIR/build.log" + nscp "$SSH_USER@$bip:/root/out/igneumd" "$SSH_USER@$bip:/root/out/igneum-miner" "$SSH_USER@$bip:/root/out/version.txt" "$BIN_DIR/" + chmod +x "$BIN_DIR/igneumd" "$BIN_DIR/igneum-miner" + cp "$BUILD_DIR/src.stamp" "$BIN_DIR/src.stamp" + log "binaries in $BIN_DIR: $(cat "$BIN_DIR/version.txt" | tr '\n' ' ')" + if [ "${KEEP_BUILDER:-0}" = 1 ]; then + log "KEEP_BUILDER=1: builder $bip stays up (it bills by the hour)" + else + log "deleting the builder VM" + if [ "$PROVIDER" = digitalocean ]; then doctl compute droplet delete -f "$PREFIX-builder"; else hcloud server delete "$PREFIX-builder" >/dev/null; fi + rm -f "$BUILD_DIR/builder.ip" + fi +} + +# ---- install ------------------------------------------------------------------------------------------------------ +do_install() { + require_nodes + [ -x "$BIN_DIR/igneumd" ] || die "no $BIN_DIR/igneumd: run ./provision.sh build" + local n; n=$(node_count); local bits; bits=$(genesis_bits_decimal) + log "installing on $n nodes (devnet suffix $DEVNET_SUFFIX, genesis bits $GENESIS_BITS = $bits, $MESH_OUT outbound peers each, $MINER_THREADS miner thread)" + while IFS=$'\t' read -r name idx reg ip; do + peers=$(peers_of "$idx") + ( + for try in 1 2 3; do nssh "$ip" "mkdir -p /opt/igneum/bin /etc/igneum" && break; sleep 10; done + nscp "$BIN_DIR/igneumd" "$BIN_DIR/igneum-miner" "$HERE/node/wrpc.py" "$HERE/node/run-igneumd.sh" "$HERE/node/blocklog.sh" "$SSH_USER@$ip:/opt/igneum/bin/" + nscp "$HERE/node/install-node.sh" "$HERE/node/igneumd.service" "$HERE/node/igneum-miner.service" "$HERE/node/igneum-blocklog.service" "$SSH_USER@$ip:/root/" + nssh "$ip" "bash /root/install-node.sh '$name' '$idx' '$DEVNET_SUFFIX' '$bits' '$peers' '$ip' '$MINER_THREADS' '$MINER_STATUS_SECS'" 2>&1 | sed "s/^/[$name] /" + ) & + # at most 10 installs at once + while [ "$(jobs -r | wc -l)" -ge 10 ]; do sleep 1; done + done < "$NODES_FILE" + wait + log "installed. Peer map:" + while IFS=$'\t' read -r name idx reg ip; do printf ' %s (%s, %s) -> %s\n' "$name" "$reg" "$ip" "$(peers_of "$idx")"; done < "$NODES_FILE" + log "next: ./start.sh" +} + +case "$what" in + all) do_build; do_install ;; + build) do_build ;; + install) do_install ;; + *) die "usage: provision.sh [all|build|install]" ;; +esac diff --git a/infra/cloud-devnet/results/.gitkeep b/infra/cloud-devnet/results/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/infra/cloud-devnet/start.sh b/infra/cloud-devnet/start.sh new file mode 100755 index 000000000..650c22b36 --- /dev/null +++ b/infra/cloud-devnet/start.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Start the network: nodes and block logs first, miners once every RPC answers. +# ./start.sh nodes, then miners +# ./start.sh nodes nodes and block logs only +# ./start.sh miners miners only +. "$(dirname "$0")/lib/common.sh" +require_nodes +what="${1:-all}" + +if [ "$what" = all ] || [ "$what" = nodes ]; then + log "starting igneumd and igneum-blocklog on $(node_count) nodes" + on_all 'systemctl restart igneumd && sleep 1 && systemctl restart igneum-blocklog && systemctl is-active igneumd' + log "waiting for the RPC on every node" + for try in $(seq 1 24); do + bad=0 + while IFS=$'\t' read -r name idx reg ip; do + nssh "$ip" "python3 /opt/igneum/bin/wrpc.py call getInfo >/dev/null 2>&1" || bad=$((bad + 1)) + done < "$NODES_FILE" + [ "$bad" = 0 ] && break + log "$bad nodes not answering yet (try $try), 5 s"; sleep 5 + done + [ "$bad" = 0 ] || log "WARNING: $bad nodes still not answering; check: ./status.sh and journalctl -u igneumd on the node" +fi + +if [ "$what" = all ] || [ "$what" = miners ]; then + log "starting the trickle miners ($MINER_THREADS thread each; the first block needs the 256 MiB cache built, about 10 to 30 s on one vCPU, approximate)" + on_all 'systemctl restart igneum-miner && systemctl is-active igneum-miner' +fi +log "started at $(date -u +%Y-%m-%dT%H:%M:%SZ). Watch with ./status.sh (every 30 s: watch -n 30 ./status.sh)" +printf '%s\tstart %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$what" >> "$(results_dir_for)/events.log" diff --git a/infra/cloud-devnet/status.sh b/infra/cloud-devnet/status.sh new file mode 100755 index 000000000..b2df55813 --- /dev/null +++ b/infra/cloud-devnet/status.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# One line per node: blocks, headers, blue score, DAA, tips, peers, difficulty, sink, miner state. Reads the +# loopback RPC through ssh (wrpc.py sample), so it costs one ssh round trip per node. +. "$(dirname "$0")/lib/common.sh" +require_nodes +printf '%-10s %-5s %-15s %8s %8s %8s %8s %4s %5s %12s %-12s %s\n' node region ip blocks headers blue daa tips peers difficulty sink miner +while IFS=$'\t' read -r name idx reg ip; do + ( + line=$(nssh "$ip" "python3 /opt/igneum/bin/wrpc.py sample 2>/dev/null; systemctl is-active igneum-miner 2>/dev/null" 2>/dev/null | tr '\n' '\t') + s=$(printf '%s' "$line" | cut -f1-9); miner=$(printf '%s' "$line" | cut -f10) + IFS=$'\t' read -r ts sink blue daa tips peers diff headers blocks <<< "$s" + printf '%-10s %-5s %-15s %8s %8s %8s %8s %4s %5s %12.0f %-12s %s\n' "$name" "$reg" "$ip" "${blocks:-?}" "${headers:-?}" "${blue:-?}" "${daa:-?}" "${tips:-?}" "${peers:-?}" "${diff:-0}" "${sink:0:12}" "${miner:-?}" + ) & +done < "$NODES_FILE" +wait diff --git a/infra/cloud-devnet/stop.sh b/infra/cloud-devnet/stop.sh new file mode 100755 index 000000000..9f464784a --- /dev/null +++ b/infra/cloud-devnet/stop.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Stop the network. The VMs keep running (and billing); destroy.sh removes them. +# ./stop.sh miners, block logs, nodes +# ./stop.sh miners miners only (the chain keeps running without new blocks) +. "$(dirname "$0")/lib/common.sh" +require_nodes +what="${1:-all}" +if [ "$what" = miners ]; then + on_all 'systemctl stop igneum-miner; systemctl is-active igneum-miner || true' +else + on_all 'systemctl stop igneum-miner igneum-blocklog igneumd; systemctl is-active igneumd || true' +fi +printf '%s\tstop %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$what" >> "$(results_dir_for)/events.log" +log "stopped ($what). The servers still bill by the hour: ./destroy.sh when the data is collected" diff --git a/infra/gpu-bench/.gitignore b/infra/gpu-bench/.gitignore new file mode 100644 index 000000000..a5362c90d --- /dev/null +++ b/infra/gpu-bench/.gitignore @@ -0,0 +1,2 @@ +# the bundle tarball; results/ is committed on purpose +build/ diff --git a/infra/gpu-bench/Dockerfile.cuda b/infra/gpu-bench/Dockerfile.cuda new file mode 100644 index 000000000..b2eb6d9b3 --- /dev/null +++ b/infra/gpu-bench/Dockerfile.cuda @@ -0,0 +1,16 @@ +# Igneum GPU bench image, NVIDIA lane. CUDA 12.8 devel (nvcc, NVRTC, cuda.h) on Ubuntu 22.04, plus the OpenCL ICD +# loader and headers for the optional OpenCL build-time measurement on NVIDIA, Python 3 for run.sh, curl for the +# upload, and sshd so the bundle can be scp'd in. RunPod: create a template with this image (or use the tag below +# directly; nvcc is in it) and tick "SSH". Blackwell (RTX 5090) needs 12.8 or newer; older toolkits do not know sm_120. +# Build and push (optional; the public tag works as-is for RunPod with run.sh installing nothing): +# docker build -f Dockerfile.cuda -t /igneum-gpu-bench:cuda12.8 . +FROM nvidia/cuda:12.8.1-devel-ubuntu22.04 +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get update && apt-get install -y --no-install-recommends \ + python3 curl ca-certificates openssh-server clinfo ocl-icd-opencl-dev opencl-headers perl build-essential \ + && rm -rf /var/lib/apt/lists/* \ + && mkdir -p /run/sshd /workspace +# NVIDIA's OpenCL ICD is provided by the host driver through the container runtime; it appears when the pod sets +# NVIDIA_DRIVER_CAPABILITIES=compute,utility (RunPod does). clinfo shows whether it is there; run.sh skips if not. +WORKDIR /workspace +CMD ["/usr/sbin/sshd", "-D"] diff --git a/infra/gpu-bench/Dockerfile.rocm b/infra/gpu-bench/Dockerfile.rocm new file mode 100644 index 000000000..1b5c2f7e6 --- /dev/null +++ b/infra/gpu-bench/Dockerfile.rocm @@ -0,0 +1,15 @@ +# Igneum GPU bench image, AMD lane: ROCm 6 with the OpenCL runtime (rocm-opencl) and headers, so proto-opencl/host.c +# builds with plain cc -lOpenCL and clinfo lists the gfx device. Needs /dev/kfd and /dev/dri passed in (a provider +# that offers AMD cards does that). RunPod lists no AMD consumer cards and Vast.ai none in general (approximate, +# 3 Oct 2026); this recipe is for whichever host has an RX 7900 XTX (gfx1100), or a datacenter MI300X as a stand-in. +# The tag is approximate: use the newest rocm/dev-ubuntu-22.04:6.x-complete the host's kernel driver supports. +# docker build -f Dockerfile.rocm -t /igneum-gpu-bench:rocm6 . +# docker run --device=/dev/kfd --device=/dev/dri --group-add video -it /igneum-gpu-bench:rocm6 +FROM rocm/dev-ubuntu-22.04:6.3-complete +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get update && apt-get install -y --no-install-recommends \ + python3 curl ca-certificates openssh-server clinfo ocl-icd-opencl-dev opencl-headers perl build-essential \ + && rm -rf /var/lib/apt/lists/* \ + && mkdir -p /run/sshd /workspace +WORKDIR /workspace +CMD ["/usr/sbin/sshd", "-D"] diff --git a/infra/gpu-bench/README.md b/infra/gpu-bench/README.md new file mode 100644 index 000000000..75f3d1e98 --- /dev/null +++ b/infra/gpu-bench/README.md @@ -0,0 +1,72 @@ +# Igneum GPU bench on rented cards + +One pod per card on RunPod (primary; Vast.ai variant below), each running the same bundle: the CUDA and OpenCL +test harnesses from `proto-cuda/` and `proto-opencl/` with the three checked-in packs, built on the pod with +`--serve` support, then a fixed sequence of measurements and one results row. Nothing is cloned on the pod; the +bundle is a tarball made here. Nothing here spends until a pod is started. + +## The command sequence + +``` +cd infra/gpu-bench +./make-bundle.sh # build/igneum-gpu-bench.tar.gz (sources + scripts, about 1 MB) +# RunPod console: Pods -> Deploy -> pick the card -> template "nvidia/cuda:12.8.1-devel-ubuntu22.04" (or the image from +# Dockerfile.cuda), Community Cloud, tick SSH, 20 GB container disk -> Deploy. Note host and port from "Connect". +scp -P build/igneum-gpu-bench.tar.gz root@:/workspace/ +ssh -p root@ 'cd /workspace && tar xzf igneum-gpu-bench.tar.gz && cd bundle && LABEL=rtx4090 ./run.sh' +# about 45 minutes: build 2 to 5 min, gate, 10 min raw, sweep, inline, recompile timings; the row prints as RESULT +# and lands in bundle/results.md and results-/row.md; the log is uploaded under label gpubench-rtx4090 +scp -P -r root@:/workspace/bundle/results-* results/ # keep the raw outputs here +# RunPod console: Stop and Terminate the pod (billing per minute while it runs; storage bills while stopped) +node ../../tools/logs.mjs # the uploaded log, if the pod could not be reached again +``` + +Repeat for each card (the same bundle). Paste the rows into `docs/bench-log.md` with `results-template.md`. +`CUDA_ARCH=sm_86` overrides `-arch=native` if the driver in a pod is older than its toolkit. + +## Cards and cost (RunPod pricing page, read 3 Oct 2026, USD per hour) + +| Card | Community Cloud | Secure Cloud | One 45-minute run, community | Note | +|---|---|---|---|---| +| RTX 3060 | not on RunPod's table | not on RunPod's table | about 0.05 to 0.10 on Vast.ai (approximate) | the entry card; Vast.ai is the host for it | +| RTX 3090 | 0.22 | 0.50 | about 0.17 | Ampere sm_86, 24 GB, 6 MB L2 (approximate) | +| RTX 4090 | 0.34 | 0.74 | about 0.26 | Ada sm_89, 24 GB, 72 MB L2 (approximate) | +| RTX 5090 | 0.69 | 0.99 | about 0.52 | Blackwell sm_120, 32 GB, 96 MB L2 (bench-log); the project lead's own card is the reference row | +| RX 7900 XTX | not offered | not offered | n/a | RunPod has no AMD consumer cards; Vast.ai lists none in general (approximate). `Dockerfile.rocm` and the OpenCL lane of `run.sh` are ready for any host that has one (or the project lead's own AMD box) | + +Four NVIDIA cards: about USD 2 on community cloud, USD 3.50 on secure cloud, plus cents of storage. Per-minute +billing; a pod left running costs the hourly rate. + +Vast.ai variant: search for the card with "CUDA 12.8" and "verified" hosts, rent with the image +`nvidia/cuda:12.8.1-devel-ubuntu22.04` and "SSH" as the launch mode, then the same scp and ssh lines (Vast prints its +own port and host). Prices on Vast.ai are per host and change by the hour; the 3060 is usually under USD 0.10. + +## What run.sh measures and why + +| Step | Output | Ledger | +|---|---|---| +| Vectors gate (`--batches 3`): cache check, dataset self-test, 3 warps standalone and in batch | `gate.txt`, must say `OVERALL: PASS` or the run stops | the cross-vendor proof per card (bench-log, "What PASS means") | +| Raw bench, about 10 minutes of 2^24-hash batches at 1 GiB | `raw.txt`: Mhash/s, seconds | the per-card hash rate; sustained, so thermals and neighbours show | +| Sweep 64, 128, 256, 512, 1024 MiB, 20 batches each | `sweep-*.txt`, the 64 MiB over 1 GiB ratio | M1: the L2 cliff per card (the 5090 gave 5.9x) | +| Inline-dataset shortcut (`make-inline.sh`: every `ds[...]` load becomes `mh_word(ds, ...)` from the cache) | `inline.txt`: Mhash/s and the ratio to the raw rate | M16: the recompute attacker's rate; the M5 Max gave 0.21. The 64 MiB-cache variant inside the 5090's L2 needs a pack exported with a smaller cache (proto-metal has no flag for it yet) and is a follow-up | +| Second pack `igneum-hourly` (closed form, 128 loads) | `pack2.txt` | comparison with the 5090 and Metal tables in the bench log | +| `nvcc -cubin` of kernel.cu x3 (the worker's `prepare` path, host.cu line 554), NVRTC in process x3 (`nvrtc-time.cu`), `clBuildProgram` of kernel.cl x3 (`clbuild-time.c`, when an OpenCL ICD is visible) | `nvrtc.txt`, `clbuild.txt`, medians in the row | M11, M17: the hourly program change on real drivers | +| `--serve` ready line (`printf quit \| worker --serve`) | `serve-ready.txt` | proves the pod's binary is the worker the miner drives (ready line shows `prepare 1` when nvcc is on PATH) | + +The inline binary fails host.cu's dataset self-test by construction (the dataset buffer holds the cache) and must +pass the vectors; `run.sh` reads its rate and vector lines and ignores its OVERALL. + +Results row (also in `results-template.md`): card, driver and toolkit, date, Mhash/s at 1 GiB over the 10 minutes, +seconds, second-pack Mhash/s, the five sweep rates, the cliff ratio, inline Mhash/s, inline over honest, nvcc ms, +NVRTC ms, OpenCL ms, vectors, build ms. + +## Files + +`run.sh` (on the pod), `make-bundle.sh` (on the Mac), `make-inline.sh`, `nvrtc-time.cu`, `clbuild-time.c`, +`upload.sh` (the intake URL and key of `proto-cuda/windows-miner/upload-log.bat` as a curl line), `Dockerfile.cuda`, +`Dockerfile.rocm`, `results-template.md`, `results/` (raw outputs copied back, one directory per run). + +Not tested here: there is no NVIDIA or AMD card on this Mac, so `run.sh`, `nvrtc-time.cu` and `clbuild-time.c` were +checked by read-through and `bash -n` only; the harness binaries they drive ran on the 5090 and the gfx1036 on 3 Oct +2026 (bench-log). The first pod run is the real test; if `nvrtc-time` fails to compile the kernel (a header NVRTC +rejects), the nvcc column still carries the out-of-process figure the worker uses today. diff --git a/infra/gpu-bench/clbuild-time.c b/infra/gpu-bench/clbuild-time.c new file mode 100644 index 000000000..08070bee6 --- /dev/null +++ b/infra/gpu-bench/clbuild-time.c @@ -0,0 +1,60 @@ +/* Times clBuildProgram of a pack's kernel.cl N times on the first GPU device: the OpenCL half of the hourly-recompile + * figure (ledger M11). Build options mirror proto-opencl/host.c for the local-memory exchange path + * (-cl-std=CL1.2 -D IGNEUM_GROUP=32 -D IGNEUM_EXCHANGE=0). + * cc -std=c99 -O2 -o clbuild-time clbuild-time.c -lOpenCL + * ./clbuild-time [iterations=3] [extra build options] + * Prints one line per iteration and "median ". */ +#define CL_TARGET_OPENCL_VERSION 120 +#include +#include +#include +#include +#include + +static double ms(void) { struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t); return t.tv_sec * 1000.0 + t.tv_nsec / 1e6; } +static int cmp(const void* a, const void* b) { double x = *(const double*)a, y = *(const double*)b; return (x > y) - (x < y); } + +int main(int argc, char** argv) { + if (argc < 2) { fprintf(stderr, "usage: clbuild-time [iterations] [extra options]\n"); return 2; } + int iters = argc > 2 ? atoi(argv[2]) : 3; + FILE* f = fopen(argv[1], "rb"); if (!f) { perror(argv[1]); return 2; } + fseek(f, 0, SEEK_END); long n = ftell(f); fseek(f, 0, SEEK_SET); + char* src = malloc(n + 1); if (fread(src, 1, n, f) != (size_t)n) { fprintf(stderr, "short read\n"); return 2; } src[n] = 0; fclose(f); + + cl_uint np = 0; clGetPlatformIDs(0, NULL, &np); if (!np) { fprintf(stderr, "no OpenCL platform\n"); return 2; } + cl_platform_id* ps = malloc(np * sizeof *ps); clGetPlatformIDs(np, ps, NULL); + cl_device_id dev = NULL; char name[256] = "?", drv[128] = "?"; + for (cl_uint i = 0; i < np && !dev; ++i) { + cl_uint nd = 0; if (clGetDeviceIDs(ps[i], CL_DEVICE_TYPE_GPU, 0, NULL, &nd) != CL_SUCCESS || !nd) continue; + clGetDeviceIDs(ps[i], CL_DEVICE_TYPE_GPU, 1, &dev, NULL); + } + if (!dev) { fprintf(stderr, "no OpenCL GPU device\n"); return 2; } + clGetDeviceInfo(dev, CL_DEVICE_NAME, sizeof name, name, NULL); clGetDeviceInfo(dev, CL_DRIVER_VERSION, sizeof drv, drv, NULL); + cl_int err; cl_context ctx = clCreateContext(NULL, 1, &dev, NULL, NULL, &err); if (err != CL_SUCCESS) { fprintf(stderr, "clCreateContext %d\n", err); return 2; } + char opts[1024]; snprintf(opts, sizeof opts, "-cl-std=CL1.2 -D IGNEUM_GROUP=32 -D IGNEUM_EXCHANGE=0 %s", argc > 3 ? argv[3] : ""); + printf("device %s, driver %s, options \"%s\", source %ld bytes\n", name, drv, opts, n); + + double* times = malloc(iters * sizeof *times); + for (int i = 0; i < iters; ++i) { + double t0 = ms(); + const char* s = src; size_t len = (size_t)n; + cl_program prog = clCreateProgramWithSource(ctx, 1, &s, &len, &err); + if (err != CL_SUCCESS) { fprintf(stderr, "clCreateProgramWithSource %d\n", err); return 1; } + err = clBuildProgram(prog, 1, &dev, opts, NULL, NULL); + double t1 = ms(); + if (err != CL_SUCCESS) { + size_t ls = 0; clGetProgramBuildInfo(prog, dev, CL_PROGRAM_BUILD_LOG, 0, NULL, &ls); + char* log = malloc(ls + 1); clGetProgramBuildInfo(prog, dev, CL_PROGRAM_BUILD_LOG, ls, log, NULL); log[ls] = 0; + fprintf(stderr, "build failed (%d):\n%s\n", err, log); return 1; + } + cl_kernel k = clCreateKernel(prog, "igneum_hash", &err); + if (err != CL_SUCCESS) fprintf(stderr, "igneum_hash not found (%d)\n", err); else clReleaseKernel(k); + clReleaseProgram(prog); + times[i] = t1 - t0; + printf("iteration %d: build %.1f ms\n", i + 1, times[i]); + } + qsort(times, iters, sizeof *times, cmp); + printf("median %.1f ms (clBuildProgram, %d iterations)\n", times[iters / 2], iters); + clReleaseContext(ctx); + return 0; +} diff --git a/infra/gpu-bench/make-bundle.sh b/infra/gpu-bench/make-bundle.sh new file mode 100755 index 000000000..e20bfa435 --- /dev/null +++ b/infra/gpu-bench/make-bundle.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# On the Mac: pack the sources the pod needs into build/igneum-gpu-bench.tar.gz. Nothing is cloned on the pod. +# Contents: proto-cuda/host.cu, the three checked-in packs, proto-opencl/host.c, and this directory's scripts. +# DEVNET_PACK= adds a pack exported from the live node (igneum-miner export-pack), which carries kernel_bound.cu +# for a --serve job test; optional. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)"; REPO="$(cd "$HERE/../.." && pwd)" +mkdir -p "$HERE/build" +stage="$HERE/build/bundle"; rm -rf "$stage"; mkdir -p "$stage/proto-cuda/packs" "$stage/proto-opencl" +cp "$REPO/proto-cuda/host.cu" "$stage/proto-cuda/" +for p in igneum-genesis-mh igneum-hourly igneum-genesis; do cp -R "$REPO/proto-cuda/packs/$p" "$stage/proto-cuda/packs/"; done +[ -n "${DEVNET_PACK:-}" ] && cp -R "$DEVNET_PACK" "$stage/proto-cuda/packs/igneum-devnet" +cp "$REPO/proto-opencl/host.c" "$stage/proto-opencl/" +cp -R "$REPO/proto-opencl/emu" "$stage/proto-opencl/" 2>/dev/null || true # emu_opencl.h is included by kernel.cl when no OpenCL compiler is present +cp "$HERE/run.sh" "$HERE/make-inline.sh" "$HERE/nvrtc-time.cu" "$HERE/clbuild-time.c" "$HERE/upload.sh" "$HERE/results-template.md" "$stage/" +chmod +x "$stage"/*.sh +git -C "$REPO" rev-parse --short HEAD > "$stage/SOURCE_COMMIT" 2>/dev/null || true +tar -C "$HERE/build" -czf "$HERE/build/igneum-gpu-bench.tar.gz" bundle +rm -rf "$stage" +echo "wrote $HERE/build/igneum-gpu-bench.tar.gz ($(du -h "$HERE/build/igneum-gpu-bench.tar.gz" | cut -f1))" +echo "on the pod (RunPod SSH, port and host from the pod's Connect panel):" +echo " scp -P $HERE/build/igneum-gpu-bench.tar.gz root@:/workspace/" +echo " ssh -p root@ 'cd /workspace && tar xzf igneum-gpu-bench.tar.gz && cd bundle && LABEL=rtx4090 ./run.sh'" diff --git a/infra/gpu-bench/make-inline.sh b/infra/gpu-bench/make-inline.sh new file mode 100755 index 000000000..6cdfae440 --- /dev/null +++ b/infra/gpu-bench/make-inline.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# Build the inline-dataset shortcut variant of a CUDA pack: the measurement of proto-metal/MEMHARD.md section 2.2 +# (ledger M16) on NVIDIA. usage: make-inline.sh +# Two text edits on kernel.cu, nothing else: +# 1. every dataset load `ds[expr]` in igneum_hash becomes `mh_word(ds, expr)`: the word is recomputed from the +# cache through memhard.h's mh_item (8 dependent cache reads and 9 mixer applications per word) instead of read +# 2. igneum_build copies the 256 MiB cache into the head of the dataset buffer instead of building items, so the +# `ds` pointer the hash kernel receives points at the cache (mh_word indexes words below 2^IGNEUM_CACHE_LOG2_WORDS) +# The dataset self-test of host.cu then FAILS by construction (it reads ds words expecting items) and the vectors +# PASS (mh_word gives the true words). The rate line is the number; run.sh checks the vector lines and ignores OVERALL. +set -euo pipefail +src="$1"; out="$2" +mkdir -p "$out" +cp "$src"/*.h "$out/" +perl -0pe ' + s/ds\[([^\]]*)\]/mh_word(ds, $1)/g; + s/__global__ void igneum_build\(uint32_t\* ds, const uint32_t\* cache, uint32_t nItems\) \{.*?\n\}\n/__global__ void igneum_build(uint32_t* ds, const uint32_t* cache, uint32_t nItems) {\n \/\/ INLINE VARIANT (infra\/gpu-bench\/make-inline.sh): copy the cache into the dataset buffer; igneum_hash recomputes words from it\n uint32_t t = blockIdx.x * blockDim.x + threadIdx.x;\n if (t < nItems && t < (1u << (IGNEUM_CACHE_LOG2_WORDS - 4u))) { for (uint32_t i = 0u; i < 16u; ++i) ds[(size_t)t * 16u + i] = cache[(size_t)t * 16u + i]; }\n}\n/s; +' "$src/kernel.cu" > "$out/kernel.cu" +n=$(grep -c 'mh_word(ds,' "$out/kernel.cu" || true) +grep -q 'INLINE VARIANT' "$out/kernel.cu" || { echo "make-inline: igneum_build not rewritten (pack layout changed?)"; exit 1; } +[ "$n" -gt 0 ] || { echo "make-inline: no dataset loads rewritten"; exit 1; } +echo "make-inline: $n loads rewritten to mh_word, build kernel replaced -> $out/kernel.cu" diff --git a/infra/gpu-bench/nvrtc-time.cu b/infra/gpu-bench/nvrtc-time.cu new file mode 100644 index 000000000..25d88b45e --- /dev/null +++ b/infra/gpu-bench/nvrtc-time.cu @@ -0,0 +1,82 @@ +// Times an in-process NVRTC compile of a pack's kernel (the device part of kernel.cu with program.h and memhard.h +// as in-memory headers), N times, then loads the cubin through the driver API to prove it is usable. +// This is the "hourly JIT in the miner process" figure of ledger M11 and M17; the worker today runs nvcc out of +// process (proto-cuda/host.cu, prepareCompile), which run.sh times separately. +// nvcc -O2 -std=c++17 -o nvrtc-time nvrtc-time.cu -lnvrtc -lcuda +// ./nvrtc-time [iterations=3] +// Prints one line per iteration (compile ms, cubin bytes, load ms) and "median ". +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static std::string readFile(const std::string& p) { + std::ifstream f(p); if (!f) { std::fprintf(stderr, "cannot read %s\n", p.c_str()); std::exit(2); } + std::stringstream ss; ss << f.rdbuf(); return ss.str(); +} +// Drop the includes NVRTC cannot see and the host launch wrappers; the typedefs replace . +static std::string deviceOnly(const std::string& src, bool cutHost) { + std::istringstream in(src); std::string line, out = "typedef unsigned int uint32_t;\ntypedef unsigned long long uint64_t;\ntypedef unsigned char uint8_t;\ntypedef unsigned long size_t;\n"; + while (std::getline(in, line)) { + if (cutHost && line.rfind("cudaError_t igneum_launch", 0) == 0) break; + if (line.find("#include ") != std::string::npos || line.find("#include ") != std::string::npos || + line.find("#include ") != std::string::npos || line.find("#include ") != std::string::npos) { out += "\n"; continue; } + out += line + "\n"; + } + return out; +} +static double ms() { return std::chrono::duration(std::chrono::steady_clock::now().time_since_epoch()).count(); } + +int main(int argc, char** argv) { + if (argc < 2) { std::fprintf(stderr, "usage: nvrtc-time [iterations]\n"); return 2; } + std::string pack = argv[1]; int iters = argc > 2 ? std::atoi(argv[2]) : 3; + std::string kernel = deviceOnly(readFile(pack + "/kernel.cu"), true); + std::string program = deviceOnly(readFile(pack + "/program.h"), false); + std::string memhard = deviceOnly(readFile(pack + "/memhard.h"), false); + const char* headers[2] = { program.c_str(), memhard.c_str() }; + const char* names[2] = { "program.h", "memhard.h" }; + + if (cuInit(0) != CUDA_SUCCESS) { std::fprintf(stderr, "cuInit failed\n"); return 2; } + CUdevice dev; cuDeviceGet(&dev, 0); int major = 0, minor = 0; + cuDeviceGetAttribute(&major, CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR, dev); + cuDeviceGetAttribute(&minor, CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR, dev); + CUcontext ctx; cuCtxCreate(&ctx, 0, dev); + char arch[64]; std::snprintf(arch, sizeof arch, "--gpu-architecture=sm_%d%d", major, minor); + const char* opts[] = { arch, "-std=c++17", "-O3", "-default-device" }; + int nvrtcMajor = 0, nvrtcMinor = 0; nvrtcVersion(&nvrtcMajor, &nvrtcMinor); + std::printf("nvrtc %d.%d, device sm_%d%d, kernel source %zu bytes\n", nvrtcMajor, nvrtcMinor, major, minor, kernel.size()); + + std::vector times; + for (int i = 0; i < iters; ++i) { + double t0 = ms(); + nvrtcProgram prog; + if (nvrtcCreateProgram(&prog, kernel.c_str(), "kernel.cu", 2, headers, names) != NVRTC_SUCCESS) { std::fprintf(stderr, "nvrtcCreateProgram failed\n"); return 1; } + nvrtcResult r = nvrtcCompileProgram(prog, 4, opts); + double t1 = ms(); + size_t logSize = 0; nvrtcGetProgramLogSize(prog, &logSize); + if (r != NVRTC_SUCCESS) { + std::string log(logSize, '\0'); nvrtcGetProgramLog(prog, &log[0]); + std::fprintf(stderr, "nvrtc compile failed:\n%s\n", log.c_str()); return 1; + } + size_t cubinSize = 0; nvrtcGetCUBINSize(prog, &cubinSize); + std::vector cubin(cubinSize); nvrtcGetCUBIN(prog, cubin.data()); + nvrtcDestroyProgram(&prog); + double t2 = ms(); + CUmodule mod; CUfunction fn; + if (cuModuleLoadData(&mod, cubin.data()) != CUDA_SUCCESS || cuModuleGetFunction(&fn, mod, "_Z11igneum_hashPKjPyjj") != CUDA_SUCCESS) { + std::fprintf(stderr, "cubin load or igneum_hash lookup failed (name mangling differs?)\n"); + } else { cuModuleUnload(mod); } + double t3 = ms(); + std::printf("iteration %d: compile %.1f ms, cubin %zu bytes, load %.1f ms\n", i + 1, t1 - t0, cubinSize, t3 - t2); + times.push_back(t1 - t0); + } + std::sort(times.begin(), times.end()); + std::printf("median %.1f ms (nvrtc compile of the device kernel, %d iterations)\n", times[times.size() / 2], iters); + return 0; +} diff --git a/infra/gpu-bench/results-template.md b/infra/gpu-bench/results-template.md new file mode 100644 index 000000000..c26512ed4 --- /dev/null +++ b/infra/gpu-bench/results-template.md @@ -0,0 +1,26 @@ +## October 2026, rented GPUs: RTX 3060, 3090, 4090, 5090 on RunPod (infra/gpu-bench/run.sh; AMD RX 7900 XTX pending a host) + +Image: nvidia/cuda:12.8.1-devel-ubuntu22.04 (RunPod, SSH), bundle of proto-cuda/host.cu plus the three packs and +proto-opencl/host.c at commit , nvcc -arch=native, 1 warp per block. Pack igneum-genesis-mh (memory-hard, 104 +loads per hash), 1 GiB dataset, batches of 2^24; the raw column is one run sized to about 10 minutes. Sweep sizes 64 +to 1024 MiB. Inline = the shortcut kernel of make-inline.sh (every dataset load recomputed from the 256 MiB cache +through mh_word), vectors PASS, dataset self-test FAIL by construction. Recompile = nvcc -cubin of kernel.cu out of +process (the worker's prepare path), NVRTC in process (nvrtc-time.cu), clBuildProgram of kernel.cl where NVIDIA's +OpenCL ICD was present. Cost: RunPod community cloud, USD for the four pods (pricing page: 3090 0.22/h, 4090 +0.34/h, 5090 0.69/h; 3060 ). + +| card | driver / toolkit | date | Mhash/s at 1 GiB (10 min) | s | igneum-hourly Mhash/s | sweep MiB:Mhash/s 64 / 128 / 256 / 512 / 1024 | 64 MiB over 1 GiB | inline Mhash/s | inline / honest | nvcc cubin ms | NVRTC ms | OpenCL build ms | vectors | build ms | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| RTX 3060 | | | | | | | | | | | | | | | +| RTX 3090 | | | | | | | | | | | | | | | +| RTX 4090 | | | | | | | | | | | | | | | +| RTX 5090 | | | | | | | | | | | | | | | +| RTX 5090 (the project lead's PC, 3 Oct 2026, for reference) | 13.4 / 12.8 | 2026-10-03 | 229 (memory-hard) | | 185.3 (closed form) | 1340 / n/a / 270 / 242 / 229 (closed form) | 5.9 | n/a | n/a | n/a | n/a | n/a | 96/96 PASS | | +| Apple M5 Max (Metal, reference) | | 2026-10-03 | 45.2 | | 36.6 | 569 (4 MiB) / 183 / 94 / 69 / 44 | | 9.49 | 0.21 | 20 to 52 (Metal compile) | | | PASS | | + +Reading per column: the 1 GiB rate is the mining rate of the card on today's hash; the 64 MiB over 1 GiB ratio is +the L2 cliff (ledger M1: the on-chip-cache advantage a chip would have to buy in DRAM); inline over honest is the +recompute attacker's rate relative to an honest miner (ledger M16; the M5 Max gives 0.21; a value above 1 on any card +means the shortcut beats the honest kernel there); the three compile columns are the hourly program change on real +NVIDIA drivers (ledger M11, M17; the 90 s gap of the Windows launcher was a rebuild, not a compile). The rows are +raw bench numbers from a rented host with whatever neighbours it had; rerun before quoting a figure outside this log. diff --git a/infra/gpu-bench/results/.gitkeep b/infra/gpu-bench/results/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/infra/gpu-bench/run.sh b/infra/gpu-bench/run.sh new file mode 100755 index 000000000..379ac1686 --- /dev/null +++ b/infra/gpu-bench/run.sh @@ -0,0 +1,141 @@ +#!/usr/bin/env bash +# Igneum GPU benchmark on a rented card. Runs on the pod, inside the unpacked bundle (make-bundle.sh). Clones nothing: +# the sources are in the bundle. Needs nvcc (CUDA 12.8 image) for NVIDIA, or an OpenCL ICD plus a C compiler for AMD. +# +# ./run.sh everything below, results in results-/ and one row appended to results.md +# MINUTES=10 PACK=igneum-genesis-mh PACK2=igneum-hourly LABEL=rtx4090 UPLOAD=1 ./run.sh +# +# Steps (NVIDIA lane): build the memory-hard pack and the closed-form second pack with --serve support (host.cu has +# it), the inline-shortcut variant (make-inline.sh), the NVRTC timer; gate on the vectors (96/96 PASS or stop); raw +# bench for MINUTES at 1 GiB; sweep 64, 128, 256, 512, 1024 MiB; inline run; recompile timings (nvcc -cubin as the +# worker's prepare path does it, NVRTC in process, OpenCL build if an ICD is present); results row; upload. +# AMD lane (ROCm image): the same through proto-opencl/host.c, minus NVRTC and the inline variant (OpenCL packs have +# no inline kernel yet; the sed of make-inline.sh is CUDA-only). +set -uo pipefail +cd "$(dirname "$0")" +MINUTES="${MINUTES:-10}"; PACK="${PACK:-igneum-genesis-mh}"; PACK2="${PACK2:-igneum-hourly}" +LABEL="${LABEL:-}"; UPLOAD="${UPLOAD:-1}"; ITER="${ITER:-3}" +STAMP=$(date -u +%Y%m%d-%H%M%S); OUT="results-$STAMP"; mkdir -p "$OUT" +exec > >(tee "$OUT/run.log") 2>&1 +log() { printf '%s %s\n' "$(date -u +%H:%M:%S)" "$*"; } +rate_of() { grep -E '^\s*(GPU|device)\b.*Mhash/s|^\s*rate' "$1" | grep -oE '[0-9]+\.[0-9]+ Mhash/s' | head -1 | cut -d' ' -f1; } +rate_wall() { grep -oE '[0-9]+\.[0-9]+ Mhash/s' "$1" | tail -1 | cut -d' ' -f1; } +now_ms() { python3 -c 'import time; print(int(time.time()*1000))'; } + +PACKDIR="proto-cuda/packs/$PACK"; PACK2DIR="proto-cuda/packs/$PACK2" +[ -d "$PACKDIR" ] || { log "no $PACKDIR in the bundle"; exit 2; } + +if command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L >/dev/null 2>&1; then LANE=cuda; else LANE=opencl; fi +log "lane: $LANE; pack $PACK (memory-hard), second pack $PACK2; $MINUTES min raw bench" + +# ---- device facts -------------------------------------------------------------------------------------------------- +if [ "$LANE" = cuda ]; then + nvidia-smi --query-gpu=name,driver_version,memory.total,clocks.max.sm,clocks.max.mem --format=csv | tee "$OUT/device.txt" + nvcc --version | tail -2 | tee -a "$OUT/device.txt" + CARD=$(nvidia-smi --query-gpu=name --format=csv,noheader | head -1 | sed 's/NVIDIA //; s/GeForce //') + DRIVER=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1) + CUDAV=$(nvcc --version | grep -oE 'release [0-9.]+' | cut -d' ' -f2) +else + clinfo 2>/dev/null | grep -E 'Device Name|Driver Version|Device Version|Global memory size|Max compute units' | head -12 | tee "$OUT/device.txt" + CARD=$(clinfo 2>/dev/null | grep -m1 'Device Name' | sed 's/.*Device Name *//'); DRIVER=$(clinfo 2>/dev/null | grep -m1 'Driver Version' | sed 's/.*Version *//'); CUDAV="opencl" + rocminfo 2>/dev/null | grep -m1 -E 'gfx[0-9]+' | tee -a "$OUT/device.txt" || true +fi +[ -n "$LABEL" ] || LABEL=$(printf '%s' "$CARD" | tr 'A-Z ' 'a-z-' | tr -cd 'a-z0-9-') +log "card: $CARD, driver $DRIVER, toolkit $CUDAV, label $LABEL" + +# ---- build --------------------------------------------------------------------------------------------------------- +t0=$(now_ms) +if [ "$LANE" = cuda ]; then + ARCH="${CUDA_ARCH:-native}" + log "nvcc -arch=$ARCH: bench-$PACK, bench-$PACK2, bench-inline, nvrtc-time" + nvcc -O3 -std=c++17 -arch="$ARCH" -I "$PACKDIR" -o "bench-$PACK" proto-cuda/host.cu "$PACKDIR/kernel.cu" 2>&1 | tail -5 || { log "build of $PACK failed"; exit 2; } + nvcc -O3 -std=c++17 -arch="$ARCH" -I "$PACK2DIR" -o "bench-$PACK2" proto-cuda/host.cu "$PACK2DIR/kernel.cu" 2>&1 | tail -5 || log "build of $PACK2 failed (continuing)" + ./make-inline.sh "$PACKDIR" "$OUT/inline-pack" && nvcc -O3 -std=c++17 -arch="$ARCH" -I "$OUT/inline-pack" -o bench-inline proto-cuda/host.cu "$OUT/inline-pack/kernel.cu" 2>&1 | tail -5 || log "inline variant did not build (continuing)" + nvcc -O2 -std=c++17 -o nvrtc-time nvrtc-time.cu -lnvrtc -lcuda 2>&1 | tail -3 || log "nvrtc-time did not build (continuing)" + WORKER="./bench-$PACK" +else + log "cc: bench-cl-$PACK (OpenCL at runtime), clbuild-time" + cc -std=c99 -O2 -I "$PACKDIR" -DIGNEUM_KERNEL_PATH="\"$PACKDIR/kernel.cl\"" -o "bench-cl-$PACK" proto-opencl/host.c -lOpenCL -ldl 2>&1 | tail -5 || { log "build failed"; exit 2; } + cc -std=c99 -O2 -I "$PACK2DIR" -DIGNEUM_KERNEL_PATH="\"$PACK2DIR/kernel.cl\"" -o "bench-cl-$PACK2" proto-opencl/host.c -lOpenCL -ldl 2>&1 | tail -5 || true + cc -std=c99 -O2 -o clbuild-time clbuild-time.c -lOpenCL 2>&1 | tail -3 || true + WORKER="./bench-cl-$PACK" +fi +BUILD_MS=$(( $(now_ms) - t0 )); log "build took $BUILD_MS ms" +# --serve support is in host.cu and host.c (ready line "ready cuda|opencl ... prepare N"); prove it answers: +printf 'quit\n' | timeout 120 "$WORKER" --serve 2>&1 | head -3 | tee "$OUT/serve-ready.txt" || true + +# ---- gate: vectors ------------------------------------------------------------------------------------------------- +log "gate: vectors on $PACK (3 batches)" +"$WORKER" --batches 3 > "$OUT/gate.txt" 2>&1 +grep -E 'verify warp|cache check|dataset self-test|OVERALL' "$OUT/gate.txt" +if ! grep -q 'OVERALL: PASS' "$OUT/gate.txt"; then log "GATE FAIL: the card does not reproduce the Mac's vectors; stopping (send $OUT/gate.txt)"; VECTORS=FAIL; else VECTORS="96/96 PASS"; fi +[ "$VECTORS" = FAIL ] && exit 1 + +# ---- raw bench, MINUTES at 1 GiB ------------------------------------------------------------------------------------- +r0=$(rate_of "$OUT/gate.txt"); [ -n "$r0" ] || r0=$(rate_wall "$OUT/gate.txt") +batches=$(python3 -c "import math; r=float('${r0:-50}'); print(max(5, min(100000, int(math.ceil($MINUTES*60*r*1e6/2**24)))))") +log "raw: $batches batches of 2^24 at about $r0 Mhash/s (about $MINUTES min)" +t0=$(now_ms); "$WORKER" --batches "$batches" > "$OUT/raw.txt" 2>&1; RAW_S=$(( ($(now_ms) - t0) / 1000 )) +RAW=$(rate_of "$OUT/raw.txt"); [ -n "$RAW" ] || RAW=$(rate_wall "$OUT/raw.txt") +grep -E 'Mhash/s|OVERALL' "$OUT/raw.txt" | head -4 +log "raw: $RAW Mhash/s over $RAW_S s" + +# ---- sweep --------------------------------------------------------------------------------------------------------- +SWEEP="" +for mib in 64 128 256 512 1024; do + "$WORKER" --dataset-mib "$mib" --batches 20 > "$OUT/sweep-$mib.txt" 2>&1 + r=$(rate_of "$OUT/sweep-$mib.txt"); [ -n "$r" ] || r=$(rate_wall "$OUT/sweep-$mib.txt") + SWEEP="$SWEEP $mib:${r:-n/a}"; log "sweep $mib MiB: ${r:-n/a} Mhash/s" +done +R64=$(printf '%s' "$SWEEP" | grep -oE ' 64:[0-9.n/a]+' | cut -d: -f2); R1024=$(printf '%s' "$SWEEP" | grep -oE '1024:[0-9.n/a]+' | cut -d: -f2) +CLIFF=$(python3 -c " +try: print('%.1f' % (float('$R64') / float('$R1024'))) +except Exception: print('n/a')") + +# ---- second pack (closed form, for comparison with the Mac's and the 5090's tables) ----------------------------------- +R2="n/a" +if [ -x "./bench-$PACK2" ] || [ -x "./bench-cl-$PACK2" ]; then + W2="./bench-$PACK2"; [ -x "$W2" ] || W2="./bench-cl-$PACK2" + "$W2" --batches 20 > "$OUT/pack2.txt" 2>&1; R2=$(rate_of "$OUT/pack2.txt"); [ -n "$R2" ] || R2=$(rate_wall "$OUT/pack2.txt") + log "$PACK2: $R2 Mhash/s, $(grep -c 'PASS' "$OUT/pack2.txt") PASS lines" +fi + +# ---- inline shortcut (CUDA only) ------------------------------------------------------------------------------------- +INLINE="n/a"; RATIO="n/a" +if [ -x ./bench-inline ]; then + # By construction the inline binary FAILS the dataset self-test (the dataset buffer holds the cache copy) and must + # PASS the vectors (mh_word recomputes the true words). Only the rate line and the vector lines count. + ./bench-inline --batches 20 > "$OUT/inline.txt" 2>&1 || true + INLINE=$(rate_of "$OUT/inline.txt"); [ -n "$INLINE" ] || INLINE=$(rate_wall "$OUT/inline.txt") + IV=$(grep -c 'verify warp.*: PASS' "$OUT/inline.txt"); log "inline: ${INLINE:-n/a} Mhash/s, $IV vector PASS lines (6 expected)" + [ "$IV" -ge 3 ] || { log "inline vectors did not pass: ratio discarded"; INLINE="n/a(vectors)"; } + RATIO=$(python3 -c " +try: print('%.3f' % (float('$INLINE') / float('$RAW'))) +except Exception: print('n/a')") +fi + +# ---- recompile timings --------------------------------------------------------------------------------------------- +NVCC_MS="n/a"; NVRTC_MS="n/a"; CL_MS="n/a" +if [ "$LANE" = cuda ]; then + ARCHSM=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1 | tr -d '.') + xs=""; for i in $(seq 1 "$ITER"); do t0=$(now_ms); nvcc -cubin -O3 -std=c++17 -arch="sm_$ARCHSM" -allow-unsupported-compiler -I "$PACKDIR" -o "$OUT/kernel.cubin" "$PACKDIR/kernel.cu" >/dev/null 2>&1; xs="$xs $(( $(now_ms) - t0 ))"; done + NVCC_MS=$(python3 -c "xs=sorted(int(x) for x in '$xs'.split()); print(xs[len(xs)//2] if xs else 'n/a')"); log "nvcc -cubin (the worker's prepare path): $xs ms, median $NVCC_MS" + if [ -x ./nvrtc-time ]; then ./nvrtc-time "$PACKDIR" "$ITER" | tee "$OUT/nvrtc.txt"; NVRTC_MS=$(grep -oE 'median [0-9.]+' "$OUT/nvrtc.txt" | cut -d' ' -f2); fi +fi +if command -v clinfo >/dev/null 2>&1 && clinfo 2>/dev/null | grep -q 'Device Name'; then + [ -x ./clbuild-time ] || cc -std=c99 -O2 -o clbuild-time clbuild-time.c -lOpenCL 2>/dev/null || true + [ -x ./clbuild-time ] && { ./clbuild-time "$PACKDIR/kernel.cl" "$ITER" | tee "$OUT/clbuild.txt"; CL_MS=$(grep -oE 'median [0-9.]+' "$OUT/clbuild.txt" | cut -d' ' -f2); } +fi + +# ---- results row ----------------------------------------------------------------------------------------------------- +L2=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -1 | grep -qE '5090' && echo 96 || echo "?") +ROW="| $CARD | $DRIVER / $CUDAV | $(date -u +%Y-%m-%d) | $RAW | $RAW_S | $R2 | $(printf '%s' "$SWEEP" | sed 's/^ //; s/ / \/ /g') | $CLIFF | $INLINE | $RATIO | $NVCC_MS | $NVRTC_MS | $CL_MS | $VECTORS | $BUILD_MS |" +HEAD="| card | driver / toolkit | date | Mhash/s at 1 GiB ($MINUTES min) | s | $PACK2 Mhash/s | sweep MiB:Mhash/s 64 / 128 / 256 / 512 / 1024 | 64 MiB over 1 GiB | inline Mhash/s | inline / honest | nvcc cubin ms | NVRTC ms | OpenCL build ms | vectors | build ms |" +[ -f results.md ] || { printf '%s\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n' "$HEAD" > results.md; } +printf '%s\n' "$ROW" >> results.md +printf '%s\n%s\n' "$HEAD" "$ROW" > "$OUT/row.md" +log "RESULT $ROW" + +# ---- upload ---------------------------------------------------------------------------------------------------------- +if [ "$UPLOAD" = 1 ]; then ./upload.sh "$OUT/run.log" "gpubench-$LABEL" "gpubench-$LABEL-$STAMP" || log "upload failed (the row is in results.md and $OUT/row.md)"; fi +log "done: $OUT/ (run.log, gate.txt, raw.txt, sweep-*.txt, inline.txt, nvrtc.txt, clbuild.txt, row.md)" diff --git a/infra/gpu-bench/upload.sh b/infra/gpu-bench/upload.sh new file mode 100755 index 000000000..7d416f2e7 --- /dev/null +++ b/infra/gpu-bench/upload.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# Upload a log (its last 256 KB) to the Igneum log intake, the same endpoint and key as +# proto-cuda/windows-miner/upload-log.bat (the key only authorises log uploads and ships inside the packages). +# Read back on the Mac with `node tools/logs.mjs`. +# ./upload.sh