diff --git a/docs/plans/build-server.md b/docs/plans/build-server.md index 1b1cc697..d3195938 100644 --- a/docs/plans/build-server.md +++ b/docs/plans/build-server.md @@ -91,6 +91,11 @@ Also logged for context: the Mac's Linux cross-build with zig (`infra/cross/buil | An all-identical overlay listed only directories | the first run's touch pipeline got an empty file list and failed | files only are counted and re-stamped (lib.sh bs_overlay_dir) | | bash 3.2 on the Mac treats an empty array as unbound under `set -u` | run-from-mac.sh died on `PASS[*]` | a string instead of an array | | `grep -q` plus `pipefail` turned a strings hit into a miss | the commit-string gate failed a stamped igneumd on its first use (SIGPIPE on `strings`) | `grep -c` | +| A path dependency inside a vendor repo (the shipper's proving build, 6 Oct 2026) | proving/igneum-prove depends on vendor/igneum-node-exec/igneum/evm-types, a MEMBER of the fork's workspace (it inherits `thiserror` from the fork's root manifest); the first design synced that one directory, so cargo found no workspace root on the box ("failed to load manifest for workspace member"), and the shipper cross-built the Linux prove-host on the Mac with cargo-zigbuild meanwhile | lib.sh groups path dependencies by git top level: one under vendor/ is a whole repository, pushed to its mirror (a fork worktree such as igneum-node-exec goes to /srv/igneum-node.git, which already held its branch; a repository of its own gets /srv/.git, created on first use) and checked out whole at /srv/builds//vendor/; run-from-mac.sh wires every vendor repo the Cargo.toml files reach (today only igneum-node-exec). The detector's first version tested "under BS_TOP" before "own repository" and missed it, since vendor/ lies under the igneum top level on disk. Then `libprotobuf-dev` was missing (sp1-prover-types's build script imports google/protobuf/empty.proto); added to provision.sh. Proof: `cd proving/igneum-prove && tools/build-remote.sh -- build --release -p igneum-prove-host`: igneum-prove-host 71,943,192 B, sha256 e9213e3a6c979512d7859f6d8e848105bab53f4355e99fb0d30fb4a72c2d5714, ELF x86-64, 1 min 05 s warm (the cold run compiled 605 crates in 55 s before protoc stopped it). A Mac worktree has no vendor/ of its own, so a worktree that builds proving needs `git -C vendor/igneum-node worktree add /vendor/igneum-node-exec execution-layer` first, as the fork worktrees do | +| One remote build lost its ssh session after 75 s (18:30:50 UTC, the first full proving build) | the remote bash died with it (slot line left behind, no JSONL line); no OOM, no reboot, the retry a minute later passed | the dashboard collector's one-pass unit finished within a second of the drop, so it was tested: a 90 s remote session through the same ControlMaster path survived two collector passes triggered by hand; the collector only reads (/proc, lock files, `flock -n`, `sccache --show-stats`, `kill(pid, 0)`). One event, no cause in the journal, the retry passed. A build that must survive a dropped connection would need the remote command under setsid with the Mac reconnecting to wait; not done, open if it happens again | +| A stamp file at a nested path failed every checkout of /srv/builds/igneum (18:31 to 18:5x UTC, the shipper's 18:52 run) | a repo-kind crate keeps its `.build-remote-sha-` and `target/` inside the crate dir; the checkout mode's clean-tree test only excused them at the tree root | the test is depth-agnostic and uses `--untracked-files=all` (a wholly untracked directory is otherwise collapsed to `?? dir/`); the self-test carries a nested stamp, a nested target dir and a stale `.git/index.lock`, which the mode now removes when no git runs there | +| Two runs on one worktree at once (the shipper, 18:48:56Z) | the second run's checkout replaced the first's sources mid-cargo; both died | lib.sh takes a per-worktree lock on the box (`/srv/builds/_locks/wt-`, mkdir-atomic, holder line) across sync, build and fetch; a second run waits up to 2 h (a line every minute), a lock older than 3 h is taken over; released on EXIT. The build slot (`build-`) is unchanged | +| The 0.3.15 prover pair for the PCs needs `--features igneum-prove-host/cuda` (the PCs run SP1_PROVER=cuda) | my first proving build named no feature | built on the box from master e1b5bc9: igneum-prove-host 73,161,528 B sha256 71bc2438856bb141f6cad3d18489f708568144fad5a06a002fbadefb9ce256f9, igneum-prove-export 3,609,360 B sha256 263bf4cef70af4a13a45b2e79b8dbab373282ea4c571f624d02ddb5791935361 (52 s warm, no CUDA needed at build time); handed to the shipper | | Let's Encrypt saw NXDOMAIN for build.igneum.network | the deSEC record was minutes old; Ubuntu's Caddy then fell back to ZeroSSL and failed with HTTP 422 for ever | issuer pinned to Let's Encrypt; the retry got the certificate | ## 6. What the box does not do yet diff --git a/docs/plans/hands-on-build-1.md b/docs/plans/hands-on-build-1.md new file mode 100644 index 00000000..fbe4d5c4 --- /dev/null +++ b/docs/plans/hands-on-build-1.md @@ -0,0 +1,93 @@ +# The devnet hands move to igneum-build-1 (node 1 and the observer) + +the project lead's decision, 6 October 2026: the Mac runs nothing the network depends on. After the 0.3.15 cut tonight, node 1 and the +observer leave the Mac's launchd agents and run on igneum-build-1 (188.40.146.49, Falkenstein, the build server of +docs/plans/build-server.md) as systemd units. The shipper (owner of infra/devnet/restart-hand-nodes.sh and the exec recovery +recipe) and the build-server agent run it together on the shipper's "0.3.15 live" line. + +## 1. What runs on the Mac today (read 6 Oct 2026, 19:5x UK) + +| Hand | How it runs | Flags that matter | Data | +|---|---|---|---| +| node 1 | launchd `network.igneum.devnet.node1`, KeepAlive, under `caffeinate -dims`, binary `igneum-wt-ship0314/vendor/igneum-node-0314/target-integration/release/igneumd` (0.3.14), log `~/Library/Logs/Igneum/node1.out` | `--devnet --enable-unsynced-mining --appdir=/tmp/igneum-devnet/node1 --rpclisten=0.0.0.0:26610 --listen=0.0.0.0:26611 --evm-rpclisten=127.0.0.1:26791 --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611 --override-params-file=/tmp/igneum-devnet/override-v3.json --igneum-exec-snapshot=/tmp/igneum-devnet/node1-copy-snapshot.bin,ac101f13... --nodnsseed --disable-upnp --nologfiles --yes` | 856 MB (`consensus`, `evm` with exec-snapshot.bin 127,564,588 B and .prev, `meta`) | +| observer node | launchd `network.igneum.devnet.observer`, KeepAlive, same binary, log `observer.out` | `--appdir=/tmp/igneum-devnet/observer-v4 --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 --listen=127.0.0.1:26641 --addpeer=127.0.0.1:26611 --addpeer=188.245.5.161:26611 --addpeer=192.168.68.67:26611` + the same override and snapshot flags; no EVM listener | 822 MB | +| observer process | `tools/observer/run.sh` loop (nohup, since 5 Oct 20:39) running `node tools/observer/observer.mjs` from the shared checkout, `IGNEUM_RPC=ws://127.0.0.1:28640`, `IGNEUM_EVM_RPC` default `http://127.0.0.1:26800` (the Miner app's node), `DATABASE_URL` from `~/.config/igneum/env`; `tools/observer/autosync.sh` (since 4 Oct) fast-forwards the shared checkout and restarts it on observer changes | writes Neon (`live_*` tables); `/api/live` and `/live` read Neon only | + +Also found: the Mac's own miner (`igneum-miner`, pid 7721) holds a gRPC connection to node 1 on 26610. The override file today: +`{"difficulty_v2_activation_daa":33000,"proving_v0_activation_daa":84100,"fees_v1_activation_daa":210000,"finality_v3_activation_daa":135200,"program_class_v3_activation_daa":154800,"proving_v1_activation_daa":154800,"proving_v1_segment_blocks":8,"proving_v1_unproven_daa":600,"proving_v1_aggregator_share_bps":1000,"proving_v1_fresh_rule_daa":198000,"exec_restart_number":27276,"exec_restart_hash":"bb45cf0d...","exec_restart_trust_daa":200000}`. +Node 1's last exec lines: `exec state loaded from a snapshot: tip 141700 ...` and `exec sync: resumed from .../node1/igneum-devnet/datadir/evm/exec-snapshot.bin (tip 141700, 127564588 bytes, sha256 0x67637c55...)`. + +## 2. What runs on the box afterwards + +| Unit | Runs | Ports | Files | +|---|---|---|---| +| `igneum-node1.service` | `/srv/hands/bin/run-node1.sh` -> `/srv/hands/bin/igneumd` (0.3.15 Linux, the box's own build), `--appdir=/srv/hands/node1`, `--externalip=188.40.146.49` | p2p `0.0.0.0:26611` (ufw open); gRPC `127.0.0.1:26610`, wRPC JSON `127.0.0.1:28610`, EVM `127.0.0.1:26791` on loopback | `/srv/hands/hands.env` (IGNEUMD, OVERRIDE, SNAPSHOT, EXTERNAL_IP, PEERS), `/srv/hands/override.json`, `/srv/hands/node1-copy-snapshot.bin` | +| `igneum-observer-node.service` | `run-observer-node.sh`, `--appdir=/srv/hands/observer-node`, peers node 1 on the box and the seed | p2p `127.0.0.1:26641`, gRPC `127.0.0.1:26640`, wRPC JSON `127.0.0.1:28640`, EVM `127.0.0.1:26840` (new: the observer's proving feed came from the Miner app's node on the Mac) | same env and override | +| `igneum-observer.service` | `/usr/local/bin/node tools/observer/observer.mjs` in `/srv/observer/igneum` (a clone of the mirror `/srv/igneum.git`, master), `EnvironmentFile=/srv/observer/env` (mode 600, owner build: DATABASE_URL copied from the Mac's `~/.config/igneum/env`, `IGNEUM_RPC=ws://127.0.0.1:28640`, `IGNEUM_EVM_RPC=http://127.0.0.1:26840`; never in the repo), Restart=always 3 s (run.sh's loop) | outbound to Neon only | `/srv/observer/env` | +| `igneum-observer-sync.timer` | every 5 min `observer-sync.sh`: fast-forward the clone from the mirror, restart the observer when `tools/observer` or `site/lib` changed (autosync.sh's job; the mirror is fed by every build-remote.sh and run-from-mac.sh push) | | | + +All units: `Restart=always`, journald (`journalctl -u igneum-node1 -f`), `MemoryMax=24G` on the nodes, enabled for reboot, run as user +`build` (one uid on a single-purpose box; the units are the separation). Installer: `infra/build-server/hands/install-hands.sh` +(idempotent, inert: enables, never starts). Mover: `infra/build-server/hands/move-hand.sh` (dry run by default, `--go` executes). + +## 3. Ports, names, firewall + +| Port | On the Mac today | On the box | ufw | +|---|---|---|---| +| 26611 TCP | node 1 p2p, open | node 1 p2p, open, `--externalip` so peers learn it | open already | +| 26610 TCP | node 1 gRPC on `0.0.0.0` (the Mac's miner connects to it) | gRPC on loopback; a Mac tool reaches it through `ssh -L 26610:127.0.0.1:26610 build@188.40.146.49` | not opened (decision for main: open it to a fixed IP list if a miner must feed node 1 from outside) | +| 26791 TCP | node 1 EVM RPC, loopback | loopback | closed | +| 26640, 28640, 26641 TCP | observer node, loopback | loopback (+ EVM 26840) | closed | +| 26811 TCP | (the TESTNET seed p2p port, not an RPC port: infra/seed-nodes/config.sh) | unused by the hands | open from provision; harmless | + +DNS (deSEC, main adds): `node1.devnet.igneum.network A 188.40.146.49`, `observer.devnet.igneum.network A 188.40.146.49`. The +observer node listens on loopback only, so its name is for the future public endpoint and for the runbooks' wording. The public +API (`/api/live`, `/live`) reads Neon and is unchanged. + +## 4. The move, one hand at a time (run on the shipper's "0.3.15 live" line) + +| Step | Command (Mac) | What happens | Proof | +|---|---|---|---| +| 0 | `ssh root@188.40.146.49 'bash -s' < infra/build-server/hands/install-hands.sh` | units, run scripts, dirs, the observer clone; nothing started (done 6 Oct, see section 6) | `systemd-analyze verify` clean, units enabled and inactive | +| 1 | `move-hand.sh binary --node /Users/joshm/Projects/igneum-wt-ship0315/vendor/igneum-node-0315` | the 0.3.15 Linux igneumd built ON the box (tools/build-remote.sh), installed as `/srv/hands/bin/igneumd-0.3.15-`, commit string checked; the Mac's override file and the exec snapshot copied; hands.env pointed at them | `igneumd --version`, commit in strings, sha256 lines | +| 2 | `move-hand.sh observer-node --go` | hot rsync of `/tmp/igneum-devnet/observer-v4` (822 MB) while the Mac node runs; `launchctl bootout` of the Mac's observer agent (KeepAlive would restart a killed pid); final rsync (the delta, seconds, node stopped so RocksDB is consistent); `systemctl start igneum-observer-node` | the unit's first `[igneum-exec] exec sync: resumed from ...` line and its first `Accepted N blocks` lines, printed by the script | +| 3 | `move-hand.sh observer --go` | `/srv/observer/env` written (scp, mode 600, owner build); the Mac's autosync.sh, run.sh and observer.mjs stopped FIRST (two writers would duplicate Neon rows), then `systemctl start igneum-observer` | the observer's first journal lines (`rpc load`, events); `/api/live` fresh within a minute | +| 4 | `move-hand.sh node1 --go` | the same as step 2 for node 1 (856 MB); node 1 is the last node the Mac serves, the observer node on the box already peers with the seed, so the network never loses both hands | node 1's first exec line and `PoW accepted` lines on the box | +| 5 | `move-hand.sh unload --go` | the two plists moved aside so a login never brings the Mac hands back; the Mac's miner loses node 1's RPC (section 5) | `pgrep igneumd` on the Mac shows only the wallet's node | +| 6 | `move-hand.sh status`, 10 minutes later | both nodes at the network tip, observer writing, `/live` current | the status output in this plan's section 6 | + +Exec recovery on the box: each node resumes from its data dir's `evm/exec-snapshot.bin` (copied with the data dir); if that file +is bad or missing, `--igneum-exec-snapshot=/srv/hands/node1-copy-snapshot.bin,` (the Mac's recovery snapshot, copied in +step 1) is taken, as the launchd agents do today; the override's `exec_restart_number`/`exec_restart_hash`/`exec_restart_trust_daa` +travel unchanged. A snapshot from the seed is the fallback the shipper's recipe names; the p2p snapshot path refuses one below the +node's tip (CLAUDE.md, Devnet 2 rules). + +Rollback at any step: the Mac's plist is still in `~/Library/LaunchAgents` until step 5; `launchctl bootstrap gui/$(id -u) ` +brings a hand back on the Mac within 10 s, and the box unit is stopped with `systemctl stop`. Data dirs are copies; nothing is deleted +on the Mac. + +## 5. Decisions and consequences for main (DECIDED by main, 6 October 2026, 19:1x UTC) + +| Question | Decision | +|---|---| +| The Mac's miner on node 1's gRPC | stops (paused by the project lead's order anyway; the Mac mines nothing) | +| The LAN peer 192.168.68.67 (a PC) | the shipper adds `--addpeer=188.40.146.49:26611` to the PCs' and the Mac's app node args in the 0.3.15 update; the seeds carry the public peers already | +| node 1's gRPC 26610 | loopback only on the box; Mac tools tunnel | +| DNS | `node1.devnet.igneum.network` and `observer.devnet.igneum.network` -> 188.40.146.49 exist in deSEC (checked: ns1.desec.io answers both) | + +The table below is the reasoning that led to them. + + +| Finding | Means | Proposed | +|---|---|---| +| The Mac's own miner (`igneum-miner` pid 7721) mines through node 1's gRPC 26610 | after step 4 it loses its node; the project lead's rule says the Mac runs nothing the network depends on, and a miner is hashrate, not a dependency | either it stops with node 1, or it follows through an ssh tunnel to the box (`ssh -L 26610:...`); main decides, default: it stops | +| `192.168.68.67:26611` is a LAN peer of both hands | unreachable from the box; the box peers with the seed 188.245.5.161 and the two hands peer with each other | hands.env `PEERS=188.245.5.161:26611`; whoever runs 192.168.68.67 adds `--addpeer=188.40.146.49:26611` if it relied on node 1 | +| CLAUDE.md "Running agents on this Mac" says the box never hosts a live-devnet node (my R6, 6 Oct 18:xx) | contradicted by the project lead's decision the same evening | rewritten in this commit: the box hosts the two hands as units; it still holds no secret beyond `/srv/observer/env` (DATABASE_URL, mode 600) | +| The observer's proving feed on the Mac read the Miner app's node (26800) | on the box there is no app node | the observer node gets `--evm-rpclisten=127.0.0.1:26840` (0.3.15 runs the proving build) and the observer reads it; if its shard plans lag node 1's, point `IGNEUM_EVM_RPC` at node 1's 26791 | +| autosync.sh followed origin/master from GitHub; the box has no GitHub credential | the box's clone follows the MIRROR, which moves only when a Mac agent pushes (every build-remote.sh and run-from-mac.sh run pushes the branch it builds; run-from-mac.sh pushes every branch) | enough today; a read-only deploy key on the box (generated there, added by main to the repository) would make it follow GitHub directly, open | +| Two hands stop for one to three minutes each during the move (the final rsync and the start) | the other hand serves throughout; the observer feed pauses once for about a minute (step 3) | accepted by the order above | +| Data dirs are rsynced hot then with the node stopped | the hot pass moves 99 percent of 1.7 GB with the hands up; the stopped pass is the delta | the Mac's upload rate decides the hot pass (minutes); measured in section 6 | + +## 6. Run log (filled as it happens) + +RUNLOG diff --git a/docs/plans/pool.md b/docs/plans/pool.md new file mode 100644 index 00000000..30c21215 --- /dev/null +++ b/docs/plans/pool.md @@ -0,0 +1,238 @@ +# Pool v0: what it does, what it does not, what was measured, and the listing checklist + +5 October 2026, branch `pool-v0` (main worktree `../igneum-wt-pool`, fork worktree `vendor/igneum-node-pool` on the +fork's `pool-v0` from `release-0.3.6` a24ab01a). REBASED 6 October 2026 (section 9): branch `pool-v0-rebase` on master, fork +branch `pool-v0-rebase` on `release-0.3.14-node` 4c6b129d (worktree `../igneum-wt-pool-rebase`, fork worktree +`vendor/igneum-node-pr`); the share re-check hashes the template's program class, never a fixed one. the project lead's ask: a public mining pool with the stats API WhatToMine and +HiveOS read, "the biggest step toward being listable". Built against `docs/spec/09-pool-protocol.md`, the miner's +template subscription and worker protocol (`igneum/miner/src/main.rs`), the Hive package's `local` mode +(`packaging/hive`), ledger F10 and G6 (pools and votes), and `docs/plans/explorer.md` section 6. Nothing is deployed. + +## 1. Rust, and why + +Spec 9.3 fixes the member protocol as newline-delimited JSON over TLS, not the node's gRPC. The miner is Rust and +talks gRPC to its node; the pool must verify every share on the CPU warp verifier (spec 9.8 item 5), which is the +node's own `IgneumEngine` in `consensus/pow/src/igneum.rs` (same program, cache and init words as block validation), +and it must build templates and submit blocks over the node's gRPC. Both are Rust crates by path. A Node service +would have had to reimplement the verifier or shell out per share; the Rust pool calls it in process at 0.44 ms per +share (measured, section 5). JSON lines are three serde derives. So: Rust, `pool/`, its own Cargo workspace reading +the fork by path. The miner side is a module in the fork (`igneum/miner/src/pool.rs`), because the identity, voter, +template and worker code it reuses lives there. + +## 2. What v0 does + +| Area | v0 | Spec section | +|---|---|---| +| Transport | newline JSON over plain TCP on 4463 (devnet), 4462, 4461; one object per line, `"t"` names the type, unknown fields ignored, 4 MiB line cap | 9.3, 9.5 | +| Session | `hello` / `welcome` (version, chain id, pool address, modes, `vote_mode: member`, `share_scheme` with fee, window and min payout), `authorize` with the member's BLS pubkey and proof of possession (verified with `verify_pop`), payout address and worker name / `authorized` | 9.3, 9.5 | +| Templates | mode A, one template per member per tip (NewBlockTemplate subscription, 1 s refresh, on demand): the member's `vote_key_hash` in the header, its key reveal and the pool's `IGNA` payout address in the coinbase extra data; sent as the node's `RpcRawBlock` JSON so a member with a node can submit it there | 9.4, 9.4.1 items 1 and 2, 9.6 item 2 | +| Seeds | `seeds` on connection and on every change, from the template's `pow_epoch` (current and next epoch seed, boundary DAA, day, the schedule; since 6 October 2026 also `program_class`, `next_program_class`, `era_seed`, `era_index`, `genesis_day_index`, `genesis_dataset_log2`, `program_class_v3_activation_daa`, so a member without a node builds the same program and day cache as the pool's node) | 9.9 | +| Jobs | `job` with prehash, `target64`, `share_target64`, a random 2^32 nonce space, the seed pair, `program_class` and `era_seed` (6 October 2026: the member refuses a job without a class, code `seeds`, and never hashes a guessed one), `clean` on a tip change | 9.5 | +| Shares | `share` = a lane hash at or below `share_target64`; the pool evaluates the lane on the CPU and answers `share_result` with `ok`, `stale` (superseded job past the 2,000 ms grace), `duplicate`, `above_target`, `wrong_hash`, `unknown_job`; a share at or below `target64` is a block, submitted to every node | 9.8 items 1, 5, 7 | +| Vardiff | per member, one share per 10 s, shift `s` with `share_target = target64 << s` never saturated; first correction sized from the measured rate (up to 8 steps), then one step per 30 s at most; idle members eased one step per 3 intervals | 9.8 items 2 and 4 (one deviation: the sized first step, below) | +| Weight | `2^-s` of a block per share | 9.8 item 3 | +| Payment | PPLNS over a window of N blocks of weight (default 2); the split is snapshotted when a block is found and credited when it is confirmed blue (a chain block or in a chain block's mergeset blues, the set the execution layer pays); orphans pay nobody; fee (default 1%) off the top | 9.8 item 8 | +| Payouts | EIP-1559 transfers from the pool's coinbase address through the node's eth_ JSON-RPC, one per payee at or above the minimum, at most 16 per round, receipts checked, a failed receipt re-credits; `--dry-run` records and sends nothing | design 1.1 (execution layer rewards) | +| The 20% | untouched: the execution layer credits the proving escrow by rule from consensus data (`executor.rs`); the pool's templates name only its own address and receive the 80% | 2.5, 5.3 | +| Votes | the member votes through its own node with its own key, as in solo mode; the pool holds no key and names each member's key in its header | 9.6, 9.7 items 1 and 2 | +| Member checks | own key in the header, the pool's address and own key reveal in the coinbase (refused with `vote_key`, `payout`, `reveal`); the pool's epoch seed, program class and era seed against the member's node (`seeds`; the class and era joined the check on 6 October 2026); a custodial pool (`vote_mode` not `member`) refused by default | 9.4.1 items 1 and 2, 9.9 item 2, 9.6 item 5 | +| Found blocks | the member submits to its own node and sends `solution`; the pool submits to every node | 9.2 | +| API | `/api/stats`, `/api/blocks`, `/api/miners/
`, `/api/payments`, `/api/pool-stats` (Hive-style flat object); field contracts in `pool/src/api.rs`, fixtures from the measured run | explorer.md section 6 | +| Page | `pool/web/index.html` under the site's tokens: tiles, connect card with the exact miner flags, address lookup, blocks, payments | | +| Persistence | `state.json` snapshot every 15 s and at shutdown (balances, blocks, payments, PPLNS window, lifetime counters) | | +| Miner | `igneum-miner mine