4 KiB
The rented GPU fleet: the standing rule and the one-shot rule
the project lead's ruling, 6 October 2026, 19:50 UK ("keep rented cards up"), written as the fleet agent runs it. Tooling in
tools/fleet/ on branch gpu-fleet; every box operation goes through tools/fleet/lib/ (the box library) and the
box-side scripts it ships.
Two kinds of box
| Kind | Rule | Examples |
|---|---|---|
| Standing | stays up, never destroyed on a job's end; replaced in the same shape when its host dies; node under a supervisor; version follows the live manifest; never joins an experiment | the 13 live-devnet boxes of 6 October (USD 3.50/h), the Devnet 2 set (seed, 3 miners, 2 provers) |
| One-shot | rented for one measurement, destroyed the minute its measurement is in (a running meter is a bug) | the memory-matrix cards, the 8x rigs, the wave pods, the RISC Zero box |
The standing set's size and cost: 16 on the live devnet (a mix: 5090, 4090s, 3090s, 4070, 3080, A5000, an L4, at least two AMD when Vast reopens) plus 6 on Devnet 2, about USD 8 an hour, USD 200 a day, inside the daily budget (USD 250 to 1,000). Tonight's 13 cost USD 3.50 an hour (USD 84 a day); the L4, the AMD pair and the Devnet 2 six are owed to the roster as providers free cards (RunPod gave no pod of 8 card types from 18:59Z, Vast refuses every new rent).
What a standing box carries
| Piece | Where | What it does |
|---|---|---|
box-standing.sh (the supervisor) |
/root/fleet/in/, started once by lib/standing.py install |
every 60 s restarts the node, the miner loop (CUDA worker, pack re-exported on seed change) and the prover when gone; every 10 min writes a RESULT standing line (uptime, node pid and binary, blocks, DAA, peers, synced, exec tip, miner rate, prover) to /root/fleet/out/standing.log; runs the shipper's recovery recipe (box-exec-snapshot.sh) when consensus is synced above 1,000 blocks and the exec tip reads 0 |
/root/fleet/standing.node |
the box | the node binary the supervisor runs; the supervisor adopts whatever node already runs when the file is absent (a canary's newer binary is never downgraded) and restarts the node only when a publish rewrites the pointer |
| the registry row | ~/Desktop/fleet/boxes.json |
standing: true, `role: live |
lib/standing.py |
the Mac | roster (role, card, price, uptime), install, check (ssh alive, supervisor up, synced, exec moving, prover, node binary against the wanted sha), update (the manifest's hive package onto the box, pointer rewritten), rerent (same card and provider, label suffixed -r<n>, old row marked destroyed), loop (check every 10 min, re-rent after two dead checks, report what is behind) |
| the fleet page | dl.igneum.network/fleet-22adafa34bc2/ |
a standing block (count, USD/h, USD/day, roles, the rule) and standing, role, uptime_h on every box row |
A publish on the standing set is a script, not the loop: tools/fleet/publish-0315.py is the first (the Linux igneumd,
igneum-miner and the generator-4 workers over the package's, the pointer rewritten, the read-back table of version line,
digest and worker sha per box). The loop reports a box behind its wanted sha; it does not move binaries by itself.
Ports
A standing box should carry a mapped p2p port whenever the provider allows it (RunPod: request 26611/tcp at rent;
Vast: none of tonight's offers mapped one), so that the live devnet stops being a star in which every rented node dials
only the two hand nodes and the hub: the 18:39Z to 19:4xZ finality pause coincided with the hands being down, and run A
(10 blocks/s on a star, 77 to 84 percent red blocks) showed the same weakness on Devnet 2. Tonight's inbound-capable
nodes were igneum-build-1 (26611) and dn2-seed's one mapped port; a true mesh needs pods rented with a p2p port.
Clock
- 19:41Z: the 13 live boxes converted (supervisor started, every node synced, every prover up).
- Owed: the Devnet 2 six after the block-rate runs; the L4 and the AMD pair when a provider reopens; the ports on re-rent.