44 lines
4 KiB
Markdown
44 lines
4 KiB
Markdown
# The rented GPU fleet: the standing rule and the one-shot rule
|
|
|
|
the project lead's ruling, 6 October 2026, 19:50 UK ("keep rented cards up"), written as the fleet agent runs it. Tooling in
|
|
`tools/fleet/` on branch `gpu-fleet`; every box operation goes through `tools/fleet/lib/` (the box library) and the
|
|
box-side scripts it ships.
|
|
|
|
## Two kinds of box
|
|
|
|
| Kind | Rule | Examples |
|
|
|---|---|---|
|
|
| Standing | stays up, never destroyed on a job's end; replaced in the same shape when its host dies; node under a supervisor; version follows the live manifest; never joins an experiment | the 13 live-devnet boxes of 6 October (USD 3.50/h), the Devnet 2 set (seed, 3 miners, 2 provers) |
|
|
| One-shot | rented for one measurement, destroyed the minute its measurement is in (a running meter is a bug) | the memory-matrix cards, the 8x rigs, the wave pods, the RISC Zero box |
|
|
|
|
The standing set's size and cost: 16 on the live devnet (a mix: 5090, 4090s, 3090s, 4070, 3080, A5000, an L4, at least
|
|
two AMD when Vast reopens) plus 6 on Devnet 2, about USD 8 an hour, USD 200 a day, inside the daily budget (USD 250 to
|
|
1,000). Tonight's 13 cost USD 3.50 an hour (USD 84 a day); the L4, the AMD pair and the Devnet 2 six are owed to the
|
|
roster as providers free cards (RunPod gave no pod of 8 card types from 18:59Z, Vast refuses every new rent).
|
|
|
|
## What a standing box carries
|
|
|
|
| Piece | Where | What it does |
|
|
|---|---|---|
|
|
| `box-standing.sh` (the supervisor) | `/root/fleet/in/`, started once by `lib/standing.py install` | every 60 s restarts the node, the miner loop (CUDA worker, pack re-exported on seed change) and the prover when gone; every 10 min writes a `RESULT standing` line (uptime, node pid and binary, blocks, DAA, peers, synced, exec tip, miner rate, prover) to `/root/fleet/out/standing.log`; runs the shipper's recovery recipe (`box-exec-snapshot.sh`) when consensus is synced above 1,000 blocks and the exec tip reads 0 |
|
|
| `/root/fleet/standing.node` | the box | the node binary the supervisor runs; the supervisor adopts whatever node already runs when the file is absent (a canary's newer binary is never downgraded) and restarts the node only when a publish rewrites the pointer |
|
|
| the registry row | `~/Desktop/fleet/boxes.json` | `standing: true`, `role: live|dn2`, `standing_since`, `node_sha16_wanted` after a publish |
|
|
| `lib/standing.py` | the Mac | `roster` (role, card, price, uptime), `install`, `check` (ssh alive, supervisor up, synced, exec moving, prover, node binary against the wanted sha), `update` (the manifest's hive package onto the box, pointer rewritten), `rerent` (same card and provider, label suffixed `-r<n>`, old row marked destroyed), `loop` (check every 10 min, re-rent after two dead checks, report what is behind) |
|
|
| the fleet page | `dl.igneum.network/fleet-22adafa34bc2/` | a `standing` block (count, USD/h, USD/day, roles, the rule) and `standing`, `role`, `uptime_h` on every box row |
|
|
|
|
A publish on the standing set is a script, not the loop: `tools/fleet/publish-0315.py` is the first (the Linux igneumd,
|
|
igneum-miner and the generator-4 workers over the package's, the pointer rewritten, the read-back table of version line,
|
|
digest and worker sha per box). The loop reports a box behind its wanted sha; it does not move binaries by itself.
|
|
|
|
## Ports
|
|
|
|
A standing box should carry a mapped p2p port whenever the provider allows it (RunPod: request `26611/tcp` at rent;
|
|
Vast: none of tonight's offers mapped one), so that the live devnet stops being a star in which every rented node dials
|
|
only the two hand nodes and the hub: the 18:39Z to 19:4xZ finality pause coincided with the hands being down, and run A
|
|
(10 blocks/s on a star, 77 to 84 percent red blocks) showed the same weakness on Devnet 2. Tonight's inbound-capable
|
|
nodes were igneum-build-1 (26611) and dn2-seed's one mapped port; a true mesh needs pods rented with a p2p port.
|
|
|
|
## Clock
|
|
|
|
- 19:41Z: the 13 live boxes converted (supervisor started, every node synced, every prover up).
|
|
- Owed: the Devnet 2 six after the block-rate runs; the L4 and the AMD pair when a provider reopens; the ports on re-rent.
|