65 lines
6.7 KiB
Markdown
65 lines
6.7 KiB
Markdown
# The rented GPU fleet: the standing rule and the one-shot rule
|
|
|
|
the project lead's ruling, 6 October 2026, 19:50 UK ("keep rented cards up"), written as the fleet agent runs it. Tooling in
|
|
`tools/fleet/` on branch `gpu-fleet`; every box operation goes through `tools/fleet/lib/` (the box library) and the
|
|
box-side scripts it ships.
|
|
|
|
## Two kinds of box
|
|
|
|
| Kind | Rule | Examples |
|
|
|---|---|---|
|
|
| Standing | stays up, never destroyed on a job's end; replaced in the same shape when its host dies; node under a supervisor; version follows the live manifest; never joins an experiment | the 13 live-devnet boxes of 6 October (USD 3.50/h), the Devnet 2 set (seed, 3 miners, 2 provers) |
|
|
| One-shot | rented for one measurement, destroyed the minute its measurement is in (a running meter is a bug) | the memory-matrix cards, the 8x rigs, the wave pods, the RISC Zero box |
|
|
|
|
The standing set's size and cost: 16 on the live devnet (a mix: 5090, 4090s, 3090s, 4070, 3080, A5000, an L4, at least
|
|
two AMD when Vast reopens) plus 6 on Devnet 2, about USD 8 an hour, USD 200 a day, inside the daily budget (USD 250 to
|
|
1,000). Tonight's 13 cost USD 3.50 an hour (USD 84 a day); the L4, the AMD pair and the Devnet 2 six are owed to the
|
|
roster as providers free cards (RunPod gave no pod of 8 card types from 18:59Z, Vast refuses every new rent).
|
|
|
|
## What a standing box carries
|
|
|
|
| Piece | Where | What it does |
|
|
|---|---|---|
|
|
| `box-standing.sh` (the supervisor) | `/root/fleet/in/`, started once by `lib/standing.py install` | every 60 s restarts the node, the miner loop (CUDA worker, pack re-exported on seed change) and the prover when gone; every 10 min writes a `RESULT standing` line (uptime, node pid and binary, blocks, DAA, peers, synced, exec tip, miner rate, prover) to `/root/fleet/out/standing.log`; runs the shipper's recovery recipe (`box-exec-snapshot.sh`) when consensus is synced above 1,000 blocks and the exec tip reads 0 |
|
|
| `/root/fleet/standing.node` | the box | the node binary the supervisor runs; the supervisor adopts whatever node already runs when the file is absent (a canary's newer binary is never downgraded) and restarts the node only when a publish rewrites the pointer |
|
|
| the registry row | `~/Desktop/fleet/boxes.json` | `standing: true`, `role: live|dn2`, `standing_since`, `node_sha16_wanted` after a publish |
|
|
| `lib/standing.py` | the Mac | `roster` (role, card, price, uptime), `install`, `check` (ssh alive, supervisor up, synced, exec moving, prover, node binary against the wanted sha), `update` (the manifest's hive package onto the box, pointer rewritten), `rerent` (same card and provider, label suffixed `-r<n>`, old row marked destroyed), `loop` (check every 10 min, re-rent after two dead checks, report what is behind) |
|
|
| the fleet page | `dl.igneum.network/fleet-22adafa34bc2/` | a `standing` block (count, USD/h, USD/day, roles, the rule) and `standing`, `role`, `uptime_h` on every box row |
|
|
|
|
A publish on the standing set is a script, not the loop: `tools/fleet/publish-0315.py` is the first (the Linux igneumd,
|
|
igneum-miner and the generator-4 workers over the package's, the pointer rewritten, the read-back table of version line,
|
|
digest and worker sha per box). The loop reports a box behind its wanted sha; it does not move binaries by itself.
|
|
|
|
## Ports
|
|
|
|
A standing box should carry a mapped p2p port whenever the provider allows it (RunPod: request `26611/tcp` at rent;
|
|
Vast: none of tonight's offers mapped one), so that the live devnet stops being a star in which every rented node dials
|
|
only the two hand nodes and the hub: the 18:39Z to 19:4xZ finality pause coincided with the hands being down, and run A
|
|
(10 blocks/s on a star, 77 to 84 percent red blocks) showed the same weakness on Devnet 2. Tonight's inbound-capable
|
|
nodes were igneum-build-1 (26611) and dn2-seed's one mapped port; a true mesh needs pods rented with a p2p port.
|
|
|
|
## Clock
|
|
|
|
- 19:41Z: the 13 live boxes converted (supervisor started, every node synced, every prover up).
|
|
- 20:00Z: the 10 percent rule, after the finality pause the fleet caused (`lib/standing.py weight_check`).
|
|
- 20:42Z to 20:48Z: the disk class (the prover's exports) found on the hub's third death; the supervisor prunes, `disk-sweep.py` watches, box-prover.py caps its own directory.
|
|
- 20:58Z: Devnet 2 standing at 1 block/s: the seed on igneum-build-1 (188.40.146.49:26611), miners dn2-seed, dn2-1, dn2-2, dn2-3, provers on dn2-1 and dn2-2 (the floor host copied from p1-4090); the gate owed until the rehearsal chain ends and dn2-seed's mapped port is free.
|
|
- 21:44Z to 21:53Z: publish 1 of 0.3.15 on the 14 live standing boxes (`publish-0315.py`: node 7f0bde70 = f1ea7a38, miner, generator-4 workers; the thirteen-field object, digest b18ed271).
|
|
- 22:18Z to 22:24Z: the forced poisoned-peer confirmation on the live network (relay on 6615571c over a 1026 datadir, a 713ef876 miner into it, the f1ea7a38 target refusing six relayed 1026 blocks with the old rule's line and never itself rejected).
|
|
- 22:49Z: publish 2 (the sixteen-field object: exec_restart_state_root, class v4 floor 831,600, window 86,400) on the 14 live standing boxes by `~/Desktop/fleet/move-16.py` (file to /root/fleet/override.json, the previous kept as override-13.json, node restarted by the supervisor, miner with it); expected digest eada4bda; `~/Desktop/fleet/expected-digest` rewritten so the five-minute check posts on any box left on b18ed271. The Devnet 2 five stay on their own object (4a0b8726).
|
|
- Owed: the L4 and the AMD pair when a provider reopens; the ports on re-rent.
|
|
|
|
## The 10 percent rule (6 October 2026, 20:00Z, from the finality pause)
|
|
|
|
What happened: the class v4 rehearsal job took the GPUs of 13 live-devnet miners between 18:27Z and 18:30Z (their
|
|
live nodes stayed up and synced, but a voter's weight is its blue blocks, and a miner that stops mining stops being
|
|
a voter), and with seven earlier leavers that was 42.7 percent of the frozen voter table; finality rule v3 then holds
|
|
the pause for one full window, and the first lock after 18:39:36Z is expected at about 20:40Z. The fleet's read that
|
|
"the boxes never left the devnet" was true of the nodes and wrong about the weight.
|
|
|
|
The rule: never remove more than 10 percent of the live devnet's 30-day weight in any hour. Weight is counted by blue
|
|
blocks per key over the window, read from the hub; any experiment that borrows live miners does it in slices with an
|
|
hour between slices; a standing box's miner is never stopped (or its GPU shared with a second worker) by a job
|
|
without that check. `lib/standing.py weight_check <labels>` is the gate: it reads the hub's blue blocks per payout key
|
|
over the window, sums the share of the labels asked for, and refuses the job when the share since the last hour's
|
|
removals exceeds 10 percent; every fleet job that touches a standing box's miner calls it first.
|