7.4 KiB
7.4 KiB
Build queue: the nine build boxes and their standing work
Set up 9 October 2026, 09:1x UK, on the order relayed by the coordinator: every box has one owner lane and a standing queue, a reader on build-1 reads this file every 30 minutes against the merged workers.json and reports an idle box as a fault. The live copy is build-1:/srv/queue/build-queue.md (the reader reads that one; this file is its source on master, landed through the gate).
Rules that bind every entry
- Kill by recorded pid or pid file only, never by a name or a pattern (
pkillandkillallrefuse on every box, exit 97; tools/ci/kill-by-name-check.sh reads the tree). Every job writes its pid file before it starts; a stop reads that file. - A box never builds for the fleet: fleet binaries come only from the cross-build kit (x86-64-v3, the ISA gate at 0 AVX-512 lines).
- Nothing on the chain by hand; the devnet, hub-1 and every node are the node and fleet lanes' through their own harnesses.
- An entry carries its owner lane, its box, its pid-file path and its clock (UK). A box with no live pid file from its queue and a load under 1.0 for a 30-minute read is a fault, reported by the reader to the coordinator (/srv/queue/faults.log).
- A lane that finishes an entry replaces it with the next or hands the box back here with a line; an empty box is the fault.
- A box over 90 percent memory is a fault line from the reader to the coordinator (9 October 2026, 10:0x UK), with the box-1 rule above.
- Main's rule (9 October 2026, 09:5x UK): an owner starts its matrix job on its box under /srv/queue/pids within 15 minutes of a read that found the box idle, or the reader marks the box unclaimed (open to any lane, written in status.json and faults.log); a box idle for 30 minutes is a red against its owner on the steward's board. A shared box names every owner and none ends another's pid.
Owners (one lane per box)
| box | threads | owner lane | standing use |
|---|---|---|---|
| build-1 | 96 | build-server lane | cuts and kits, the hands (observer-node, node1), the devnet-4 seed and hand, the light-reader, the capacity fuzz slices, the workers page, the queue reader. MEMORY RULE (the OOM of 09:54 to 10:04 UK, 9 Oct 2026: a fresh sync with a public listener reached 85 GB and the kernel killed nodes, Caddy, cron and rsyslogd): one fresh sync at a time, with its listener on loopback until within ten; one read node at most beside the hands, a second is a refusal; the queue reader faults any box over 90 percent memory |
| build-2 | 96 | site lane and the adversary lane (shared; neither ends the other's pid) | the site gate (Playwright), the scene-parity suites; X5's dram design beside them |
| build-3 | 32 | node lane | the long consensus fuzz and property suites (kaspa-consensus, kaspa-consensus-core) |
| build-4 | 96 | adversary lane | the chip model (OpenROAD, kepler-formal): the 20 to 25 percent floorplan for the converged SPEF row |
| build-5 | 32 vCPU | adversary lane (X5 hyb50), the TV-02 lane a light guest for minutes | X5's opponent-sweep variant; TV-02's last comparison run beside it (one core, under five minutes) |
| build-6 | 32 vCPU | adversary lane (X5 hyb75) and the HEAL lane (shared) | X5's opponent-sweep variant; the long p2p and exec property suites (kaspa-p2p-flows, igneum-exec) |
| build-7 | 96 | node lane and the relay lane (shared; neither ends the other's pid) | the 2.0.3 flows items: their suites and known-failed tests; the heal-on harness; the standing Windows canary VM (docker winvm, 4 vCPU, 8 GB, /srv/winvm/up.pid) |
| build-8 | 96 | fleet lane | 2.0.3 kit canaries the moment a chain moves; the heal-off harness; the dn4 roll node |
| build-9 | 96 | steward | the board's rows that need only a box (ZKP, EVM, VER fixtures), the pow fuzz on the side |
Queue (first entries, 9 October 2026)
| # | entry | owner | box | pid file | clock (UK) |
|---|---|---|---|---|---|
| 1 | 2.0.3 flows items: the suites and the known-failed tests on release-2.0.3-node-k6 (bf60948a and after) | node lane | build-7 | /srv/queue/pids/build-7-node-flows.pid | from 09:30, continuous |
| 2 | long fuzz and property suites, consensus (kaspa-consensus, kaspa-consensus-core) | node lane | build-3 | /srv/queue/pids/build-3-consensus-fuzz.pid | from 09:45, continuous |
| 3 | long fuzz and property suites, exec and p2p (igneum-exec, kaspa-p2p-flows, kaspa-p2p-lib) | HEAL lane | build-6 | /srv/queue/pids/build-6-exec-p2p-fuzz.pid | from 09:45, continuous |
| 4 | long fuzz and property suites, pow (kaspa-pow, igneum-pow mixer and scratch) | node lane | build-9 | /srv/queue/pids/build-9-pow-fuzz.pid | from 09:45, continuous |
| 5 | TV-02: two independent supply-replay implementations, CPU only: DONE 09:35 UK (A on build-5 in 38 s, B on build-6 in 3 min); the last comparison on the TV-01 spec runs as a guest under build-5-tv02.pid and build-6-tv02.pid by 11:30 | TV-02 lane | build-5, build-6 | /srv/queue/pids/build-5-tv02.pid, build-6-tv02.pid | by 11:30, minutes |
| 6 | chip model: the 20 to 25 percent floorplan for the converged SPEF row (about four host-hours) | adversary lane | build-4 | /srv/queue/pids/build-4-floorplan.pid | running (OpenROAD 844081, kepler-formal 1797623), about 13:30 |
| 7 | the board's rows that need only a box: ZKP, EVM and VER fixtures (the steward names them) | steward | build-9 | /srv/queue/pids/build-9-board-rows.pid | from 10:00, by row |
| 8 | 2.0.3 kit canaries the moment a chain moves (the evidence kits under /srv/workers/fleet) | fleet lane | build-8 | /srv/queue/pids/build-8-kit-canary.pid | on the chain's move |
| 9 | the site gate and the scene-parity suites | site lane | build-2 | /srv/queue/pids/build-2-site-gate.pid | on each landing |
| 11 | the standing Windows canary VM on build-7 (docker winvm; its collector posts to the intake as the Windows canary card) | relay lane | build-7 | /srv/queue/pids/build-7-winvm.pid (mirrors /srv/winvm/up.pid) | from 10:00, standing |
| 10 | cuts, kits, the hands, the capacity fuzz slices, the workers page, the queue reader | build-server lane | build-1 | /srv/queue/queue-reader.pid and the cut pid files under the cutter's scratch | continuous |
A lane starts its entry by writing its pid file (<pid> <start UTC> <label>, the format of the slot files) under /srv/queue/pids on
build-1 (or on its own box, mirrored there by the lane) and ends it by removing the file and replacing the entry.
Fault lines (build-1's guards)
- 9 October 2026, 11:32 UK (the network watch): the memory guard stopped the devnet-4 seed and hand by their units as "a second fresh sync" while all three were synced. The reference tip feed's Mac-side writer (hub1-tip-feed.sh) had died with the lanes at the founder's stop, so /srv/canary/hub1-tip.txt froze at 29,482 for thirteen minutes while the three live nodes read more than 100 past it; the guard counted three fresh syncs and kept the earliest. /api/live was stale for nine minutes. Fix (landed from this line): the guard reads the feed's age before any height compare (a feed older than three minutes is a FEED FAULT line, once per ten minutes, never a sync count); a unit counts only on two consecutive passes (one backward feed sample or one timed-out read is never a count); a unit whose datadir's oldest file is older than 30 minutes is a resume, never a fresh sync, never stopped. The feed writer runs under its pid file on the Mac and the network watch keeps it alive; the guard's one-fresh-sync rule itself stands.