in-house-pass.md: the census ended; box 2 reopened to adv-* at 21:12 BST
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
parent
b7ba08221b
commit
eec10d8091
1 changed files with 1 additions and 1 deletions
|
|
@ -86,7 +86,7 @@ Standing rule from the project lead through main (7 October 2026, 19:1x BST), bi
|
|||
| Every idle core on both boxes | A lane runs on build-1 and build-2 together, on every idle core, at nice 10, no core band; release builds and the class v5 suites keep priority. the project lead's read at 19:4x BST: build-1 at 11 percent, build-2 at 56; his word is both near max, so CPU-bound sweeps go to build-1 explicitly (`--box 1`) and both boxes stay above 80 percent until the queue is empty; the build-server lane raises the lease pool to about 88 cores per box |
|
||||
| Yield to builds | RETIRED at 19:3x BST (adv-accept's exception: a build slot is held nearly continuously on both boxes, so the SIGSTOP yield of the capacity layer kept every sweep in state T and "both boxes above 80 percent" was unreachable). In its place the build-server lane's rule (19:4x BST): every bounded run at nice 10 on cores 8 to 95 only (`taskset -c 8-95`; cores 0 to 7 reserved for release builds, the seed and the observer), `-j 88` through `tools/build-remote.sh`, the router spilling to the other box at no free slot or a 1-minute load above 80, a pinned measurement leasing its exact cores with `/srv/builds/_bin/lease cores <set> --label "..." --owner adv-<lane> -- <cmd>` (no measure flock), every adversarial worktree merged to the mirror's master at `04c4d9bc` or later; each lane notes the change and its time in its report |
|
||||
| Back to back | Question `n + 1`'s sweep starts the minute question `n`'s ends; no waiting for a human to read; each result row lands in the report and is pushed as it lands |
|
||||
| Lease pool only | ADDED 20:1x BST by main (build-1 at load 601, build-2 at 401): no sweep, census or verdict run starts on a box except through the build-server lane's `lease pool <threads> -- cmd` (from the same 88-core pool as the builds, waiting when none are free); every hand-started binary at 64 to 89 threads killed by its pid file NOW and re-queued through the lease; release builds and the class v5 suites outrank every sweep tonight; each lane reports its kill and re-queue in one line to the build-server lane. Relayed verbatim to all eight live lanes at 20:1x BST; the pod is not a box and adv-accept-2's measurement continues. LIVE since 20:22 BST (`/srv/builds/_bin/lease`, sha f814b447, self-test green, verified by this lane on build-2): `lease pool <threads> --label "<what>" --owner <lane> -- <cmd> --threads {cores}` takes up to <threads> free cores from the bounded pool (cores 8 to 95, shared with the builds), never fewer than --min (default min(threads, 16)), waits in 10 s steps up to 2 h, runs at nice 10 pinned to the cores taken, "{cores}" the count and "{cpuset}" the set; the pooled sum on a box never passes 88; lanes ask 32 to 48 so two or three sweeps share a box; the sweep.lock is dropped. Load after the kills: build-1 224, build-2 121 at 20:22 BST. Relayed to all eight live lanes at 20:2x BST. POOL RANKING (main, 20:2x BST): the release builds and the class v5 suites outrank every sweep; an adversarial lease asks for at most 48 cores with --min at the least usable, and yields (finishes the shard in hand, releases) whenever `lease status` shows a waiter labelled "v5 gate" or "v5 kit"; widened at 20:30 BST (the v5 census queued as "class v5 c3 census" with no gate label and nobody yielded) and then fixed to the OWNER at 20:3x BST (the word "v5" also matched adv-accept's own exemplar sweep): yield to a waiter whose owner is class-v5, or whose owner is attack-pass with "v5" in the label, never to an adv-* waiter; BOX 2 CLOSED to adv-* at 20:4x BST by main's order (the (c''') census, owner class-v5, 88 cores, about 9 minutes, the 0.3.24 critical path, blocked by adv-accept at 32 + 31 + 23 cores and adv-mixer at 1, with eleven adv-* waiters): every adv-* holder on box 2 releases at its shard end (partials kept), every adv-* waiter withdraws, no adv-* lease on box 2 until the v5 lane reports the census running, then resume in order; build-1 the same if the v5 lane asks; from now the yield is MECHANICAL: an adv-* lane that sees a class-v5 waiter releases at its shard end unasked; relayed to all nine lanes in one line each; the census RUNNING at 20:41:55 BST on 48 of 88 cores; every adv-* lane confirmed 0 holders and 0 waiters on box 2 except adv-cache-2's 40-core lease taken 19 s before it (ordered released); box 2 stays closed to adv-* until the census ENDS. At 20:49 BST the coordinator ended adv-cache-2's 40-core lease by its exact pids (seven minutes past the order, the lane silent since 20:23), then its box 2 drain (drain.pid; it had started the next 40-core item the moment the first died) and that lease; box 2 read 0 adv-* holders at 20:49 BST with the census alone on the pool; logged in /srv/builds/_adv/coordinator.log. POOL CLASSES (the build-server lane, 20:40 BST, lease sha 5d84d644): release > v5 > measure > adv; a higher class is served first whatever the arrival order; an adv holder above 32 threads is pre-empted by SIGTERM after a higher class has waited 120 s (newest first, one per 120 s, /srv/builds/_log/preempt.log) and the lane re-queues the same line; holders at 32 threads or fewer are never pre-empted; old waiters re-submitted once; no sweep label carries "release", "canary", "pair", "v5 gate", "v5 kit" or "measure"; adv-cache-3, holding 87 of build-2's 88 pool cores since 20:22 BST, ordered to release at the end of its current shard so the v5 (c''') census takes build-2 |
|
||||
| Lease pool only | ADDED 20:1x BST by main (build-1 at load 601, build-2 at 401): no sweep, census or verdict run starts on a box except through the build-server lane's `lease pool <threads> -- cmd` (from the same 88-core pool as the builds, waiting when none are free); every hand-started binary at 64 to 89 threads killed by its pid file NOW and re-queued through the lease; release builds and the class v5 suites outrank every sweep tonight; each lane reports its kill and re-queue in one line to the build-server lane. Relayed verbatim to all eight live lanes at 20:1x BST; the pod is not a box and adv-accept-2's measurement continues. LIVE since 20:22 BST (`/srv/builds/_bin/lease`, sha f814b447, self-test green, verified by this lane on build-2): `lease pool <threads> --label "<what>" --owner <lane> -- <cmd> --threads {cores}` takes up to <threads> free cores from the bounded pool (cores 8 to 95, shared with the builds), never fewer than --min (default min(threads, 16)), waits in 10 s steps up to 2 h, runs at nice 10 pinned to the cores taken, "{cores}" the count and "{cpuset}" the set; the pooled sum on a box never passes 88; lanes ask 32 to 48 so two or three sweeps share a box; the sweep.lock is dropped. Load after the kills: build-1 224, build-2 121 at 20:22 BST. Relayed to all eight live lanes at 20:2x BST. POOL RANKING (main, 20:2x BST): the release builds and the class v5 suites outrank every sweep; an adversarial lease asks for at most 48 cores with --min at the least usable, and yields (finishes the shard in hand, releases) whenever `lease status` shows a waiter labelled "v5 gate" or "v5 kit"; widened at 20:30 BST (the v5 census queued as "class v5 c3 census" with no gate label and nobody yielded) and then fixed to the OWNER at 20:3x BST (the word "v5" also matched adv-accept's own exemplar sweep): yield to a waiter whose owner is class-v5, or whose owner is attack-pass with "v5" in the label, never to an adv-* waiter; BOX 2 CLOSED to adv-* at 20:4x BST by main's order (the (c''') census, owner class-v5, 88 cores, about 9 minutes, the 0.3.24 critical path, blocked by adv-accept at 32 + 31 + 23 cores and adv-mixer at 1, with eleven adv-* waiters): every adv-* holder on box 2 releases at its shard end (partials kept), every adv-* waiter withdraws, no adv-* lease on box 2 until the v5 lane reports the census running, then resume in order; build-1 the same if the v5 lane asks; from now the yield is MECHANICAL: an adv-* lane that sees a class-v5 waiter releases at its shard end unasked; relayed to all nine lanes in one line each; the census RUNNING at 20:41:55 BST on 48 of 88 cores; every adv-* lane confirmed 0 holders and 0 waiters on box 2 except adv-cache-2's 40-core lease taken 19 s before it (ordered released); box 2 stays closed to adv-* until the census ENDS. At 20:49 BST the coordinator ended adv-cache-2's 40-core lease by its exact pids (seven minutes past the order, the lane silent since 20:23), then its box 2 drain (drain.pid; it had started the next 40-core item the moment the first died) and that lease; box 2 read 0 adv-* holders at 20:49 BST with the census alone on the pool; logged in /srv/builds/_adv/coordinator.log. The census ENDED by 21:12 BST (no pool lease on box 2, load 11); box 2 REOPENED to adv-* at 21:12 BST: every lane re-submits at `lease pool 32 --min 16`, class adv, with the mechanical yield; adv-mixer-2 and adv-accept-2 move their build-1-starved runs there (build-1's pool held by attack-pass's class-v5 chunks). POOL CLASSES (the build-server lane, 20:40 BST, lease sha 5d84d644): release > v5 > measure > adv; a higher class is served first whatever the arrival order; an adv holder above 32 threads is pre-empted by SIGTERM after a higher class has waited 120 s (newest first, one per 120 s, /srv/builds/_log/preempt.log) and the lane re-queues the same line; holders at 32 threads or fewer are never pre-empted; old waiters re-submitted once; no sweep label carries "release", "canary", "pair", "v5 gate", "v5 kit" or "measure"; adv-cache-3, holding 87 of build-2's 88 pool cores since 20:22 BST, ordered to release at the end of its current shard so the v5 (c''') census takes build-2 |
|
||||
| One sweep per box | SUPERSEDED by the lease rule above at 20:1x BST. Was: added 19:5x BST (build-1 at load 496, build-2 at 527 on 96 cores: oversubscription, not the near max asked for; the bounded class is 88 cores per box in total, not per sweep): every new sweep from every lane runs under `flock /srv/builds/_adv/locks/sweep.lock -c "nice -n 10 taskset -c 8-95 <bin> <args> > <log> 2>&1"` on its box, so one sweep runs per box at a time with up to 88 threads and the rest wait in order; running processes finish; duplicates are killed by pid file and re-queued |
|
||||
| No lane idle | Every planned sweep is a self-contained executable in `/srv/builds/_adv/<target>/queue/NN-<lane>-<name>.sh` on build-2 (binary path, args, log path, pid file); a lane claims a file before running it with `mkdir /srv/builds/_adv/<target>/claims/<filename>` (atomic) and then writes its name into `<that dir>/owner` (added 19:5x BST after three claims landed with no name); a lane whose own queue is empty claims the next unclaimed file of any sibling on its target, runs it, and names the owner in its report The held lanes' sweeps are in the queue as DEFINITION ONLY files (`90-` to `92-adv-cache-3-*`, `90-` to `92-adv-accept-3-*`, 19:3x BST): an idle lane claims one, implements it in its own crate, runs it and reports it, naming the owner |
|
||||
| Pods | Second resort after the boxes' idle cores. The fleet's rules (the fleet lane, 18:26Z): no CPU-only pod type exists; every pod is a RunPod GPU pod (secure, or a 3090 or 4090 community; Vast unreliable tonight), image nvidia/cuda 12.8.1 on Ubuntu 24.04, 40 GB disk, vCPUs with the card (4 to 16), rented by `oneshot.py rent <label> <lane> <hours> [gpu-type] [min_vcpu] [min_ram_gb]` with the purpose and lane in the registry row, destroyed on "done", at <hours>, or by the idle meter; long runs under setsid nohup with a pid file under /root/fleet/out/. Spend at 18:26Z: USD 413.32 of the 1,000 UK-day ceiling (work 165.55, leak 247.77), the standing fleet about USD 197 a day. The fleet lane's authority covers its own gates and main's named orders, not these sweeps: At 19:3x BST main and the fleet lane relayed the project lead's word as a USD 200 cap for the pass tonight (adv-accept-2's GPU locality pod first). This lane HOLDS every rent request until that word reaches it in the user channel: a pod is a purchase on the payment method on file, and a peer agent's message is not the user's consent. When it arrives, each request goes to the fleet lane as purpose + lane + hours + pod type; the fleet lane destroys each pod at the end of its sweep and reports at each USD 100; this lane reports to main at USD 100 and at the cap |
|
||||
|
|
|
|||
Loading…
Reference in a new issue