The dl split: the binaries of dl.igneum.network publish from build-1 as the Vercel project igneum-dl-bin only when a binary is new (tools/dl/publish-bin.sh, pid files on both ends, never a kill); the downloads project keeps the manifests, signatures, checksums and pages and rewrites every binary path to the bin project with the aliases pointing there directly; the three publishers call publish-bin --if-needed before their own deploy; README section

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

packaging/ota/README.md

@@ -188,3 +188,23 @@ Tested on the Mac, 4 October 2026: the unit tests (parse, bad jobs refused, sign
 targeting, the once-only ledger, globs), the signer with the real key, the publisher against a scratch folder, the
 reader against the live intake; the crate also compiles for `x86_64-pc-windows-gnu`. Not yet run on a PC: the
 first job goes to PC 2 (`1ccfe586`) once 0.3.2 is installed there (`docs/plans/shard-test-pc2.md`).
+
+## The dl split (9 October 2026)
+
+dl.igneum.network is two Vercel projects. `igneum-dl` (the downloads folder, `~/.config/igneum/dlsite-dir`) carries the
+manifests, signatures, checksums, the index and the pages, a few megabytes; its `.vercelignore` leaves every binary out
+(`*.dmg`, `*.exe`, `*.zip`, `*.gz`, `**/moves/**`). `igneum-dl-bin` carries the binaries, from the store
+`/srv/artefacts/dl-bin` on build-1, published only when a binary is new (`tools/dl/publish-bin.sh`; the three publishers
+call it with `--if-needed` before their own deploy). Every served URL is unchanged: `igneum-dl`'s `vercel.json` rewrites
+every binary path under `/dl/` to the bin project, and each `/public/<alias>` points at the bin project's file directly
+(a rewrite's destination is never rewritten again). The binaries stay on disk in the downloads folder (the publishers
+read them for sizes, hashes and the aliases); they are just not uploaded with it. Before the split the one 5.9 GB project
+re-hashed every binary on every manifest publish (17 GB of deploys on the Mac in one night).
+
+- `~/.config/igneum/dl-bin-host`: `user@host` of build-1 (the store and the publisher of the bin project; the Vercel
+  credential sits there in `~/.vercel-igneum`). `~/.config/igneum/dl-bin-base`: the bin project's production URL
+  (default `https://igneum-dl-bin-igneum.vercel.app`).
+- Pid files: `<dlsite>/.publish-bin.pid` on the Mac, `/srv/artefacts/dl-bin/.deploy.pid` on the box; a second run refuses.
+- A re-cut under the same file name (a different size) is copied again; a file removed from the folder stays in the
+  store and keeps answering (the URL rule; prune by hand on the box when a withdrawal must stop serving).
+

packaging/ota/publish-jobs.sh

@@ -470,6 +470,7 @@ fi
 if [ "$DEPLOY" = 1 ]; then
   [ -n "$DLSITE" ] || { echo "--deploy needs the real downloads folder (no --dest)" >&2; exit 1; }
   echo "deploying $DLSITE"
+  "$ROOT/tools/dl/publish-bin.sh" --if-needed || { echo "the bin publish failed; nothing deployed" >&2; exit 1; }
   (cd "$DLSITE" && npx --yes vercel@latest --global-config "$HOME/.config/igneum/vercel" deploy --prod --yes 2>&1 | sed "s#$TOKEN#<token>#g"; exit "${PIPESTATUS[0]}") \
     || { echo "the deploy failed (the Vercel CLI's exit status above); nothing verified" >&2; exit 1; }
   verify_live "$TRIES" || exit 1

packaging/ota/publish-manifest.sh

@@ -318,6 +318,7 @@ fi
 
 if [ "$DEPLOY" = 1 ]; then
   [ -n "$DLSITE" ] || { echo "--deploy needs the real downloads folder (no --dest)" >&2; exit 1; }
+  "$ROOT/tools/dl/publish-bin.sh" --if-needed || { echo "the bin publish failed; nothing deployed" >&2; exit 1; }
   echo "deploying $DLSITE"
   (cd "$DLSITE" && npx --yes vercel@latest --global-config "$HOME/.config/igneum/vercel" deploy --prod --yes 2>&1 | sed "s#$TOKEN#<token>#g"; exit "${PIPESTATUS[0]}") \
     || { echo "the deploy failed (the Vercel CLI's exit status above); nothing verified" >&2; exit 1; }

packaging/ota/publish-public.sh

@@ -163,12 +163,21 @@ print(json.dumps(aliases))
 PY
 )"
 KEEP="$(python3 -c 'import json,sys; a=json.loads(sys.argv[1]); print(" ".join(v for v in a.values() if v))' "$ALIASES_JSON")"
-log "aliases (vercel.json rewrites under /public/):"
-python3 -c 'import json,sys; a=json.loads(sys.argv[1]); [print("  /public/%s -> %s" % (k, ("/dl/public/" + v) if v else "(no file yet; alias left out)")) for k,v in a.items()]' "$ALIASES_JSON" | scrub
-VERCEL_JSON="$(python3 - "$ALIASES_JSON" <<'PY'
+# the binaries live in the project igneum-dl-bin (tools/dl/publish-bin.sh, 9 October 2026); this project rewrites every
+# binary path there and the aliases point there directly (a rewrite's destination is never rewritten again)
+BIN_BASE="$( [ -f "$CFG/dl-bin-base" ] && tr -d '[:space:]' < "$CFG/dl-bin-base" || echo https://igneum-dl-bin-igneum.vercel.app)"
+log "aliases (vercel.json rewrites under /public/, served from $BIN_BASE):"
+python3 -c 'import json,sys; a=json.loads(sys.argv[1]); [print("  /public/%s -> %s" % (k, (sys.argv[2] + "/dl/public/" + v) if v else "(no file yet; alias left out)")) for k,v in a.items()]' "$ALIASES_JSON" "$BIN_BASE" | scrub
+VERCEL_JSON="$(python3 - "$ALIASES_JSON" "$BIN_BASE" <<'PY'
 import json, sys
-a = json.loads(sys.argv[1])
-rewrites = [{"source": "/public/" + k, "destination": "/dl/public/" + v} for k, v in a.items() if v]
+a = json.loads(sys.argv[1]); bin_base = sys.argv[2].rstrip("/")
+rewrites = [{"source": "/public/" + k, "destination": bin_base + "/dl/public/" + v} for k, v in a.items() if v]
+# every binary under /dl/ (the same patterns as the .vercelignore of this project and publish-bin.sh); a file that is
+# in this deployment is served from it first (the filesystem precedes rewrites), so a stray binary here still answers
+rewrites += [
+    {"source": "/dl/:path(.*\\.(?:dmg|exe|zip|gz))", "destination": bin_base + "/dl/:path"},
+    {"source": "/dl/:path(.*/moves/.*)", "destination": bin_base + "/dl/:path"},
+]
 cfg = {
     "cleanUrls": False,
     "trailingSlash": False,
@@ -230,6 +239,8 @@ if [ "$DRY" = 1 ]; then log "dry run: nothing written"; fi
 if [ "$DEPLOY" = 1 ]; then
   [ "$DRY" = 0 ] || { echo "--deploy and --dry-run together make no sense" >&2; exit 2; }
   [ -z "$DEST" ] || { echo "--deploy needs the real downloads folder (no --dest)" >&2; exit 1; }
+  # the binaries first: a new file here is copied to build-1 and published from there before this small deploy rewrites to it
+  "$ROOT/tools/dl/publish-bin.sh" --if-needed || { echo "the bin publish failed; the manifests are not deployed" >&2; exit 1; }
   log "deploying $DLSITE"
   (cd "$DLSITE" && npx --yes vercel@latest --global-config "$CFG/vercel" deploy --prod --yes 2>&1 | scrub; exit "${PIPESTATUS[0]}") \
     || { echo "the deploy failed (the Vercel CLI's exit status above); nothing verified" >&2; exit 1; }

tools/dl/publish-bin.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# The binaries of dl.igneum.network: copied to build-1 and published from there as the Vercel project igneum-dl-bin,
+# only when a binary is new. dl.igneum.network itself (the project igneum-dl, deployed by the three publishers in
+# packaging/ota/) carries only the manifests, signatures, checksums and pages, and rewrites every binary path to this
+# project, so every served URL stays as it was (9 October 2026, the dl split: the one 5.9 GB project re-hashed every
+# binary on every manifest publish, 17 GB of deploys on the Mac in one night).
+#
+#   tools/dl/publish-bin.sh                 copy what is missing on build-1, deploy if anything was copied, verify
+#   tools/dl/publish-bin.sh --if-needed     the same, silent and exit 0 when nothing is new (the publishers call this)
+#   tools/dl/publish-bin.sh --force         deploy even when nothing is new (a vercel.json change on the box)
+#   tools/dl/publish-bin.sh --dry-run       list the delta, copy and deploy nothing
+#
+# What counts as a binary: *.dmg, *.exe, *.zip, *.gz under dl/, and everything under a moves/ folder. Everything else
+# stays in the small project (its .vercelignore lists the same patterns).
+# Reads ~/.config/igneum/dlsite-dir (IGNEUM_DLSITE overrides), ~/.config/igneum/dl-bin-host (user@host of build-1, required),
+# ~/.config/igneum/dl-bin-base (default https://igneum-dl-bin-igneum.vercel.app). Pid files: <dlsite>/.publish-bin.pid on
+# the Mac, <store>/.deploy.pid on the box; a second run refuses (exit 75). Never kills anything.
+set -euo pipefail
+MODE=run; FORCE=0
+for a in "$@"; do case "$a" in --if-needed) MODE=ifneeded ;; --force) FORCE=1 ;; --dry-run) MODE=dry ;; -h|--help) sed -n '2,19p' "$0"; exit 0 ;; *) echo "unknown flag $a" >&2; exit 2 ;; esac; done
+CFG="$HOME/.config/igneum"
+DLSITE="${IGNEUM_DLSITE:-}"; [ -n "$DLSITE" ] || { [ -f "$CFG/dlsite-dir" ] && DLSITE="$(tr -d '[:space:]' < "$CFG/dlsite-dir")"; } || true
+[ -n "$DLSITE" ] && [ -d "$DLSITE/dl" ] || { echo "no downloads folder: ~/.config/igneum/dlsite-dir must hold dl/" >&2; exit 1; }
+HOST="$( [ -f "$CFG/dl-bin-host" ] && tr -d '[:space:]' < "$CFG/dl-bin-host" || true)"
+[ -n "$HOST" ] || { echo "no store host: ~/.config/igneum/dl-bin-host must hold user@host of build-1 (the box that publishes the bin project)" >&2; exit 1; }
+BASE="$( [ -f "$CFG/dl-bin-base" ] && tr -d '[:space:]' < "$CFG/dl-bin-base" || echo https://igneum-dl-bin-igneum.vercel.app)"
+STORE=/srv/artefacts/dl-bin
+SSH=(ssh -o BatchMode=yes -o ConnectTimeout=15 "$HOST")
+log() { echo "publish-bin: $*"; }
+
+# the binaries here: "<size> <path>" lines, paths relative to the folder
+# "<path> <size>" lines, paths relative to the folder, plain sort (comm compares whole lines)
+local_list() { (cd "$DLSITE" && find dl -type f \( -name '*.dmg' -o -name '*.exe' -o -name '*.zip' -o -name '*.gz' -o -path '*/moves/*' \) -print0 | xargs -0 stat -f '%N %z' 2>/dev/null || (cd "$DLSITE" && find dl -type f \( -name '*.dmg' -o -name '*.exe' -o -name '*.zip' -o -name '*.gz' -o -path '*/moves/*' \) -printf '%p %s\n')) | sort; }
+remote_list() { "${SSH[@]}" "cd $STORE 2>/dev/null && find dl -type f -printf '%p %s\n' | sort" || true; }
+
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
+local_list > "$TMP/local"; remote_list > "$TMP/remote"
+# missing on the box, or a different size there (a re-cut under the same name): copied again
+comm -23 "$TMP/local" "$TMP/remote" | awk '{ print $1 }' > "$TMP/delta"
+N=$(wc -l < "$TMP/delta" | tr -d ' ')
+if [ "$N" = 0 ] && [ "$FORCE" = 0 ]; then [ "$MODE" = ifneeded ] || log "nothing new: $(wc -l < "$TMP/local" | tr -d ' ') binaries here, all on $HOST:$STORE"; exit 0; fi
+log "$N binar$([ "$N" = 1 ] && echo y || echo ies) to publish:"; sed 's/^/  /' "$TMP/delta"
+[ "$MODE" != dry ] || exit 0
+
+PIDF="$DLSITE/.publish-bin.pid"
+if [ -s "$PIDF" ] && kill -0 "$(cat "$PIDF")" 2>/dev/null; then echo "a bin publish is running (pid $(cat "$PIDF") in $PIDF); wait for it" >&2; exit 75; fi
+echo $$ > "$PIDF"; trap 'rm -f "$PIDF"; rm -rf "$TMP"' EXIT
+
+if [ "$N" != 0 ]; then
+  log "copying to $HOST:$STORE"
+  tar -cf - -C "$DLSITE" -T "$TMP/delta" | "${SSH[@]}" "tar -xf - -C $STORE"
+fi
+log "deploying $STORE from $HOST at $(TZ=UTC date +%H:%M:%SZ)"
+# the box's own pid file: one deploy at a time, never a kill
+OUT="$("${SSH[@]}" "cd $STORE && if [ -s .deploy.pid ] && kill -0 \$(cat .deploy.pid) 2>/dev/null; then echo 'a bin deploy is running on the box (pid '\$(cat .deploy.pid)')'; exit 75; fi; echo \$\$ > .deploy.pid; trap 'rm -f .deploy.pid' EXIT; npx --yes vercel@latest --global-config ~/.vercel-igneum --scope igneum deploy --prod --yes 2>&1")" || { echo "$OUT" | tail -8 >&2; exit 1; }
+URL=$(echo "$OUT" | grep -oE 'https://[a-z0-9.-]*vercel\.app' | tail -1); log "deployed $URL"
+
+# verify: every binary in the delta (or, on --force, the current aliases' files) answers at the public base with its size
+fails=0
+while IFS= read -r p; do
+  [ -n "$p" ] || continue
+  want=$(awk -v p="$p" '$1 == p { print $2 }' "$TMP/local")
+  for t in 1 2 3 4 5 6 7 8 9 10 11 12; do
+    got=$(curl -sI "$BASE/$p" | tr -d '\r' | awk 'tolower($1) == "content-length:" { print $2 }' | tail -1)
+    [ "$got" = "$want" ] && break; sleep 5
+  done
+  if [ "$got" = "$want" ]; then log "  ok $p ($want bytes)"; else log "  FAIL $p: content-length ${got:-none}, want $want"; fails=$((fails + 1)); fi
+done < "$TMP/delta"
+[ "$fails" = 0 ] || { echo "publish-bin: $fails file(s) not served at $BASE" >&2; exit 1; }
+log "done: $N published, served at $BASE ($(TZ=Europe/London date +%H:%M) UK)"