# Release plan: command-stream viewer

Status notes for taking the command-stream pipeline to production as the
primary viewer, with the existing AWS video-streaming pipeline as fallback.
Updated 2026-08-20.

## London CPU server (LIVE)

Box: 82.39.133.153 = london-cpu-worker.war3replays.com (Cloudflare DNS-only
A record; unproxied on purpose - WebRTC UDP cannot ride the CF proxy).
Ubuntu 24.04, 4 cores / 8 GB / 192 GB, KVM (CRIU-capable). Deployed and
verified streaming a live 1.28.5 session end to end on 2026-08-20.

- Access: ssh danny@... (key only, passwordless sudo). Root login and
  password auth disabled. ufw: 22, 80, 443, 8145, 8150/tcp + 40000-40199/udp.
  fail2ban (sshd), unattended-upgrades, 5 GB swap total.
- Service user `ubuntu` (uid 1001, no sudo, linger on) runs everything;
  paths mirror the AWS lab exactly (/home/ubuntu/w3cs-lab, /opt/war3-runtime).
  NOTE for future boxes: create `ubuntu` BEFORE any admin user, or rely on
  the now-parameterized SERVICE_UID in classic-criu-worker.sh.
- Stack: wine-staging 11.13 (pinned + apt-mark hold, matches lab),
  full runtime (war3-1285/1140/rk, classic-131 CASC, prefix), engine
  profiles, 12 staged replays with sidecars, map corpus.
- systemd: w3cs-live (relay + warm seat, Restart=always), w3cs-web (8150),
  w3cs-watchdog.timer (30 s Xvfb/relay liveness -> restart; fixes the
  lab's recurring dead-Xvfb-behind-live-relay failure), nginx, certbot.timer.
- TLS: Let's Encrypt on nginx, single origin routing /signal|/reliable|
  /frame|/control|/resource-bundle -> relay :8145, everything else ->
  no-store page server :8150. Secure context is REQUIRED off-box (WebGPU
  and crypto.randomUUID do not exist on plain-HTTP non-localhost origins).
  interactive-live.html now derives ws/http bases from the page origin.
- Test page: https://london-cpu-worker.war3replays.com/poc/cpu-test-index.html
  (replays.json generated at deploy from replay sidecars, grouped by patch).
- Deploy: experiments/d3d9-command-stream/deploy/deploy-cpu-server.sh
  (rsync app + infra, remote build with -Werror, units, nginx, cert paths).
- Replay source: staged files only. R2 on-demand fetch is the next work
  item (with catalog metadata mapping sha -> engineProfile + map, like the
  staged sidecars). No AWS credentials live on this box.

### Multi-seat (2026-08-20, live)

Three independent seats on the London box, each a full stack (Xvfb :1N +
relay :814N + own wine prefixes + pulse sink + capture/control files),
run as systemd template instances `w3cs-live@{1,2,3}` via
deploy/w3cs-seat-launch.sh. Any seat runs any replay/profile: profiles
resolve from replay metadata at session start; seats are homogeneous.

- Allocation: the live page probes same-origin `/seatN/status` (nginx maps
  /seatN/* -> 814N; no extra ports, no CORS), connects to the first free
  seat, and retries on a lost race. The relay REFUSES a second viewer
  (ws close 4001 "seat-busy") - no session takeover. All seats busy ->
  the page shows "All seats are taken" and polls every 8 s, starting
  automatically when a seat frees (verified live end to end).
- Seat freeing: viewer disconnect parks the engine (SIGSTOP, zero CPU) and
  frees the seat; replay end (page finishReplay) parks AND drops the viewer
  connection (close 4002 "replay-ended") so an open end-card tab cannot
  hold the seat. The page suppresses its auto-restart machinery once
  replayEnded is set.
- Old-engine resolution SOLVED: pre-1.28 engines size their outer window to
  exactly 0.75 x desktop and use its client area (frame 8x34) as the D3D
  backbuffer; they ignore reswidth/resheight in windowed mode, and unknown
  -width/-height args break their -loadfile parsing (strands at the menu).
  Seat Xvfb roots are therefore 1376x1070, which makes the old engines pick
  1024x768 themselves (verified: 1.14 source 1024x768, bandwidth 10.2 ->
  5.5 Mbps). 1.28.5 uses the registry stamp as before. 1.31 becomes
  1032x802 on this root (from 1440x1080); exact 1024x768 for it is a
  follow-up (it may accept explicit size args - untested).
- Throughput observed: 3 concurrent games ~= load 2.0 on 4 cores, ~1.8 GiB
  used; 3 seats is conservative for this box.
- Concurrency watch item: one simultaneous triple cold-start showed 1.28.5's
  "User Data Migration Required" dialog on one seat (shared-install race);
  staggered starts have not reproduced it.

### CRIU fast-start: PROVEN on the London box (2026-08-20)

Full round trip verified for native-1285:
boot engine gated at replay-open (classic-replay-open-gate114 preload,
32+64-bit builds) -> criu dump of the whole namespaced tree (Xvfb +
wineserver + game; 131 MiB images, sub-second) -> tree killed ->
criu restore in **142 ms** -> place a DIFFERENT replay at the gated path ->
touch release -> game loads and PLAYS it (screenshot-verified in-game at
13 s elapsed). One generic checkpoint per profile serves any replay of
that profile: the gate fires before the replay header is read, so map
load happens after restore.

Key facts learned:
- criu check: "Looks good." on kernel 6.8.0-106 (built 4.2.1 + war3-procfs
  patch via infra/runtime/build-criu-war3.sh; /usr/local/libexec/war3/criu).
- classic-criu-worker.sh launch/dump/restore works almost unmodified; the
  hardcoded uid 1000 is now SERVICE_UID=$(id -u ubuntu) (London: 1001).
- Xvfb lives INSIDE the pid+net namespace, so X sockets never cross the
  checkpoint boundary; /tmp is shared, so the host relay can still reach
  the namespaced display socket for XTest input.
- restore needs a war3.json manifest (wineServerDirectory - derived from
  prefix dev/ino hex as /tmp/.wine-<uid>/server-<dev>-<ino> - plus
  xvfb/worker/war3 log paths and sizes at dump). The builder flow should
  write it automatically; the proof wrote it by hand.
- Proof assets on the box: /opt/war3-runtime/w3cs-checkpoints/native-1285/
  {images,war3.json}, prefix clone /opt/war3-runtime/classic/prefix-criu-slot0,
  gates at /usr/local/lib/war3-replay-open-gate{32,64}.so, shims at
  /usr/local/lib/war3-shims, seccomp wrapper in /usr/local/libexec/war3.

Remaining CRIU -> production work:
1. d3d9 proxy inside the checkpoint: launch with W3_D3D9_* env + d3d9=n,b
   overrides so the capture stream exists in the image; define the capture
   file lifecycle across restore (proxy keeps its saved fd offset - decide
   truncate-and-resync vs fresh-file handshake with the relay).
2. Relay integration: a checkpoint-aware session command (restore + place
   replay + release instead of cold boot) per profile, falling back to
   lab-game-session.sh when no checkpoint exists.
3. Build checkpoints for native-124cde / native-1140 (same flow) and
   native-1311-d3d9 (win64 prefix + CASC - needs its own launch args).
4. Embedded audio mode (pulse inside the namespace) for streaming parity;
   the proof ran audio-disabled.

## Patch coverage matrix (tested end to end through the live pipeline)

| Patch | Engine profile | Result |
|---|---|---|
| 1.28.5 | native-1285 | PASS (long-session baseline, 40 FPS) |
| 1.24 c/d/e | native-124cde | PASS (38 FPS, replay patch24proof) |
| 1.24 a/b | native-124ab | NOT TESTED - overlay installed, no test replay staged |
| 1.14 | native-1140 | PASS (38 FPS, expired-patch map install path works) |
| 1.31 | native-1311-d3d9 | PASS (37 FPS, win64 prefix + CASC). Open: cursor |
| | | suppression misses 1.31's cursor atlas (cyan reticle |
| | | visible); per-patch atlas registration needed. |
| 1.17 / 1.24 via ReplayKit | replaykit | ROOT-CAUSED (fix in progress). |
| | | RK locates the game via the Blizzard registry keys; |
| | | the prefix's InstallPath/GamePath pointed at a DEAD |
| | | per-session overlay (`Z:\tmp\w3cs-game-session.*`), |
| | | so RK exited silently under -totalsilence. Working |
| | | recipe (from streaming/server.py): run RK from its |
| | | own writable install (C:\war3rk), replay at |
| | | C:\war3rk\Replays\, maps into the INSTALL Maps dir |
| | | (<=1.27 engines read maps there), registry stamped |
| | | to C:\war3rk, serialized (RK mutates its install), |
| | | args `<replay> -totalsilence -window` (no -names/ |
| | | -colors). Session scripts must stop leaking their |
| | | ephemeral GamePath into the shared prefix registry. |

Also open: the 124/1140 profiles render at 1432x1046 and 1311 at 1440x1080
instead of the requested 1024x768 - the resolution stamp only reaches the
native-1285 registry path. Harmless but should be unified (bandwidth).

## Archive coverage math (classic-readiness-v1, 27,552 replays)

| Runtime class | Ready | Lane | Status |
|---|---|---|---|
| classic-convert-128 (c28 pre-converted) | 14,441 | native-1285 | VALIDATED |
| classic-native-128 | 1,374 | native-1285 | VALIDATED |
| classic-native-131 | 66 | native-1311-d3d9 | VALIDATED |
| requires-classic-129-engine | 0 (4,143 blocked) | none | needs a 1.29 profile (acquisition item) |
| missing-conversion | 222 | offline c28 converter | run the ingest converter, not runtime RK |
| map-ambiguous / missing-map | ~6,600 | any | map-corpus curation, not engine work |

**Verdict: runtime ReplayKit is NOT release-blocking.** 57% of the archive
plays today on validated lanes; the biggest blocked slice (15%) needs a
1.29 engine profile, which no amount of RK work provides. The 222
missing-conversion replays are better served by running the existing
offline converter. RK-at-runtime remains a nice-to-have for exact-engine
fidelity; findings so far: registry InstallPath must point at C:\war3rk
(session scripts leak dead overlay paths into the shared prefix), the
prefix needs wine-mono (installed 10.1.0), and after both fixes RK v1.28b
still idles at its window under wine 11.13-staging/Xvfb without spawning
the game - next step is an A/B against a streaming worker's wine version,
or driving its UI once via xdotool to learn its state.

## Fast viewer start: CRIU checkpoint per engine profile

Goal: viewer connect starts at REPLAY LOAD (seconds), not engine boot.
Today the lab keeps ONE warm engine (SIGSTOP-parked) for ONE profile;
switching profiles is a cold boot (20-60 s, worst on 1311).

Design (reuses the streaming pipeline's proven machinery):
- classic-replay-open-gate114.c (LD_PRELOAD) blocks the game exactly at
  replay-file open; W3_REPLAY_GATE_DIR/W3_REPLAY_GATE_SUFFIX are already
  plumbed in streaming/server.py.
- Per profile, at image-build time: boot engine -> reach the gate -> CRIU
  dump (criu-4.2.1-war3-procfs.patch + criu-dri-reopen-plugin.c on the box;
  benchmark-classic-criu.sh has timings).
- Per viewer session: inject the chosen replay at the gated path -> CRIU
  restore -> release the gate -> game loads the replay directly.
- One checkpoint image per (profile, resolution). Images are immutable and
  bakeable into the VPS provisioning.
- The relay keeps the current warm-seat flow as fallback when no checkpoint
  exists for a profile.

## Production topology (Germany VPS + AWS fallback)

- Primary: command-stream relay on a dedicated VPS in Germany (NOT AWS).
  No GPU required: the proxy runs with rasterization disabled, so seat cost
  is CPU-only (~0.5 core + ~1 GB per active seat measured; an 8-core VPS
  suits ~12-15 seats). Requirements: UDP open (ICE range), CAP_SYS_ADMIN
  for CRIU, wine + profiles + checkpoints provisioned, good EU peering.
- Fallback: the existing AWS video-streaming pipeline stays live.
- Selection: war3replays.com (web tier) decides per viewer:
  1. Page attempts WebRTC to the VPS; the existing in-page latency
     admission (idle-RTT gate, entry-only) decides viability.
  2. On rejection, UDP-blocked networks, or missing patch support
     (e.g. ReplayKit-only replays until unblocked), serve the streaming
     player instead.
  3. Selection happens before seat claim; production ranks workers by
     measured RTT (the admission module already carries product copy).
- The content-addressed resource bundles (/resource-bundle/<digest>) are
  the CDN artifact when cold-start bandwidth matters: same digests across
  sessions, CloudFront in front, browser cache path already implemented.