# Phase 7.5 Design: Proactive Playout Timing This document summarizes the timing-specific findings from [ARCHITECTURE_RESILIENCE_REVIEW.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/ARCHITECTURE_RESILIENCE_REVIEW.md) and turns them into a focused bridge phase after [PHASE_7_BACKEND_LIFECYCLE_PLAYOUT_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_7_BACKEND_LIFECYCLE_PLAYOUT_DESIGN.md). Phase 7 made backend lifecycle, playout policy, ready-frame queueing, late/drop recovery, and backend playout health explicit. Phase 7.5 should use those foundations to move output production from demand-filled scheduling toward proactive, deadline-aware playout. ## Status - Phase 7.5 design package: proposed. - Phase 7.5 implementation: Step 5 in progress. - Current alignment: Phase 7 is complete. `RenderOutputQueue`, `VideoPlayoutPolicy`, `VideoPlayoutScheduler`, `VideoBackendLifecycle`, and backend playout telemetry exist. The backend worker fills the ready queue on completion demand, but render production is not yet proactively driven by queue pressure or video cadence. Current footholds: - `RenderEngine` owns normal GL work on the render thread. - `VideoBackend` owns backend lifecycle, completion processing, ready-frame queue use, and backend playout health reporting. - `RenderOutputQueue` reports depth, capacity, pushed, popped, dropped, and underrun counts. - `VideoPlayoutPolicy` names ready-frame headroom and catch-up policy. - `HealthTelemetry::BackendPlayoutSnapshot` exposes queue depth, underruns, late/drop streaks, and recovery decisions. - Step 1 adds baseline timing fields for ready-queue min/max/zero-depth samples and output render duration. - Step 2 adds a pure `OutputProductionController` for queue-pressure production decisions. - Step 3 adds a proactive output producer worker that keeps `RenderOutputQueue` warm after playback starts. - Step 4 skips non-forced preview presentation while output ready-queue depth is below target. - Step 5 makes async readback misses prefer cached output over synchronous readback after bootstrap. ## Timing Review Findings The resilience review highlights several timing risks that remain after basic render-thread and backend ownership cleanup: - playout is still effectively filled on demand instead of continuously produced ahead - output buffering is named, but queue depth is not yet tuned against measured render/readback cost - GPU readback has an asynchronous path, but the miss path can still fall back to synchronous readback - preview presentation is best-effort, but it still shares render-thread budget with playout - telemetry is improving, but render timing is still too coarse to distinguish draw, pack, fence wait, readback copy, and preview cost The practical concern is not average frame time. It is what happens during a short spike. A single slow render, readback wait, preview present, or callback scheduling delay can drain playout headroom and cause late or dropped output frames. ## Why Phase 7.5 Exists Phase 7 made the backend safer and observable, but Step 5 intentionally stopped at demand-filled queue behavior: - a completion arrives - the backend worker fills the ready queue to target depth - the backend schedules one ready frame That is better than callback-thread rendering, but it still couples frame production to output completion pressure. Phase 7.5 should make render production proactive: - keep the ready queue near target depth before the device asks for the next frame - let DeckLink consume already-prepared frames - treat queue depth as the pressure signal between render and backend - make preview and readback fallback subordinate to output deadlines ## Goals Phase 7.5 should establish: - a proactive output producer that fills `RenderOutputQueue` based on queue pressure - a clear trigger model for output production: queue-low, cadence tick, or both - a bounded sleep/yield strategy when the ready queue is full - explicit priority rules between playout, preview, screenshots, shader work, and background render requests - readback miss behavior that does not blindly return to the most timing-sensitive synchronous path - telemetry that can explain why the queue drains: render cost, readback wait, preview cost, or scheduling pressure - pure tests for producer pressure policy where possible ## Non-Goals Phase 7.5 should not require: - replacing the renderer - replacing DeckLink support - a full telemetry subsystem rewrite - perfect adaptive latency - a new UI - changing live-state layering or persistence semantics This phase is about output timing behavior, not broad subsystem redesign. ## Target Timing Model The target model is: ```text Video cadence / queue pressure -> proactive output producer request -> RenderEngine renders and reads back output frame -> RenderOutputQueue stores ready frame -> VideoBackend consumes ready frame for DeckLink scheduling ``` The important difference from Phase 7 is that output production should not wait until a completion has already created demand. The queue should usually have headroom before the completion worker needs to schedule. Suggested pressure rules: - if ready depth is below `targetReadyFrames`, request output production immediately - if ready depth is at or above `maxReadyFrames`, producer sleeps or yields - if late/drop streak grows, temporarily bias toward output production over preview - if readback is late, prefer stale/black underrun policy over blocking the deadline path - if preview is due but output queue is below target, skip or delay preview ## Proposed Collaborators ### `OutputProductionController` Small policy owner that decides when to request another output frame. Responsibilities: - evaluate ready queue depth and capacity - evaluate late/drop/underrun pressure - decide whether to produce, sleep, or yield - keep policy testable without DeckLink or GL Non-responsibilities: - GL rendering - DeckLink scheduling - live-state composition ### `OutputProducerWorker` Worker or render-thread-adjacent loop that keeps output frames ready. Responsibilities: - wake on queue-low pressure - request render-thread output production - push completed frames into `RenderOutputQueue` - stop cleanly before render/backend teardown Non-responsibilities: - device callback handling - hardware scheduling - persistent state mutation ### `RenderTimingBreakdown` Lightweight render timing sample for the output path. Initial fields: - total output render time - draw/composite time - output pack time - readback fence wait time - readback copy time - synchronous readback fallback count - preview present cost - preview skipped count This can be reported into existing telemetry first, then Phase 8 can fold it into the broader health model. ## Migration Plan ### Step 1. Snapshot Current Timing Behavior Use existing Phase 7 telemetry to capture baseline behavior before changing production cadence. Initial target: - [x] record ready queue depth over time while running - [x] record underrun count, late/drop streaks, and catch-up frames - [x] record output render duration and completion interval - [x] identify whether queue depth regularly falls to zero Exit criteria: - [x] there is a clear before/after baseline for proactive production - [x] runtime-state output exposes enough values to diagnose whether queue starvation is happening Implementation notes: - `HealthTelemetry::BackendPlayoutSnapshot` exposes current, min, max, and zero-depth ready-queue samples. - `VideoBackend` samples ready-queue depth before demand-fill, after queue fill, and after scheduling from the queue. - `VideoBackend` records last, smoothed, and max output render duration for demand-produced output frames. - Runtime-state JSON exposes the baseline under `backendPlayout.readyQueue` and `backendPlayout.outputRender`. ### Step 2. Extract Output Production Policy Introduce a pure policy helper for queue-pressure decisions. Initial target: - [x] input: ready depth, capacity, target depth, late/drop streaks, underrun count - [x] output: produce, wait, or throttle - [x] tests cover low queue, full queue, late/drop pressure, and normalized policy values Exit criteria: - [x] production cadence policy can evolve without touching DeckLink or GL code Implementation notes: - `OutputProductionController` lives in `videoio` and depends only on `VideoPlayoutPolicy`. - `OutputProductionPressure` carries ready-queue depth/capacity plus underrun and late/drop pressure. - `OutputProductionDecision` returns `Produce`, `Wait`, or `Throttle`, a requested frame count, effective target/max limits, and a reason string. - Step 2 is intentionally not wired into live playback yet. Step 3 should use this policy to drive the proactive producer loop. ### Step 3. Add A Proactive Producer Loop Move from demand-filled output production to queue-pressure production. Initial target: - [x] producer wakes when queue depth is below target - [x] producer requests render-thread output production until target depth is reached - [x] producer stops when backend stops or render thread shuts down - [x] completion worker mostly schedules from already-ready frames Exit criteria: - [x] normal playback does not depend on completion processing to fill the queue from empty - [x] callback/completion pressure and render production pressure are separate Implementation notes: - `VideoBackend` starts the completion worker before device start, then starts the output producer only after DeckLink start succeeds. This avoids fighting DeckLink preroll for the same output frame pool. - `OutputProducerWorkerMain()` periodically wakes and uses `OutputProductionController` to decide whether to produce, wait, or throttle. - Completion handling records pacing/recovery, updates producer pressure, schedules a ready frame, and wakes the producer to refill headroom. - Completion handling keeps a one-frame synchronous fallback when the ready queue is unexpectedly empty, then falls back to black underrun behavior if that also fails. - Producer shutdown is explicit and joined before video output teardown. ### Step 4. Prioritize Playout Over Preview Make preview explicitly subordinate to output playout deadlines. Initial target: - [x] skip or delay preview when ready queue depth is below target - count skipped previews - record preview present cost separately from output render cost Exit criteria: - [x] preview cannot drain output headroom invisibly - runtime telemetry shows preview skips and preview present cost Implementation notes: - `OpenGLComposite::paintGL(false)` now skips preview presentation when `VideoBackend` reports that the ready queue is below the target depth. - Forced preview paints are still allowed so resize/manual paint behavior remains intact. - Preview skip counters and present-cost telemetry remain follow-up work for this step. ### Step 5. Make Readback Miss Policy Deadline-Aware Avoid turning a late async readback fence into synchronous deadline pressure by default. Initial target: - count async readback misses - count synchronous fallback uses - [x] allow policy to prefer stale/black output over synchronous fallback when queue pressure is high - [x] keep current fallback available while behavior is measured Exit criteria: - [x] readback fallback is an explicit policy decision - [x] late GPU fences do not automatically block the most timing-sensitive path Implementation notes: - `OpenGLRenderPipeline::ReadOutputFrame()` now uses synchronous readback only to bootstrap the first cached output frame. - After cached output exists, an async readback miss copies the cached output frame into the DeckLink output frame instead of blocking on synchronous `glReadPixels`. - Async readback queueing now skips when the next PBO slot is still in flight rather than deleting an in-flight fence and overwriting it. - Miss/fallback counters remain follow-up telemetry work for this step. ### Step 6. Tune Headroom Policy Use measured behavior to choose default queue depth and latency tradeoffs. Initial target: - compare 30fps and 60fps behavior - tune `targetReadyFrames` and `maxReadyFrames` - document expected latency cost of each default - keep the setting centralized in `VideoPlayoutPolicy` Exit criteria: - default headroom values are based on observed timing, not guesswork - latency versus resilience tradeoff is documented ## Testing Strategy Recommended tests: - production policy requests work when queue is below target - production policy throttles when queue is full - late/drop pressure biases toward production - preview policy skips when output queue is below target - readback miss policy selects stale/black versus synchronous fallback according to pressure - producer shutdown drains or cancels work without touching destroyed render/backend state Useful homes: - a new `OutputProductionControllerTests` - `RenderOutputQueueTests` for pressure-adjacent queue behavior - `VideoPlayoutSchedulerTests` for recovery/pressure interactions - non-GL fakes for producer loop wake/stop behavior ## Risks ### Latency Risk More ready frames means more latency. Phase 7.5 should make that latency a visible, measured policy choice. ### Producer Runaway Risk A proactive producer must not spin when the queue is full or when output is stopped. ### Buffer Ownership Risk Ready frames must not be reused while DeckLink or the render path still owns their buffers. ### Readback Policy Risk Stale or black output may be preferable to a missed deadline, but it can be visually obvious. External keying may make stale/black fallback more sensitive. ### Preview Regression Risk Treating preview as subordinate may make desktop preview less smooth. That is acceptable only if playout quality improves and preview skips are visible. ## Phase 7.5 Exit Criteria Phase 7.5 can be considered complete once the project can say: - [ ] output production is driven by queue pressure or cadence, not only by completion demand - [ ] completion handling normally schedules already-ready frames - [ ] preview work is explicitly lower priority than playout - [ ] readback miss behavior is explicit and deadline-aware - [ ] queue depth, underruns, render timing, readback misses, and preview skips are visible - [ ] default ready-frame headroom is documented for target frame rates - [ ] production policy has non-DeckLink tests ## Open Questions - Should proactive production be driven by a timer, queue-low notifications, or both? - Should the producer live inside `VideoBackend`, `RenderEngine`, or a small playout controller between them? - Should underrun default to black, last scheduled, or newest completed output once proactive production exists? - How much latency is acceptable at 30fps and 60fps? - Should preview have a hard minimum frame rate, or be fully opportunistic under playout pressure? - Should synchronous readback fallback be disabled automatically after repeated late/drop pressure? ## Short Version Phase 7 made playout observable and safer. Phase 7.5 should make it proactive. The render side should keep the output queue warm before DeckLink needs the next frame. DeckLink should consume ready frames. Preview and synchronous readback fallback should never quietly steal the budget needed to hit output deadlines.