video-shader-toys/docs/PHASE_7_6_SYSTEM_MEMORY_PLAYOUT_BUFFER_DESIGN.md

Phase 7.6: System-Memory Playout Buffer Design

Status

In progress.

Implemented so far:

• BGRA8 SystemOutputFramePool with non-GL tests
• render/readback production now writes into app-owned system-memory slots
• DeckLink output scheduling can wrap system-memory slots with CreateVideoFrameWithBuffer()
• DeckLink completion callbacks release scheduled system-memory slots
• ready-queue discard paths release owned frames instead of leaking slots
• telemetry scaffolding exposes free, ready, and scheduled system-memory frame counts
• async PBO readback is now a deeper pipeline by default and ordinary misses no longer flush queued readbacks
• the output producer now honors requested burst production when the ready queue is below target instead of producing only one frame per wake

Still to verify/tune on hardware:

• sustained DeckLink buffer depth
• frame age at schedule/completion
• repeat/underrun policy behavior under real stalls
• whether deeper async readback reduces sawtooth buffer drain
• whether BGRA8 bandwidth is sufficient before considering v210
• whether burst filling keeps readyQueue.depth above zero and reduces the remaining short stutters

Phase 7.5 isolated the current playout timing problem around output readback and DeckLink scheduling pressure. The fast-transfer path from the DeckLink OpenGL sample is not available on the current test GPU, so the next direction is to make the normal path behave more like broadcast playout systems: render ahead, read back into system-memory frame buffers, and let DeckLink consume already-complete frames.

This phase is not a move away from rendering every frame. It is a move away from making DeckLink wait for each frame to be rendered and read back at the moment it needs to be scheduled.

SDK Finding: RGBA8 Is Not Required

DeckLink output frames do not have to be RGBA8/BGRA8.

The SDK accepts a BMDPixelFormat when creating output frames. The available formats include:

• bmdFormat8BitYUV
• bmdFormat10BitYUV
• bmdFormat10BitYUVA
• bmdFormat8BitARGB
• bmdFormat8BitBGRA
• bmdFormat10BitRGB
• bmdFormat12BitRGB

The SDK samples also use non-RGBA output paths:

• FilePlayback converts unsupported source frames to bmdFormat10BitYUV
• PlaybackStills uses bmdFormat10BitYUV
• InputLoopThrough handles bmdFormat10BitYUV and related formats
• SignalGen exposes 8-bit YUV, 10-bit YUV, 8-bit RGB, and 10-bit RGB choices
• OpenGLOutput uses BGRA for that sample path, but that is not a DeckLink API requirement

The current app already has partial support for this direction:

• DeckLinkSession probes support for bmdFormat10BitYUV and bmdFormat10BitYUVA
• VideoIOPixelFormat::V210 maps to bmdFormat10BitYUV
• VideoIOPixelFormat::Yuva10 maps to bmdFormat10BitYUVA
• RenderEngine has a 10-bit output packing path
• row-byte calculation already distinguishes bmdFormat10BitYUV, bmdFormat10BitYUVA, and BGRA-style formats

So yes: we can pack before readback later if bandwidth proves to be the remaining bottleneck. For the first Phase 7.6 implementation, keep BGRA8 as the active output format and focus on the larger architectural problem: DeckLink should schedule from completed system-memory frames instead of waiting on the current render/readback operation.

Goal

Create a buffered system-memory playout path:

• render every output frame
• keep BGRA8 as the first output/readback format
• read back into reusable CPU/system-memory frame slots
• keep a small queue of completed frames ahead of DeckLink
• schedule DeckLink from completed frames rather than from in-progress rendering
• preserve telemetry so every experiment can be compared against Phase 7.5

Non-Goals

• Do not reintroduce NVIDIA DVP or AMD pinned-memory as a required path.
• Do not hide dropped, repeated, or late frames.
• Do not make cached-output playback the default production behavior.
• Do not add a large latency buffer without making that latency explicit.
• Do not rewrite shader/effect evaluation unless profiling proves it is the bottleneck.
• Do not make v210/YUV packing part of the first implementation unless BGRA8 buffering is proven insufficient.

Architecture

Current Problem Shape

The current path is still too close to:

1. DeckLink needs a frame.
2. App renders or finalizes a frame.
3. App reads back from GL.
4. App schedules the frame.

That can work only if every step reliably fits inside the frame budget. When readback stalls or scheduling is delayed, DeckLink sees a shallow buffer and playback freezes.

Target Shape

The target path is:

1. Render producer prepares future frames.
2. GPU output is read back as BGRA8 into the selected system frame slot.
3. Readback fills a free system-memory frame slot.
4. Completed slots enter a ready queue.
5. DeckLink scheduler consumes ready slots at output cadence.
6. Completion callbacks release slots back to the pool.

This gives the scheduler a small cushion without sacrificing rendered frames.

Pixel Format Strategy

First Target: BGRA8

Use BGRA8 as the first serious output target.

Reasons:

• it is the path closest to the current renderer
• it avoids introducing color-space packing risk while the buffering architecture is still being proven
• it keeps alpha/keying behavior easier to reason about
• it lets Phase 7.6 isolate scheduling/readback ownership from pixel-format conversion

Known byte cost at 1920x1080:

• BGRA8: about 8.29 MB per frame

That is larger than v210, but the immediate hypothesis is that the freezes come from scheduling coupling and readback stalls, not only raw byte count. Prove or disprove that with the system-memory queue first.

Later Target: 10-bit YUV / v210

Keep v210 available as a later optimization.

Reasons to revisit it:

• it is a native DeckLink output format
• it can reduce 1920x1080 readback size from about 8.29 MB per BGRA8 frame to about 5.53 MB per v210 frame
• it may better match final video I/O expectations for fill-only output

Do this only after the BGRA8 system-memory queue is measured. If BGRA8 buffering keeps DeckLink healthy, v210 becomes a quality/bandwidth refinement rather than a rescue path.

Alpha / Keying

For the first implementation, BGRA8 remains the default target.

For alpha/key workflows, bmdFormat10BitYUVA may be needed, or key/fill may need to remain split depending on the device mode and keyer configuration. Phase 7.6 should make this explicit rather than assuming one format fits both.

Proposed Components

SystemOutputFramePool

Owns reusable CPU-side frame slots.

Responsibilities:

• allocate a fixed number of output slots
• expose free slots to the render/readback producer
• expose completed slots to the DeckLink scheduler
• track slot generation, frame id, frame time, and pixel format
• prevent reuse while DeckLink still owns or may read the frame

OutputFrameSlot

Represents one CPU/system-memory playout frame.

Likely contents:

• pointer to writable frame bytes
• row bytes
• width and height
• BMDPixelFormat or app-level equivalent
• frame number / stream time
• timing metadata
• completion state
• optional DeckLink frame wrapper

DeckLinkOutputFrameAdapter

Bridges app-owned memory to DeckLink output frames.

Options to evaluate:

• create DeckLink frames with app-owned buffers where supported by the SDK
• keep DeckLink-created frames in the pool and write directly into their bytes
• wrap app memory behind a small IDeckLinkVideoFrame implementation only if needed

The simplest production path should avoid an extra CPU copy between app memory and DeckLink memory.

OutputFrameProducer

Runs on or is driven by the render thread.

Responsibilities:

• acquire a free system frame slot
• render the next frame
• read back BGRA8 into the slot
• publish the slot to the ready queue
• record readback timings

DeckLinkPlayoutScheduler

Consumes completed system frames.

Responsibilities:

• keep DeckLink scheduled ahead by the configured target depth
• schedule from the ready queue
• repeat/drop according to explicit policy when the queue is empty or too deep
• release frame slots after DeckLink completion callbacks
• report buffer depth and scheduling lead

Migration Plan

Step 1: Make Output Pixel Format Explicit Everywhere

Current format selection exists, but Phase 7.6 should make it impossible to confuse render texture format, readback format, and DeckLink scheduled format.

Deliverables:

• log selected DeckLink output pixel format at startup
• expose readback bytes per frame in telemetry
• expose whether the frame was BGRA, v210, or YUVA
• make BGRA8 the default and first supported system-buffer path

Step 2: Introduce the BGRA8 System Frame Pool

Add a fixed-size pool of BGRA8 system-memory output slots.

Initial target depth:

• 3 ready/scheduled frames minimum
• 5 frames as the practical DeckLink-health target
• configurable for experiments

The pool should be testable without OpenGL or DeckLink hardware.

Step 3: Read Back BGRA8 Into Pool Slots

Move readback output away from transient buffers and into acquired frame slots.

The producer must never block DeckLink scheduling while waiting for a free slot if a safe repeat/drop policy can keep playback alive.

Step 4: Schedule From Completed Slots

Change DeckLink scheduling to consume completed system frames.

DeckLink callbacks should become the point where slots are returned to the pool.

This is the main behavioral change: scheduling no longer waits for the active render/readback operation.

Step 5: Add Playout Policies

Make underflow and overflow behavior explicit.

Possible policies:

• repeat last completed frame on underflow
• schedule black on startup only
• drop oldest completed frame if the producer gets too far ahead
• preserve most recent frame for live-control responsiveness

The default should favor stable output cadence and visible telemetry over silent correctness guesses.

Step 6: Tune Buffer Depth and Latency

Measure:

• render time
• readback time
• CPU copy time, if any
• ready queue depth
• scheduled queue depth
• frame age at schedule time
• frame age at display callback
• repeats, drops, and underruns

Then choose a default buffer depth that keeps DeckLink healthy without adding unnecessary latency.

Step 7: Optional v210 Experiment

Only after BGRA8 buffering has been measured, add a runtime option that forces:

• GPU pack to v210
• readback of packed v210 bytes
• DeckLink scheduling as bmdFormat10BitYUV

This should be compared against the completed BGRA8 system-memory path, not against the older coupled path.

Telemetry

Keep the Phase 7.5 counters and add:

• outputPixelFormat
• outputReadbackBytes
• outputPackMode
• systemFramePoolFree
• systemFramePoolReady
• systemFramePoolScheduled
• systemFrameAgeAtScheduleMs
• systemFrameAgeAtCompletionMs
• systemFrameUnderruns
• systemFrameRepeats
• systemFrameDrops
• deckLinkScheduleLeadFrames
• deckLinkScheduleLeadMs

Telemetry scaffolding can land before the frame pool itself. Until SystemOutputFramePool exists, these fields should remain producer-owned gauges/counters with default zero values in HealthTelemetry; they should not be inferred from the existing render-ready queue or DeckLink pool because those are adjacent concepts, not the final free/ready/scheduled system-memory slot model.

Existing counters that should remain useful:

• render frame time
• async queue time
• readback timing
• output queue depth
• displayed late count
• dropped count
• DeckLink buffered frame count

Tests

Add non-GL tests for:

• frame pool acquire/publish/consume/release
• slots are not reused while scheduled
• underflow repeats the last completed frame when configured
• overflow drops according to policy
• row-byte and byte-size calculation for BGRA8 first, with v210 and YUVA covered when those modes are enabled
• scheduler consumes only completed frames
• completion callback releases the expected slot

Hardware/manual tests:

• BGRA8 system-buffered output works
• DeckLink buffer depth stays healthy
• no black-frame startup longer than configured preroll
• shutdown drains or releases scheduled slots safely

Risks

• DeckLink frame ownership rules may force one extra copy if app-owned buffers are not accepted in the exact path we use.
• Buffering improves cadence but adds latency.
• If GPU readback itself remains slower than real time, buffering only delays the underflow.
• v210 remains a future optimization and may still carry color-space/keying risk when introduced.

Exit Criteria

Phase 7.6 is complete when:

• DeckLink output format is explicit and logged
• BGRA8 system-memory output slots are the default playout path
• completed system-memory frames are queued ahead of DeckLink scheduling
• DeckLink callbacks release/recycle frame slots
• ready/scheduled buffer depth is visible in telemetry
• underflow/repeat/drop behavior is explicit and tested
• the app can sustain a healthy DeckLink buffer without using cached-output playback