docs update
Some checks failed
CI / React UI Build (push) Successful in 10s
CI / Native Windows Build And Tests (push) Successful in 2m36s
CI / Windows Release Package (push) Has been cancelled

This commit is contained in:
Aiden
2026-05-11 17:16:39 +10:00
parent e5c5920ccd
commit ebc10a9925
5 changed files with 417 additions and 31 deletions

View File

@@ -7,6 +7,7 @@ Phase checklist:
- [x] Define subsystem boundaries and target architecture - [x] Define subsystem boundaries and target architecture
- [x] Introduce an internal event model - [x] Introduce an internal event model
- [x] Split `RuntimeHost` - [x] Split `RuntimeHost`
- [x] Finish live-state and service-facing coordination
- [ ] Make the render thread the sole GL owner - [ ] Make the render thread the sole GL owner
- [ ] Refactor live state layering into an explicit composition model - [ ] Refactor live state layering into an explicit composition model
- [ ] Move persistence onto a background snapshot writer - [ ] Move persistence onto a background snapshot writer
@@ -17,7 +18,8 @@ Checklist note:
- The checked Phase 1 item means the subsystem vocabulary, dependency direction, state categories, design package, and runtime implementation foothold are in place. - The checked Phase 1 item means the subsystem vocabulary, dependency direction, state categories, design package, and runtime implementation foothold are in place.
- The checked Phase 2 item means the internal event model substrate is complete enough for later phases: the typed event vocabulary, app-owned dispatcher, coalesced event pump, reload bridge events, production bridges, and pure event tests are in place. Remaining items in [PHASE_2_INTERNAL_EVENT_MODEL_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_2_INTERNAL_EVENT_MODEL_DESIGN.md) are narrow follow-ups, mainly completion/failure observations and later replacement of the runtime-store poll fallback with real file-watch events. - The checked Phase 2 item means the internal event model substrate is complete enough for later phases: the typed event vocabulary, app-owned dispatcher, coalesced event pump, reload bridge events, production bridges, and pure event tests are in place. Remaining items in [PHASE_2_INTERNAL_EVENT_MODEL_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_2_INTERNAL_EVENT_MODEL_DESIGN.md) are narrow follow-ups, mainly completion/failure observations and later replacement of the runtime-store poll fallback with real file-watch events.
- It does not mean the whole app is fully extracted. Sole-owner render threading, live-state layering, background persistence, backend lifecycle, and richer telemetry continue through later phases. - The checked Phase 3 item means the render-facing state path now has named live-state, composition, frame-state, resolver, and service-bridge boundaries. `OpenGLComposite::renderEffect()` is reduced to runtime work, frame input construction, and frame rendering. This prepares Phase 4 but does not yet move GL work onto a dedicated render thread.
- It does not mean the whole app is fully extracted. Sole-owner render threading, deeper live-state layering, background persistence, backend lifecycle, and richer telemetry continue through later phases.
## Timing Review ## Timing Review
@@ -177,7 +179,7 @@ Relevant timing code:
Why this matters: Why this matters:
- `PlayoutFrameCompleted()` currently begins an output frame, takes the shared GL path, renders, reads back, and schedules the next frame in one callback-driven flow. - the output completion path currently requests a scheduled render through `OpenGLVideoIOBridge::RenderScheduledFrame()`, which still takes the shared GL path, renders, reads back, and schedules the next frame in one callback-driven flow.
- `VideoPlayoutScheduler::AccountForCompletionResult()` currently reacts to both late and dropped frames by blindly advancing the schedule index by `2`, which is simple but not especially robust. - `VideoPlayoutScheduler::AccountForCompletionResult()` currently reacts to both late and dropped frames by blindly advancing the schedule index by `2`, which is simple but not especially robust.
- `kPrerollFrameCount` is now `12`, but `DeckLinkSession::ConfigureOutput()` still creates a fixed pool of `10` mutable output frames. That mismatch suggests the buffering model is not being sized from one coherent source of truth. - `kPrerollFrameCount` is now `12`, but `DeckLinkSession::ConfigureOutput()` still creates a fixed pool of `10` mutable output frames. That mismatch suggests the buffering model is not being sized from one coherent source of truth.
@@ -496,6 +498,10 @@ Primary design rule:
With state and coordination cleaner, move to a dedicated render-thread model. With state and coordination cleaner, move to a dedicated render-thread model.
Dedicated design note:
- [PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md)
Target behavior: Target behavior:
- one thread owns the GL context - one thread owns the GL context

View File

@@ -6,7 +6,7 @@ Phase 1 split runtime responsibilities into named subsystems. Phase 2 added the
## Status ## Status
- Phase 3 design package: proposed. - Phase 3 design package: accepted.
- Phase 3 implementation: exit criteria satisfied for the current architecture. - Phase 3 implementation: exit criteria satisfied for the current architecture.
- Current alignment: the repo now has the live-state/composer building blocks, a service bridge, and a named frame-state handoff. `OpenGLComposite::renderEffect()` still remains the app-level frame entrypoint, but the service drain, layer-state resolution, and OSC commit handoff now sit behind named helpers and frame-state data. - Current alignment: the repo now has the live-state/composer building blocks, a service bridge, and a named frame-state handoff. `OpenGLComposite::renderEffect()` still remains the app-level frame entrypoint, but the service drain, layer-state resolution, and OSC commit handoff now sit behind named helpers and frame-state data.
@@ -24,17 +24,17 @@ Current footholds:
- `RuntimeServiceLiveBridge` translates service OSC queues into render live-state updates and queues settled overlay commit requests. - `RuntimeServiceLiveBridge` translates service OSC queues into render live-state updates and queues settled overlay commit requests.
- `RuntimeEventDispatcher` now routes accepted mutations, reloads, snapshots, shader build events, backend observations, and health observations. - `RuntimeEventDispatcher` now routes accepted mutations, reloads, snapshots, shader build events, backend observations, and health observations.
The current architecture is much better than the original `RuntimeHost` shape, but the render path still has too much coordination logic stitched through `OpenGLComposite`, `RuntimeServices`, `RuntimeCoordinator`, and `RenderEngine`. The current architecture is much better than the original `RuntimeHost` shape. The remaining render risk is now mostly Phase 4 work: GL calls are still reached from callback and UI paths through a shared context lock rather than through one render-thread owner.
## Why Phase 3 Exists ## Why Phase 3 Exists
The resilience review says render-thread isolation should come after state access and control coordination are no longer centered on a large mutable runtime object. Phase 2 gives us the event substrate; Phase 3 should make the data flowing into render explicit enough that Phase 4 can make the render thread the sole GL owner without dragging service coordination and state reconciliation with it. The resilience review says render-thread isolation should come after state access and control coordination are no longer centered on a large mutable runtime object. Phase 2 gives us the event substrate; Phase 3 should make the data flowing into render explicit enough that Phase 4 can make the render thread the sole GL owner without dragging service coordination and state reconciliation with it.
The main problems Phase 3 addresses: The main problems Phase 3 addressed:
- transient OSC overlay state and persisted committed state are still reconciled by hand in `OpenGLComposite::renderEffect()` - transient OSC overlay state and persisted committed state needed a named reconciliation boundary
- `RenderEngine` both stores transient OSC overlay state and resolves final layer states - `RenderEngine` needed to move final frame-state selection and value composition out of drawing code
- `ControlServices` exposes service-side queues for pending OSC updates and completed OSC commits - service-side queues for pending OSC updates and completed OSC commits needed a bridge outside `OpenGLComposite`
- `RuntimeStore` still performs synchronous persistence directly from many state mutation paths - `RuntimeStore` still performs synchronous persistence directly from many state mutation paths
- `RuntimeUpdateController` still exists partly as compatibility glue between synchronous coordinator results and event-driven effects - `RuntimeUpdateController` still exists partly as compatibility glue between synchronous coordinator results and event-driven effects
@@ -64,11 +64,11 @@ Those are later phases. Phase 3 is about making state and service coordination c
## Current Coordination Shape ## Current Coordination Shape
`OpenGLComposite::renderEffect()` is now narrower, but it still owns the frame entrypoint: `OpenGLComposite::renderEffect()` is now the app-level frame entrypoint, but it is intentionally narrow:
1. pumps `RuntimeUpdateController::ProcessRuntimeWork()` 1. pumps `RuntimeUpdateController::ProcessRuntimeWork()`
2. asks `RuntimeServiceLiveBridge` to prepare live render layer states 2. builds a `RenderFrameInput`
3. asks `RenderEngine` to draw the layer stack 3. renders the frame through `RuntimeServiceLiveBridge`, `RenderFrameStateResolver`, and `RenderEngine`
The bridge now owns service queue draining, live automation settlement, committed/live state selection, and OSC commit handoff. `RenderFrameStateResolver` owns snapshot cache selection, parameter refresh decisions, and dynamic render-field refresh before handing a prepared frame state to `RenderEngine`. The bridge now owns service queue draining, live automation settlement, committed/live state selection, and OSC commit handoff. `RenderFrameStateResolver` owns snapshot cache selection, parameter refresh decisions, and dynamic render-field refresh before handing a prepared frame state to `RenderEngine`.
@@ -90,11 +90,11 @@ final render state = published snapshot + committed live selection + transient o
The important change is not the exact formula name. The important change is that final render-state composition has one named owner and can be tested without GL. The important change is not the exact formula name. The important change is that final render-state composition has one named owner and can be tested without GL.
## Proposed Collaborators ## Phase 3 Collaborators
### `RuntimeLiveState` ### `RuntimeLiveState`
New small runtime collaborator or equivalent read model builder. Small runtime collaborator for transient automation state.
Responsibilities: Responsibilities:
@@ -113,7 +113,7 @@ Non-responsibilities:
### `RenderStateComposer` ### `RenderStateComposer`
New pure or mostly pure collaborator. Pure or mostly pure collaborator for frame value composition.
Responsibilities: Responsibilities:
@@ -250,7 +250,7 @@ Introduce `RuntimeLiveState`, `RenderStateComposer`, or an equivalent pair of cl
Start by moving pure data operations out of frame rendering without changing behavior. Start by moving pure data operations out of frame rendering without changing behavior.
Status: started. `runtime/live/RuntimeLiveState` and `runtime/live/RenderStateComposer` now exist, are included in the build, and have a focused `RuntimeLiveStateTests` target. Status: complete for Phase 3. `runtime/live/RuntimeLiveState` and `runtime/live/RenderStateComposer` exist, are included in the build, and have a focused `RuntimeLiveStateTests` target.
### Step 2. Move OSC Overlay Bookkeeping Behind The Boundary ### Step 2. Move OSC Overlay Bookkeeping Behind The Boundary
@@ -263,7 +263,7 @@ Move these responsibilities out of the current frame orchestration:
The first implementation can still be called synchronously from the current render path. The important part is that the behavior has a named owner and tests. The first implementation can still be called synchronously from the current render path. The important part is that the behavior has a named owner and tests.
Status: mostly complete for the current architecture. `RenderEngine` still exposes compatibility methods used by the service bridge, but it now delegates overlay updates, commit completions, smoothing, generation matching, and commit-request creation to `RuntimeLiveState`/`RenderStateComposer`. Status: complete for Phase 3. `RenderEngine` still exposes compatibility methods used by the service bridge, but it delegates overlay updates, commit completions, smoothing, generation matching, and commit-request creation to `RuntimeLiveState`/`RenderStateComposer`.
### Step 3. Bridge Service Queues To Events Or Live-State Commands ### Step 3. Bridge Service Queues To Events Or Live-State Commands
@@ -276,7 +276,7 @@ Replace `OpenGLComposite::renderEffect()` queue draining with a bridge that publ
This is where the remaining Phase 2 open question about transient OSC overlay event scope should be resolved for the current architecture. This is where the remaining Phase 2 open question about transient OSC overlay event scope should be resolved for the current architecture.
Status: started. `RuntimeServiceLiveBridge` now drains pending OSC updates and completed OSC commits, applies them to render live state, and queues settled commit requests. It is still a source-local bridge rather than a fully dispatcher-driven event bridge. Status: complete for Phase 3. `RuntimeServiceLiveBridge` now drains pending OSC updates and completed OSC commits, applies them to render live state, and queues settled commit requests. It remains a source-local bridge by design until later live-state layering decides whether transient automation should enter the app-level dispatcher.
### Step 4. Narrow `OpenGLComposite::renderEffect()` ### Step 4. Narrow `OpenGLComposite::renderEffect()`
@@ -293,7 +293,7 @@ void OpenGLComposite::renderEffect()
The exact names can change. The goal is that render effect no longer manually drains services, settles overlay commits, and resolves layer values. The exact names can change. The goal is that render effect no longer manually drains services, settles overlay commits, and resolves layer values.
Status: started. `OpenGLComposite::renderEffect()` still drives frame timing, video dimensions, and drawing, but the service-drain, resolve, and commit-handoff path has moved behind `RuntimeServiceLiveBridge::PrepareLiveRenderFrameState(...)` and a named `RenderFrameInput` / `RenderFrameState` handoff. Status: complete for Phase 3. `OpenGLComposite::renderEffect()` now processes runtime work, builds `RenderFrameInput`, and calls a narrow frame-render helper. Service draining, state resolution, and commit handoff sit behind `RuntimeServiceLiveBridge::PrepareLiveRenderFrameState(...)`, `RenderFrameStateResolver`, and `RenderFrameState`.
### Step 5. Add Persistence Boundary Tests ### Step 5. Add Persistence Boundary Tests
@@ -304,6 +304,8 @@ Add behavior tests for:
- preset load/save persistence requests remain explicit - preset load/save persistence requests remain explicit
- rejected mutations do not publish persistence work - rejected mutations do not publish persistence work
Status: complete for Phase 3. `RuntimeSubsystemTests` and `RuntimeEventTypeTests` cover accepted mutation persistence requests, rejected mutations, and transient OSC overlay behavior that does not request persistence.
### Step 6. Update Docs And Phase 4 Readiness ### Step 6. Update Docs And Phase 4 Readiness
Before calling Phase 3 complete, update: Before calling Phase 3 complete, update:
@@ -312,6 +314,8 @@ Before calling Phase 3 complete, update:
- architecture review checklist - architecture review checklist
- Phase 4 assumptions about render thread input state - Phase 4 assumptions about render thread input state
Status: complete. The Phase 4 design note starts from the `RenderFrameInput` / `RenderFrameState` contract and the remaining shared-GL ownership paths.
## Testing Strategy ## Testing Strategy
Phase 3 tests should avoid GL, DeckLink, and sockets. Phase 3 tests should avoid GL, DeckLink, and sockets.
@@ -340,11 +344,11 @@ The current groundwork is intended to let these lanes proceed in parallel with l
| Lane | Primary files | Goal | | Lane | Primary files | Goal |
| --- | --- | --- | | --- | --- | --- |
| A. Live-state behavior | `runtime/live/RuntimeLiveState.*`, `tests/RuntimeLiveStateTests.cpp` | Finish stale completion tests, smoothing edge cases, trigger behavior, and overlay settle policy. | | A. Live-state behavior | `runtime/live/RuntimeLiveState.*`, `tests/RuntimeLiveStateTests.cpp` | Implemented for Phase 3: stale completion, smoothing, trigger behavior, and overlay settle policy are covered by focused tests. |
| B. Render-state composition | `runtime/live/RenderStateComposer.*`, `gl/RenderFrameStateResolver.*`, `gl/RenderEngine.*` | Keep value composition and frame-state selection outside GL drawing while keeping GL calls in `RenderEngine`. | | B. Render-state composition | `runtime/live/RenderStateComposer.*`, `gl/RenderFrameStateResolver.*`, `gl/RenderEngine.*` | Implemented for Phase 3: value composition and frame-state selection sit outside GL drawing while GL calls remain in `RenderEngine`. |
| C. Service bridge | `control/RuntimeServices.*`, `control/ControlServices.*`, possible new bridge class | Stop `OpenGLComposite::renderEffect()` from draining OSC update/completion queues directly. | | C. Service bridge | `control/RuntimeServices.*`, `control/RuntimeServiceLiveBridge.*`, `control/ControlServices.*` | Implemented for Phase 3: `OpenGLComposite::renderEffect()` no longer drains OSC update/completion queues directly. |
| D. App-frame orchestration | `gl/OpenGLComposite.*`, `gl/RuntimeUpdateController.*` | Replace render-effect glue with a narrow frame-state preparation call and commit-request handoff. | | D. App-frame orchestration | `gl/OpenGLComposite.*`, `gl/RuntimeUpdateController.*` | Implemented for Phase 3: render-effect glue is a narrow runtime-work, frame-input, render-frame sequence. |
| E. Persistence boundary | `runtime/coordination/RuntimeCoordinator.*`, `runtime/store/*`, event tests | Keep persistence request publication explicit and prepare for a later background writer without changing storage behavior yet. | | E. Persistence boundary | `runtime/coordination/RuntimeCoordinator.*`, `runtime/store/*`, event tests | Implemented for Phase 3: persistence request publication is explicit and ready for a later background writer. |
## Phase 3 Exit Criteria ## Phase 3 Exit Criteria

View File

@@ -0,0 +1,373 @@
# Phase 4 Design: Render Thread Ownership
This document expands Phase 4 of [ARCHITECTURE_RESILIENCE_REVIEW.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/ARCHITECTURE_RESILIENCE_REVIEW.md) into a concrete design target.
Phase 1 named the subsystems. Phase 2 added the typed event substrate. Phase 3 made render-facing live state explicit through `RuntimeLiveState`, `RenderStateComposer`, `RenderFrameInput`, `RenderFrameState`, `RenderFrameStateResolver`, and `RuntimeServiceLiveBridge`. Phase 4 can now focus on the core timing-risk boundary: making one render thread the only owner of OpenGL work.
## Status
- Phase 4 design package: proposed.
- Phase 4 implementation: not started.
- Current alignment: the repo has a named frame-state contract and cleaner render-state preparation, but GL work is still entered through multiple paths protected by one shared `CRITICAL_SECTION`.
Current GL ownership footholds:
- `RenderEngine` owns GL resources and the current context-binding helpers.
- `RenderFrameInput` / `RenderFrameState` provide the frame-state contract that a render thread can consume.
- `RenderFrameStateResolver` prepares the render-facing layer state before drawing.
- `OpenGLVideoIOBridge` still calls `RenderEngine::TryUploadInputFrame(...)` from the input path and `RenderEngine::RenderOutputFrame(...)` from the output path.
- `OpenGLComposite::paintGL(...)`, screenshot capture, input upload, and output rendering still reach GL through `RenderEngine` methods that bind the shared context under `pMutex`.
## Why Phase 4 Exists
The resilience review identifies shared GL ownership as the main remaining timing and failure-isolation risk. Today the shared context lock protects correctness, but it does not isolate timing:
- input callbacks can attempt texture upload
- output callbacks can trigger frame rendering and readback
- preview paint can enter the same GL context
- screenshot capture can enter the same GL context
- the DeckLink completion path is still too close to render work
That means brief input, preview, readback, or callback stalls can still collide on the most timing-sensitive path.
Phase 4 should turn GL from a shared resource guarded by a lock into a resource owned by one thread with explicit queues and handoff points.
## Goals
Phase 4 should establish:
- one render thread as the sole long-lived owner of the GL context
- non-render threads enqueue work instead of binding the GL context
- input upload requests are accepted and executed by the render thread
- output frame rendering is requested or scheduled through render-owned work
- preview and screenshot requests become render-thread commands or consumers
- `RenderFrameInput` / `RenderFrameState` become the stable data contract for frame production
- GL context entrypoints are reduced to render-thread-only code paths
- tests for queue semantics, request coalescing, and lifecycle behavior without requiring DeckLink hardware
## Non-Goals
Phase 4 should not require:
- the final producer/consumer playout queue for DeckLink
- the final DeckLink lifecycle state machine
- replacing the async readback policy
- implementing background persistence
- completing Phase 5's deeper live-state layering
- replacing every UI or backend API at once
Those are later phases or follow-on work. Phase 4 is about making GL ownership deterministic first.
## Current GL Entry Points
The current code paths that matter most are:
| Entry point | Current behavior | Phase 4 direction |
| --- | --- | --- |
| `RenderEngine::TryUploadInputFrame(...)` | attempts to take the GL lock, binds the context, uploads input texture | enqueue latest input frame; render thread uploads |
| `RenderEngine::RenderOutputFrame(...)` | takes the GL lock, binds the context, renders, packs/readbacks output | render thread executes output frame production |
| `RenderEngine::TryPresentPreview(...)` | attempts to take the GL lock and presents preview | render thread or preview presenter consumes latest completed frame |
| `RenderEngine::CaptureOutputFrameRgbaTopDown(...)` | takes the GL lock and reads output pixels | screenshot request becomes render-thread command |
| `OpenGLVideoIOBridge::UploadInputFrame(...)` | calls render upload directly | push input frame into render queue/mailbox |
| `OpenGLVideoIOBridge::RenderScheduledFrame(...)` | calls render output directly from backend path | request/consume render-produced output without callback-owned GL |
## Target Ownership Model
### Render Thread
The render thread should own:
- `wglMakeCurrent(...)` for the rendering context
- all GL resource creation/destruction
- input texture upload
- pass execution
- output pack conversion
- async readback buffers and fences
- preview presentation or preview frame publication
- screenshot readback
- temporal history and feedback resources
### Other Threads
Other threads may:
- enqueue input frames or replace the latest input frame
- publish control/runtime/backend events
- request shader build application
- request render-local resets
- request screenshots
- consume ready output frames or receive completion notifications
Other threads should not:
- call GL directly
- bind or unbind the render context
- wait on GL fences directly
- mutate render-local resource state
## Proposed Collaborators
### `RenderThread`
Owns the OS thread, wakeup primitive, lifecycle, and render-loop execution.
Responsibilities:
- start and stop the render thread
- bind the GL context for the thread lifetime or render-loop lifetime
- drain render commands
- execute frame production work
- publish lifecycle and failure observations
Non-responsibilities:
- runtime mutation policy
- DeckLink scheduling policy
- durable persistence
### `RenderCommandQueue`
Small bounded queue or command mailbox for render-thread work.
Possible commands:
- `UploadInputFrame`
- `RenderOutputFrame`
- `PrepareFrameState`
- `ApplyShaderBuild`
- `ResetTemporalHistory`
- `ResetShaderFeedback`
- `PresentPreview`
- `CaptureScreenshot`
- `Stop`
High-rate commands should be coalesced where appropriate. Input frames should likely be latest-value rather than unbounded FIFO.
### `RenderFrameCoordinator`
Optional helper that combines Phase 3's frame contract with render-thread execution.
Responsibilities:
- build or receive `RenderFrameInput`
- call `RuntimeServiceLiveBridge` and `RenderFrameStateResolver`
- hand `RenderFrameState` to `RenderEngine`
This can begin as a thin helper. The important part is that it keeps frame-state preparation explicit when `renderEffect()` stops being called directly from the callback path.
### `RenderOutputMailbox`
Optional transitional bridge for output frames.
Responsibilities:
- hold the latest completed output frame or a small bounded set
- let backend code consume output without owning GL
- report underrun/stale-frame reuse observations
This may be a Phase 4 late step or a Phase 7 playout-policy step. Phase 4 should at least avoid designing the render thread in a way that blocks it.
## Threading Contract
Phase 4 should make thread ownership visible in APIs.
Candidate naming:
- `RenderEngine::StartRenderThread(...)`
- `RenderEngine::StopRenderThread()`
- `RenderEngine::EnqueueInputFrame(...)`
- `RenderEngine::RequestOutputFrame(...)`
- `RenderEngine::RequestPreviewPresent(...)`
- `RenderEngine::RequestScreenshot(...)`
Render-thread-only methods should be private or clearly named:
- `RenderEngine::UploadInputFrameOnRenderThread(...)`
- `RenderEngine::RenderOutputFrameOnRenderThread(...)`
- `RenderEngine::CaptureOutputFrameOnRenderThread(...)`
The current `TryUploadInputFrame`, `RenderOutputFrame`, `TryPresentPreview`, and `CaptureOutputFrameRgbaTopDown` methods can remain as compatibility shims during migration, but their implementations should move toward enqueue-and-wait or enqueue-and-return behavior instead of binding GL directly from the caller's thread.
## Frame Production Shape
A target render-thread frame should look like:
1. wake for input, output demand, preview demand, shader build, reset, screenshot, or stop
2. drain bounded render commands
3. coalesce to the latest input frame and latest control/live state
4. build `RenderFrameInput`
5. prepare `RenderFrameState`
6. upload accepted input frame
7. render layer stack
8. pack output if needed
9. stage readback or output buffer
10. publish preview/screenshot/output completion as needed
11. record timing and queue metrics
The exact cadence can remain demand-driven initially. The architectural win is that the demand wakes the render thread rather than borrowing GL from the caller.
## Migration Plan
### Step 1. Name Render-Thread-Only Methods
Split existing direct GL methods into public request methods and private render-thread methods without changing behavior much.
Initial target:
- keep current synchronous behavior where callers need a result
- move GL bodies into clearly render-thread-owned helpers
- make future queue migration mechanical
### Step 2. Add Render Command Queue
Introduce a small queue/mailbox for render commands.
Start with low-risk commands:
- preview present request
- screenshot request
- render-local reset requests
Then move input upload and output render requests once the queue and wakeup behavior are proven.
### Step 3. Start A Dedicated Render Thread
Create the render thread and make it own context binding.
Transitional behavior may still allow synchronous request/response for output frames. The important change is that the caller waits for render-thread completion rather than taking the GL context itself.
### Step 4. Move Input Upload To The Render Thread
Change `OpenGLVideoIOBridge::UploadInputFrame(...)` so it enqueues or replaces the latest input frame.
Policy targets:
- bounded memory
- latest-frame wins under load
- input upload skip count is observable
- input callback never waits for GL
### Step 5. Move Output Rendering To The Render Thread
Change `OpenGLVideoIOBridge::RenderScheduledFrame(...)` so it requests render-thread output production or consumes a completed render-thread output.
Transitional option:
- synchronous request/response through the render thread
Better follow-up:
- render ahead into a bounded output queue and let backend callbacks consume ready frames
### Step 6. Decouple Preview And Screenshot Requests
Preview should become best-effort:
- request preview presentation from the render thread
- skip when render is busy or output deadline pressure is high
- record preview skips
Screenshot should become:
- queued render-thread capture request
- async disk write remains outside render thread
### Step 7. Remove Shared GL Lock From Normal Paths
Once all GL entrypoints are render-thread-owned:
- remove normal dependence on `pMutex` for render correctness
- keep assertions or diagnostics that detect wrong-thread GL calls
- leave only lifecycle synchronization where needed
## Testing Strategy
Phase 4 tests should avoid hardware where possible.
Recommended tests:
- render command queue preserves FIFO for non-coalesced commands
- latest-input mailbox drops older frames under load
- stop command wakes and drains the render thread
- screenshot request receives one completion or failure
- output render request reports timeout/failure if render thread is stopped
- render reset commands coalesce where expected
- wrong-thread render-only methods are not publicly reachable
Existing useful homes:
- `RuntimeEventTypeTests` for new render/backend observations
- `RuntimeSubsystemTests` for pure request/coalescing helpers
- a new `RenderThreadTests` target for queue/mailbox/lifecycle helpers that do not require GL
Manual verification will still be needed for:
- real DeckLink input/output
- preview interaction
- screenshot capture
- shader reload while rendering
## Telemetry Added During Phase 4
Phase 4 should add minimal metrics while moving ownership:
- render command queue depth
- input frames accepted, replaced, and dropped
- render-thread wake reason counts
- render-thread frame duration
- output request latency
- preview request skipped count
- screenshot request success/failure count
- wrong-thread GL call diagnostics if practical
Full operational reporting remains Phase 8, but these metrics make the threading migration debuggable.
## Risks
### Deadlock Risk
Synchronous request/response shims can deadlock if the caller is already on the render thread or holds a lock the render thread needs. Phase 4 should keep request waits narrow and add render-thread detection early.
### Latency Risk
Moving work through queues can hide latency. Queue depth and output request latency should be measured from the first migration step.
### Lifetime Risk
Moving context ownership changes startup and shutdown order. The render thread must stop before GL resources or window/context handles are destroyed.
### Callback Pressure Risk
If DeckLink callbacks wait too long for render-thread work, Phase 4 may improve GL ownership but still leave callback timing fragile. A synchronous bridge is acceptable as a transition, but the design should keep the path open for producer/consumer playout.
### Preview Coupling Risk
Preview can remain a hidden budget consumer if it stays in the output frame path. Phase 4 should keep preview explicitly best-effort, even if physical decoupling continues later.
## Phase 4 Exit Criteria
Phase 4 can be considered complete once the project can say:
- [ ] one render thread owns the GL context during normal operation
- [ ] input callbacks do not bind GL or wait on GL upload
- [ ] output callbacks do not bind GL directly
- [ ] preview and screenshot requests enter render through explicit render-thread requests
- [ ] `RenderFrameInput` / `RenderFrameState` remain the frame-state contract
- [ ] normal frame production no longer depends on a shared GL `CRITICAL_SECTION`
- [ ] render-thread queue/mailbox behavior has non-GL tests
- [ ] shutdown order is explicit and tested or manually verified
## Open Questions
- Should the first output migration be synchronous request/response, or should Phase 4 go directly to a small ready-frame queue?
- Should the render thread own `RuntimeServiceLiveBridge` calls, or should frame state be prepared just before enqueue?
- How much input frame memory should be copied at enqueue time versus referenced from backend-owned buffers?
- Should preview present on the render thread, or should render publish a preview image/texture to a separate presenter?
- What timeout should output callbacks use if the render thread cannot produce a frame in time?
- Should wrong-thread GL access be enforced with assertions, telemetry, or both?
## Short Version
Phase 4 should make GL ownership boring and deterministic.
One render thread owns the context. Other threads submit work or consume results. Input upload, frame rendering, readback, preview, and screenshot capture all move behind render-thread entrypoints. The first implementation can be transitional and partly synchronous, but after Phase 4 the app should no longer rely on callback and UI paths borrowing the GL context under one shared lock.

View File

@@ -39,14 +39,14 @@ The current rendering path is split across several classes:
- [OpenGLComposite.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/OpenGLComposite.cpp:86) constructs the renderer, render pipeline, shader programs, runtime services, and video bridge in one owner. - [OpenGLComposite.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/OpenGLComposite.cpp:86) constructs the renderer, render pipeline, shader programs, runtime services, and video bridge in one owner.
- [OpenGLRenderPipeline.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLRenderPipeline.cpp:31) performs pass execution, pack/readback, preview paint, and performance stat publication. - [OpenGLRenderPipeline.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLRenderPipeline.cpp:31) performs pass execution, pack/readback, preview paint, and performance stat publication.
- [OpenGLVideoIOBridge.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:58) accepts capture frames and still performs render work from the playout completion callback path. - [OpenGLVideoIOBridge.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:58) accepts capture frames and still performs render work from the playout completion callback path.
- `OpenGLComposite` still holds render-local overlay behavior and shader rebuild handling alongside runtime orchestration responsibilities. - `RenderFrameStateResolver` and `RenderStateComposer` now keep frame-state selection and live value composition outside GL drawing, while `RenderEngine` still owns the current GL resource and draw path.
That split is workable today, but it creates architectural pressure: That split is workable today, but it creates architectural pressure:
- GL ownership is thread-shared instead of sole-owned. - GL ownership is thread-shared instead of sole-owned.
- render and playout timing are still callback-coupled. - render and playout timing are still callback-coupled.
- preview and playout are produced in the same immediate path. - preview and playout are produced in the same immediate path.
- render-local transient state is too easy to leak back into runtime-facing code. - render-local transient state now has clearer Phase 3 boundaries, but GL ownership is still shared through callback and UI entrypoints.
- it is difficult to test render behavior separately from app bootstrap and hardware integration. - it is difficult to test render behavior separately from app bootstrap and hardware integration.
`RenderEngine` exists to absorb that responsibility into one subsystem with one direction of ownership. `RenderEngine` exists to absorb that responsibility into one subsystem with one direction of ownership.
@@ -170,11 +170,12 @@ Other threads should interact with the subsystem through queues, snapshots, and
## Current State ## Current State
Today GL work is still shared across callback-driven entrypoints: Today GL work is still shared across callback-driven and UI entrypoints:
- input upload occurs in [OpenGLVideoIOBridge::VideoFrameArrived()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:58) - input upload is requested through [OpenGLVideoIOBridge::UploadInputFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11)
- playout-triggered render occurs in [OpenGLVideoIOBridge::PlayoutFrameCompleted()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:95) - playout-triggered render is requested through [OpenGLVideoIOBridge::RenderScheduledFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:18)
- render-pass execution occurs in [OpenGLRenderPipeline::RenderFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLRenderPipeline.cpp:31) - render-pass execution occurs in [OpenGLRenderPipeline::RenderFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLRenderPipeline.cpp:31)
- preview and screenshot paths still enter `RenderEngine` methods that bind the shared context
The `CRITICAL_SECTION` protects correctness, but it is not the target architectural model. The `CRITICAL_SECTION` protects correctness, but it is not the target architectural model.
@@ -406,6 +407,8 @@ inside render-owned code paths instead of putting them back into runtime storage
Introduce snapshot-facing APIs so render no longer depends on broad runtime-state access for frame production. Introduce snapshot-facing APIs so render no longer depends on broad runtime-state access for frame production.
Current status: Phase 3 introduced `RenderFrameInput`, `RenderFrameState`, and `RenderFrameStateResolver`, so frame-state selection is named and no longer lives inside GL drawing. Phase 4 can build on that contract while moving GL ownership.
### Step 4. Move Uploads Onto Render Ownership ### Step 4. Move Uploads Onto Render Ownership
Input callbacks should enqueue or hand off frame data; render executes the upload. Input callbacks should enqueue or hand off frame data; render executes the upload.

View File

@@ -102,7 +102,7 @@ This work is currently split across:
- [DeckLinkSession::ConfigureInput](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/videoio/decklink/DeckLinkSession.cpp:221) - [DeckLinkSession::ConfigureInput](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/videoio/decklink/DeckLinkSession.cpp:221)
- [CaptureDelegate::VideoInputFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/videoio/decklink/DeckLinkFrameTransfer.cpp:33) - [CaptureDelegate::VideoInputFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/videoio/decklink/DeckLinkFrameTransfer.cpp:33)
- [OpenGLVideoIOBridge::VideoFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:50) - [OpenGLVideoIOBridge::UploadInputFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11)
### 3. Output Lifecycle and Output Callback Handling ### 3. Output Lifecycle and Output Callback Handling
@@ -307,7 +307,7 @@ Today the callback path effectively does this:
That path is visible in: That path is visible in:
- [OpenGLVideoIOBridge::PlayoutFrameCompleted](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:83) - [OpenGLVideoIOBridge::RenderScheduledFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:18)
This couples output timing directly to render work. This couples output timing directly to render work.
@@ -354,7 +354,7 @@ Recommended default policy for live playout:
The current "skip upload if the GL bridge is busy" behavior is directionally correct for live timing: The current "skip upload if the GL bridge is busy" behavior is directionally correct for live timing:
- [OpenGLVideoIOBridge::VideoFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:57) - [OpenGLVideoIOBridge::UploadInputFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11)
But in the target architecture that decision should move out of GL lock acquisition and into an explicit backend-to-render handoff queue policy. But in the target architecture that decision should move out of GL lock acquisition and into an explicit backend-to-render handoff queue policy.