docs update
This commit is contained in:
@@ -7,6 +7,7 @@ Phase checklist:
|
||||
- [x] Define subsystem boundaries and target architecture
|
||||
- [x] Introduce an internal event model
|
||||
- [x] Split `RuntimeHost`
|
||||
- [x] Finish live-state and service-facing coordination
|
||||
- [ ] Make the render thread the sole GL owner
|
||||
- [ ] Refactor live state layering into an explicit composition model
|
||||
- [ ] Move persistence onto a background snapshot writer
|
||||
@@ -17,7 +18,8 @@ Checklist note:
|
||||
|
||||
- The checked Phase 1 item means the subsystem vocabulary, dependency direction, state categories, design package, and runtime implementation foothold are in place.
|
||||
- The checked Phase 2 item means the internal event model substrate is complete enough for later phases: the typed event vocabulary, app-owned dispatcher, coalesced event pump, reload bridge events, production bridges, and pure event tests are in place. Remaining items in [PHASE_2_INTERNAL_EVENT_MODEL_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_2_INTERNAL_EVENT_MODEL_DESIGN.md) are narrow follow-ups, mainly completion/failure observations and later replacement of the runtime-store poll fallback with real file-watch events.
|
||||
- It does not mean the whole app is fully extracted. Sole-owner render threading, live-state layering, background persistence, backend lifecycle, and richer telemetry continue through later phases.
|
||||
- The checked Phase 3 item means the render-facing state path now has named live-state, composition, frame-state, resolver, and service-bridge boundaries. `OpenGLComposite::renderEffect()` is reduced to runtime work, frame input construction, and frame rendering. This prepares Phase 4 but does not yet move GL work onto a dedicated render thread.
|
||||
- It does not mean the whole app is fully extracted. Sole-owner render threading, deeper live-state layering, background persistence, backend lifecycle, and richer telemetry continue through later phases.
|
||||
|
||||
## Timing Review
|
||||
|
||||
@@ -177,7 +179,7 @@ Relevant timing code:
|
||||
|
||||
Why this matters:
|
||||
|
||||
- `PlayoutFrameCompleted()` currently begins an output frame, takes the shared GL path, renders, reads back, and schedules the next frame in one callback-driven flow.
|
||||
- the output completion path currently requests a scheduled render through `OpenGLVideoIOBridge::RenderScheduledFrame()`, which still takes the shared GL path, renders, reads back, and schedules the next frame in one callback-driven flow.
|
||||
- `VideoPlayoutScheduler::AccountForCompletionResult()` currently reacts to both late and dropped frames by blindly advancing the schedule index by `2`, which is simple but not especially robust.
|
||||
- `kPrerollFrameCount` is now `12`, but `DeckLinkSession::ConfigureOutput()` still creates a fixed pool of `10` mutable output frames. That mismatch suggests the buffering model is not being sized from one coherent source of truth.
|
||||
|
||||
@@ -496,6 +498,10 @@ Primary design rule:
|
||||
|
||||
With state and coordination cleaner, move to a dedicated render-thread model.
|
||||
|
||||
Dedicated design note:
|
||||
|
||||
- [PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md)
|
||||
|
||||
Target behavior:
|
||||
|
||||
- one thread owns the GL context
|
||||
|
||||
@@ -6,7 +6,7 @@ Phase 1 split runtime responsibilities into named subsystems. Phase 2 added the
|
||||
|
||||
## Status
|
||||
|
||||
- Phase 3 design package: proposed.
|
||||
- Phase 3 design package: accepted.
|
||||
- Phase 3 implementation: exit criteria satisfied for the current architecture.
|
||||
- Current alignment: the repo now has the live-state/composer building blocks, a service bridge, and a named frame-state handoff. `OpenGLComposite::renderEffect()` still remains the app-level frame entrypoint, but the service drain, layer-state resolution, and OSC commit handoff now sit behind named helpers and frame-state data.
|
||||
|
||||
@@ -24,17 +24,17 @@ Current footholds:
|
||||
- `RuntimeServiceLiveBridge` translates service OSC queues into render live-state updates and queues settled overlay commit requests.
|
||||
- `RuntimeEventDispatcher` now routes accepted mutations, reloads, snapshots, shader build events, backend observations, and health observations.
|
||||
|
||||
The current architecture is much better than the original `RuntimeHost` shape, but the render path still has too much coordination logic stitched through `OpenGLComposite`, `RuntimeServices`, `RuntimeCoordinator`, and `RenderEngine`.
|
||||
The current architecture is much better than the original `RuntimeHost` shape. The remaining render risk is now mostly Phase 4 work: GL calls are still reached from callback and UI paths through a shared context lock rather than through one render-thread owner.
|
||||
|
||||
## Why Phase 3 Exists
|
||||
|
||||
The resilience review says render-thread isolation should come after state access and control coordination are no longer centered on a large mutable runtime object. Phase 2 gives us the event substrate; Phase 3 should make the data flowing into render explicit enough that Phase 4 can make the render thread the sole GL owner without dragging service coordination and state reconciliation with it.
|
||||
|
||||
The main problems Phase 3 addresses:
|
||||
The main problems Phase 3 addressed:
|
||||
|
||||
- transient OSC overlay state and persisted committed state are still reconciled by hand in `OpenGLComposite::renderEffect()`
|
||||
- `RenderEngine` both stores transient OSC overlay state and resolves final layer states
|
||||
- `ControlServices` exposes service-side queues for pending OSC updates and completed OSC commits
|
||||
- transient OSC overlay state and persisted committed state needed a named reconciliation boundary
|
||||
- `RenderEngine` needed to move final frame-state selection and value composition out of drawing code
|
||||
- service-side queues for pending OSC updates and completed OSC commits needed a bridge outside `OpenGLComposite`
|
||||
- `RuntimeStore` still performs synchronous persistence directly from many state mutation paths
|
||||
- `RuntimeUpdateController` still exists partly as compatibility glue between synchronous coordinator results and event-driven effects
|
||||
|
||||
@@ -64,11 +64,11 @@ Those are later phases. Phase 3 is about making state and service coordination c
|
||||
|
||||
## Current Coordination Shape
|
||||
|
||||
`OpenGLComposite::renderEffect()` is now narrower, but it still owns the frame entrypoint:
|
||||
`OpenGLComposite::renderEffect()` is now the app-level frame entrypoint, but it is intentionally narrow:
|
||||
|
||||
1. pumps `RuntimeUpdateController::ProcessRuntimeWork()`
|
||||
2. asks `RuntimeServiceLiveBridge` to prepare live render layer states
|
||||
3. asks `RenderEngine` to draw the layer stack
|
||||
2. builds a `RenderFrameInput`
|
||||
3. renders the frame through `RuntimeServiceLiveBridge`, `RenderFrameStateResolver`, and `RenderEngine`
|
||||
|
||||
The bridge now owns service queue draining, live automation settlement, committed/live state selection, and OSC commit handoff. `RenderFrameStateResolver` owns snapshot cache selection, parameter refresh decisions, and dynamic render-field refresh before handing a prepared frame state to `RenderEngine`.
|
||||
|
||||
@@ -90,11 +90,11 @@ final render state = published snapshot + committed live selection + transient o
|
||||
|
||||
The important change is not the exact formula name. The important change is that final render-state composition has one named owner and can be tested without GL.
|
||||
|
||||
## Proposed Collaborators
|
||||
## Phase 3 Collaborators
|
||||
|
||||
### `RuntimeLiveState`
|
||||
|
||||
New small runtime collaborator or equivalent read model builder.
|
||||
Small runtime collaborator for transient automation state.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
@@ -113,7 +113,7 @@ Non-responsibilities:
|
||||
|
||||
### `RenderStateComposer`
|
||||
|
||||
New pure or mostly pure collaborator.
|
||||
Pure or mostly pure collaborator for frame value composition.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
@@ -250,7 +250,7 @@ Introduce `RuntimeLiveState`, `RenderStateComposer`, or an equivalent pair of cl
|
||||
|
||||
Start by moving pure data operations out of frame rendering without changing behavior.
|
||||
|
||||
Status: started. `runtime/live/RuntimeLiveState` and `runtime/live/RenderStateComposer` now exist, are included in the build, and have a focused `RuntimeLiveStateTests` target.
|
||||
Status: complete for Phase 3. `runtime/live/RuntimeLiveState` and `runtime/live/RenderStateComposer` exist, are included in the build, and have a focused `RuntimeLiveStateTests` target.
|
||||
|
||||
### Step 2. Move OSC Overlay Bookkeeping Behind The Boundary
|
||||
|
||||
@@ -263,7 +263,7 @@ Move these responsibilities out of the current frame orchestration:
|
||||
|
||||
The first implementation can still be called synchronously from the current render path. The important part is that the behavior has a named owner and tests.
|
||||
|
||||
Status: mostly complete for the current architecture. `RenderEngine` still exposes compatibility methods used by the service bridge, but it now delegates overlay updates, commit completions, smoothing, generation matching, and commit-request creation to `RuntimeLiveState`/`RenderStateComposer`.
|
||||
Status: complete for Phase 3. `RenderEngine` still exposes compatibility methods used by the service bridge, but it delegates overlay updates, commit completions, smoothing, generation matching, and commit-request creation to `RuntimeLiveState`/`RenderStateComposer`.
|
||||
|
||||
### Step 3. Bridge Service Queues To Events Or Live-State Commands
|
||||
|
||||
@@ -276,7 +276,7 @@ Replace `OpenGLComposite::renderEffect()` queue draining with a bridge that publ
|
||||
|
||||
This is where the remaining Phase 2 open question about transient OSC overlay event scope should be resolved for the current architecture.
|
||||
|
||||
Status: started. `RuntimeServiceLiveBridge` now drains pending OSC updates and completed OSC commits, applies them to render live state, and queues settled commit requests. It is still a source-local bridge rather than a fully dispatcher-driven event bridge.
|
||||
Status: complete for Phase 3. `RuntimeServiceLiveBridge` now drains pending OSC updates and completed OSC commits, applies them to render live state, and queues settled commit requests. It remains a source-local bridge by design until later live-state layering decides whether transient automation should enter the app-level dispatcher.
|
||||
|
||||
### Step 4. Narrow `OpenGLComposite::renderEffect()`
|
||||
|
||||
@@ -293,7 +293,7 @@ void OpenGLComposite::renderEffect()
|
||||
|
||||
The exact names can change. The goal is that render effect no longer manually drains services, settles overlay commits, and resolves layer values.
|
||||
|
||||
Status: started. `OpenGLComposite::renderEffect()` still drives frame timing, video dimensions, and drawing, but the service-drain, resolve, and commit-handoff path has moved behind `RuntimeServiceLiveBridge::PrepareLiveRenderFrameState(...)` and a named `RenderFrameInput` / `RenderFrameState` handoff.
|
||||
Status: complete for Phase 3. `OpenGLComposite::renderEffect()` now processes runtime work, builds `RenderFrameInput`, and calls a narrow frame-render helper. Service draining, state resolution, and commit handoff sit behind `RuntimeServiceLiveBridge::PrepareLiveRenderFrameState(...)`, `RenderFrameStateResolver`, and `RenderFrameState`.
|
||||
|
||||
### Step 5. Add Persistence Boundary Tests
|
||||
|
||||
@@ -304,6 +304,8 @@ Add behavior tests for:
|
||||
- preset load/save persistence requests remain explicit
|
||||
- rejected mutations do not publish persistence work
|
||||
|
||||
Status: complete for Phase 3. `RuntimeSubsystemTests` and `RuntimeEventTypeTests` cover accepted mutation persistence requests, rejected mutations, and transient OSC overlay behavior that does not request persistence.
|
||||
|
||||
### Step 6. Update Docs And Phase 4 Readiness
|
||||
|
||||
Before calling Phase 3 complete, update:
|
||||
@@ -312,6 +314,8 @@ Before calling Phase 3 complete, update:
|
||||
- architecture review checklist
|
||||
- Phase 4 assumptions about render thread input state
|
||||
|
||||
Status: complete. The Phase 4 design note starts from the `RenderFrameInput` / `RenderFrameState` contract and the remaining shared-GL ownership paths.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Phase 3 tests should avoid GL, DeckLink, and sockets.
|
||||
@@ -340,11 +344,11 @@ The current groundwork is intended to let these lanes proceed in parallel with l
|
||||
|
||||
| Lane | Primary files | Goal |
|
||||
| --- | --- | --- |
|
||||
| A. Live-state behavior | `runtime/live/RuntimeLiveState.*`, `tests/RuntimeLiveStateTests.cpp` | Finish stale completion tests, smoothing edge cases, trigger behavior, and overlay settle policy. |
|
||||
| B. Render-state composition | `runtime/live/RenderStateComposer.*`, `gl/RenderFrameStateResolver.*`, `gl/RenderEngine.*` | Keep value composition and frame-state selection outside GL drawing while keeping GL calls in `RenderEngine`. |
|
||||
| C. Service bridge | `control/RuntimeServices.*`, `control/ControlServices.*`, possible new bridge class | Stop `OpenGLComposite::renderEffect()` from draining OSC update/completion queues directly. |
|
||||
| D. App-frame orchestration | `gl/OpenGLComposite.*`, `gl/RuntimeUpdateController.*` | Replace render-effect glue with a narrow frame-state preparation call and commit-request handoff. |
|
||||
| E. Persistence boundary | `runtime/coordination/RuntimeCoordinator.*`, `runtime/store/*`, event tests | Keep persistence request publication explicit and prepare for a later background writer without changing storage behavior yet. |
|
||||
| A. Live-state behavior | `runtime/live/RuntimeLiveState.*`, `tests/RuntimeLiveStateTests.cpp` | Implemented for Phase 3: stale completion, smoothing, trigger behavior, and overlay settle policy are covered by focused tests. |
|
||||
| B. Render-state composition | `runtime/live/RenderStateComposer.*`, `gl/RenderFrameStateResolver.*`, `gl/RenderEngine.*` | Implemented for Phase 3: value composition and frame-state selection sit outside GL drawing while GL calls remain in `RenderEngine`. |
|
||||
| C. Service bridge | `control/RuntimeServices.*`, `control/RuntimeServiceLiveBridge.*`, `control/ControlServices.*` | Implemented for Phase 3: `OpenGLComposite::renderEffect()` no longer drains OSC update/completion queues directly. |
|
||||
| D. App-frame orchestration | `gl/OpenGLComposite.*`, `gl/RuntimeUpdateController.*` | Implemented for Phase 3: render-effect glue is a narrow runtime-work, frame-input, render-frame sequence. |
|
||||
| E. Persistence boundary | `runtime/coordination/RuntimeCoordinator.*`, `runtime/store/*`, event tests | Implemented for Phase 3: persistence request publication is explicit and ready for a later background writer. |
|
||||
|
||||
## Phase 3 Exit Criteria
|
||||
|
||||
|
||||
373
docs/PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md
Normal file
373
docs/PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md
Normal file
@@ -0,0 +1,373 @@
|
||||
# Phase 4 Design: Render Thread Ownership
|
||||
|
||||
This document expands Phase 4 of [ARCHITECTURE_RESILIENCE_REVIEW.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/ARCHITECTURE_RESILIENCE_REVIEW.md) into a concrete design target.
|
||||
|
||||
Phase 1 named the subsystems. Phase 2 added the typed event substrate. Phase 3 made render-facing live state explicit through `RuntimeLiveState`, `RenderStateComposer`, `RenderFrameInput`, `RenderFrameState`, `RenderFrameStateResolver`, and `RuntimeServiceLiveBridge`. Phase 4 can now focus on the core timing-risk boundary: making one render thread the only owner of OpenGL work.
|
||||
|
||||
## Status
|
||||
|
||||
- Phase 4 design package: proposed.
|
||||
- Phase 4 implementation: not started.
|
||||
- Current alignment: the repo has a named frame-state contract and cleaner render-state preparation, but GL work is still entered through multiple paths protected by one shared `CRITICAL_SECTION`.
|
||||
|
||||
Current GL ownership footholds:
|
||||
|
||||
- `RenderEngine` owns GL resources and the current context-binding helpers.
|
||||
- `RenderFrameInput` / `RenderFrameState` provide the frame-state contract that a render thread can consume.
|
||||
- `RenderFrameStateResolver` prepares the render-facing layer state before drawing.
|
||||
- `OpenGLVideoIOBridge` still calls `RenderEngine::TryUploadInputFrame(...)` from the input path and `RenderEngine::RenderOutputFrame(...)` from the output path.
|
||||
- `OpenGLComposite::paintGL(...)`, screenshot capture, input upload, and output rendering still reach GL through `RenderEngine` methods that bind the shared context under `pMutex`.
|
||||
|
||||
## Why Phase 4 Exists
|
||||
|
||||
The resilience review identifies shared GL ownership as the main remaining timing and failure-isolation risk. Today the shared context lock protects correctness, but it does not isolate timing:
|
||||
|
||||
- input callbacks can attempt texture upload
|
||||
- output callbacks can trigger frame rendering and readback
|
||||
- preview paint can enter the same GL context
|
||||
- screenshot capture can enter the same GL context
|
||||
- the DeckLink completion path is still too close to render work
|
||||
|
||||
That means brief input, preview, readback, or callback stalls can still collide on the most timing-sensitive path.
|
||||
|
||||
Phase 4 should turn GL from a shared resource guarded by a lock into a resource owned by one thread with explicit queues and handoff points.
|
||||
|
||||
## Goals
|
||||
|
||||
Phase 4 should establish:
|
||||
|
||||
- one render thread as the sole long-lived owner of the GL context
|
||||
- non-render threads enqueue work instead of binding the GL context
|
||||
- input upload requests are accepted and executed by the render thread
|
||||
- output frame rendering is requested or scheduled through render-owned work
|
||||
- preview and screenshot requests become render-thread commands or consumers
|
||||
- `RenderFrameInput` / `RenderFrameState` become the stable data contract for frame production
|
||||
- GL context entrypoints are reduced to render-thread-only code paths
|
||||
- tests for queue semantics, request coalescing, and lifecycle behavior without requiring DeckLink hardware
|
||||
|
||||
## Non-Goals
|
||||
|
||||
Phase 4 should not require:
|
||||
|
||||
- the final producer/consumer playout queue for DeckLink
|
||||
- the final DeckLink lifecycle state machine
|
||||
- replacing the async readback policy
|
||||
- implementing background persistence
|
||||
- completing Phase 5's deeper live-state layering
|
||||
- replacing every UI or backend API at once
|
||||
|
||||
Those are later phases or follow-on work. Phase 4 is about making GL ownership deterministic first.
|
||||
|
||||
## Current GL Entry Points
|
||||
|
||||
The current code paths that matter most are:
|
||||
|
||||
| Entry point | Current behavior | Phase 4 direction |
|
||||
| --- | --- | --- |
|
||||
| `RenderEngine::TryUploadInputFrame(...)` | attempts to take the GL lock, binds the context, uploads input texture | enqueue latest input frame; render thread uploads |
|
||||
| `RenderEngine::RenderOutputFrame(...)` | takes the GL lock, binds the context, renders, packs/readbacks output | render thread executes output frame production |
|
||||
| `RenderEngine::TryPresentPreview(...)` | attempts to take the GL lock and presents preview | render thread or preview presenter consumes latest completed frame |
|
||||
| `RenderEngine::CaptureOutputFrameRgbaTopDown(...)` | takes the GL lock and reads output pixels | screenshot request becomes render-thread command |
|
||||
| `OpenGLVideoIOBridge::UploadInputFrame(...)` | calls render upload directly | push input frame into render queue/mailbox |
|
||||
| `OpenGLVideoIOBridge::RenderScheduledFrame(...)` | calls render output directly from backend path | request/consume render-produced output without callback-owned GL |
|
||||
|
||||
## Target Ownership Model
|
||||
|
||||
### Render Thread
|
||||
|
||||
The render thread should own:
|
||||
|
||||
- `wglMakeCurrent(...)` for the rendering context
|
||||
- all GL resource creation/destruction
|
||||
- input texture upload
|
||||
- pass execution
|
||||
- output pack conversion
|
||||
- async readback buffers and fences
|
||||
- preview presentation or preview frame publication
|
||||
- screenshot readback
|
||||
- temporal history and feedback resources
|
||||
|
||||
### Other Threads
|
||||
|
||||
Other threads may:
|
||||
|
||||
- enqueue input frames or replace the latest input frame
|
||||
- publish control/runtime/backend events
|
||||
- request shader build application
|
||||
- request render-local resets
|
||||
- request screenshots
|
||||
- consume ready output frames or receive completion notifications
|
||||
|
||||
Other threads should not:
|
||||
|
||||
- call GL directly
|
||||
- bind or unbind the render context
|
||||
- wait on GL fences directly
|
||||
- mutate render-local resource state
|
||||
|
||||
## Proposed Collaborators
|
||||
|
||||
### `RenderThread`
|
||||
|
||||
Owns the OS thread, wakeup primitive, lifecycle, and render-loop execution.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- start and stop the render thread
|
||||
- bind the GL context for the thread lifetime or render-loop lifetime
|
||||
- drain render commands
|
||||
- execute frame production work
|
||||
- publish lifecycle and failure observations
|
||||
|
||||
Non-responsibilities:
|
||||
|
||||
- runtime mutation policy
|
||||
- DeckLink scheduling policy
|
||||
- durable persistence
|
||||
|
||||
### `RenderCommandQueue`
|
||||
|
||||
Small bounded queue or command mailbox for render-thread work.
|
||||
|
||||
Possible commands:
|
||||
|
||||
- `UploadInputFrame`
|
||||
- `RenderOutputFrame`
|
||||
- `PrepareFrameState`
|
||||
- `ApplyShaderBuild`
|
||||
- `ResetTemporalHistory`
|
||||
- `ResetShaderFeedback`
|
||||
- `PresentPreview`
|
||||
- `CaptureScreenshot`
|
||||
- `Stop`
|
||||
|
||||
High-rate commands should be coalesced where appropriate. Input frames should likely be latest-value rather than unbounded FIFO.
|
||||
|
||||
### `RenderFrameCoordinator`
|
||||
|
||||
Optional helper that combines Phase 3's frame contract with render-thread execution.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- build or receive `RenderFrameInput`
|
||||
- call `RuntimeServiceLiveBridge` and `RenderFrameStateResolver`
|
||||
- hand `RenderFrameState` to `RenderEngine`
|
||||
|
||||
This can begin as a thin helper. The important part is that it keeps frame-state preparation explicit when `renderEffect()` stops being called directly from the callback path.
|
||||
|
||||
### `RenderOutputMailbox`
|
||||
|
||||
Optional transitional bridge for output frames.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- hold the latest completed output frame or a small bounded set
|
||||
- let backend code consume output without owning GL
|
||||
- report underrun/stale-frame reuse observations
|
||||
|
||||
This may be a Phase 4 late step or a Phase 7 playout-policy step. Phase 4 should at least avoid designing the render thread in a way that blocks it.
|
||||
|
||||
## Threading Contract
|
||||
|
||||
Phase 4 should make thread ownership visible in APIs.
|
||||
|
||||
Candidate naming:
|
||||
|
||||
- `RenderEngine::StartRenderThread(...)`
|
||||
- `RenderEngine::StopRenderThread()`
|
||||
- `RenderEngine::EnqueueInputFrame(...)`
|
||||
- `RenderEngine::RequestOutputFrame(...)`
|
||||
- `RenderEngine::RequestPreviewPresent(...)`
|
||||
- `RenderEngine::RequestScreenshot(...)`
|
||||
|
||||
Render-thread-only methods should be private or clearly named:
|
||||
|
||||
- `RenderEngine::UploadInputFrameOnRenderThread(...)`
|
||||
- `RenderEngine::RenderOutputFrameOnRenderThread(...)`
|
||||
- `RenderEngine::CaptureOutputFrameOnRenderThread(...)`
|
||||
|
||||
The current `TryUploadInputFrame`, `RenderOutputFrame`, `TryPresentPreview`, and `CaptureOutputFrameRgbaTopDown` methods can remain as compatibility shims during migration, but their implementations should move toward enqueue-and-wait or enqueue-and-return behavior instead of binding GL directly from the caller's thread.
|
||||
|
||||
## Frame Production Shape
|
||||
|
||||
A target render-thread frame should look like:
|
||||
|
||||
1. wake for input, output demand, preview demand, shader build, reset, screenshot, or stop
|
||||
2. drain bounded render commands
|
||||
3. coalesce to the latest input frame and latest control/live state
|
||||
4. build `RenderFrameInput`
|
||||
5. prepare `RenderFrameState`
|
||||
6. upload accepted input frame
|
||||
7. render layer stack
|
||||
8. pack output if needed
|
||||
9. stage readback or output buffer
|
||||
10. publish preview/screenshot/output completion as needed
|
||||
11. record timing and queue metrics
|
||||
|
||||
The exact cadence can remain demand-driven initially. The architectural win is that the demand wakes the render thread rather than borrowing GL from the caller.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Step 1. Name Render-Thread-Only Methods
|
||||
|
||||
Split existing direct GL methods into public request methods and private render-thread methods without changing behavior much.
|
||||
|
||||
Initial target:
|
||||
|
||||
- keep current synchronous behavior where callers need a result
|
||||
- move GL bodies into clearly render-thread-owned helpers
|
||||
- make future queue migration mechanical
|
||||
|
||||
### Step 2. Add Render Command Queue
|
||||
|
||||
Introduce a small queue/mailbox for render commands.
|
||||
|
||||
Start with low-risk commands:
|
||||
|
||||
- preview present request
|
||||
- screenshot request
|
||||
- render-local reset requests
|
||||
|
||||
Then move input upload and output render requests once the queue and wakeup behavior are proven.
|
||||
|
||||
### Step 3. Start A Dedicated Render Thread
|
||||
|
||||
Create the render thread and make it own context binding.
|
||||
|
||||
Transitional behavior may still allow synchronous request/response for output frames. The important change is that the caller waits for render-thread completion rather than taking the GL context itself.
|
||||
|
||||
### Step 4. Move Input Upload To The Render Thread
|
||||
|
||||
Change `OpenGLVideoIOBridge::UploadInputFrame(...)` so it enqueues or replaces the latest input frame.
|
||||
|
||||
Policy targets:
|
||||
|
||||
- bounded memory
|
||||
- latest-frame wins under load
|
||||
- input upload skip count is observable
|
||||
- input callback never waits for GL
|
||||
|
||||
### Step 5. Move Output Rendering To The Render Thread
|
||||
|
||||
Change `OpenGLVideoIOBridge::RenderScheduledFrame(...)` so it requests render-thread output production or consumes a completed render-thread output.
|
||||
|
||||
Transitional option:
|
||||
|
||||
- synchronous request/response through the render thread
|
||||
|
||||
Better follow-up:
|
||||
|
||||
- render ahead into a bounded output queue and let backend callbacks consume ready frames
|
||||
|
||||
### Step 6. Decouple Preview And Screenshot Requests
|
||||
|
||||
Preview should become best-effort:
|
||||
|
||||
- request preview presentation from the render thread
|
||||
- skip when render is busy or output deadline pressure is high
|
||||
- record preview skips
|
||||
|
||||
Screenshot should become:
|
||||
|
||||
- queued render-thread capture request
|
||||
- async disk write remains outside render thread
|
||||
|
||||
### Step 7. Remove Shared GL Lock From Normal Paths
|
||||
|
||||
Once all GL entrypoints are render-thread-owned:
|
||||
|
||||
- remove normal dependence on `pMutex` for render correctness
|
||||
- keep assertions or diagnostics that detect wrong-thread GL calls
|
||||
- leave only lifecycle synchronization where needed
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Phase 4 tests should avoid hardware where possible.
|
||||
|
||||
Recommended tests:
|
||||
|
||||
- render command queue preserves FIFO for non-coalesced commands
|
||||
- latest-input mailbox drops older frames under load
|
||||
- stop command wakes and drains the render thread
|
||||
- screenshot request receives one completion or failure
|
||||
- output render request reports timeout/failure if render thread is stopped
|
||||
- render reset commands coalesce where expected
|
||||
- wrong-thread render-only methods are not publicly reachable
|
||||
|
||||
Existing useful homes:
|
||||
|
||||
- `RuntimeEventTypeTests` for new render/backend observations
|
||||
- `RuntimeSubsystemTests` for pure request/coalescing helpers
|
||||
- a new `RenderThreadTests` target for queue/mailbox/lifecycle helpers that do not require GL
|
||||
|
||||
Manual verification will still be needed for:
|
||||
|
||||
- real DeckLink input/output
|
||||
- preview interaction
|
||||
- screenshot capture
|
||||
- shader reload while rendering
|
||||
|
||||
## Telemetry Added During Phase 4
|
||||
|
||||
Phase 4 should add minimal metrics while moving ownership:
|
||||
|
||||
- render command queue depth
|
||||
- input frames accepted, replaced, and dropped
|
||||
- render-thread wake reason counts
|
||||
- render-thread frame duration
|
||||
- output request latency
|
||||
- preview request skipped count
|
||||
- screenshot request success/failure count
|
||||
- wrong-thread GL call diagnostics if practical
|
||||
|
||||
Full operational reporting remains Phase 8, but these metrics make the threading migration debuggable.
|
||||
|
||||
## Risks
|
||||
|
||||
### Deadlock Risk
|
||||
|
||||
Synchronous request/response shims can deadlock if the caller is already on the render thread or holds a lock the render thread needs. Phase 4 should keep request waits narrow and add render-thread detection early.
|
||||
|
||||
### Latency Risk
|
||||
|
||||
Moving work through queues can hide latency. Queue depth and output request latency should be measured from the first migration step.
|
||||
|
||||
### Lifetime Risk
|
||||
|
||||
Moving context ownership changes startup and shutdown order. The render thread must stop before GL resources or window/context handles are destroyed.
|
||||
|
||||
### Callback Pressure Risk
|
||||
|
||||
If DeckLink callbacks wait too long for render-thread work, Phase 4 may improve GL ownership but still leave callback timing fragile. A synchronous bridge is acceptable as a transition, but the design should keep the path open for producer/consumer playout.
|
||||
|
||||
### Preview Coupling Risk
|
||||
|
||||
Preview can remain a hidden budget consumer if it stays in the output frame path. Phase 4 should keep preview explicitly best-effort, even if physical decoupling continues later.
|
||||
|
||||
## Phase 4 Exit Criteria
|
||||
|
||||
Phase 4 can be considered complete once the project can say:
|
||||
|
||||
- [ ] one render thread owns the GL context during normal operation
|
||||
- [ ] input callbacks do not bind GL or wait on GL upload
|
||||
- [ ] output callbacks do not bind GL directly
|
||||
- [ ] preview and screenshot requests enter render through explicit render-thread requests
|
||||
- [ ] `RenderFrameInput` / `RenderFrameState` remain the frame-state contract
|
||||
- [ ] normal frame production no longer depends on a shared GL `CRITICAL_SECTION`
|
||||
- [ ] render-thread queue/mailbox behavior has non-GL tests
|
||||
- [ ] shutdown order is explicit and tested or manually verified
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should the first output migration be synchronous request/response, or should Phase 4 go directly to a small ready-frame queue?
|
||||
- Should the render thread own `RuntimeServiceLiveBridge` calls, or should frame state be prepared just before enqueue?
|
||||
- How much input frame memory should be copied at enqueue time versus referenced from backend-owned buffers?
|
||||
- Should preview present on the render thread, or should render publish a preview image/texture to a separate presenter?
|
||||
- What timeout should output callbacks use if the render thread cannot produce a frame in time?
|
||||
- Should wrong-thread GL access be enforced with assertions, telemetry, or both?
|
||||
|
||||
## Short Version
|
||||
|
||||
Phase 4 should make GL ownership boring and deterministic.
|
||||
|
||||
One render thread owns the context. Other threads submit work or consume results. Input upload, frame rendering, readback, preview, and screenshot capture all move behind render-thread entrypoints. The first implementation can be transitional and partly synchronous, but after Phase 4 the app should no longer rely on callback and UI paths borrowing the GL context under one shared lock.
|
||||
@@ -39,14 +39,14 @@ The current rendering path is split across several classes:
|
||||
- [OpenGLComposite.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/OpenGLComposite.cpp:86) constructs the renderer, render pipeline, shader programs, runtime services, and video bridge in one owner.
|
||||
- [OpenGLRenderPipeline.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLRenderPipeline.cpp:31) performs pass execution, pack/readback, preview paint, and performance stat publication.
|
||||
- [OpenGLVideoIOBridge.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:58) accepts capture frames and still performs render work from the playout completion callback path.
|
||||
- `OpenGLComposite` still holds render-local overlay behavior and shader rebuild handling alongside runtime orchestration responsibilities.
|
||||
- `RenderFrameStateResolver` and `RenderStateComposer` now keep frame-state selection and live value composition outside GL drawing, while `RenderEngine` still owns the current GL resource and draw path.
|
||||
|
||||
That split is workable today, but it creates architectural pressure:
|
||||
|
||||
- GL ownership is thread-shared instead of sole-owned.
|
||||
- render and playout timing are still callback-coupled.
|
||||
- preview and playout are produced in the same immediate path.
|
||||
- render-local transient state is too easy to leak back into runtime-facing code.
|
||||
- render-local transient state now has clearer Phase 3 boundaries, but GL ownership is still shared through callback and UI entrypoints.
|
||||
- it is difficult to test render behavior separately from app bootstrap and hardware integration.
|
||||
|
||||
`RenderEngine` exists to absorb that responsibility into one subsystem with one direction of ownership.
|
||||
@@ -170,11 +170,12 @@ Other threads should interact with the subsystem through queues, snapshots, and
|
||||
|
||||
## Current State
|
||||
|
||||
Today GL work is still shared across callback-driven entrypoints:
|
||||
Today GL work is still shared across callback-driven and UI entrypoints:
|
||||
|
||||
- input upload occurs in [OpenGLVideoIOBridge::VideoFrameArrived()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:58)
|
||||
- playout-triggered render occurs in [OpenGLVideoIOBridge::PlayoutFrameCompleted()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:95)
|
||||
- input upload is requested through [OpenGLVideoIOBridge::UploadInputFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11)
|
||||
- playout-triggered render is requested through [OpenGLVideoIOBridge::RenderScheduledFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:18)
|
||||
- render-pass execution occurs in [OpenGLRenderPipeline::RenderFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLRenderPipeline.cpp:31)
|
||||
- preview and screenshot paths still enter `RenderEngine` methods that bind the shared context
|
||||
|
||||
The `CRITICAL_SECTION` protects correctness, but it is not the target architectural model.
|
||||
|
||||
@@ -406,6 +407,8 @@ inside render-owned code paths instead of putting them back into runtime storage
|
||||
|
||||
Introduce snapshot-facing APIs so render no longer depends on broad runtime-state access for frame production.
|
||||
|
||||
Current status: Phase 3 introduced `RenderFrameInput`, `RenderFrameState`, and `RenderFrameStateResolver`, so frame-state selection is named and no longer lives inside GL drawing. Phase 4 can build on that contract while moving GL ownership.
|
||||
|
||||
### Step 4. Move Uploads Onto Render Ownership
|
||||
|
||||
Input callbacks should enqueue or hand off frame data; render executes the upload.
|
||||
|
||||
@@ -102,7 +102,7 @@ This work is currently split across:
|
||||
|
||||
- [DeckLinkSession::ConfigureInput](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/videoio/decklink/DeckLinkSession.cpp:221)
|
||||
- [CaptureDelegate::VideoInputFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/videoio/decklink/DeckLinkFrameTransfer.cpp:33)
|
||||
- [OpenGLVideoIOBridge::VideoFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:50)
|
||||
- [OpenGLVideoIOBridge::UploadInputFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11)
|
||||
|
||||
### 3. Output Lifecycle and Output Callback Handling
|
||||
|
||||
@@ -307,7 +307,7 @@ Today the callback path effectively does this:
|
||||
|
||||
That path is visible in:
|
||||
|
||||
- [OpenGLVideoIOBridge::PlayoutFrameCompleted](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:83)
|
||||
- [OpenGLVideoIOBridge::RenderScheduledFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:18)
|
||||
|
||||
This couples output timing directly to render work.
|
||||
|
||||
@@ -354,7 +354,7 @@ Recommended default policy for live playout:
|
||||
|
||||
The current "skip upload if the GL bridge is busy" behavior is directionally correct for live timing:
|
||||
|
||||
- [OpenGLVideoIOBridge::VideoFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:57)
|
||||
- [OpenGLVideoIOBridge::UploadInputFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11)
|
||||
|
||||
But in the target architecture that decision should move out of GL lock acquisition and into an explicit backend-to-render handoff queue policy.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user