diff --git a/docs/ARCHITECTURE_RESILIENCE_REVIEW.md b/docs/ARCHITECTURE_RESILIENCE_REVIEW.md index fde5224..5fc28ae 100644 --- a/docs/ARCHITECTURE_RESILIENCE_REVIEW.md +++ b/docs/ARCHITECTURE_RESILIENCE_REVIEW.md @@ -7,6 +7,7 @@ Phase checklist: - [x] Define subsystem boundaries and target architecture - [x] Introduce an internal event model - [x] Split `RuntimeHost` +- [x] Finish live-state and service-facing coordination - [ ] Make the render thread the sole GL owner - [ ] Refactor live state layering into an explicit composition model - [ ] Move persistence onto a background snapshot writer @@ -17,7 +18,8 @@ Checklist note: - The checked Phase 1 item means the subsystem vocabulary, dependency direction, state categories, design package, and runtime implementation foothold are in place. - The checked Phase 2 item means the internal event model substrate is complete enough for later phases: the typed event vocabulary, app-owned dispatcher, coalesced event pump, reload bridge events, production bridges, and pure event tests are in place. Remaining items in [PHASE_2_INTERNAL_EVENT_MODEL_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_2_INTERNAL_EVENT_MODEL_DESIGN.md) are narrow follow-ups, mainly completion/failure observations and later replacement of the runtime-store poll fallback with real file-watch events. -- It does not mean the whole app is fully extracted. Sole-owner render threading, live-state layering, background persistence, backend lifecycle, and richer telemetry continue through later phases. +- The checked Phase 3 item means the render-facing state path now has named live-state, composition, frame-state, resolver, and service-bridge boundaries. `OpenGLComposite::renderEffect()` is reduced to runtime work, frame input construction, and frame rendering. This prepares Phase 4 but does not yet move GL work onto a dedicated render thread. +- It does not mean the whole app is fully extracted. Sole-owner render threading, deeper live-state layering, background persistence, backend lifecycle, and richer telemetry continue through later phases. ## Timing Review @@ -177,7 +179,7 @@ Relevant timing code: Why this matters: -- `PlayoutFrameCompleted()` currently begins an output frame, takes the shared GL path, renders, reads back, and schedules the next frame in one callback-driven flow. +- the output completion path currently requests a scheduled render through `OpenGLVideoIOBridge::RenderScheduledFrame()`, which still takes the shared GL path, renders, reads back, and schedules the next frame in one callback-driven flow. - `VideoPlayoutScheduler::AccountForCompletionResult()` currently reacts to both late and dropped frames by blindly advancing the schedule index by `2`, which is simple but not especially robust. - `kPrerollFrameCount` is now `12`, but `DeckLinkSession::ConfigureOutput()` still creates a fixed pool of `10` mutable output frames. That mismatch suggests the buffering model is not being sized from one coherent source of truth. @@ -496,6 +498,10 @@ Primary design rule: With state and coordination cleaner, move to a dedicated render-thread model. +Dedicated design note: + +- [PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md) + Target behavior: - one thread owns the GL context diff --git a/docs/PHASE_3_LIVE_STATE_SERVICE_COORDINATION_DESIGN.md b/docs/PHASE_3_LIVE_STATE_SERVICE_COORDINATION_DESIGN.md index 2914ac2..caa5d97 100644 --- a/docs/PHASE_3_LIVE_STATE_SERVICE_COORDINATION_DESIGN.md +++ b/docs/PHASE_3_LIVE_STATE_SERVICE_COORDINATION_DESIGN.md @@ -6,7 +6,7 @@ Phase 1 split runtime responsibilities into named subsystems. Phase 2 added the ## Status -- Phase 3 design package: proposed. +- Phase 3 design package: accepted. - Phase 3 implementation: exit criteria satisfied for the current architecture. - Current alignment: the repo now has the live-state/composer building blocks, a service bridge, and a named frame-state handoff. `OpenGLComposite::renderEffect()` still remains the app-level frame entrypoint, but the service drain, layer-state resolution, and OSC commit handoff now sit behind named helpers and frame-state data. @@ -24,17 +24,17 @@ Current footholds: - `RuntimeServiceLiveBridge` translates service OSC queues into render live-state updates and queues settled overlay commit requests. - `RuntimeEventDispatcher` now routes accepted mutations, reloads, snapshots, shader build events, backend observations, and health observations. -The current architecture is much better than the original `RuntimeHost` shape, but the render path still has too much coordination logic stitched through `OpenGLComposite`, `RuntimeServices`, `RuntimeCoordinator`, and `RenderEngine`. +The current architecture is much better than the original `RuntimeHost` shape. The remaining render risk is now mostly Phase 4 work: GL calls are still reached from callback and UI paths through a shared context lock rather than through one render-thread owner. ## Why Phase 3 Exists The resilience review says render-thread isolation should come after state access and control coordination are no longer centered on a large mutable runtime object. Phase 2 gives us the event substrate; Phase 3 should make the data flowing into render explicit enough that Phase 4 can make the render thread the sole GL owner without dragging service coordination and state reconciliation with it. -The main problems Phase 3 addresses: +The main problems Phase 3 addressed: -- transient OSC overlay state and persisted committed state are still reconciled by hand in `OpenGLComposite::renderEffect()` -- `RenderEngine` both stores transient OSC overlay state and resolves final layer states -- `ControlServices` exposes service-side queues for pending OSC updates and completed OSC commits +- transient OSC overlay state and persisted committed state needed a named reconciliation boundary +- `RenderEngine` needed to move final frame-state selection and value composition out of drawing code +- service-side queues for pending OSC updates and completed OSC commits needed a bridge outside `OpenGLComposite` - `RuntimeStore` still performs synchronous persistence directly from many state mutation paths - `RuntimeUpdateController` still exists partly as compatibility glue between synchronous coordinator results and event-driven effects @@ -64,11 +64,11 @@ Those are later phases. Phase 3 is about making state and service coordination c ## Current Coordination Shape -`OpenGLComposite::renderEffect()` is now narrower, but it still owns the frame entrypoint: +`OpenGLComposite::renderEffect()` is now the app-level frame entrypoint, but it is intentionally narrow: 1. pumps `RuntimeUpdateController::ProcessRuntimeWork()` -2. asks `RuntimeServiceLiveBridge` to prepare live render layer states -3. asks `RenderEngine` to draw the layer stack +2. builds a `RenderFrameInput` +3. renders the frame through `RuntimeServiceLiveBridge`, `RenderFrameStateResolver`, and `RenderEngine` The bridge now owns service queue draining, live automation settlement, committed/live state selection, and OSC commit handoff. `RenderFrameStateResolver` owns snapshot cache selection, parameter refresh decisions, and dynamic render-field refresh before handing a prepared frame state to `RenderEngine`. @@ -90,11 +90,11 @@ final render state = published snapshot + committed live selection + transient o The important change is not the exact formula name. The important change is that final render-state composition has one named owner and can be tested without GL. -## Proposed Collaborators +## Phase 3 Collaborators ### `RuntimeLiveState` -New small runtime collaborator or equivalent read model builder. +Small runtime collaborator for transient automation state. Responsibilities: @@ -113,7 +113,7 @@ Non-responsibilities: ### `RenderStateComposer` -New pure or mostly pure collaborator. +Pure or mostly pure collaborator for frame value composition. Responsibilities: @@ -250,7 +250,7 @@ Introduce `RuntimeLiveState`, `RenderStateComposer`, or an equivalent pair of cl Start by moving pure data operations out of frame rendering without changing behavior. -Status: started. `runtime/live/RuntimeLiveState` and `runtime/live/RenderStateComposer` now exist, are included in the build, and have a focused `RuntimeLiveStateTests` target. +Status: complete for Phase 3. `runtime/live/RuntimeLiveState` and `runtime/live/RenderStateComposer` exist, are included in the build, and have a focused `RuntimeLiveStateTests` target. ### Step 2. Move OSC Overlay Bookkeeping Behind The Boundary @@ -263,7 +263,7 @@ Move these responsibilities out of the current frame orchestration: The first implementation can still be called synchronously from the current render path. The important part is that the behavior has a named owner and tests. -Status: mostly complete for the current architecture. `RenderEngine` still exposes compatibility methods used by the service bridge, but it now delegates overlay updates, commit completions, smoothing, generation matching, and commit-request creation to `RuntimeLiveState`/`RenderStateComposer`. +Status: complete for Phase 3. `RenderEngine` still exposes compatibility methods used by the service bridge, but it delegates overlay updates, commit completions, smoothing, generation matching, and commit-request creation to `RuntimeLiveState`/`RenderStateComposer`. ### Step 3. Bridge Service Queues To Events Or Live-State Commands @@ -276,7 +276,7 @@ Replace `OpenGLComposite::renderEffect()` queue draining with a bridge that publ This is where the remaining Phase 2 open question about transient OSC overlay event scope should be resolved for the current architecture. -Status: started. `RuntimeServiceLiveBridge` now drains pending OSC updates and completed OSC commits, applies them to render live state, and queues settled commit requests. It is still a source-local bridge rather than a fully dispatcher-driven event bridge. +Status: complete for Phase 3. `RuntimeServiceLiveBridge` now drains pending OSC updates and completed OSC commits, applies them to render live state, and queues settled commit requests. It remains a source-local bridge by design until later live-state layering decides whether transient automation should enter the app-level dispatcher. ### Step 4. Narrow `OpenGLComposite::renderEffect()` @@ -293,7 +293,7 @@ void OpenGLComposite::renderEffect() The exact names can change. The goal is that render effect no longer manually drains services, settles overlay commits, and resolves layer values. -Status: started. `OpenGLComposite::renderEffect()` still drives frame timing, video dimensions, and drawing, but the service-drain, resolve, and commit-handoff path has moved behind `RuntimeServiceLiveBridge::PrepareLiveRenderFrameState(...)` and a named `RenderFrameInput` / `RenderFrameState` handoff. +Status: complete for Phase 3. `OpenGLComposite::renderEffect()` now processes runtime work, builds `RenderFrameInput`, and calls a narrow frame-render helper. Service draining, state resolution, and commit handoff sit behind `RuntimeServiceLiveBridge::PrepareLiveRenderFrameState(...)`, `RenderFrameStateResolver`, and `RenderFrameState`. ### Step 5. Add Persistence Boundary Tests @@ -304,6 +304,8 @@ Add behavior tests for: - preset load/save persistence requests remain explicit - rejected mutations do not publish persistence work +Status: complete for Phase 3. `RuntimeSubsystemTests` and `RuntimeEventTypeTests` cover accepted mutation persistence requests, rejected mutations, and transient OSC overlay behavior that does not request persistence. + ### Step 6. Update Docs And Phase 4 Readiness Before calling Phase 3 complete, update: @@ -312,6 +314,8 @@ Before calling Phase 3 complete, update: - architecture review checklist - Phase 4 assumptions about render thread input state +Status: complete. The Phase 4 design note starts from the `RenderFrameInput` / `RenderFrameState` contract and the remaining shared-GL ownership paths. + ## Testing Strategy Phase 3 tests should avoid GL, DeckLink, and sockets. @@ -340,11 +344,11 @@ The current groundwork is intended to let these lanes proceed in parallel with l | Lane | Primary files | Goal | | --- | --- | --- | -| A. Live-state behavior | `runtime/live/RuntimeLiveState.*`, `tests/RuntimeLiveStateTests.cpp` | Finish stale completion tests, smoothing edge cases, trigger behavior, and overlay settle policy. | -| B. Render-state composition | `runtime/live/RenderStateComposer.*`, `gl/RenderFrameStateResolver.*`, `gl/RenderEngine.*` | Keep value composition and frame-state selection outside GL drawing while keeping GL calls in `RenderEngine`. | -| C. Service bridge | `control/RuntimeServices.*`, `control/ControlServices.*`, possible new bridge class | Stop `OpenGLComposite::renderEffect()` from draining OSC update/completion queues directly. | -| D. App-frame orchestration | `gl/OpenGLComposite.*`, `gl/RuntimeUpdateController.*` | Replace render-effect glue with a narrow frame-state preparation call and commit-request handoff. | -| E. Persistence boundary | `runtime/coordination/RuntimeCoordinator.*`, `runtime/store/*`, event tests | Keep persistence request publication explicit and prepare for a later background writer without changing storage behavior yet. | +| A. Live-state behavior | `runtime/live/RuntimeLiveState.*`, `tests/RuntimeLiveStateTests.cpp` | Implemented for Phase 3: stale completion, smoothing, trigger behavior, and overlay settle policy are covered by focused tests. | +| B. Render-state composition | `runtime/live/RenderStateComposer.*`, `gl/RenderFrameStateResolver.*`, `gl/RenderEngine.*` | Implemented for Phase 3: value composition and frame-state selection sit outside GL drawing while GL calls remain in `RenderEngine`. | +| C. Service bridge | `control/RuntimeServices.*`, `control/RuntimeServiceLiveBridge.*`, `control/ControlServices.*` | Implemented for Phase 3: `OpenGLComposite::renderEffect()` no longer drains OSC update/completion queues directly. | +| D. App-frame orchestration | `gl/OpenGLComposite.*`, `gl/RuntimeUpdateController.*` | Implemented for Phase 3: render-effect glue is a narrow runtime-work, frame-input, render-frame sequence. | +| E. Persistence boundary | `runtime/coordination/RuntimeCoordinator.*`, `runtime/store/*`, event tests | Implemented for Phase 3: persistence request publication is explicit and ready for a later background writer. | ## Phase 3 Exit Criteria diff --git a/docs/PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md b/docs/PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md new file mode 100644 index 0000000..93b9d16 --- /dev/null +++ b/docs/PHASE_4_RENDER_THREAD_OWNERSHIP_DESIGN.md @@ -0,0 +1,373 @@ +# Phase 4 Design: Render Thread Ownership + +This document expands Phase 4 of [ARCHITECTURE_RESILIENCE_REVIEW.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/ARCHITECTURE_RESILIENCE_REVIEW.md) into a concrete design target. + +Phase 1 named the subsystems. Phase 2 added the typed event substrate. Phase 3 made render-facing live state explicit through `RuntimeLiveState`, `RenderStateComposer`, `RenderFrameInput`, `RenderFrameState`, `RenderFrameStateResolver`, and `RuntimeServiceLiveBridge`. Phase 4 can now focus on the core timing-risk boundary: making one render thread the only owner of OpenGL work. + +## Status + +- Phase 4 design package: proposed. +- Phase 4 implementation: not started. +- Current alignment: the repo has a named frame-state contract and cleaner render-state preparation, but GL work is still entered through multiple paths protected by one shared `CRITICAL_SECTION`. + +Current GL ownership footholds: + +- `RenderEngine` owns GL resources and the current context-binding helpers. +- `RenderFrameInput` / `RenderFrameState` provide the frame-state contract that a render thread can consume. +- `RenderFrameStateResolver` prepares the render-facing layer state before drawing. +- `OpenGLVideoIOBridge` still calls `RenderEngine::TryUploadInputFrame(...)` from the input path and `RenderEngine::RenderOutputFrame(...)` from the output path. +- `OpenGLComposite::paintGL(...)`, screenshot capture, input upload, and output rendering still reach GL through `RenderEngine` methods that bind the shared context under `pMutex`. + +## Why Phase 4 Exists + +The resilience review identifies shared GL ownership as the main remaining timing and failure-isolation risk. Today the shared context lock protects correctness, but it does not isolate timing: + +- input callbacks can attempt texture upload +- output callbacks can trigger frame rendering and readback +- preview paint can enter the same GL context +- screenshot capture can enter the same GL context +- the DeckLink completion path is still too close to render work + +That means brief input, preview, readback, or callback stalls can still collide on the most timing-sensitive path. + +Phase 4 should turn GL from a shared resource guarded by a lock into a resource owned by one thread with explicit queues and handoff points. + +## Goals + +Phase 4 should establish: + +- one render thread as the sole long-lived owner of the GL context +- non-render threads enqueue work instead of binding the GL context +- input upload requests are accepted and executed by the render thread +- output frame rendering is requested or scheduled through render-owned work +- preview and screenshot requests become render-thread commands or consumers +- `RenderFrameInput` / `RenderFrameState` become the stable data contract for frame production +- GL context entrypoints are reduced to render-thread-only code paths +- tests for queue semantics, request coalescing, and lifecycle behavior without requiring DeckLink hardware + +## Non-Goals + +Phase 4 should not require: + +- the final producer/consumer playout queue for DeckLink +- the final DeckLink lifecycle state machine +- replacing the async readback policy +- implementing background persistence +- completing Phase 5's deeper live-state layering +- replacing every UI or backend API at once + +Those are later phases or follow-on work. Phase 4 is about making GL ownership deterministic first. + +## Current GL Entry Points + +The current code paths that matter most are: + +| Entry point | Current behavior | Phase 4 direction | +| --- | --- | --- | +| `RenderEngine::TryUploadInputFrame(...)` | attempts to take the GL lock, binds the context, uploads input texture | enqueue latest input frame; render thread uploads | +| `RenderEngine::RenderOutputFrame(...)` | takes the GL lock, binds the context, renders, packs/readbacks output | render thread executes output frame production | +| `RenderEngine::TryPresentPreview(...)` | attempts to take the GL lock and presents preview | render thread or preview presenter consumes latest completed frame | +| `RenderEngine::CaptureOutputFrameRgbaTopDown(...)` | takes the GL lock and reads output pixels | screenshot request becomes render-thread command | +| `OpenGLVideoIOBridge::UploadInputFrame(...)` | calls render upload directly | push input frame into render queue/mailbox | +| `OpenGLVideoIOBridge::RenderScheduledFrame(...)` | calls render output directly from backend path | request/consume render-produced output without callback-owned GL | + +## Target Ownership Model + +### Render Thread + +The render thread should own: + +- `wglMakeCurrent(...)` for the rendering context +- all GL resource creation/destruction +- input texture upload +- pass execution +- output pack conversion +- async readback buffers and fences +- preview presentation or preview frame publication +- screenshot readback +- temporal history and feedback resources + +### Other Threads + +Other threads may: + +- enqueue input frames or replace the latest input frame +- publish control/runtime/backend events +- request shader build application +- request render-local resets +- request screenshots +- consume ready output frames or receive completion notifications + +Other threads should not: + +- call GL directly +- bind or unbind the render context +- wait on GL fences directly +- mutate render-local resource state + +## Proposed Collaborators + +### `RenderThread` + +Owns the OS thread, wakeup primitive, lifecycle, and render-loop execution. + +Responsibilities: + +- start and stop the render thread +- bind the GL context for the thread lifetime or render-loop lifetime +- drain render commands +- execute frame production work +- publish lifecycle and failure observations + +Non-responsibilities: + +- runtime mutation policy +- DeckLink scheduling policy +- durable persistence + +### `RenderCommandQueue` + +Small bounded queue or command mailbox for render-thread work. + +Possible commands: + +- `UploadInputFrame` +- `RenderOutputFrame` +- `PrepareFrameState` +- `ApplyShaderBuild` +- `ResetTemporalHistory` +- `ResetShaderFeedback` +- `PresentPreview` +- `CaptureScreenshot` +- `Stop` + +High-rate commands should be coalesced where appropriate. Input frames should likely be latest-value rather than unbounded FIFO. + +### `RenderFrameCoordinator` + +Optional helper that combines Phase 3's frame contract with render-thread execution. + +Responsibilities: + +- build or receive `RenderFrameInput` +- call `RuntimeServiceLiveBridge` and `RenderFrameStateResolver` +- hand `RenderFrameState` to `RenderEngine` + +This can begin as a thin helper. The important part is that it keeps frame-state preparation explicit when `renderEffect()` stops being called directly from the callback path. + +### `RenderOutputMailbox` + +Optional transitional bridge for output frames. + +Responsibilities: + +- hold the latest completed output frame or a small bounded set +- let backend code consume output without owning GL +- report underrun/stale-frame reuse observations + +This may be a Phase 4 late step or a Phase 7 playout-policy step. Phase 4 should at least avoid designing the render thread in a way that blocks it. + +## Threading Contract + +Phase 4 should make thread ownership visible in APIs. + +Candidate naming: + +- `RenderEngine::StartRenderThread(...)` +- `RenderEngine::StopRenderThread()` +- `RenderEngine::EnqueueInputFrame(...)` +- `RenderEngine::RequestOutputFrame(...)` +- `RenderEngine::RequestPreviewPresent(...)` +- `RenderEngine::RequestScreenshot(...)` + +Render-thread-only methods should be private or clearly named: + +- `RenderEngine::UploadInputFrameOnRenderThread(...)` +- `RenderEngine::RenderOutputFrameOnRenderThread(...)` +- `RenderEngine::CaptureOutputFrameOnRenderThread(...)` + +The current `TryUploadInputFrame`, `RenderOutputFrame`, `TryPresentPreview`, and `CaptureOutputFrameRgbaTopDown` methods can remain as compatibility shims during migration, but their implementations should move toward enqueue-and-wait or enqueue-and-return behavior instead of binding GL directly from the caller's thread. + +## Frame Production Shape + +A target render-thread frame should look like: + +1. wake for input, output demand, preview demand, shader build, reset, screenshot, or stop +2. drain bounded render commands +3. coalesce to the latest input frame and latest control/live state +4. build `RenderFrameInput` +5. prepare `RenderFrameState` +6. upload accepted input frame +7. render layer stack +8. pack output if needed +9. stage readback or output buffer +10. publish preview/screenshot/output completion as needed +11. record timing and queue metrics + +The exact cadence can remain demand-driven initially. The architectural win is that the demand wakes the render thread rather than borrowing GL from the caller. + +## Migration Plan + +### Step 1. Name Render-Thread-Only Methods + +Split existing direct GL methods into public request methods and private render-thread methods without changing behavior much. + +Initial target: + +- keep current synchronous behavior where callers need a result +- move GL bodies into clearly render-thread-owned helpers +- make future queue migration mechanical + +### Step 2. Add Render Command Queue + +Introduce a small queue/mailbox for render commands. + +Start with low-risk commands: + +- preview present request +- screenshot request +- render-local reset requests + +Then move input upload and output render requests once the queue and wakeup behavior are proven. + +### Step 3. Start A Dedicated Render Thread + +Create the render thread and make it own context binding. + +Transitional behavior may still allow synchronous request/response for output frames. The important change is that the caller waits for render-thread completion rather than taking the GL context itself. + +### Step 4. Move Input Upload To The Render Thread + +Change `OpenGLVideoIOBridge::UploadInputFrame(...)` so it enqueues or replaces the latest input frame. + +Policy targets: + +- bounded memory +- latest-frame wins under load +- input upload skip count is observable +- input callback never waits for GL + +### Step 5. Move Output Rendering To The Render Thread + +Change `OpenGLVideoIOBridge::RenderScheduledFrame(...)` so it requests render-thread output production or consumes a completed render-thread output. + +Transitional option: + +- synchronous request/response through the render thread + +Better follow-up: + +- render ahead into a bounded output queue and let backend callbacks consume ready frames + +### Step 6. Decouple Preview And Screenshot Requests + +Preview should become best-effort: + +- request preview presentation from the render thread +- skip when render is busy or output deadline pressure is high +- record preview skips + +Screenshot should become: + +- queued render-thread capture request +- async disk write remains outside render thread + +### Step 7. Remove Shared GL Lock From Normal Paths + +Once all GL entrypoints are render-thread-owned: + +- remove normal dependence on `pMutex` for render correctness +- keep assertions or diagnostics that detect wrong-thread GL calls +- leave only lifecycle synchronization where needed + +## Testing Strategy + +Phase 4 tests should avoid hardware where possible. + +Recommended tests: + +- render command queue preserves FIFO for non-coalesced commands +- latest-input mailbox drops older frames under load +- stop command wakes and drains the render thread +- screenshot request receives one completion or failure +- output render request reports timeout/failure if render thread is stopped +- render reset commands coalesce where expected +- wrong-thread render-only methods are not publicly reachable + +Existing useful homes: + +- `RuntimeEventTypeTests` for new render/backend observations +- `RuntimeSubsystemTests` for pure request/coalescing helpers +- a new `RenderThreadTests` target for queue/mailbox/lifecycle helpers that do not require GL + +Manual verification will still be needed for: + +- real DeckLink input/output +- preview interaction +- screenshot capture +- shader reload while rendering + +## Telemetry Added During Phase 4 + +Phase 4 should add minimal metrics while moving ownership: + +- render command queue depth +- input frames accepted, replaced, and dropped +- render-thread wake reason counts +- render-thread frame duration +- output request latency +- preview request skipped count +- screenshot request success/failure count +- wrong-thread GL call diagnostics if practical + +Full operational reporting remains Phase 8, but these metrics make the threading migration debuggable. + +## Risks + +### Deadlock Risk + +Synchronous request/response shims can deadlock if the caller is already on the render thread or holds a lock the render thread needs. Phase 4 should keep request waits narrow and add render-thread detection early. + +### Latency Risk + +Moving work through queues can hide latency. Queue depth and output request latency should be measured from the first migration step. + +### Lifetime Risk + +Moving context ownership changes startup and shutdown order. The render thread must stop before GL resources or window/context handles are destroyed. + +### Callback Pressure Risk + +If DeckLink callbacks wait too long for render-thread work, Phase 4 may improve GL ownership but still leave callback timing fragile. A synchronous bridge is acceptable as a transition, but the design should keep the path open for producer/consumer playout. + +### Preview Coupling Risk + +Preview can remain a hidden budget consumer if it stays in the output frame path. Phase 4 should keep preview explicitly best-effort, even if physical decoupling continues later. + +## Phase 4 Exit Criteria + +Phase 4 can be considered complete once the project can say: + +- [ ] one render thread owns the GL context during normal operation +- [ ] input callbacks do not bind GL or wait on GL upload +- [ ] output callbacks do not bind GL directly +- [ ] preview and screenshot requests enter render through explicit render-thread requests +- [ ] `RenderFrameInput` / `RenderFrameState` remain the frame-state contract +- [ ] normal frame production no longer depends on a shared GL `CRITICAL_SECTION` +- [ ] render-thread queue/mailbox behavior has non-GL tests +- [ ] shutdown order is explicit and tested or manually verified + +## Open Questions + +- Should the first output migration be synchronous request/response, or should Phase 4 go directly to a small ready-frame queue? +- Should the render thread own `RuntimeServiceLiveBridge` calls, or should frame state be prepared just before enqueue? +- How much input frame memory should be copied at enqueue time versus referenced from backend-owned buffers? +- Should preview present on the render thread, or should render publish a preview image/texture to a separate presenter? +- What timeout should output callbacks use if the render thread cannot produce a frame in time? +- Should wrong-thread GL access be enforced with assertions, telemetry, or both? + +## Short Version + +Phase 4 should make GL ownership boring and deterministic. + +One render thread owns the context. Other threads submit work or consume results. Input upload, frame rendering, readback, preview, and screenshot capture all move behind render-thread entrypoints. The first implementation can be transitional and partly synchronous, but after Phase 4 the app should no longer rely on callback and UI paths borrowing the GL context under one shared lock. diff --git a/docs/subsystems/RenderEngine.md b/docs/subsystems/RenderEngine.md index 16bdff4..bc30ebc 100644 --- a/docs/subsystems/RenderEngine.md +++ b/docs/subsystems/RenderEngine.md @@ -39,14 +39,14 @@ The current rendering path is split across several classes: - [OpenGLComposite.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/OpenGLComposite.cpp:86) constructs the renderer, render pipeline, shader programs, runtime services, and video bridge in one owner. - [OpenGLRenderPipeline.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLRenderPipeline.cpp:31) performs pass execution, pack/readback, preview paint, and performance stat publication. - [OpenGLVideoIOBridge.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:58) accepts capture frames and still performs render work from the playout completion callback path. -- `OpenGLComposite` still holds render-local overlay behavior and shader rebuild handling alongside runtime orchestration responsibilities. +- `RenderFrameStateResolver` and `RenderStateComposer` now keep frame-state selection and live value composition outside GL drawing, while `RenderEngine` still owns the current GL resource and draw path. That split is workable today, but it creates architectural pressure: - GL ownership is thread-shared instead of sole-owned. - render and playout timing are still callback-coupled. - preview and playout are produced in the same immediate path. -- render-local transient state is too easy to leak back into runtime-facing code. +- render-local transient state now has clearer Phase 3 boundaries, but GL ownership is still shared through callback and UI entrypoints. - it is difficult to test render behavior separately from app bootstrap and hardware integration. `RenderEngine` exists to absorb that responsibility into one subsystem with one direction of ownership. @@ -170,11 +170,12 @@ Other threads should interact with the subsystem through queues, snapshots, and ## Current State -Today GL work is still shared across callback-driven entrypoints: +Today GL work is still shared across callback-driven and UI entrypoints: -- input upload occurs in [OpenGLVideoIOBridge::VideoFrameArrived()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:58) -- playout-triggered render occurs in [OpenGLVideoIOBridge::PlayoutFrameCompleted()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:95) +- input upload is requested through [OpenGLVideoIOBridge::UploadInputFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11) +- playout-triggered render is requested through [OpenGLVideoIOBridge::RenderScheduledFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:18) - render-pass execution occurs in [OpenGLRenderPipeline::RenderFrame()](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLRenderPipeline.cpp:31) +- preview and screenshot paths still enter `RenderEngine` methods that bind the shared context The `CRITICAL_SECTION` protects correctness, but it is not the target architectural model. @@ -406,6 +407,8 @@ inside render-owned code paths instead of putting them back into runtime storage Introduce snapshot-facing APIs so render no longer depends on broad runtime-state access for frame production. +Current status: Phase 3 introduced `RenderFrameInput`, `RenderFrameState`, and `RenderFrameStateResolver`, so frame-state selection is named and no longer lives inside GL drawing. Phase 4 can build on that contract while moving GL ownership. + ### Step 4. Move Uploads Onto Render Ownership Input callbacks should enqueue or hand off frame data; render executes the upload. diff --git a/docs/subsystems/VideoBackend.md b/docs/subsystems/VideoBackend.md index 38c9f9d..dfec491 100644 --- a/docs/subsystems/VideoBackend.md +++ b/docs/subsystems/VideoBackend.md @@ -102,7 +102,7 @@ This work is currently split across: - [DeckLinkSession::ConfigureInput](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/videoio/decklink/DeckLinkSession.cpp:221) - [CaptureDelegate::VideoInputFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/videoio/decklink/DeckLinkFrameTransfer.cpp:33) -- [OpenGLVideoIOBridge::VideoFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:50) +- [OpenGLVideoIOBridge::UploadInputFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11) ### 3. Output Lifecycle and Output Callback Handling @@ -307,7 +307,7 @@ Today the callback path effectively does this: That path is visible in: -- [OpenGLVideoIOBridge::PlayoutFrameCompleted](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:83) +- [OpenGLVideoIOBridge::RenderScheduledFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:18) This couples output timing directly to render work. @@ -354,7 +354,7 @@ Recommended default policy for live playout: The current "skip upload if the GL bridge is busy" behavior is directionally correct for live timing: -- [OpenGLVideoIOBridge::VideoFrameArrived](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:57) +- [OpenGLVideoIOBridge::UploadInputFrame](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/gl/pipeline/OpenGLVideoIOBridge.cpp:11) But in the target architecture that decision should move out of GL lock acquisition and into an explicit backend-to-render handoff queue policy.