aiden/video-shader-toys
1
0
Fork 1
Code Issues Pull Requests Actions Releases 5 Wiki Activity

Docs update
Some checks failed
CI / React UI Build (push) Successful in 11s
Details
CI / Native Windows Build And Tests (push) Failing after 2m11s
Details
CI / Windows Release Package (push) Has been skipped
Details

Browse Source
This commit is contained in:
Aiden
2026-05-21 17:14:29 +10:00
parent 5c46eaf18a
commit 3fc78d5bb8
7 changed files with 204 additions and 559 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
README.md
View File

@@ -17,11 +17,13 @@ The app loads shader packages from `shaders/`, compiles Slang to GLSL at runtime
Native app internals are grouped by boundary:
- `videoio/`: backend-neutral video I/O contracts, formats, and playout timing.
- `videoio/decklink/`: DeckLink-specific device adapter, callbacks, and SDK bindings.
- `gl/renderer/`: low-level OpenGL resources and extension helpers.
- `gl/pipeline/`: frame pipeline, render passes, video I/O bridge, preview/readback, and screenshots.
- `gl/shader/`: shader compilation, texture/text assets, UBO packing, and shader program ownership.
- `app/`: startup/shutdown orchestration and runtime layer controller.
- `control/`: HTTP/WebSocket server, command parsing, and runtime-state JSON presentation.
- `frames/`: system-memory frame exchange and input mailbox handoff.
- `render/`: render thread, readback, runtime render scene, and shared-context shader program preparation.
- `runtime/`: shader catalog support, layer model, Slang build bridge, font atlas build, and runtime-state persistence.
- `shader/`: shader package parsing and Slang compilation helpers.
- `video/`: DeckLink input/output edges, format helpers, and scheduling.
## Requirements
@@ -72,10 +74,8 @@ The native app serves `ui/dist` when it exists, otherwise it falls back to the s
The control UI provides:
- A searchable shader library for adding layers.
- Compact parameter rows with inline descriptions and OSC copy controls.
- Stack save/recall presets.
- Compact parameter rows with inline descriptions and intended OSC route copy controls.
- Manual shader reload.
- Screenshot capture from the final output render target.
## Package
@@ -138,7 +138,6 @@ Current native test coverage includes:
- Parameter normalization and preset filename safety.
- Shader manifest parsing, temporal manifest validation, and package registry scanning.
- Video I/O format helpers, v210/Ay10 row-byte math, v210 pack/unpack math, playout scheduler timing, and fake backend contract coverage.
- OSC packet parsing.
- Slang validation for every available shader package.
## Runtime Configuration
@@ -172,11 +171,13 @@ The control UI is available at:
http://127.0.0.1:<serverPort>
```
## Runtime State And Presets
## Runtime State
The current layer stack is autosaved to `runtime/runtime_state.json` whenever layers, shader assignments, bypass state, ordering, or parameter values change. On startup, the host reloads that file before compiling the stack, so the last working stack should come back automatically.
The current layer stack is autosaved to `runtime/runtime_state.json` whenever durable UI/API layer changes are accepted: add/remove, shader assignment, bypass state, ordering, parameter updates, parameter reset, and reload compatibility refreshes. Saves are debounced and written on a background worker, with a final flush during shutdown.
Manual stack presets are still available from the control UI and are saved under `runtime/stack_presets/*.json`. Presets are useful for named looks, while `runtime_state.json` is the latest working state for the local machine.
On startup, the host tries to reload `runtime/runtime_state.json` before compiling the stack. Valid saved layers are rebuilt in saved order, with shader id, bypass state, and parameter values restored. Missing shader packages are skipped, invalid saved parameter values fall back to shader defaults, and if the file is missing or unusable the app falls back to the configured default shader.
Manual stack preset and screenshot routes are still present in the UI/OpenAPI surface, but they are not implemented by the current native command path yet. `runtime_state.json` is the supported latest-working-state mechanism for now.
## Control API
@@ -199,13 +200,15 @@ A Swagger UI page is available at:
http://127.0.0.1:<serverPort>/docs
```
Use those docs to inspect the `/api/state`, layer control, stack preset, and reload endpoints. Live state updates are also sent over the `/ws` WebSocket.
Use those docs to inspect the `/api/state`, layer control, and reload endpoints. Live state updates are also sent over the `/ws` WebSocket.
The control UI has a **Reload shaders** button. It rescans `shaders/`, re-reads manifests, queues shader compilation, refreshes shader availability/errors, and keeps the previous working shader stack running if a changed shader fails to compile.
The control UI has a **Reload shaders** button. It rescans `shaders/`, re-reads manifests, refreshes shader availability/errors, updates active layer parameter definitions from changed manifests, and queues recompilation for every catalog-valid layer in the active stack. Missing shader packages are marked failed, and the previous working render stack remains active where possible until replacement builds commit successfully.
Each parameter row also includes a small **OSC** button. Clicking it copies that parameter's OSC route to the clipboard.
Each parameter row still exposes the intended OSC route in the UI, but OSC ingress is not wired in the current native host.
The control UI also has a **Screenshot** button. It queues a capture of the final output render target and writes a PNG under:
The control UI currently still shows preset and screenshot controls from the intended route surface. Those endpoints return an unimplemented action result in the native host until their backend paths are wired.
The planned screenshot output directory is:
```text
runtime/screenshots/
@@ -213,13 +216,15 @@ runtime/screenshots/
## OSC Control
The native host also listens for OSC parameter control on the configured `oscBindAddress` and `oscPort`:
OSC fields are present in `config/runtime-host.json` and `/api/state` for compatibility with the intended control surface, but the current native host does not start an OSC listener yet.
The intended route shape is:
```text
/VideoShaderToys/{LayerNameOrID}/{ParameterNameOrID}
```
For example, `/VideoShaderToys/VHS/intensity` updates the `intensity` parameter on the first matching `VHS` layer. The listener accepts float, integer, string, and boolean OSC values, and validates them through the same shader parameter path as the REST API. OSC updates are coalesced and applied once per render tick, UI state broadcasts are throttled, and OSC-driven parameter changes are not autosaved to `runtime/runtime_state.json`. `oscSmoothing` adds a small per-frame easing amount for numeric OSC controls such as floats, `vec2`, and `color`, while booleans, enums, text, and triggers stay immediate. The default bind address is `127.0.0.1`; set `oscBindAddress` to `0.0.0.0` to accept OSC on all IPv4 interfaces. See `docs/OSC_CONTROL.md` for details.
For now, use the REST layer parameter endpoints or the control UI for live parameter changes. Future OSC-driven parameter changes should stay out of autosave unless an explicit persistence policy is added.
## Shader Packages
@@ -243,8 +248,8 @@ Runtime-generated files are intentionally ignored:
- `runtime/shader_cache/active_shader.raw.frag`
- `runtime/shader_cache/active_shader.frag`
- `runtime/runtime_state.json` autosaved latest stack and parameter state.
- `runtime/stack_presets/*.json`
- `runtime/screenshots/*.png` screenshots captured from the final output render target.
- `runtime/stack_presets/*.json` planned manual preset output; preset routes are not implemented in the native host yet.
- `runtime/screenshots/*.png` planned screenshot output; screenshot capture is not implemented in the native host yet.
Only `runtime/templates/` and `runtime/README.md` are tracked.

11
docs/ARCHITECTURE_RESILIENCE_REVIEW.md
View File

@@ -263,14 +263,13 @@ Recommended direction:
Status: addressed by Phase 6.
Relevant code:
Relevant current `RenderCadenceCompositor` code:
- `RuntimeCoordinator.cpp`
- `RuntimeUpdateController.cpp`
- `RuntimeStore.cpp`
- `PersistenceWriter.cpp`
- `src/app/RuntimeLayerControllerControls.cpp`
- `src/runtime/RuntimeStatePersistence.cpp`
- `src/runtime/RuntimeStatePersistence.h`
Runtime-state persistence now flows from accepted coordinator mutations to typed persistence events, then into a debounced background writer. The store still owns serialization and preset IO, while the writer owns temp-file replacement, coalescing, result reporting, and shutdown flushing.
Runtime-state persistence now flows from accepted durable layer-stack mutations into a debounced background writer. The layer controller owns the current snapshot source, while `RuntimeStatePersistenceWriter` owns serialization, temp-file replacement, coalescing, result reporting, and shutdown flushing.
The remaining architecture concern is broader persistence policy, not direct mutation-path disk writes:

637
docs/CURRENT_SYSTEM_ARCHITECTURE.md
View File

@@ -1,540 +1,175 @@
# Current System Architecture
This document describes how the application currently works.
This document describes how the current `RenderCadenceCompositor` app works.
It replaces the phase-by-phase design trail as the best entry point for understanding the repo. The older phase documents remain useful history, but they mix implementation notes, experiments, and target designs. This document is organized by current runtime behavior and subsystem ownership instead.
The active plan for tightening render-thread ownership is:
- [Render Thread Ownership Plan](RENDER_THREAD_OWNERSHIP_PLAN.md)
The plan for building a fresh modular app around the proven probe architecture is:
- [RenderCadenceCompositor README](../apps/RenderCadenceCompositor/README.md)
- [Render Cadence Golden Rules](RENDER_CADENCE_GOLDEN_RULES.md)
`NEW_RENDER_CADENCE_APP_PLAN.md` remains as historical planning context, but the README and golden rules are the current contract for the new cadence-first app.
The implementation is cadence-first: the render/output path owns frame timing, while shader compilation, HTTP control, preview, persistence, and video I/O edges stay outside the render cadence loop wherever possible. The guardrails in [Render Cadence Golden Rules](RENDER_CADENCE_GOLDEN_RULES.md) are the operating contract.
## Application Shape
The app is a live OpenGL compositor with DeckLink input/output, runtime control services, persistent layer-stack state, live state overlays, health telemetry, and a small internal event model.
The app is a native C++ OpenGL compositor with:
At runtime the major subsystems are:
- optional DeckLink input
- optional DeckLink scheduled output
- a render-thread-owned OpenGL context
- runtime Slang shader packages from `shaders/`
- a configurable active layer stack
- a local HTTP/WebSocket control server
- optional Win32/GDI preview from system-memory output frames
- background runtime-state persistence
- cadence telemetry and log output
- `OpenGLComposite`
- `RuntimeStore`
- `RuntimeCoordinator`
- `RuntimeSnapshotProvider`
- `RuntimeServices`
- `RuntimeUpdateController`
- `RenderEngine`
- `VideoBackend`
- `DeckLinkSession`
- `HealthTelemetry`
- `RuntimeEventDispatcher`
- `PersistenceWriter`
Primary source areas:
The key architectural rule is:
- `src/app`: startup/shutdown orchestration, config loading, runtime layer controller
- `src/render`: render thread, readback, runtime GL scene, shared-context shader prepare worker
- `src/frames`: system-memory frame exchange
- `src/video`: DeckLink input/output edges and scheduling
- `src/runtime`: shader catalog support, layer model, Slang build bridge, font atlas build, runtime-state persistence
- `src/control`: HTTP routing, command parsing, OpenAPI state JSON
- `src/preview`: optional non-consuming preview window
- `src/telemetry` and `src/logging`: runtime observation and logging
- runtime/control subsystems decide what state should exist
- render subsystems decide how to draw that state
- video subsystems decide how frames move to and from hardware
- telemetry observes behavior without becoming a control plane
## Startup
## Process Startup
Startup broadly proceeds as:
The Win32 app creates the window, chooses a pixel format, creates an OpenGL context, initializes COM, and constructs `OpenGLComposite`.
1. Load `config/runtime-host.json` through `AppConfigProvider`, then apply CLI overrides.
2. Load the supported shader catalog from the configured `shaderLibrary`.
3. Start the runtime-state persistence writer.
4. Try to restore `runtime/runtime_state.json`.
5. If restore fails or no usable state exists, create one layer from the configured default shader.
6. Start the render thread.
7. Queue background Slang builds for every pending active layer.
8. Build a small completed-frame reserve.
9. Start optional preview, optional DeckLink output, telemetry, and HTTP control.
`OpenGLComposite` owns the high-level assembly of the runtime:
The runtime-state restore is intentionally app/control side work. The render thread does not read JSON, inspect the shader library, or decide what to compile.
- runtime store
- runtime coordinator
- runtime services
- runtime update controller
- render engine
- video backend
## Runtime Layer State
Startup proceeds broadly as:
`RuntimeLayerController` owns the app-side layer model and coordinates:
1. COM and OpenGL are initialized by the Win32 app.
2. `OpenGLComposite::InitDeckLink()` discovers/configures DeckLink and runtime state.
3. Runtime services are started.
4. Shader programs and GL resources are initialized.
5. The render thread is started.
6. The video backend starts output preroll and playback.
- supported shader catalog loading
- layer add/remove/reorder/bypass/shader assignment
- parameter update/reset
- startup restore
- reload reconciliation
- background Slang builds
- render-layer publication
- runtime-state persistence requests
The normal VS Code debug launch currently sets:
`RuntimeLayerModel` owns the in-memory active stack:
```text
VST_DISABLE_INPUT_CAPTURE=1
```
- layer id
- shader id and display name
- build state and message
- bypass state
- manifest parameter definitions
- current parameter values
- render-ready artifacts
That disables DeckLink input capture for output-timing isolation while keeping the output path active.
The current durable runtime state is stored in `runtime/runtime_state.json`. It contains the active stack order, shader ids, bypass flags, and parameter values. On startup, valid saved layers are restored in order. Missing shader packages are skipped, invalid saved parameter values fall back to manifest defaults, and a missing or unusable file falls back to the configured default shader.
## Runtime State
### `RuntimeStore`
`RuntimeStore` owns durable runtime data and file-backed state.
It owns:
- runtime host configuration
- stored layer stack data
- persisted parameter values
- stack presets
- shader package catalog metadata
- runtime state presentation data
- persistence requests
It does not own render-thread resources, DeckLink timing, control ingress, or mutation policy.
### `CommittedLiveState`
`CommittedLiveState` owns current session/operator layer state that is live but not necessarily persisted as the durable base state.
It gives the renderer and snapshot builder a named read model for current committed layer state.
### `RuntimeCoordinator`
`RuntimeCoordinator` is the mutation policy boundary.
It validates and applies runtime mutations, classifies whether changes are persisted/committed/transient, emits persistence requests, and produces render reset/reload decisions.
It keeps mutation decisions out of:
- the render engine
- control services
- video backend
- telemetry
### `RuntimeSnapshotProvider`
`RuntimeSnapshotProvider` publishes render-facing snapshots.
It owns the currently published render snapshot and gives the render path a stable read boundary. Rendering does not read mutable store objects directly.
## Live State And Layering
The current render state is built from named layers of state:
- persisted layer/package/default state from the runtime store
- committed live/session state
- transient live overlays from OSC/control input
- render-local state owned by the renderer
`RuntimeStateLayerModel` names these categories. `RenderStateComposer` and `RuntimeLiveState` combine live values into render-facing state.
`RenderFrameInput` and `RenderFrameState` are the frame contract:
- `RenderFrameInput` describes what kind of frame is being built
- `RenderFrameState` describes the resolved state used to draw that frame
The renderer should not ask global state systems which snapshot or layer state to use midway through drawing.
## Control And Events
### `RuntimeServices`
`RuntimeServices` owns runtime-facing services such as OSC/control integration and service lifecycle.
It connects control ingress to the coordinator and live-state bridge.
### `ControlServices`
`ControlServices` handles OSC/control ingress, buffering, and polling/wake behavior.
It does not own runtime mutation policy. It normalizes ingress and asks the coordinator/runtime services to apply changes.
### `RuntimeEventDispatcher`
The app uses typed runtime events for internal coordination and observation.
Events are used for:
- runtime state broadcast requests
- shader build lifecycle
- backend state changes
- input/output frame observations
- timing samples
- health and queue observations
Events say what happened. Commands/request methods still exist where a caller needs an immediate success/failure answer.
Manual stack preset routes are present in the UI/OpenAPI surface but are not implemented in the current native command path yet. `runtime_state.json` is the supported latest-working-state mechanism.
## Persistence
Persistence is handled by `PersistenceWriter`.
`RuntimeStatePersistenceWriter` performs debounced background writes to `runtime/runtime_state.json`.
Runtime mutations can enqueue persistence requests without blocking the render/output path. Shutdown performs a bounded persistence flush.
Durable UI/API mutations request persistence after they are accepted:
The store owns durable state; the writer owns background write execution.
- add/remove layer
- reorder layer
- set bypass
- set shader
- update parameter
- reset parameters
- reload compatibility refresh
## Render System
The mutation path snapshots the current layer model and hands serialized state to the writer. File IO happens on the persistence worker, not on the render thread or cadence path. Shutdown flushes the latest pending snapshot.
### `RenderEngine`
OSC-driven changes are intentionally not part of this autosave path yet.
`RenderEngine` owns normal runtime OpenGL work.
## Shader Reload
It starts a dedicated render thread and binds the GL context on that thread. Runtime GL work enters through render-thread requests or render command queues.
`POST /api/reload` and the control UI reload button:
The render thread handles:
- rescan `shaders/`
- re-read manifests
- rebuild the supported shader catalog
- update active layer metadata and parameter definitions from changed manifests
- preserve compatible parameter values
- default new or incompatible parameter values
- queue recompilation for every catalog-valid layer in the active stack
- output frame rendering
- input frame upload
- preview present
- screenshot capture
- render-local resets
- shader/rebuild application
- temporal history and shader feedback resources
Reload does not compile every package in the shader library. A package is compiled when it is part of the active layer stack. If an active layer references a shader that no longer exists, that layer is marked failed and skipped. Existing render output remains active where possible until replacement builds are ready.
Startup initialization still happens before the render thread starts while the app explicitly owns the context. Normal runtime work is routed through `RenderEngine`.
`autoReload` is still exposed in config/state for compatibility, but automatic file watching is not currently wired.
### Current Render-Thread Limitation
## Render Ownership
The current render thread is a shared GL executor, not a pure output-only cadence thread.
The render thread owns the app OpenGL context during normal operation.
This means output render can still be delayed by:
The render path consumes published render-layer snapshots. It does not:
- input upload work
- preview present requests
- screenshot capture
- render reset commands
- shader/resource update work
- synchronous render-thread request queue wait
- parse JSON
- scan files
- launch Slang
- run font atlas generation
- perform persistence
- handle HTTP or OSC
- call DeckLink discovery/setup APIs
For output-timing diagnosis, input capture can be disabled with:
When a runtime shader build completes, the app publishes a render-layer artifact. The render thread-owned runtime scene diffs the snapshot and queues changed pass programs to the shared-context prepare worker. The render thread swaps in an already-prepared render plan at a frame boundary.
```text
VST_DISABLE_INPUT_CAPTURE=1
## Video And Preview
DeckLink input and output are optional edges.
Input captures BGRA8 directly where possible, or raw UYVY8 into `InputFrameMailbox` for render-thread GPU decode. The input edge does not call GL, render, preview, screenshot, shader, or output scheduling code.
Output consumes completed system-memory frames from `SystemFrameExchange` and schedules them to DeckLink. If DeckLink output is unavailable, the app continues running render cadence, control, preview, telemetry, and logging.
`PreviewWindowThread` is optional and uses a non-consuming system-memory tap. It paints with Win32/GDI on its own thread and skips preview ticks instead of blocking the frame exchange.
Screenshot routes are present in the UI/OpenAPI surface but are not implemented in the current native command path yet.
## Control Surface
The HTTP server runs on its own thread. It serves:
- UI assets
- OpenAPI/Swagger docs
- `GET /api/state`
- `/ws` state updates
- layer mutation POST routes
- `/api/reload`
Known but not implemented in the current native command path:
- `/api/layers/move`
- `/api/stack-presets/save`
- `/api/stack-presets/load`
- `/api/screenshot`
Unsupported routes return an action response with `ok: false`.
## Tests
Native tests cover the main non-GL contracts:
- JSON parsing/serialization
- runtime parameter normalization
- shader package registry and Slang validation
- supported shader catalog
- runtime layer restore/reload behavior
- runtime-state persistence writer
- HTTP command parsing
- frame exchange and input mailbox behavior
- video format and scheduling helpers
Run:
```powershell
cmake --build --preset build-debug --target RUN_TESTS --parallel
```
When enabled, the backend skips DeckLink input configuration/start and `HasInputSource()` reports false.
### `OpenGLRenderPipeline`
`OpenGLRenderPipeline` draws the frame and performs output packing/readback.
The current output path:
1. binds the composite framebuffer
2. calls the render effect callback
3. blits/composes into the output framebuffer
4. packs the output for the configured pixel format
5. flushes GL
6. reads output into the provided system-memory output frame
7. records render/readback timing
For BGRA8 output, the pipeline uses a BGRA-compatible pack framebuffer and async PBO readback by default.
## Video Backend
### `VideoBackend`
`VideoBackend` owns app-level video device lifecycle, output production, system-memory frame slots, and backend playout health.
It owns:
- backend lifecycle state
- output production worker
- output completion worker
- system-memory output frame pool
- ready/completed output queue
- render cadence controller
- playout policy
- output frame scheduling into `VideoIODevice`
- backend timing and queue telemetry
It does not own GL drawing. It asks `OpenGLVideoIOBridge` / `RenderEngine` to render into system-memory output frames.
### Lifecycle
The current backend lifecycle includes:
- discovery
- configuring
- configured
- prerolling
- running
- degraded
- stopping
- stopped
- failed
Startup now separates output schedule preparation from scheduled playback:
1. prepare the DeckLink output schedule
2. start output completion worker
3. start output producer worker
4. warm up rendered system-memory preroll frames
5. optionally start input streams
6. start DeckLink scheduled playback
### Output Production
The output producer is cadence-driven.
`RenderCadenceController` tracks the selected output frame duration and decides when the producer should render another frame.
The render producer attempts to render one output frame per selected output tick. It does not speed up just because DeckLink is empty.
If render/GPU work is late enough, the cadence controller can skip late ticks according to policy.
### System-Memory Frame Pool
`SystemOutputFramePool` owns reusable system-memory output slots.
Slots have four states:
- `Free`
- `Rendering`
- `Completed`
- `Scheduled`
In the current legacy app, completed-but-unscheduled frames are treated as a latest-N cache. The newer `RenderCadenceCompositor` uses a bounded FIFO completed reserve instead; see its README for the cadence-first contract.
Scheduled frames are protected until DeckLink reports completion.
### Output Queue
`RenderOutputQueue` holds completed unscheduled output frames waiting to be scheduled.
In the legacy app it is bounded and latest-N:
- pushing beyond capacity releases/drops the oldest ready frame
- `DropOldestFrame()` is used when the frame pool needs to recycle old completed work
### Scheduling
`VideoBackend::ScheduleReadyOutputFramesToTarget()` schedules completed system-memory frames up to the configured preroll/scheduled target.
DeckLink scheduling is capped by the current app-owned scheduled count. Real DeckLink buffered-frame telemetry is also recorded.
### Completion Handling
DeckLink completion callbacks do not render.
The callback path reports completion into `VideoBackend`, which processes completions on a backend worker. Completion processing:
- releases the system-memory slot by buffer pointer
- records pacing
- accounts for late/drop/flushed/completed result
- records telemetry
- wakes the output producer
## DeckLink Integration
### `DeckLinkSession`
`DeckLinkSession` is the DeckLink implementation of `VideoIODevice`.
It owns:
- DeckLink discovery
- input/output mode selection
- DeckLink input/output interfaces
- keyer configuration
- capture and playout delegates
- schedule-time generation through `VideoPlayoutScheduler`
- DeckLink frame scheduling
- actual buffered-frame telemetry
For output, system-memory frames are scheduled through DeckLink `CreateVideoFrameWithBuffer()`.
When a system-memory frame is scheduled, `DeckLinkSession` records a map from the DeckLink frame object back to the app-owned system-memory buffer pointer. On completion, the buffer pointer is returned so `VideoBackend` can release the matching slot.
### Actual DeckLink Buffer Telemetry
`DeckLinkSession` calls `GetBufferedVideoFrameCount()` after schedule/completion where available.
Telemetry separates:
- actual DeckLink buffered frames
- app-owned scheduled system-memory slots
- synthetic schedule/completion counters
- late/drop/flushed completion results
## Output Timing Experiments And Current Finding
The repo includes `DeckLinkRenderCadenceProbe`, a small standalone test app under:
```text
apps/DeckLinkRenderCadenceProbe
```
The probe does not use the main runtime, shader system, preview path, input upload path, or shared render engine. It uses:
- one OpenGL render thread with its own hidden GL context
- simple BGRA8 motion rendering
- async PBO readback
- legacy latest-N system-memory frame slots; bounded FIFO completed reserve in `RenderCadenceCompositor`
- a playout thread that feeds DeckLink
- real rendered warmup before scheduled playback
The first hardware result was smooth at roughly 59.94/60 fps with:
- `renderFps` near 59.9
- `scheduleFps` near 59.9
- DeckLink actual buffered frames stable at 4
- no late frames
- no dropped frames
- no PBO misses
- no completed-frame drops
That proves the clean architecture can work on the test machine. Remaining main-app timing issues are therefore likely integration/ownership issues in the main app rather than a fundamental DeckLink/OpenGL/BGRA8 limitation.
The highest-value current suspects are:
- input upload sharing the output render thread
- shared render-thread task queue contention
- preview/screenshot work
- runtime/render-state work on the output path
## Health Telemetry
`HealthTelemetry` owns app-visible health and timing observations.
It records:
- signal/input status
- performance/render timing
- event queue timing
- backend lifecycle/playout state
- output render queue wait
- output render/readback timing
- system-memory frame counts
- actual DeckLink buffer depth
- late/drop/flushed/completed frame counters
- schedule-call timing/failure counts
Several hot-path telemetry calls use try-lock variants so observation does not become a major timing dependency.
Runtime state presentation exposes telemetry through the runtime JSON/open API surface.
## Preview And Screenshot
Preview is best-effort.
`OpenGLComposite::paintGL()` skips preview when the backend reports output pressure. Preview presentation is requested through the render thread.
Screenshot capture is also a render-thread request. It reads pixels from the output framebuffer and writes PNG asynchronously after capture.
Both preview and screenshot share GL execution with output render, so they are secondary to output timing.
## Output Readback Modes
The output readback path supports environment-selected modes:
```text
VST_OUTPUT_READBACK_MODE=async_pbo
VST_OUTPUT_READBACK_MODE=sync
VST_OUTPUT_READBACK_MODE=cached_only
```
Default behavior is `async_pbo`.
Experiment findings:
- direct synchronous readback was slower on the sampled machine
- cached-only recovered timing but is visually invalid for live motion
- BGRA8 pack framebuffer plus async PBO removed the earlier large readback stall
## Current Debug/Experiment Launches
VS Code launch configurations include:
- `Debug LoopThroughWithOpenGLCompositing`
- `Debug LoopThroughWithOpenGLCompositing - sync readback experiment`
- `Debug LoopThroughWithOpenGLCompositing - cached output experiment`
- `Debug DeckLinkRenderCadenceProbe`
The default main-app debug launch currently disables input capture with `VST_DISABLE_INPUT_CAPTURE=1` so output timing can be tested without input upload interference.
## Current Ownership Summary
| Area | Current Owner |
| --- | --- |
| Durable runtime config/state | `RuntimeStore` |
| Current committed live layer state | `CommittedLiveState` |
| Mutation validation/policy | `RuntimeCoordinator` |
| Render snapshot publication | `RuntimeSnapshotProvider` |
| OSC/control ingress | `RuntimeServices` / `ControlServices` |
| Internal event dispatch | `RuntimeEventDispatcher` |
| Background persistence writes | `PersistenceWriter` |
| GL context and normal GL work | `RenderEngine` render thread |
| Render-pass execution and output readback | `OpenGLRenderPipeline` |
| Device lifecycle and output production | `VideoBackend` |
| DeckLink API integration | `DeckLinkSession` |
| Operational health/timing | `HealthTelemetry` |
## Current Runtime Flow Summary
### Control Mutation
```text
OSC/API/control input
-> RuntimeServices / ControlServices
-> RuntimeCoordinator
-> RuntimeStore / CommittedLiveState / RuntimeLiveState
-> RuntimeSnapshotProvider publication or live overlay update
-> RuntimeEventDispatcher observations
```
### Output Render
```text
VideoBackend output producer
-> RenderCadenceController tick
-> SystemOutputFramePool acquire rendering slot
-> OpenGLVideoIOBridge::RenderScheduledFrame
-> RenderEngine::RequestOutputFrame
-> render thread
-> OpenGLRenderPipeline::RenderFrame
-> system-memory output slot
-> RenderOutputQueue completed frame
```
### DeckLink Playout
```text
RenderOutputQueue completed frame
-> VideoBackend schedules to target
-> DeckLinkSession::ScheduleOutputFrame
-> CreateVideoFrameWithBuffer
-> ScheduleVideoFrame
-> DeckLink playback
-> completion callback
-> VideoBackend completion worker
-> release scheduled system-memory slot
```
### Input Capture
When input capture is enabled:
```text
DeckLink input callback
-> VideoBackend::HandleInputFrame
-> OpenGLVideoIOBridge::UploadInputFrame
-> RenderEngine::QueueInputFrame
-> render thread upload
```
When `VST_DISABLE_INPUT_CAPTURE=1`, this flow is skipped.
## Known Current Constraints
- The main app render thread still handles multiple kinds of GL work.
- Output render still uses a synchronous request/response call into the render thread.
- Input upload can contend with output render when input capture is enabled.
- Preview and screenshot share the render thread.
- Phase/experiment documents still exist as historical notes, but this document is the current architecture summary.
## Practical Rules
- Keep one owner for each kind of state.
- Keep GL work on the render thread.
- Keep DeckLink completion callbacks passive.
- In the legacy app, treat completed unscheduled output frames as latest-N cache entries; in `RenderCadenceCompositor`, preserve completed frames as a bounded FIFO reserve.
- Protect scheduled output frames until DeckLink completion.
- Keep output timing more important than preview/screenshot.
- Measure timing by domain instead of adding fallback branches blindly.

6
docs/OSC_CONTROL.md
View File

@@ -1,6 +1,6 @@
# OSC Control
Video Shader Toys can listen for local OSC messages and map them onto shader layer parameters.
This is the intended OSC control contract, but OSC ingress is not wired in the current `RenderCadenceCompositor` native host yet. The config fields and UI copy buttons are present for compatibility; use the REST layer parameter endpoints or the control UI for live parameter changes today.
## Configuration
@@ -14,9 +14,7 @@ Set the UDP port in `config/runtime-host.json`:
}
```
Set `oscPort` to `0` to disable the OSC listener.
Set `oscBindAddress` to `127.0.0.1` to keep OSC local to the host, or `0.0.0.0` to listen on all IPv4 interfaces.
Set `oscSmoothing` to a value from `0.0` to `1.0` to add a subtle per-frame easing amount for numeric OSC controls. `0.0` disables smoothing, and larger values respond more quickly.
When OSC ingress is implemented, `oscPort: 0` should disable the OSC listener, `oscBindAddress: "127.0.0.1"` should keep OSC local to the host, and `oscBindAddress: "0.0.0.0"` should listen on all IPv4 interfaces. `oscSmoothing` is reserved for a per-frame easing amount on numeric OSC controls.
## Address Pattern

16
docs/openapi.yaml
View File

@@ -23,7 +23,7 @@ tags:
- name: Layers
description: Layer stack control.
- name: Stack Presets
description: Save and recall layer stack presets.
description: Planned save/recall layer stack presets. The current native host exposes the routes but returns an unimplemented action result.
- name: Runtime
description: Runtime actions.
paths:
@@ -233,7 +233,7 @@ paths:
post:
tags: [Layers]
summary: Move a layer by direction
description: Moves a layer up or down by one or more positions, depending on the signed direction value.
description: Planned relative move route. The current native command path supports `/api/layers/reorder`; this route currently returns an unimplemented action result.
operationId: moveLayer
requestBody:
required: true
@@ -329,7 +329,8 @@ paths:
/api/stack-presets/save:
post:
tags: [Stack Presets]
summary: Save the current layer stack as a preset
summary: Planned preset save route
description: Planned preset route. The current native command path currently returns an unimplemented action result; latest-working-state autosave uses `runtime/runtime_state.json`.
operationId: saveStackPreset
requestBody:
required: true
@@ -345,7 +346,8 @@ paths:
/api/stack-presets/load:
post:
tags: [Stack Presets]
summary: Load a layer stack preset
summary: Planned preset load route
description: Planned preset route. The current native command path currently returns an unimplemented action result; latest-working-state startup restore uses `runtime/runtime_state.json`.
operationId: loadStackPreset
requestBody:
required: true
@@ -362,7 +364,7 @@ paths:
post:
tags: [Runtime]
summary: Reload shaders
description: Rescans the shader library, re-reads manifests, queues shader compilation, and refreshes shader availability/errors. If a changed shader fails, the previous working stack remains active where possible.
description: Rescans the shader library, re-reads manifests, refreshes shader availability/errors, reconciles active layer parameter definitions, and queues recompilation for every catalog-valid layer in the active stack. It does not compile every shader package in the library; packages are compiled when they are active layers. If a rebuild fails, the previous working render stack remains active where possible.
operationId: reloadShaders
requestBody:
required: false
@@ -379,8 +381,8 @@ paths:
/api/screenshot:
post:
tags: [Runtime]
summary: Queue a PNG screenshot of the final output render target
description: Captures the next completed output render target and writes it under `runtime/screenshots/`.
summary: Planned screenshot route
description: Planned screenshot route. The current native command path currently returns an unimplemented action result.
operationId: queueScreenshot
requestBody:
required: false

6
runtime/README.md
View File

@@ -17,9 +17,9 @@ Generated files:
- `shader_cache/active_shader_wrapper.slang`: generated Slang wrapper for the most recently compiled shader pass.
- `shader_cache/active_shader.raw.frag`: raw GLSL emitted by `slangc` for the most recently compiled pass.
- `shader_cache/active_shader.frag`: patched GLSL consumed by the OpenGL path for the most recently compiled pass.
- `runtime_state.json`: autosaved latest layer stack, layer order, bypass state, shader assignments, and parameter values. The host reloads this file on startup.
- `stack_presets/*.json`: user-saved layer stack presets.
- `screenshots/*.png`: screenshots captured from the final output render target through the control UI/API.
- `runtime_state.json`: debounced autosave of the latest layer stack, layer order, bypass state, shader assignments, and parameter values. The host reloads this file on startup and falls back to the configured default shader if the file is missing or unusable.
- `stack_presets/*.json`: planned user-saved layer stack presets. Preset routes are present in the API surface but not implemented in the current native host.
- `screenshots/*.png`: planned screenshot output. Screenshot capture is present in the API surface but not implemented in the current native host.
Git policy:

42
src/README.md
View File

@@ -78,6 +78,8 @@ Included now:
- local HTTP control server matching the OpenAPI route surface
- HTTP layer controls for add, remove, reorder, bypass, shader change, parameter update, and parameter reset
- trigger parameters as latest-pulse controls with shader-visible count/time
- startup restore from `runtime/runtime_state.json`
- debounced background autosave for durable layer-stack UI/API changes
- startup config provider for `config/runtime-host.json`
- quiet telemetry health monitor
- optional preview window fed from completed system-memory frames on its own thread
@@ -89,12 +91,10 @@ Intentionally not included yet:
- additional input format conversion/scaling
- temporal/history/feedback shader storage
- texture/LUT asset upload
- runtime state
- manual stack preset save/load
- OSC control
- persistent control/state writes
- trigger event history for stacked repeated pulses
- screenshots
- persistence
Those features should be ported only after the cadence spine is stable.
@@ -143,8 +143,10 @@ This tracks parity with `apps/LoopThroughWithOpenGLCompositing`.
- [x] CPU-side MSDF/MTSDF font atlas generation cache
- [x] Single-line text parameter rasterization and GL binding
- [ ] Trigger history/event buffers for overlapping repeated trigger effects
- [ ] Full runtime state store/read model
- [ ] Persistent layer stack/config writes
- [x] Startup restore from latest runtime layer state
- [x] Debounced background autosave for durable layer-stack changes
- [ ] Manual stack preset save/load
- [ ] Persistent config writes
- [ ] OSC ingress
- [x] Preview output from a non-consuming system-memory tap
- [ ] Screenshot capture
@@ -195,9 +197,9 @@ Currently consumed fields:
- `serverPort`
- `shaderLibrary`
- `oscBindAddress`
- `oscPort`
- `oscSmoothing`
- `oscBindAddress` (reported for compatibility; OSC ingress is not wired yet)
- `oscPort` (reported for compatibility; OSC ingress is not wired yet)
- `oscSmoothing` (reported for compatibility; OSC ingress is not wired yet)
- `inputVideoFormat`
- `inputFrameRate`
- `outputVideoFormat`
@@ -212,6 +214,8 @@ When `previewEnabled` is true, the preview window runs on `PreviewWindowThread`.
The loaded config is treated as a read-only startup snapshot. Subsystems that need config should receive this snapshot or a narrowed config struct from app orchestration; they should not reload files independently.
`autoReload` is exposed in config and state for compatibility, but automatic file watching is not currently wired. Use `POST /api/reload` or the control UI reload button to rescan shader files and manifests.
Supported CLI overrides:
- `--shader <shader-id>`
@@ -236,9 +240,10 @@ Current endpoints:
- `GET /docs/openapi.yaml` and `GET /openapi.yaml`: serves the OpenAPI document
- `GET /docs`: serves Swagger UI
- `POST /api/layers/add`, `/remove`, `/reorder`, `/set-bypass`, `/set-shader`, `/update-parameter`, and `/reset-parameters` use the shared runtime control-command path
- `POST /api/reload`: rescans the shader library, refreshes manifests, and queues recompilation for every catalog-valid layer in the active stack
- other OpenAPI POST routes are present but return `{ "ok": false, "error": "Endpoint is not implemented in RenderCadenceCompositor yet." }`
The HTTP server runs on its own thread. It serves static UI/docs files, samples/copies telemetry through callbacks, and translates POST bodies into runtime control commands. Command execution is app-owned, so future OSC ingress can create the same commands without depending on HTTP route code. Control commands may update the display layer model, start background shader builds, or publish an already-built render-layer snapshot, but they do not call render work or DeckLink scheduling directly.
The HTTP server runs on its own thread. It serves static UI/docs files, samples/copies telemetry through callbacks, and translates POST bodies into runtime control commands. Command execution is app-owned, so future OSC ingress can create the same commands without depending on HTTP route code. Control commands may update the display layer model, request debounced runtime-state persistence, start background shader builds, or publish an already-built render-layer snapshot, but they do not call render work or DeckLink scheduling directly.
## Optional DeckLink Output
@@ -332,17 +337,19 @@ Healthy first-run signs:
- `deckLinkScheduleRealignments` does not increase continuously
- `late` and `dropped` do not increase continuously
- `scheduleFailures` does not increase
- `shaderCommitted` becomes `1` after the background Happy Accident compile completes
- `shaderCommitted` increases after the restored/default runtime layer compiles and commits
- `shaderFailures` remains `0`
`completedPollMisses` means the DeckLink scheduling thread woke up before a completed frame was available. It is not a DeckLink playout underrun by itself. Treat it as healthy polling noise when `scheduled`, `decklinkBuffered`, `late`, `dropped`, and `scheduleFailures` remain stable.
## Runtime Slang Shader Test
## Runtime Slang Shader Stack
On startup the app begins compiling the selected shader package on a background thread owned by the app orchestration layer. The default is `shaders/happy-accident`.
On startup the app first tries to restore `runtime/runtime_state.json`. Valid saved layers are rebuilt in saved order, including shader id, bypass state, and parameter values. Missing shader packages are skipped, invalid saved parameter values fall back to manifest defaults, and if the runtime-state file is missing or unusable the app falls back to the configured shader package. The default configured shader is `shaders/happy-accident`.
The render thread keeps drawing the simple motion renderer while Slang compiles. It does not choose packages, launch Slang, or track build lifecycle. Once a completed shader artifact is published, the render-thread-owned runtime scene queues changed layers to a shared-context GL prepare worker. That worker compiles/links runtime shader programs off the cadence thread. The render thread only swaps in an already-prepared GL program at a frame boundary. If either the Slang build or GL preparation fails, the app keeps rendering the current renderer or simple motion fallback.
`POST /api/reload` rescans `shaders/`, re-reads manifests, refreshes supported shader metadata, reconciles active layer parameters against changed definitions, and queues recompilation for every catalog-valid layer in the active stack. It does not compile every package in the shader library; packages are compiled when they are part of the active stack.
Current runtime shader support is deliberately limited to stateless full-frame packages:
- one or more named passes
@@ -371,7 +378,7 @@ Shader source semantics:
The `/api/state` shader list uses the same support rules as runtime shader compilation and reports only packages this app can run today. Unsupported manifest feature sets such as temporal, feedback, and texture-backed shaders are hidden from the control UI for now.
Runtime shaders are exposed through `RuntimeLayerModel` as display layers with manifest parameter definitions, current parameter values, build status, and render-ready artifacts. POST controls mutate this app-owned model and may start background shader builds when the selected shader changes.
Runtime shaders are exposed through `RuntimeLayerModel` as display layers with manifest parameter definitions, current parameter values, build status, and render-ready artifacts. POST controls mutate this app-owned model and may start background shader builds when the selected shader changes or when `/api/reload` is requested. Durable UI/API layer changes request a debounced background write to `runtime/runtime_state.json`. OSC ingress is not wired yet; when it is added, OSC-driven changes should stay out of autosave unless an explicit persistence policy is introduced.
When a layer becomes render-ready, the app publishes the ready render-layer snapshot to the render thread. The render thread owns the GL-side `RuntimeRenderScene`, diffs that snapshot at a frame boundary, queues new or changed pass programs to the shared-context prepare worker, swaps in a full prepared render plan only after every pass is ready, removes obsolete GL programs, and renders ready layers in order. Stacked stateless full-frame shaders render through internal ping-pong targets so each layer can sample the previous layer through `gLayerInput`; multipass shaders route named intermediate outputs through their manifest-declared pass inputs, and the final ready layer renders to the output target.
@@ -429,7 +436,7 @@ This app keeps the same core behavior but splits it into modules that can grow:
- `render/readback/`: PBO-backed BGRA8 readback and completed-frame publication
- `render/runtime/RuntimeRenderScene`: render-thread-owned GL scene for ready runtime shader layers
- `render/runtime/RuntimeShaderPrepareWorker`: shared-context runtime shader program compile/link worker
- `runtime/`: app-owned shader layer readiness model, runtime Slang build bridge, and completed artifact handoff
- `runtime/`: app-owned shader layer readiness model, runtime Slang build bridge, runtime-state persistence, and completed artifact handoff
- `control/`: control action results and runtime-state JSON presentation
- `control/http/`: local HTTP API, static UI serving, OpenAPI serving, and WebSocket updates
- `json/`: compact JSON serialization helpers
@@ -445,7 +452,6 @@ Only after this app matches the probe's smooth output:
1. replace `SimpleMotionRenderer` with a render-scene interface
2. port shader package rendering
3. port runtime snapshots/live state
4. add control services
5. add preview/screenshot from system-memory frames
6. add scaling and additional input format support after the BGRA8/raw-UYVY8 input edge is stable
3. broaden runtime snapshots/live state toward OSC and presets
4. add screenshot capture from system-memory frames
5. add scaling and additional input format support after the BGRA8/raw-UYVY8 input edge is stable