video-shader-toys/docs/CURRENT_SYSTEM_ARCHITECTURE.md

Current System Architecture

This document describes how the current RenderCadenceCompositor app works.

The implementation is cadence-first: the render/output path owns frame timing, while shader compilation, HTTP control, preview, persistence, and video I/O edges stay outside the render cadence loop wherever possible. The guardrails in Render Cadence Golden Rules are the operating contract.

Application Shape

The app is a native C++ OpenGL compositor with:

• optional DeckLink input
• optional DeckLink scheduled output
• a render-thread-owned OpenGL context
• a pluggable app-side runtime-content controller
• the default runtime Slang shader package stack from shaders/
• a local HTTP/WebSocket control server
• optional Win32/GDI preview from system-memory output frames
• background runtime-state persistence
• cadence telemetry and log output

Primary source areas:

• src/app: startup/shutdown orchestration, config loading, video backend factory, runtime-content controller boundary, and the default shader runtime-content controller
• src/render: cadence clock, input texture upload, render-content boundary, readback, and runtime GL support
• src/render/thread: render thread lifecycle, cadence loop, metrics, and runtime shader commit mailbox
• src/render/runtime: render-thread-owned runtime shader scene, renderer, text texture upload cache, and shared-context shader prepare worker
• src/frames: system-memory frame exchange
• src/video/core: generic video IO edge contracts, mode descriptions, formats, and output scheduling thread
• src/video/decklink: current DeckLink input/output backend
• src/video/playout: backend-adjacent playout policy, queues, frame pools, and scheduling helpers
• src/video/legacy: older backend pipeline pieces kept separate from the current app path
• src/runtime/catalog: supported shader catalog and package filtering
• src/runtime/layers: app-side runtime layer model, restore, reload, and render snapshot construction
• src/runtime/shader: background Slang build bridge and prepared shader artifact types
• src/runtime/state: runtime JSON helpers, parameter normalization, and debounced runtime-state persistence
• src/runtime/text: MSDF/MTSDF font atlas build and CPU-side prepared text texture composition
• src/control: command parsing, HTTP/WebSocket transport helpers, OSC status stub, OpenAPI state JSON
• src/app/RenderCadenceHttpRoutes.*: this app's current HTTP endpoint map
• src/preview: optional non-consuming preview window
• src/telemetry and src/logging: runtime observation and logging

Startup

Startup broadly proceeds as:

1. Load config/runtime-host.json through AppConfigProvider, then apply CLI overrides.
2. Initialize the active IRuntimeContentController. The checked-in app uses ShaderRuntimeContentController, which loads the supported shader catalog, starts runtime-state persistence, and tries to restore runtime/runtime_state.json.
3. If restore fails or no usable state exists, the shader controller falls back to the optional configured startup shader. The checked-in config leaves runtimeShaderId empty, so a fresh host keeps the simple fallback renderer.
4. Start the render thread.
5. Start the active runtime-content controller. The shader controller queues background Slang builds for every pending active layer.
6. Build a small completed-frame reserve.
7. Start optional preview, optional video output, telemetry, and HTTP control.

The runtime-state restore is intentionally app/control side work. The render thread does not read JSON, inspect the shader library, or decide what to compile.

Runtime Content And Shader Layer State

RenderCadenceApp owns startup, video output, preview, telemetry, OSC status, and HTTP server lifetime. It does not own the Slang shader stack directly. Runtime content plugs in through IRuntimeContentController.

The checked-in implementation is ShaderRuntimeContentController. It wraps RuntimeLayerController, exposes shader catalog/layer JSON for /api/state, handles shader layer POST commands, and publishes render-ready shader layer snapshots to the render thread.

RuntimeLayerController owns the app-side layer model and coordinates:

• supported shader catalog loading
• layer add/remove/reorder/bypass/shader assignment
• parameter update/reset
• startup restore
• reload reconciliation
• background Slang builds
• render-layer publication
• runtime-state persistence requests

RuntimeLayerModel owns the in-memory active stack:

• layer id
• shader id and display name
• build state and message
• bypass state
• manifest parameter definitions
• optional shader-declared custom UI metadata
• current parameter values
• render-ready artifacts

The current durable runtime state is stored in runtime/runtime_state.json. It contains the active stack order, shader ids, bypass flags, and parameter values. On startup, valid saved layers are restored in order. Missing shader packages are skipped, invalid saved parameter values fall back to manifest defaults, and a missing or unusable file falls back to the optional configured startup shader.

Manual stack preset routes are present in the UI/OpenAPI surface but are not implemented in the current native command path yet. runtime_state.json is the supported latest-working-state mechanism.

Persistence

RuntimeStatePersistenceWriter performs debounced background writes to runtime/runtime_state.json.

Durable UI/API mutations request persistence after they are accepted:

• add/remove layer
• reorder layer
• set bypass
• set shader
• update parameter
• reset parameters
• reload compatibility refresh

The mutation path snapshots the current layer model and hands serialized state to the writer. File IO happens on the persistence worker, not on the render thread or cadence path. Shutdown flushes the latest pending snapshot.

OSC-driven changes are intentionally not part of this autosave path yet.

The host configuration editor is separate from runtime layer persistence. The UI reads active and saved startup config through /api/config, saves config/runtime-host.json through /api/config/save, and requests a native host restart through /api/app/restart. Render cadence, video input/output selection, resolution, frame rate, output pixel format, HTTP port, and preview settings are still startup-owned; they are not hot-swapped inside the cadence path.

Shader Reload

POST /api/reload and the control UI reload button:

• rescan shaders/
• re-read manifests
• rebuild the supported shader catalog
• refresh optional shader custom UI metadata
• update active layer metadata and parameter definitions from changed manifests
• preserve compatible parameter values
• default new or incompatible parameter values
• queue recompilation for every catalog-valid layer in the active stack

Reload does not compile every package in the shader library. A package is compiled when it is part of the active layer stack. If an active layer references a shader that no longer exists, that layer is marked failed and skipped. Existing render output remains active where possible until replacement builds are ready.

autoReload is still exposed in config/state for compatibility, but automatic file watching is not currently wired.

Render Ownership

The render thread owns the app OpenGL context during normal operation.

The render path consumes published render-layer snapshots. It does not:

• parse JSON
• scan files
• launch Slang
• run font atlas generation
• perform persistence
• handle HTTP or OSC
• call DeckLink discovery/setup APIs

When a runtime shader build completes, the default shader runtime-content controller publishes a render-layer artifact. The render thread forwards pending layer snapshots to the active render-content adapter. The default RuntimeShaderRenderContent owns the runtime scene, diffs the snapshot, and queues changed pass programs to the shared-context prepare worker. The render thread swaps in an already-prepared render plan at a frame boundary through that adapter.

Video And Preview

Video input and output are optional edges. input.backend and output.backend select the concrete backend through the app-side backend factory. DeckLink and NDI are the current concrete backends, and none disables an edge. input and output also carry the device selector plus resolution/frame-rate settings. Configured video modes are represented in src/video/core and translated to backend-specific modes only inside the concrete edge.

The input edge writes CPU frames into InputFrameMailbox. The current DeckLink backend captures BGRA8 directly where possible, or raw UYVY8 for render-thread GPU decode. The input edge does not call GL, render, preview, screenshot, shader, or output scheduling code.

The output edge consumes completed system-memory frames from SystemFrameExchange. The render thread owns output pixel packing before readback: BGRA8 is read directly, and UYVY8 is packed on the GPU into a half-width RGBA8 target before async PBO readback. DeckLink and NDI output then schedule/send those completed CPU frames without invoking GL or converting pixels. If video output is unavailable, the app continues running render cadence, control, preview, telemetry, and logging.

Runtime state exposes backend-neutral output telemetry through videoOutput. Portable fields such as enabled, backend, and scheduleFailures stay at that level; backend-specific counters live under videoOutput.backendMetrics.

PreviewWindowThread is optional and uses a non-consuming system-memory tap. It paints BGRA8 directly, decodes UYVY8 only for preview presentation, and skips preview ticks instead of blocking the frame exchange.

Screenshot routes are present in the UI/OpenAPI surface but are not implemented in the current native command path yet.

Control Surface

The HTTP server runs on its own thread. HttpControlServer owns socket lifetime, HTTP parsing, static asset helpers, OpenAPI/Swagger helper serving, and WebSocket state transport. RenderCadenceHttpRoutes owns this app's current endpoint map:

• UI assets
• shader package custom UI assets under /shader-assets/{shaderId}/...
• OpenAPI/Swagger docs
• GET /api/state
• /ws state updates
• layer mutation POST routes, dispatched to the active runtime-content controller
• /api/reload

Known but not implemented in the current native command path:

• /api/layers/move
• /api/stack-presets/save
• /api/stack-presets/load
• /api/screenshot

Unsupported routes return an action response with ok: false.

Forks can reuse the HTTP/WebSocket shell without keeping these endpoints by installing a different route callback.

OscControlServer is currently a lifecycle/status stub. It consumes startup OSC config, exposes configured/disabled/not-listening state through /api/state, and leaves UDP socket receive, OSC decode, and runtime dispatch unimplemented until that ingress boundary is built deliberately.

Tests

Native tests cover the main non-GL contracts:

• JSON parsing/serialization
• runtime parameter normalization
• shader package registry and Slang validation
• supported shader catalog
• runtime layer restore/reload behavior
• runtime-state persistence writer
• HTTP transport and app-route dispatch
• frame exchange and input mailbox behavior
• video format and scheduling helpers

Run:

cmake --build --preset build-debug --target RUN_TESTS --parallel