video-shader-toys/README.md

# Video Shader

Native video shader host with an OpenGL render path, pluggable video I/O boundary, DeckLink and NDI backends, Slang shader packages, and a local React control UI.

The app loads shader packages from `shaders/`, compiles Slang to GLSL at runtime, renders a configurable layer stack, and exposes a browser-based control surface over a local HTTP/WebSocket server. Shader compilation is prepared off the frame path where possible, then committed on the render thread so editing shader files does not block video output for the whole compile.

## Repository Layout

- `src/`: native C++ host app.
- `shaders/`: shader packages, each with `shader.json` and `shader.slang`.
- `ui/`: Vite/React control UI.
- `config/runtime-host.json`: runtime configuration.
- `runtime/templates/`: tracked shader wrapper templates.
- `runtime/`: ignored generated runtime cache/state output. See `runtime/README.md`.
- `tests/`: focused native tests for pure runtime logic.
- `docs/FORKING_RENDER_CADENCE_BASE.md`: notes for forking the cadence/video I/O base while replacing the GPU-rendered content.
- `.gitea/workflows/ci.yml`: Gitea Actions CI for Windows native tests and Ubuntu UI build.

Native app internals are grouped by boundary:

- `app/`: startup/shutdown orchestration and runtime layer controller.
- `control/`: HTTP/WebSocket server, command parsing, and runtime-state JSON presentation.
- `frames/`: system-memory frame exchange and input mailbox handoff.
- `render/`: render thread, readback, runtime render scene, and shared-context shader program preparation.
- `runtime/`: shader catalog support, layer model, Slang build bridge, font atlas build, and runtime-state persistence.
- `shader/`: shader package parsing and Slang compilation helpers.
- `video/core/`: backend-neutral video IO handoff contracts, mode descriptions, pixel formats, and output scheduling thread.
- `video/decklink/`: DeckLink input/output backend.
- `video/ndi/`: NDI input/output backend.
- `video/playout/`: backend-adjacent playout policy, queues, frame pools, and scheduling helpers.
- `video/legacy/`: older backend pipeline pieces kept separate while the new edge model settles.

The runtime shader stack is plugged into the host through `src/app/RuntimeContentController.h`. The checked-in app wires `ShaderRuntimeContentController` into that slot; a fork can provide another runtime-content controller for a larger render engine while keeping the cadence/video I/O shell.

## Requirements

- Windows with Visual Studio 2022 C++ tooling.
- CMake 3.24 or newer.
- Node.js and npm for the control UI.
- Blackmagic Desktop Video drivers and a DeckLink device for the current production video I/O backend.
- Blackmagic DeckLink SDK 16.0 for DeckLink SDK reference files and redistribution notices.
- NDI 6 SDK for NDI input/output builds. CMake expects `Include/`, `Lib/x64/Processing.NDI.Lib.x64.lib`, `Bin/x64/Processing.NDI.Lib.x64.dll`, and the NDI license/notice files.
- Slang 2026.8 Windows x86_64 binary release with `bin/slangc.exe`, `bin/slang-compiler.dll`, `bin/slang-glslang.dll`, and `LICENSE`.
- `msdf-atlas-gen` Windows binary release with `msdf-atlas-gen.exe`, `LICENSE.txt`, `README.md`, and any adjacent runtime DLLs for font atlas generation.

### Third-party SDK bundle

Org members can initialize the private SDK bundle submodule:

```powershell
git submodule update --init --recursive
```

When present, CMake defaults to this private bundle:

```text
video-io-3rdParty/
  Blackmagic DeckLink SDK 16.0/
  NDI 6 SDK/
  slang-2026.8-windows-x86_64/
  msdf-atlas-gen/
```

The parent repository is public, but this bundle is private. External builders need to obtain the SDKs from their vendors and place them in an ignored local `3rdParty/` folder with the same layout, or pass explicit CMake paths.

Fallback local Slang path:

```text
3rdParty/slang-2026.8-windows-x86_64
```

Fallback local `msdf-atlas-gen` path:

```text
3rdParty/msdf-atlas-gen
```

Fallback local NDI SDK path:

```text
3rdParty/NDI 6 SDK
```

Fallback local Blackmagic DeckLink SDK path:

```text
3rdParty/Blackmagic DeckLink SDK 16.0
```

Single-root override example:

```powershell
cmake --preset vs2022-x64-debug -DTHIRD_PARTY_ROOT="D:/SDKs/video-io-3rdParty"
```

Individual override example:

```powershell
cmake --preset vs2022-x64-debug `
  -DSLANG_ROOT="D:/SDKs/slang-2026.8-windows-x86_64" `
  -DMSDF_ATLAS_GEN_ROOT="D:/SDKs/msdf-atlas-gen" `
  -DNDI_SDK_ROOT="D:/SDKs/NDI 6 SDK" `
  -DDECKLINK_SDK_ROOT="D:/SDKs/Blackmagic DeckLink SDK 16.0"
```

At runtime, Slang compilation follows the same root order: `SLANG_ROOT`, `THIRD_PARTY_ROOT`, repo `video-io-3rdParty/`, then repo or packaged `3rdParty/`. The packaged layout uses `3rdParty/slang/bin/slangc.exe`; development bundles can use `slang-2026.8-windows-x86_64/bin/slangc.exe`.

## Build

Configure and build the native app:

```powershell
cmake --preset vs2022-x64-debug
cmake --build --preset build-debug --parallel
```

Build the React control UI:

```powershell
cd ui
npm ci
npm run build
```

The native app serves `ui/dist` when it exists, otherwise it falls back to the source UI directory during development.

The control UI provides:

- A searchable shader library for adding layers.
- Compact parameter rows with inline descriptions and intended OSC route copy controls.
- Manual shader reload.

## Package

Build the UI, build the native Release target, then install into a portable runtime folder:

```powershell
cd ui
npm ci
npm run build
cd ..
cmake --preset vs2022-x64-release
cmake --build --preset build-release --parallel
cmake --install build/vs2022-x64-release --config Release --prefix dist/VideoShader
```

The package folder will contain:

```text
dist/VideoShader/
  RenderCadenceCompositor.exe
  Processing.NDI.Lib.x64.dll
  config/
  shaders/
  3rdParty/slang/bin/
  3rdParty/msdf-atlas-gen/
  ui/dist/
  docs/
  SHADER_CONTRACT.md
  runtime/templates/
  third_party_notices/
```

You can run `RenderCadenceCompositor.exe` directly from that folder. In packaged mode, the app resolves `config/`, `shaders/`, `3rdParty/slang/bin/slangc.exe`, `3rdParty/msdf-atlas-gen/msdf-atlas-gen.exe`, `Processing.NDI.Lib.x64.dll`, `ui/dist/`, and `runtime/templates/` relative to the exe folder. In development mode, it still falls back to repo-root discovery.

The install step copies only the Slang runtime files required by the shader compiler (`slangc.exe`, `slang-compiler.dll`, and `slang-glslang.dll`) plus `third_party_notices/SLANG_LICENSE.txt`. It also copies `msdf-atlas-gen.exe`, any adjacent `msdf-atlas-gen` DLLs, `Processing.NDI.Lib.x64.dll`, and the `third_party_notices/MSDF_ATLAS_GEN_LICENSE.txt`, `third_party_notices/MSDF_ATLAS_GEN_README.md`, `third_party_notices/NDI_SDK_LICENSE_AGREEMENT.pdf`, and `third_party_notices/NDI_RUNTIME_LICENSES.txt` notice files. It does not copy full third-party release folders.

Create a zip for distribution:

```powershell
Compress-Archive -Path dist/VideoShader/* -DestinationPath dist/VideoShader.zip -Force
```

## Tests

Run native tests:

```powershell
cmake --build --preset build-debug --target RUN_TESTS --parallel
```

Run the UI production build check:

```powershell
cd ui
npm run build
```

Current native test coverage includes:

- JSON parsing and serialization.
- Parameter normalization and preset filename safety.
- Shader manifest parsing, temporal manifest validation, and package registry scanning.
- Video I/O format helpers, v210/Ay10 row-byte math, v210 pack/unpack math, playout scheduler timing, and fake backend contract coverage.
- Slang validation for every available shader package.

## Runtime Configuration

`config/runtime-host.json` controls host behavior:

```json
{
  "shaderLibrary": "shaders",
  "serverPort": 8080,
  "oscBindAddress": "127.0.0.1",
  "oscPort": 9000,
  "oscSmoothing": 0.18,
  "input": {
    "backend": "decklink",
    "device": "default",
    "resolution": "1080p",
    "frameRate": "59.94"
  },
  "output": {
    "backend": "decklink",
    "device": "default",
    "resolution": "1080p",
    "frameRate": "59.94",
    "pixelFormat": "auto",
    "keying": {
      "external": true,
      "alphaRequired": true
    }
  },
  "autoReload": true,
  "maxTemporalHistoryFrames": 12
}
```

`input.backend` and `output.backend` select the concrete video I/O backend. Today the app supports `decklink`, `ndi`, and `none`; future backends such as Spout or file playback can be added behind the same factory boundary. `device` is a backend-neutral selector placeholder: DeckLink still chooses the first compatible device, NDI input uses it as the source selector, and NDI output uses it as the sender name.

`input.resolution`/`input.frameRate` select the video capture mode. `output.resolution`/`output.frameRate` select the playout mode through a backend-neutral mode description; the current DeckLink backend maps that mode to a `BMDDisplayMode` at the DeckLink boundary. Supported modes still depend on the installed card and driver. The shader stack runs at input resolution and the final rendered frame is scaled once into the configured output mode. Common examples include `720p`/`50`, `720p`/`59.94`, `1080i`/`50`, `1080i`/`59.94`, `1080p`/`25`, `1080p`/`50`, `1080p`/`59.94`, and `2160p`/`59.94`.

The checked-in config uses the nested `input` and `output` objects as the supported shape. When `input.backend` is `ndi`, the host-config editor uses NDI discovery to offer source-name suggestions in the `input.device` field while still allowing manual entry. The control UI presents `output.keying.external` and `output.keying.alphaRequired` as one **Output alpha** control; DeckLink maps that to external keying, while NDI only uses it to request an alpha-carrying system-frame format.

The control UI is available at:

```text
http://127.0.0.1:<serverPort>
```

`/api/state` exposes backend-neutral output telemetry in `videoOutput`. Use `videoOutput.enabled`, `videoOutput.backend`, and `videoOutput.scheduleFailures` for portable status. Backend-specific counters live in `videoOutput.backendMetrics`.

## Runtime State

The current layer stack is autosaved to `runtime/runtime_state.json` whenever durable UI/API layer changes are accepted: add/remove, shader assignment, bypass state, ordering, parameter updates, parameter reset, and reload compatibility refreshes. Saves are debounced and written on a background worker, with a final flush during shutdown.

On startup, the host tries to reload `runtime/runtime_state.json` before compiling the stack. Valid saved layers are rebuilt in saved order, with shader id, bypass state, and parameter values restored. Missing shader packages are skipped, invalid saved parameter values fall back to shader defaults, and if the file is missing or unusable the app falls back to the configured default shader.

Manual stack preset and screenshot routes are still present in the UI/OpenAPI surface, but they are not implemented by the current native command path yet. `runtime_state.json` is the supported latest-working-state mechanism for now.

## Control API

The local REST control API is documented as an OpenAPI/Swagger spec:

```text
docs/openapi.yaml
```

When the control server is running, the same spec is also served at:

```text
http://127.0.0.1:<serverPort>/docs/openapi.yaml
http://127.0.0.1:<serverPort>/openapi.yaml
```

A Swagger UI page is available at:

```text
http://127.0.0.1:<serverPort>/docs
```

Use those docs to inspect the `/api/state`, layer control, and reload endpoints. Live state updates are also sent over the `/ws` WebSocket.

The control UI has a **Reload shaders** button. It rescans `shaders/`, re-reads manifests, refreshes shader availability/errors, updates active layer parameter definitions from changed manifests, and queues recompilation for every catalog-valid layer in the active stack. Missing shader packages are marked failed, and the previous working render stack remains active where possible until replacement builds commit successfully.

Each parameter row still exposes the intended OSC route in the UI. The native host has an OSC service stub that reports the configured bind/port in state, but it does not open a UDP listener or dispatch OSC messages yet.

The control UI currently still shows preset and screenshot controls from the intended route surface. Those endpoints return an unimplemented action result in the native host until their backend paths are wired.

The planned screenshot output directory is:

```text
runtime/screenshots/
```

## OSC Control

OSC fields are present in `config/runtime-host.json` and `/api/state` for compatibility with the intended control surface, but the current native host does not start an OSC listener yet.

The intended route shape is:

```text
/VideoShaderToys/{LayerNameOrID}/{ParameterNameOrID}
```

For now, use the REST layer parameter endpoints or the control UI for live parameter changes. Future OSC-driven parameter changes should stay out of autosave unless an explicit persistence policy is added.

## Shader Packages

Each shader package lives under:

```text
shaders/<id>/
  shader.json
  shader.slang
  optional-extra-pass.slang
  optional-font-or-texture-assets
```

See `SHADER_CONTRACT.md` for the manifest schema, parameter types, texture assets, font/text assets, temporal history support, optional render-pass declarations, and the Slang entry point contract. `shaders/text-overlay/` is the reference live text package and bundles Roboto Regular with its OFL license. Broken shader packages are shown as unavailable in the selector with their error text instead of preventing the app from launching.

## Generated Files

Runtime-generated files are intentionally ignored:

- `runtime/shader_cache/active_shader_wrapper.slang`
- `runtime/shader_cache/active_shader.raw.frag`
- `runtime/shader_cache/active_shader.frag`
- `runtime/runtime_state.json` autosaved latest stack and parameter state.
- `runtime/stack_presets/*.json` planned manual preset output; preset routes are not implemented in the native host yet.
- `runtime/screenshots/*.png` planned screenshot output; screenshot capture is not implemented in the native host yet.

Only `runtime/templates/` and `runtime/README.md` are tracked.

## CI

The Gitea workflow expects two act runners:

- `windows-2022`: builds the native app and runs native tests.
- `ubuntu-latest`: installs UI dependencies and runs the Vite build.

The Windows jobs validate native third-party dependencies before configuring CMake. Because `3rdParty/` is ignored, configure this path on the runner or in a Gitea repository variable:

- `THIRD_PARTY_ROOT`: path to a bundle containing the expected SDK folder layout.
- `SLANG_ROOT`: path to the Slang binary release folder containing `bin/slangc.exe`.
- `MSDF_ATLAS_GEN_ROOT`: path to the `msdf-atlas-gen` binary release folder containing `msdf-atlas-gen.exe`.
- `NDI_SDK_ROOT`: path to the NDI SDK folder containing `Include/`, `Lib/x64/`, and `Bin/x64/`.
- `DECKLINK_SDK_ROOT`: path to the Blackmagic DeckLink SDK folder containing `Win/include/DeckLinkAPI.idl`.

The Windows runner also needs the Visual Studio ATL component installed. In Visual Studio Build Tools 2022, add `C++ ATL for latest v143 build tools (x86 & x64)`, component ID `Microsoft.VisualStudio.Component.VC.ATL`.

Example runner paths:

```text
D:\SDKs\slang-2026.8-windows-x86_64
D:\SDKs\msdf-atlas-gen
D:\SDKs\NDI 6 SDK
D:\SDKs\Blackmagic DeckLink SDK 16.0
```

If these variables are not set, CMake first looks under the private `video-io-3rdParty/` submodule and then falls back to repo-local defaults under ignored `3rdParty/`.

## Still Todo

- Audio.
- Genlock.
- Logs.
- Support a separate sound shader `.slang` file in shader packages. (https://www.shadertoy.com/view/XsBXWt)
- Add WebView2 for an embedded native control surface.
- More shader-library organisation and filtering as the built-in library grows.
- Optional linear-light compositing mode.
- compute shaders or a small 1x1 or nx1 RGBA16f render target for arbitrary data storage
- allow shaders to read other shaders data store based on name? or output over OSC
- Mipmapping for shader-declared textures
- Anotate included shaders
- allow 3 vector exposed controls
- add nearest sampling to the extra shader pass
- add spout input/output (https://github.com/leadedge/Spout2)
- Add Aja input and output (Assuming i can get a hold of an aja card)
- Add bluefish input and output (Assuming again card acess)
- Endpoint to show OSC paths seperatly instead of a part of the control UI


## Custom shader UI
Extend the shader manifest contract
Add optional UI metadata:

"ui": {
  "type": "webComponent",
  "entry": "ui/controls.js",
  "tag": "my-shader-controls"
}
Keep this optional. No custom UI means current default controls.

Expose UI metadata in /api/state
Add the parsed ui block to each shader/layer summary so the React app knows whether a layer has a custom control panel.

Serve shader package UI assets safely
Add a route like:

/shader-assets/{shaderId}/ui/controls.js
It should only serve files inside that shader package folder.

Add a React host component
Create something like ShaderCustomPanel.jsx that:

dynamically imports/registers the custom element
passes layer, parameters, and setParameter
catches load/render failures
falls back to the normal ParameterField grid
Define the custom element API
Keep it small and stable:

element.layer = layer;
element.parameters = layer.parameters;
element.setParameter = (id, value) => {};
element.requestReset = () => {};
Custom UI should never bypass manifest validation.

Add fallback and escape hatch
Even if custom UI loads, provide a “Default controls” toggle per layer. That is the life raft.

Add tests
Backend:

manifest parser accepts valid UI blocks
rejects unsafe paths like ../
/api/state includes UI metadata
asset route refuses files outside the shader package
Frontend:

falls back when custom component fails
calls /api/layers/update-parameter through the same path as default controls
Document the contract
Add a section to shaders/SHADER_CONTRACT.md with:

manifest example
custom element lifecycle
available properties/functions
rule: all controls must map to declared parameters