video-shader-toys/README.md

# Video Shader

Native video shader host with an OpenGL/DeckLink render path, Slang shader packages, and a local React control UI.

The app loads shader packages from `shaders/`, compiles Slang to GLSL at runtime, renders a configurable layer stack, and exposes a browser-based control surface over a local HTTP/WebSocket server.

## Repository Layout

- `apps/LoopThroughWithOpenGLCompositing/`: native C++ host app.
- `shaders/`: shader packages, each with `shader.json` and `shader.slang`.
- `ui/`: Vite/React control UI.
- `config/runtime-host.json`: runtime configuration.
- `runtime/templates/`: tracked shader wrapper templates.
- `runtime/`: ignored generated runtime cache/state output. See `runtime/README.md`.
- `tests/`: focused native tests for pure runtime logic.
- `.gitea/workflows/ci.yml`: Gitea Actions CI for Windows native tests and Ubuntu UI build.

## Requirements

- Windows with Visual Studio 2022 C++ tooling.
- CMake 3.24 or newer.
- Node.js and npm for the control UI.
- Blackmagic DeckLink SDK 16.0 with the NVIDIA GPUDirect sample files available locally.
- Slang binary release with `slangc.exe`, `slang-compiler.dll`, `slang-glslang.dll`, and `LICENSE`.

The Blackmagic/GPUDirect SDK should not be committed to this repository. `CMakeLists.txt` exposes `GPUDIRECT_DIR` as a cache path so local machines and CI runners can point at their installed SDK location.

Default expected SDK path:

```text
3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/NVIDIA_GPUDirect
```

Override example:

```powershell
cmake --preset vs2022-x64-debug -DGPUDIRECT_DIR="D:/SDKs/Blackmagic DeckLink SDK 16.0/Win/Samples/NVIDIA_GPUDirect"
```

Default expected Slang path:

```text
3rdParty/slang-2026.8-windows-x86_64
```

Override example:

```powershell
cmake --preset vs2022-x64-debug -DSLANG_ROOT="D:/SDKs/slang-2026.8-windows-x86_64"
```

## Build

Configure and build the native app:

```powershell
cmake --preset vs2022-x64-debug
cmake --build --preset build-debug
```

Build the React control UI:

```powershell
cd ui
npm ci
npm run build
```

The native app serves `ui/dist` when it exists, otherwise it falls back to the source UI directory during development.

## Package

Build the UI, build the native Release target, then install into a portable runtime folder:

```powershell
cd ui
npm ci
npm run build
cd ..
cmake --preset vs2022-x64-release
cmake --build --preset build-release
cmake --install build/vs2022-x64-release --config Release --prefix dist/VideoShader
```

The package folder will contain:

```text
dist/VideoShader/
  LoopThroughWithOpenGLCompositing.exe
  dvp.dll
  config/
  shaders/
  3rdParty/slang/bin/
  ui/dist/
  runtime/templates/
  third_party_notices/
```

You can run `LoopThroughWithOpenGLCompositing.exe` directly from that folder. In packaged mode, the app resolves `config/`, `shaders/`, `3rdParty/slang/bin/slangc.exe`, `ui/dist/`, and `runtime/templates/` relative to the exe folder. In development mode, it still falls back to repo-root discovery.

The install step copies only the Slang runtime files required by the shader compiler (`slangc.exe`, `slang-compiler.dll`, and `slang-glslang.dll`) plus `third_party_notices/SLANG_LICENSE.txt`. It does not copy the full Slang release folder.

Create a zip for distribution:

```powershell
Compress-Archive -Path dist/VideoShader/* -DestinationPath dist/VideoShader.zip -Force
```

## Tests

Run native tests:

```powershell
cmake --build --preset build-debug --target RUN_TESTS
```

Run the UI production build check:

```powershell
cd ui
npm run build
```

Current native test coverage includes:

- JSON parsing and serialization.
- Parameter normalization and preset filename safety.
- Shader manifest parsing, temporal manifest validation, and package registry scanning.

## Runtime Configuration

`config/runtime-host.json` controls host behavior:

```json
{
  "shaderLibrary": "shaders",
  "serverPort": 8080,
  "oscPort": 9000,
  "inputVideoFormat": "1080p",
  "inputFrameRate": "59.94",
  "outputVideoFormat": "1080p",
  "outputFrameRate": "59.94",
  "autoReload": true,
  "maxTemporalHistoryFrames": 12,
  "enableExternalKeying": true
}
```

`inputVideoFormat`/`inputFrameRate` select the DeckLink capture mode. `outputVideoFormat`/`outputFrameRate` select the playout mode. The shader stack runs at input resolution and the final rendered frame is scaled once into the configured output mode. Common examples include `720p`/`50`, `720p`/`59.94`, `1080i`/`50`, `1080i`/`59.94`, `1080p`/`25`, `1080p`/`50`, `1080p`/`59.94`, and `2160p`/`59.94`, depending on card support.

Legacy `videoFormat` and `frameRate` keys are still accepted and apply to both input and output unless the explicit input/output keys are present.

The control UI is available at:

```text
http://127.0.0.1:<serverPort>
```

## Runtime State And Presets

The current layer stack is autosaved to `runtime/runtime_state.json` whenever layers, shader assignments, bypass state, ordering, or parameter values change. On startup, the host reloads that file before compiling the stack, so the last working stack should come back automatically.

Manual stack presets are still available from the control UI and are saved under `runtime/stack_presets/*.json`. Presets are useful for named looks, while `runtime_state.json` is the latest working state for the local machine.

## Control API

The local REST control API is documented as an OpenAPI/Swagger spec:

```text
docs/openapi.yaml
```

When the control server is running, the same spec is also served at:

```text
http://127.0.0.1:<serverPort>/docs/openapi.yaml
http://127.0.0.1:<serverPort>/openapi.yaml
```

A Swagger UI page is available at:

```text
http://127.0.0.1:<serverPort>/docs
```

Use those docs to inspect the `/api/state`, layer control, stack preset, and reload endpoints. Live state updates are also sent over the `/ws` WebSocket.

## OSC Control

The native host also listens for local OSC parameter control on the configured `oscPort`:

```text
/VideoShaderToys/{LayerNameOrID}/{ParameterNameOrID}
```

For example, `/VideoShaderToys/VHS/intensity` updates the `intensity` parameter on the first matching `VHS` layer. The listener accepts float, integer, string, and boolean OSC values, and validates them through the same shader parameter path as the REST API. See `docs/OSC_CONTROL.md` for details.

## Shader Packages

Each shader package lives under:

```text
shaders/<id>/
  shader.json
  shader.slang
  optional-font-or-texture-assets
```

See `SHADER_CONTRACT.md` for the manifest schema, parameter types, texture assets, font/text assets, temporal history support, and the Slang entry point contract. `shaders/text-overlay/` is the reference live text package and bundles Roboto Regular with its OFL license.

## Generated Files

Runtime-generated files are intentionally ignored:

- `runtime/shader_cache/active_shader_wrapper.slang`
- `runtime/shader_cache/active_shader.raw.frag`
- `runtime/shader_cache/active_shader.frag`
- `runtime/runtime_state.json` autosaved latest stack and parameter state.
- `runtime/stack_presets/*.json`

Only `runtime/templates/` and `runtime/README.md` are tracked.

## CI

The Gitea workflow expects two act runners:

- `windows-2022`: builds the native app and runs native tests.
- `ubuntu-latest`: installs UI dependencies and runs the Vite build.

The Windows jobs validate native third-party dependencies before configuring CMake. Because `3rdParty/` is ignored, configure these paths on the runner or in Gitea repository variables:

- `GPUDIRECT_DIR`: path to `Blackmagic DeckLink SDK 16.0/Win/Samples/NVIDIA_GPUDirect`.
- `SLANG_ROOT`: path to the Slang binary release folder containing `bin/slangc.exe`.

The Windows runner also needs the Visual Studio ATL component installed. In Visual Studio Build Tools 2022, add `C++ ATL for latest v143 build tools (x86 & x64)`, component ID `Microsoft.VisualStudio.Component.VC.ATL`.

Example runner paths:

```text
D:\SDKs\Blackmagic DeckLink SDK 16.0\Win\Samples\NVIDIA_GPUDirect
D:\SDKs\slang-2026.8-windows-x86_64
```

If neither variable is set, the workflow falls back to the repo-local defaults under `3rdParty/`.

## Still Todo

- Audio.
- Genlock.
- Find a better UI library for react.
- Logs.
- Continue source cleanup/refactoring. Pass 2 done
- Support a separate sound shader `.slang` file in shader packages. (https://www.shadertoy.com/view/XsBXWt)
- Add WebView2
- move to MSDF, typography rasterisation
- better shader search UI
- LUT applicator
- More comprehensive greenscreen shader
- 16bit float render target, linear compositing?
- add 10bit output path decklink
- compute shaders or a small 1x1 or nx1 RGBA16f render target for abritary data store
- allow shaders to read other shaders data store based on name? or putput over OSC
- Mipmappong needed?
- unwrap a fish eyelens and mirror it and map it to equirectangulr for environmnet map purposes