506 lines
13 KiB
Markdown
506 lines
13 KiB
Markdown
# Shader Package Contract
|
|
|
|
This document explains how to create shaders for the Video Shader runtime.
|
|
|
|
Each shader is a small package under `shaders/<id>/`:
|
|
|
|
```text
|
|
shaders/my-effect/
|
|
shader.json
|
|
shader.slang
|
|
optional-texture.png
|
|
```
|
|
|
|
The runtime reads `shader.json`, generates a Slang wrapper from `runtime/templates/shader_wrapper.slang.in`, includes your `shader.slang`, compiles the result to GLSL, and exposes the shader in the local control UI.
|
|
|
|
## Quick Start
|
|
|
|
Create a folder:
|
|
|
|
```text
|
|
shaders/my-effect/
|
|
```
|
|
|
|
Add `shader.json`:
|
|
|
|
```json
|
|
{
|
|
"id": "my-effect",
|
|
"name": "My Effect",
|
|
"description": "A simple starter shader.",
|
|
"category": "Custom",
|
|
"entryPoint": "shadeVideo",
|
|
"parameters": [
|
|
{
|
|
"id": "strength",
|
|
"label": "Strength",
|
|
"type": "float",
|
|
"default": 0.5,
|
|
"min": 0.0,
|
|
"max": 1.0,
|
|
"step": 0.01
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Add `shader.slang`:
|
|
|
|
```slang
|
|
float4 shadeVideo(ShaderContext context)
|
|
{
|
|
float4 color = context.sourceColor;
|
|
color.rgb = lerp(color.rgb, 1.0 - color.rgb, strength);
|
|
return saturate(color);
|
|
}
|
|
```
|
|
|
|
With `autoReload` enabled in `config/runtime-host.json`, edits to shader source, manifests, and declared texture assets are picked up automatically.
|
|
|
|
## Manifest Fields
|
|
|
|
`shader.json` is the runtime-facing description of the shader.
|
|
|
|
Required fields:
|
|
|
|
- `id`: package ID used by state/presets. Hyphenated names are OK here, for example `my-effect`.
|
|
- `name`: display name in the UI.
|
|
- `parameters`: array of exposed controls. Use `[]` if there are no user parameters.
|
|
|
|
Optional fields:
|
|
|
|
- `description`: display/help text for the shader library.
|
|
- `category`: UI grouping label.
|
|
- `entryPoint`: Slang function to call. Defaults to `shadeVideo`.
|
|
- `textures`: texture assets to load and expose as samplers.
|
|
- `fonts`: packaged font assets for live text parameters.
|
|
- `temporal`: history-buffer requirements.
|
|
|
|
Shader-visible identifiers must be valid Slang-style identifiers:
|
|
|
|
- `entryPoint`
|
|
- parameter `id`
|
|
- texture `id`
|
|
- font `id`
|
|
|
|
Use letters, numbers, and underscores only, and start with a letter or underscore. For example, `logoTexture` is valid; `logo-texture` is not valid as a shader-visible texture ID.
|
|
|
|
## Slang Entry Point
|
|
|
|
Your shader file must implement the manifest `entryPoint`.
|
|
|
|
Default:
|
|
|
|
```slang
|
|
float4 shadeVideo(ShaderContext context)
|
|
{
|
|
return context.sourceColor;
|
|
}
|
|
```
|
|
|
|
The runtime owns the real fragment shader entry point. Your function is called from the wrapper, and the runtime handles final bypass/mix behavior:
|
|
|
|
```slang
|
|
return lerp(context.sourceColor, effectedColor, mixValue);
|
|
```
|
|
|
|
That means:
|
|
|
|
- Return the fully effected color from your function.
|
|
- Respect alpha if your shader produces an overlay or sprite.
|
|
- The runtime will blend your result with the source according to `mixAmount` and bypass state.
|
|
|
|
## ShaderContext
|
|
|
|
Your entry point receives:
|
|
|
|
```slang
|
|
struct ShaderContext
|
|
{
|
|
float2 uv;
|
|
float4 sourceColor;
|
|
float2 inputResolution;
|
|
float2 outputResolution;
|
|
float time;
|
|
float utcTimeSeconds;
|
|
float utcOffsetSeconds;
|
|
float frameCount;
|
|
float mixAmount;
|
|
float bypass;
|
|
int sourceHistoryLength;
|
|
int temporalHistoryLength;
|
|
};
|
|
```
|
|
|
|
Fields:
|
|
|
|
- `uv`: normalized texture coordinates, usually `0..1`.
|
|
- `sourceColor`: decoded RGBA source video at `uv`.
|
|
- `inputResolution`: decoded input video resolution in pixels.
|
|
- `outputResolution`: shader render resolution in pixels. The current pipeline renders the shader stack at input resolution, then scales the final frame to the configured DeckLink output mode.
|
|
- `time`: elapsed runtime time in seconds.
|
|
- `utcTimeSeconds`: current UTC time of day from the host PC clock, expressed as seconds since UTC midnight.
|
|
- `utcOffsetSeconds`: host PC local UTC offset in seconds. Add this to `utcTimeSeconds` and wrap to `0..86400` to get local time of day.
|
|
- `frameCount`: incrementing frame counter.
|
|
- `mixAmount`: runtime mix amount.
|
|
- `bypass`: `1.0` when the layer is bypassed, otherwise `0.0`.
|
|
- `sourceHistoryLength`: number of usable source-history frames currently available.
|
|
- `temporalHistoryLength`: number of usable temporal frames currently available for this layer.
|
|
|
|
## Helper Functions
|
|
|
|
The wrapper provides:
|
|
|
|
```slang
|
|
float4 sampleVideo(float2 uv);
|
|
float4 sampleSourceHistory(int framesAgo, float2 uv);
|
|
float4 sampleTemporalHistory(int framesAgo, float2 uv);
|
|
```
|
|
|
|
`sampleVideo` samples the live decoded source video.
|
|
|
|
`sampleSourceHistory` samples previous decoded source frames. `framesAgo` is clamped into the available range. If no history is available, it falls back to `sampleVideo`.
|
|
|
|
`sampleTemporalHistory` samples previous pre-layer input frames for temporal shaders that request `preLayerInput` history. `framesAgo` is clamped into the available range. If no temporal history is available, it falls back to `sampleVideo`.
|
|
|
|
Example:
|
|
|
|
```slang
|
|
float4 shadeVideo(ShaderContext context)
|
|
{
|
|
float4 previous = sampleSourceHistory(1, context.uv);
|
|
return lerp(context.sourceColor, previous, 0.35);
|
|
}
|
|
```
|
|
|
|
## Parameters
|
|
|
|
Manifest parameters are exposed to Slang as global values with the same `id`.
|
|
|
|
Supported types:
|
|
|
|
| Manifest type | Slang type | JSON value |
|
|
| --- | --- | --- |
|
|
| `float` | `float` | number |
|
|
| `vec2` | `float2` | `[x, y]` |
|
|
| `color` | `float4` | `[r, g, b, a]` |
|
|
| `bool` | `bool` | `true` or `false` |
|
|
| `enum` | `int` | selected option index |
|
|
| `text` | generated texture/helper | string |
|
|
| `trigger` | `int <id>`, `float <id>Time` | pulse/count |
|
|
|
|
Float example:
|
|
|
|
```json
|
|
{
|
|
"id": "brightness",
|
|
"label": "Brightness",
|
|
"type": "float",
|
|
"default": 1.0,
|
|
"min": 0.0,
|
|
"max": 2.0,
|
|
"step": 0.01
|
|
}
|
|
```
|
|
|
|
```slang
|
|
color.rgb *= brightness;
|
|
```
|
|
|
|
Vector example:
|
|
|
|
```json
|
|
{
|
|
"id": "offset",
|
|
"label": "Offset",
|
|
"type": "vec2",
|
|
"default": [0.0, 0.0],
|
|
"min": [-0.2, -0.2],
|
|
"max": [0.2, 0.2],
|
|
"step": [0.001, 0.001]
|
|
}
|
|
```
|
|
|
|
```slang
|
|
float2 uv = clamp(context.uv + offset, float2(0.0), float2(1.0));
|
|
```
|
|
|
|
Color example:
|
|
|
|
```json
|
|
{
|
|
"id": "tint",
|
|
"label": "Tint",
|
|
"type": "color",
|
|
"default": [1.0, 1.0, 1.0, 1.0]
|
|
}
|
|
```
|
|
|
|
```slang
|
|
color *= tint;
|
|
```
|
|
|
|
Boolean example:
|
|
|
|
```json
|
|
{
|
|
"id": "invert",
|
|
"label": "Invert",
|
|
"type": "bool",
|
|
"default": false
|
|
}
|
|
```
|
|
|
|
```slang
|
|
if (invert)
|
|
color.rgb = 1.0 - color.rgb;
|
|
```
|
|
|
|
Enum example:
|
|
|
|
```json
|
|
{
|
|
"id": "mode",
|
|
"label": "Mode",
|
|
"type": "enum",
|
|
"default": "normal",
|
|
"options": [
|
|
{ "value": "normal", "label": "Normal" },
|
|
{ "value": "luma", "label": "Luma" },
|
|
{ "value": "posterize", "label": "Posterize" }
|
|
]
|
|
}
|
|
```
|
|
|
|
Enums are stored in presets/state by their string `value`, but exposed to Slang as a zero-based integer index in option order:
|
|
|
|
```slang
|
|
if (mode == 1)
|
|
{
|
|
float luma = dot(color.rgb, float3(0.2126, 0.7152, 0.0722));
|
|
color.rgb = float3(luma);
|
|
}
|
|
else if (mode == 2)
|
|
{
|
|
color.rgb = floor(color.rgb * 4.0) / 4.0;
|
|
}
|
|
```
|
|
|
|
Text example:
|
|
|
|
```json
|
|
{
|
|
"fonts": [
|
|
{ "id": "inter", "path": "fonts/Inter-Regular.ttf" }
|
|
],
|
|
"parameters": [
|
|
{
|
|
"id": "titleText",
|
|
"label": "Title",
|
|
"type": "text",
|
|
"default": "LIVE",
|
|
"font": "inter",
|
|
"maxLength": 64
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Text parameters are runtime-owned strings. They are not emitted as uniform values. Instead, the runtime renders the current string into a single-line SDF mask texture and the shader wrapper exposes helpers based on the parameter id:
|
|
|
|
```slang
|
|
float mask = sampleTitleText(textUv);
|
|
float4 premultipliedText = drawTitleText(textUv, float4(1.0, 1.0, 1.0, 1.0));
|
|
```
|
|
|
|
Text is currently limited to printable ASCII. `maxLength` defaults to `64` and is clamped to `1..256`. The optional `font` field references a packaged font declared in `fonts`; if no font is specified, the runtime uses its fallback sans-serif renderer.
|
|
|
|
Trigger example:
|
|
|
|
```json
|
|
{
|
|
"id": "flash",
|
|
"label": "Flash",
|
|
"type": "trigger"
|
|
}
|
|
```
|
|
|
|
A trigger appears as a button in the control UI. Pressing it increments the shader-visible integer `flash` and records the runtime time in `flashTime`:
|
|
|
|
```slang
|
|
float age = context.time - flashTime;
|
|
float intensity = flash > 0 ? exp(-age * 5.0) : 0.0;
|
|
color.rgb += intensity;
|
|
```
|
|
|
|
Triggers are useful for one-shot shader reactions such as flashes, ripples, cuts, or randomized looks. They do not execute arbitrary CPU code; they only update uniforms consumed by the shader.
|
|
|
|
Parameter validation:
|
|
|
|
- Float values are clamped to `min`/`max` if provided.
|
|
- `vec2` must have exactly 2 numbers.
|
|
- `color` must have exactly 4 numbers.
|
|
- Enum defaults must match one of the declared option values.
|
|
- Text defaults must be strings. Non-printable characters are dropped and values are clamped to `maxLength`.
|
|
- Trigger values are incremented by the host when triggered. The shader sees the trigger count and last trigger time.
|
|
- Non-finite numeric values are rejected.
|
|
|
|
## Texture Assets
|
|
|
|
Declare texture assets in the manifest:
|
|
|
|
```json
|
|
{
|
|
"textures": [
|
|
{
|
|
"id": "logoTexture",
|
|
"path": "logo.png"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Rules:
|
|
|
|
- `id` must be a valid shader identifier.
|
|
- `path` is relative to the shader package directory.
|
|
- The file must exist when the manifest is loaded.
|
|
- Texture asset changes trigger shader reload.
|
|
|
|
Texture IDs become `Sampler2D<float4>` globals:
|
|
|
|
```slang
|
|
float4 logo = logoTexture.Sample(logoUv);
|
|
```
|
|
|
|
For sprite or overlay shaders, return premultiplied-looking output if you want clean composition:
|
|
|
|
```slang
|
|
float alpha = logo.a;
|
|
return float4(logo.rgb * alpha, alpha);
|
|
```
|
|
|
|
See `shaders/dvd-bounce/` for a complete texture-driven example.
|
|
|
|
## Font Assets
|
|
|
|
Declare packaged font assets in the manifest:
|
|
|
|
```json
|
|
{
|
|
"fonts": [
|
|
{
|
|
"id": "inter",
|
|
"path": "fonts/Inter-Regular.ttf"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Rules:
|
|
|
|
- `id` must be a valid shader identifier.
|
|
- `path` is relative to the shader package directory.
|
|
- The file must exist when the manifest is loaded.
|
|
- Font asset changes trigger shader reload.
|
|
- V1 text layout is single-line; shaders position and scale the generated text texture themselves.
|
|
|
|
See `shaders/text-overlay/` for a complete live text example. The sample bundles Roboto Regular and includes its OFL license beside the font file.
|
|
|
|
## Temporal Shaders
|
|
|
|
Temporal shaders can request access to previous frames.
|
|
|
|
Manifest example:
|
|
|
|
```json
|
|
{
|
|
"temporal": {
|
|
"enabled": true,
|
|
"historySource": "preLayerInput",
|
|
"historyLength": 12
|
|
}
|
|
}
|
|
```
|
|
|
|
Supported `historySource` values:
|
|
|
|
- `source`: decoded source-video history from previous frames.
|
|
- `preLayerInput`: history of the input arriving at this layer before the shader runs.
|
|
|
|
`historyLength` is the requested frame count. The runtime clamps it by `maxTemporalHistoryFrames` in `config/runtime-host.json`.
|
|
|
|
Temporal history resets when:
|
|
|
|
- layers are added, removed, or reordered
|
|
- a layer bypass state changes
|
|
- a layer changes shader
|
|
- a shader is reloaded or recompiled
|
|
- render dimensions change
|
|
|
|
Use the available history lengths to avoid assuming history is ready on the first frame:
|
|
|
|
```slang
|
|
float4 shadeVideo(ShaderContext context)
|
|
{
|
|
if (context.temporalHistoryLength <= 0)
|
|
return context.sourceColor;
|
|
|
|
float4 oldFrame = sampleTemporalHistory(3, context.uv);
|
|
return lerp(context.sourceColor, oldFrame, 0.4);
|
|
}
|
|
```
|
|
|
|
See `shaders/temporal-ghost-trail/` and `shaders/temporal-low-fps/` for examples.
|
|
|
|
## Coordinate And Color Notes
|
|
|
|
- `uv` is normalized.
|
|
- Use `context.outputResolution` for pixel-sized effects.
|
|
- Use `context.inputResolution` when sampling source video by input pixel size.
|
|
- `sourceColor` and `sampleVideo` return RGBA values in normalized `0..1` range.
|
|
- Prefer `saturate(color)` or explicit `clamp` before returning if your math can overshoot.
|
|
|
|
Pixel-size example:
|
|
|
|
```slang
|
|
float2 pixel = 1.0 / max(context.outputResolution, float2(1.0));
|
|
float4 right = sampleVideo(context.uv + float2(pixel.x, 0.0));
|
|
```
|
|
|
|
## Reload And Generated Files
|
|
|
|
When a shader compiles, the runtime writes generated files under `runtime/shader_cache/`:
|
|
|
|
- `active_shader_wrapper.slang`
|
|
- `active_shader.raw.frag`
|
|
- `active_shader.frag`
|
|
|
|
These files are ignored by git and are useful for debugging compiler output. If a shader fails to compile, inspect the wrapper first; it shows the exact generated Slang code including your included shader.
|
|
|
|
## Common Pitfalls
|
|
|
|
- Do not use hyphens in parameter IDs, texture IDs, or entry point names.
|
|
- Do not declare your own `ShaderContext`, `GlobalParams`, `sampleVideo`, `sampleSourceHistory`, or `sampleTemporalHistory`.
|
|
- Do not write a `[shader("fragment")]` entry point in `shader.slang`; the runtime provides it.
|
|
- Remember enum globals are integer indexes, not strings.
|
|
- Declare every texture in `shader.json`; undeclared texture samplers will not be bound.
|
|
- Declare packaged fonts in `shader.json` when text parameters should use a specific font.
|
|
- Keep temporal history requests modest. They consume texture units and memory and are capped by runtime config.
|
|
- If a parameter appears in the UI but not in Slang, the shader may still compile, but the control has no effect.
|
|
- If a Slang name collides with a generated global, rename your parameter or local symbol.
|
|
|
|
## Minimal Package Checklist
|
|
|
|
Before committing a new shader package:
|
|
|
|
- `shader.json` is valid JSON.
|
|
- `id` is unique across `shaders/`.
|
|
- `entryPoint`, parameter IDs, and texture IDs are valid identifiers.
|
|
- `shader.slang` implements the configured entry point.
|
|
- Texture files referenced by `textures` exist.
|
|
- Font files referenced by `fonts` exist.
|
|
- Enum defaults are present in their `options`.
|
|
- Temporal shaders handle short or empty history gracefully.
|
|
- The app can reload and compile the shader without errors.
|