Files
video-shader-toys/SHADER_CONTRACT.md
Aiden 27bf2ae45c
All checks were successful
CI / React UI Build (push) Successful in 11s
CI / Native Windows Build And Tests (push) Successful in 2m22s
CI / Windows Release Package (push) Successful in 2m27s
doc updates
2026-05-08 18:49:27 +10:00

661 lines
22 KiB
Markdown

# Shader Package Contract
This document explains how to create shaders for the Video Shader runtime.
Each shader is a small package under `shaders/<id>/`:
```text
shaders/my-effect/
shader.json
shader.slang
optional-texture.png
```
The runtime reads `shader.json`, generates a Slang wrapper from `runtime/templates/shader_wrapper.slang.in`, includes your `shader.slang`, compiles the result to GLSL, and exposes the shader in the local control UI.
## Quick Start
Create a folder:
```text
shaders/my-effect/
```
Add `shader.json`:
```json
{
"id": "my-effect",
"name": "My Effect",
"description": "A simple starter shader.",
"category": "Custom",
"entryPoint": "shadeVideo",
"parameters": [
{
"id": "strength",
"label": "Strength",
"type": "float",
"default": 0.5,
"min": 0.0,
"max": 1.0,
"step": 0.01
}
]
}
```
Add `shader.slang`:
```slang
float4 shadeVideo(ShaderContext context)
{
float4 color = context.sourceColor;
color.rgb = lerp(color.rgb, 1.0 - color.rgb, strength);
return saturate(color);
}
```
With `autoReload` enabled in `config/runtime-host.json`, edits to shader source, manifests, and declared texture assets are picked up automatically. You can also use **Reload shaders** in the control UI to manually rescan the shader library.
## Guidance For Shaders
When generating a new shader package, prefer matching the existing runtime contract over copying code verbatim from Shadertoy, GLSL sandbox sites, or WebGL demos.
Important rules:
- Generate a complete package: `shaders/<id>/shader.json` and `shaders/<id>/shader.slang`.
- Use `float4 shadeVideo(ShaderContext context)` unless the manifest explicitly sets a different `entryPoint`.
- Do not create `mainImage`, `main`, `fragColor`, `iResolution`, `iTime`, `iChannel0`, or a fragment shader attribute layout. The runtime wrapper provides the real fragment entry point.
- Replace Shadertoy `fragCoord` with `context.uv * context.outputResolution`.
- Replace `iResolution.xy` with `context.outputResolution`.
- Replace `iTime` with `context.time`.
- Replace `iFrame` with `context.frameCount`.
- Replace source-video `iChannel0` sampling with `sampleVideo(uv)` or `context.sourceColor`.
- Use Slang/HLSL names and syntax: `float2`, `float3`, `float4`, `float2x2`, `lerp`, `frac`, `saturate`, and `mul(matrix, vector)`.
- Do not use GLSL-only types/functions such as `vec2`, `vec3`, `vec4`, `mat2`, `mix`, `fract`, `mod`, `texture`, or `mainImage`.
- Keep parameter IDs, texture IDs, font IDs, and function entry points as valid shader identifiers: letters, numbers, and underscores only, starting with a letter or underscore.
- Add only controls that are actually used by the shader.
- Prefer a small number of clear controls with conservative defaults.
- Keep shaders deterministic unless randomness is an explicit feature. For stable process-level variation, use `context.startupRandom`; for per-pixel pseudo-randomness, hash from `uv`, pixel coordinates, `frameCount`, or trigger values.
- If adapting third-party code, include attribution and source URL in the manifest description when the license allows adaptation.
- If the source license is unclear or incompatible, do not add the shader package.
Before finishing, compile-check the shader through the runtime wrapper or launch the app and verify the shader appears without an error in the selector. CI also runs shader validation, so every available package in `shaders/` should compile successfully. Intentionally broken examples should stay visibly marked as broken rather than pretending to be production shaders.
## Manifest Fields
`shader.json` is the runtime-facing description of the shader.
Required fields:
- `id`: package ID used by state/presets. Hyphenated names are OK here, for example `my-effect`.
- `name`: display name in the UI.
- `parameters`: array of exposed controls. Use `[]` if there are no user parameters.
Optional fields:
- `description`: display/help text for the shader library.
- `category`: UI grouping label.
- `entryPoint`: Slang function to call. Defaults to `shadeVideo`.
- `passes`: advanced render-pass declarations. Omit this for normal single-pass shaders.
- `textures`: texture assets to load and expose as samplers.
- `fonts`: packaged font assets for live text parameters.
- `temporal`: history-buffer requirements.
Parameter objects may also include an optional `description` string. The control UI displays it as one-line helper text with the full text available on hover, so use it for short operational guidance rather than long documentation.
Shader-visible identifiers must be valid Slang-style identifiers:
- `entryPoint`
- parameter `id`
- texture `id`
- font `id`
Use letters, numbers, and underscores only, and start with a letter or underscore. For example, `logoTexture` is valid; `logo-texture` is not valid as a shader-visible texture ID.
## Render Passes
Most shaders should omit `passes`. The runtime then creates one implicit pass:
```json
{
"id": "main",
"source": "shader.slang",
"entryPoint": "shadeVideo",
"output": "layerOutput"
}
```
Advanced shaders may declare explicit passes. All passes may live in one `.slang` file by using different `entryPoint` values, or they may be split across multiple source files:
```json
{
"passes": [
{
"id": "blurX",
"source": "blur-x.slang",
"entryPoint": "blurHorizontal",
"inputs": ["layerInput"],
"output": "blurredX"
},
{
"id": "final",
"source": "final.slang",
"entryPoint": "finish",
"inputs": ["blurredX"],
"output": "layerOutput"
}
]
}
```
Pass fields:
- `id`: required pass identifier. It must be a valid shader identifier and unique inside the package.
- `source`: required Slang source path relative to the package directory.
- `entryPoint`: optional Slang function for this pass. Defaults to the package-level `entryPoint`.
- `inputs`: optional list of named inputs. The first input is used as the pass input texture.
- `output`: optional output name. Use `layerOutput` for the final visible layer result.
Pass input names:
- `layerInput`: the input to this layer, before any of its passes run.
- `previousPass`: the previous pass output in this layer. If there is no previous pass, this falls back to `layerInput`.
- Any earlier pass `id` or `output` name from the same layer.
If `inputs` is omitted, the first pass samples `layerInput` and later passes sample `previousPass`.
Single-file multipass example:
```json
{
"passes": [
{
"id": "mask",
"source": "shader.slang",
"entryPoint": "makeMask",
"output": "maskBuffer"
},
{
"id": "final",
"source": "shader.slang",
"entryPoint": "finish",
"inputs": ["maskBuffer"],
"output": "layerOutput"
}
]
}
```
Pass output names:
- `layerOutput`: the final visible output of this layer.
- Any other name creates an intermediate 16-bit float render target that later passes may sample.
If the final declared pass does not explicitly output `layerOutput`, the runtime still treats that final pass as the visible layer output. Existing single-pass shaders are unaffected.
## Slang Entry Point
Your shader file must implement the manifest `entryPoint`.
Default:
```slang
float4 shadeVideo(ShaderContext context)
{
return context.sourceColor;
}
```
The runtime owns the real fragment shader entry point. Your function is called from the wrapper, and the runtime handles final bypass/mix behavior:
```slang
return lerp(context.sourceColor, effectedColor, mixValue);
```
That means:
- Return the fully effected color from your function.
- Respect alpha if your shader produces an overlay or sprite.
- The runtime will blend your result with the source according to `mixAmount` and bypass state.
## ShaderContext
Your entry point receives:
```slang
struct ShaderContext
{
float2 uv;
float4 sourceColor;
float2 inputResolution;
float2 outputResolution;
float time;
float utcTimeSeconds;
float utcOffsetSeconds;
float startupRandom;
float frameCount;
float mixAmount;
float bypass;
int sourceHistoryLength;
int temporalHistoryLength;
};
```
Fields:
- `uv`: normalized texture coordinates, usually `0..1`.
- `sourceColor`: decoded RGBA source video at `uv`.
- `inputResolution`: decoded input video resolution in pixels.
- `outputResolution`: shader render resolution in pixels. The current pipeline renders the shader stack at input resolution, then scales the final frame to the configured video I/O output mode.
- `time`: elapsed runtime time in seconds.
- `utcTimeSeconds`: current UTC time of day from the host PC clock, expressed as seconds since UTC midnight.
- `utcOffsetSeconds`: host PC local UTC offset in seconds. Add this to `utcTimeSeconds` and wrap to `0..86400` to get local time of day.
- `startupRandom`: random `0..1` value generated once when the host process starts. It stays constant for the lifetime of the app and changes on the next launch.
- `frameCount`: incrementing frame counter.
- `mixAmount`: runtime mix amount.
- `bypass`: `1.0` when the layer is bypassed, otherwise `0.0`.
- `sourceHistoryLength`: number of usable source-history frames currently available.
- `temporalHistoryLength`: number of usable temporal frames currently available for this layer.
Color/precision notes:
- `context.sourceColor`, `sampleVideo()`, and temporal history samples are display-referred Rec.709-like RGB, not linear-light RGB.
- The current DeckLink backend prefers 10-bit YUV capture and output when the card/mode supports it, with automatic 8-bit fallback. If external keying is enabled, output prefers 10-bit YUVA (`Ay10`) when supported so shader alpha can drive the key signal, then falls back to 8-bit BGRA.
- Internal decoded, layer, composite, output, and temporal render targets are 16-bit floating point, so gradients and LUT work have more headroom than packed byte video I/O formats.
- Do not add extra Rec.709 or linear conversions unless the shader intentionally documents that behavior.
## Helper Functions
The wrapper provides:
```slang
float4 sampleVideo(float2 uv);
float4 sampleSourceHistory(int framesAgo, float2 uv);
float4 sampleTemporalHistory(int framesAgo, float2 uv);
```
`sampleVideo` samples the live decoded source video.
`sampleSourceHistory` samples previous decoded source frames. `framesAgo` is clamped into the available range. If no history is available, it falls back to `sampleVideo`.
`sampleTemporalHistory` samples previous pre-layer input frames for temporal shaders that request `preLayerInput` history. `framesAgo` is clamped into the available range. If no temporal history is available, it falls back to `sampleVideo`.
Example:
```slang
float4 shadeVideo(ShaderContext context)
{
float4 previous = sampleSourceHistory(1, context.uv);
return lerp(context.sourceColor, previous, 0.35);
}
```
## Parameters
Manifest parameters are exposed to Slang as global values with the same `id`.
Supported types:
| Manifest type | Slang type | JSON value |
| --- | --- | --- |
| `float` | `float` | number |
| `vec2` | `float2` | `[x, y]` |
| `color` | `float4` | `[r, g, b, a]` |
| `bool` | `bool` | `true` or `false` |
| `enum` | `int` | selected option index |
| `text` | generated texture/helper | string |
| `trigger` | `int <id>`, `float <id>Time` | pulse/count |
Float example:
```json
{
"id": "brightness",
"label": "Brightness",
"type": "float",
"default": 1.0,
"min": 0.0,
"max": 2.0,
"step": 0.01
}
```
```slang
color.rgb *= brightness;
```
Vector example:
```json
{
"id": "offset",
"label": "Offset",
"type": "vec2",
"default": [0.0, 0.0],
"min": [-0.2, -0.2],
"max": [0.2, 0.2],
"step": [0.001, 0.001]
}
```
```slang
float2 uv = clamp(context.uv + offset, float2(0.0), float2(1.0));
```
Color example:
```json
{
"id": "tint",
"label": "Tint",
"type": "color",
"default": [1.0, 1.0, 1.0, 1.0]
}
```
```slang
color *= tint;
```
Boolean example:
```json
{
"id": "invert",
"label": "Invert",
"type": "bool",
"default": false
}
```
```slang
if (invert)
color.rgb = 1.0 - color.rgb;
```
Enum example:
```json
{
"id": "mode",
"label": "Mode",
"type": "enum",
"default": "normal",
"options": [
{ "value": "normal", "label": "Normal" },
{ "value": "luma", "label": "Luma" },
{ "value": "posterize", "label": "Posterize" }
]
}
```
Enums are stored in presets/state by their string `value`, but exposed to Slang as a zero-based integer index in option order:
```slang
if (mode == 1)
{
float luma = dot(color.rgb, float3(0.2126, 0.7152, 0.0722));
color.rgb = float3(luma);
}
else if (mode == 2)
{
color.rgb = floor(color.rgb * 4.0) / 4.0;
}
```
Text example:
```json
{
"fonts": [
{ "id": "inter", "path": "fonts/Inter-Regular.ttf" }
],
"parameters": [
{
"id": "titleText",
"label": "Title",
"type": "text",
"default": "LIVE",
"font": "inter",
"maxLength": 64
}
]
}
```
Text parameters are runtime-owned strings. They are not emitted as uniform values. Instead, the runtime renders the current string into a single-line SDF mask texture and the shader wrapper exposes helpers based on the parameter id:
```slang
float mask = sampleTitleText(textUv);
float4 premultipliedText = drawTitleText(textUv, float4(1.0, 1.0, 1.0, 1.0));
```
Text is currently limited to printable ASCII. `maxLength` defaults to `64` and is clamped to `1..256`. The optional `font` field references a packaged font declared in `fonts`; if no font is specified, the runtime uses its fallback sans-serif renderer.
Trigger example:
```json
{
"id": "flash",
"label": "Flash",
"type": "trigger"
}
```
A trigger appears as a button in the control UI. Pressing it increments the shader-visible integer `flash` and records the runtime time in `flashTime`:
```slang
float age = context.time - flashTime;
float intensity = flash > 0 ? exp(-age * 5.0) : 0.0;
color.rgb += intensity;
```
Triggers are useful for one-shot shader reactions such as flashes, ripples, cuts, or randomized looks. They do not execute arbitrary CPU code; they only update uniforms consumed by the shader.
Parameter validation:
- Float values are clamped to `min`/`max` if provided.
- `vec2` must have exactly 2 numbers.
- `color` must have exactly 4 numbers.
- Enum defaults must match one of the declared option values.
- Text defaults must be strings. Non-printable characters are dropped and values are clamped to `maxLength`.
- Trigger values are incremented by the host when triggered. The shader sees the trigger count and last trigger time.
- Non-finite numeric values are rejected.
## Texture Assets
Declare texture assets in the manifest:
```json
{
"textures": [
{
"id": "logoTexture",
"path": "logo.png"
}
]
}
```
Rules:
- `id` must be a valid shader identifier.
- `path` is relative to the shader package directory.
- The file must exist when the manifest is loaded.
- Texture asset changes trigger shader reload.
Texture IDs become `Sampler2D<float4>` globals:
```slang
float4 logo = logoTexture.Sample(logoUv);
```
For sprite or overlay shaders, return premultiplied-looking output if you want clean composition:
```slang
float alpha = logo.a;
return float4(logo.rgb * alpha, alpha);
```
See `shaders/dvd-bounce/` for a complete texture-driven example.
## Font Assets
Declare packaged font assets in the manifest:
```json
{
"fonts": [
{
"id": "inter",
"path": "fonts/Inter-Regular.ttf"
}
]
}
```
Rules:
- `id` must be a valid shader identifier.
- `path` is relative to the shader package directory.
- The file must exist when the manifest is loaded.
- Font asset changes trigger shader reload.
- V1 text layout is single-line; shaders position and scale the generated text texture themselves.
See `shaders/text-overlay/` for a complete live text example. The sample bundles Roboto Regular and includes its OFL license beside the font file.
## Temporal Shaders
Temporal shaders can request access to previous frames.
Manifest example:
```json
{
"temporal": {
"enabled": true,
"historySource": "preLayerInput",
"historyLength": 12
}
}
```
Supported `historySource` values:
- `source`: decoded source-video history from previous frames.
- `preLayerInput`: history of the input arriving at this layer before the shader runs.
`historyLength` is the requested frame count. The runtime clamps it by `maxTemporalHistoryFrames` in `config/runtime-host.json`.
Temporal history resets when:
- layers are added, removed, or reordered
- a layer bypass state changes
- a layer changes shader
- a shader is reloaded or recompiled
- render dimensions change
Use the available history lengths to avoid assuming history is ready on the first frame:
```slang
float4 shadeVideo(ShaderContext context)
{
if (context.temporalHistoryLength <= 0)
return context.sourceColor;
float4 oldFrame = sampleTemporalHistory(3, context.uv);
return lerp(context.sourceColor, oldFrame, 0.4);
}
```
See `shaders/temporal-ghost-trail/` and `shaders/temporal-low-fps/` for examples.
## Coordinate And Color Notes
- `uv` is normalized.
- Use `context.outputResolution` for pixel-sized effects.
- Use `context.inputResolution` when sampling source video by input pixel size.
- `sourceColor` and `sampleVideo` return RGBA values in normalized `0..1` range.
- Prefer `saturate(color)` or explicit `clamp` before returning if your math can overshoot.
- For generated calibration charts, test patterns, gradients, and exposure ramps, state whether patch values are linear-light, display-referred gamma encoded, Rec.709 encoded, or intentionally artistic.
- For one-stop exposure patches, each patch should normally be `baseLevel * 2^patchIndex` before any display/tone encoding.
- For Rec.709 OETF encoding, use:
```slang
float rec709Oetf(float linearLevel)
{
float value = saturate(linearLevel);
if (value < 0.018)
return 4.5 * value;
return 1.099 * pow(value, 0.45) - 0.099;
}
```
Pixel-size example:
```slang
float2 pixel = 1.0 / max(context.outputResolution, float2(1.0));
float4 right = sampleVideo(context.uv + float2(pixel.x, 0.0));
```
## Animation And Timing Notes
- `context.time` is elapsed runtime time in seconds and is the default animation source for generative shaders.
- `context.frameCount` increments once per rendered output frame and is useful when an effect must be frame-locked.
- Avoid expensive CPU-like timing logic in the shader; animation should usually be a simple function of `context.time`, `context.frameCount`, trigger uniforms, or parameters.
- If a shader appears to judder only while animated, first test whether freezing its time removes the issue. That usually separates animation cadence issues from rendering or transfer issues.
- Do not add custom timer uniforms to the wrapper. Use the fields already in `ShaderContext`.
## Performance Notes
The app has to meet a fixed video frame cadence, so avoid shader code that only looks good in unconstrained browser demos.
Guidelines:
- Keep loops bounded with compile-time constants where possible.
- Avoid very high per-pixel raymarch counts by default. If a heavy loop is needed, expose a quality/steps control with a safe default.
- Prefer early exits only when they are simple; highly divergent branches can be expensive across a full frame.
- Avoid repeated texture sampling in large loops unless the effect really needs it.
- Use `context.outputResolution` carefully. A 1080p frame is over 2 million fragments; a tiny extra loop can become expensive.
- The UI render time may measure CPU command submission rather than true GPU execution time, so visual frame issues can still be GPU-related even when reported render time is small.
- Do not write debug files, allocate resources, or assume CPU-side work can happen from `shader.slang`. Shader code is GPU-only.
## Reload And Generated Files
When a shader compiles, the runtime writes generated files under `runtime/shader_cache/`:
- `active_shader_wrapper.slang`
- `active_shader.raw.frag`
- `active_shader.frag`
These files are ignored by git and are useful for debugging compiler output. If a shader fails to compile, inspect the wrapper first; it shows the exact generated Slang code including your included shader.
For multipass shaders, these files reflect the most recently compiled pass. If a package has several passes, the reported compile error and pass name are usually more useful than assuming the cache contains the first pass.
## Common Pitfalls
- Do not use hyphens in parameter IDs, texture IDs, or entry point names.
- Do not declare your own `ShaderContext`, `GlobalParams`, `sampleVideo`, `sampleSourceHistory`, or `sampleTemporalHistory`.
- Do not write a `[shader("fragment")]` entry point in `shader.slang`; the runtime provides it.
- Remember enum globals are integer indexes, not strings.
- Declare every texture in `shader.json`; undeclared texture samplers will not be bound.
- Declare packaged fonts in `shader.json` when text parameters should use a specific font.
- Keep temporal history requests modest. They consume texture units and memory and are capped by runtime config.
- If a parameter appears in the UI but not in Slang, the shader may still compile, but the control has no effect.
- If a Slang name collides with a generated global, rename your parameter or local symbol.
## Minimal Package Checklist
Before committing a new shader package:
- `shader.json` is valid JSON.
- `id` is unique across `shaders/`.
- `entryPoint`, parameter IDs, and texture IDs are valid identifiers.
- `shader.slang` implements the configured entry point.
- Texture files referenced by `textures` exist.
- Font files referenced by `fonts` exist.
- Enum defaults are present in their `options`.
- Temporal shaders handle short or empty history gracefully.
- The app can reload and compile the shader without errors.