575 lines
18 KiB
Markdown
575 lines
18 KiB
Markdown
# Shader Package Contract
|
|
|
|
This document explains how to create shaders for the Video Shader runtime.
|
|
|
|
Each shader is a small package under `shaders/<id>/`:
|
|
|
|
```text
|
|
shaders/my-effect/
|
|
shader.json
|
|
shader.slang
|
|
optional-texture.png
|
|
```
|
|
|
|
The runtime reads `shader.json`, generates a Slang wrapper from `runtime/templates/shader_wrapper.slang.in`, includes your `shader.slang`, compiles the result to GLSL, and exposes the shader in the local control UI.
|
|
|
|
## Quick Start
|
|
|
|
Create a folder:
|
|
|
|
```text
|
|
shaders/my-effect/
|
|
```
|
|
|
|
Add `shader.json`:
|
|
|
|
```json
|
|
{
|
|
"id": "my-effect",
|
|
"name": "My Effect",
|
|
"description": "A simple starter shader.",
|
|
"category": "Custom",
|
|
"entryPoint": "shadeVideo",
|
|
"parameters": [
|
|
{
|
|
"id": "strength",
|
|
"label": "Strength",
|
|
"type": "float",
|
|
"default": 0.5,
|
|
"min": 0.0,
|
|
"max": 1.0,
|
|
"step": 0.01
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Add `shader.slang`:
|
|
|
|
```slang
|
|
float4 shadeVideo(ShaderContext context)
|
|
{
|
|
float4 color = context.sourceColor;
|
|
color.rgb = lerp(color.rgb, 1.0 - color.rgb, strength);
|
|
return saturate(color);
|
|
}
|
|
```
|
|
|
|
With `autoReload` enabled in `config/runtime-host.json`, edits to shader source, manifests, and declared texture assets are picked up automatically.
|
|
|
|
## Guidance For Shaders
|
|
|
|
When generating a new shader package, prefer matching the existing runtime contract over copying code verbatim from Shadertoy, GLSL sandbox sites, or WebGL demos.
|
|
|
|
Important rules:
|
|
|
|
- Generate a complete package: `shaders/<id>/shader.json` and `shaders/<id>/shader.slang`.
|
|
- Use `float4 shadeVideo(ShaderContext context)` unless the manifest explicitly sets a different `entryPoint`.
|
|
- Do not create `mainImage`, `main`, `fragColor`, `iResolution`, `iTime`, `iChannel0`, or a fragment shader attribute layout. The runtime wrapper provides the real fragment entry point.
|
|
- Replace Shadertoy `fragCoord` with `context.uv * context.outputResolution`.
|
|
- Replace `iResolution.xy` with `context.outputResolution`.
|
|
- Replace `iTime` with `context.time`.
|
|
- Replace `iFrame` with `context.frameCount`.
|
|
- Replace source-video `iChannel0` sampling with `sampleVideo(uv)` or `context.sourceColor`.
|
|
- Use Slang/HLSL names and syntax: `float2`, `float3`, `float4`, `float2x2`, `lerp`, `frac`, `saturate`, and `mul(matrix, vector)`.
|
|
- Do not use GLSL-only types/functions such as `vec2`, `vec3`, `vec4`, `mat2`, `mix`, `fract`, `mod`, `texture`, or `mainImage`.
|
|
- Keep parameter IDs, texture IDs, font IDs, and function entry points as valid shader identifiers: letters, numbers, and underscores only, starting with a letter or underscore.
|
|
- Add only controls that are actually used by the shader.
|
|
- Prefer a small number of clear controls with conservative defaults.
|
|
- Keep shaders deterministic unless randomness is an explicit feature. For stable process-level variation, use `context.startupRandom`; for per-pixel pseudo-randomness, hash from `uv`, pixel coordinates, `frameCount`, or trigger values.
|
|
- If adapting third-party code, include attribution and source URL in the manifest description when the license allows adaptation.
|
|
- If the source license is unclear or incompatible, do not add the shader package.
|
|
|
|
Before finishing, compile-check the shader through the runtime wrapper or launch the app and verify the shader appears without an error in the selector.
|
|
|
|
## Manifest Fields
|
|
|
|
`shader.json` is the runtime-facing description of the shader.
|
|
|
|
Required fields:
|
|
|
|
- `id`: package ID used by state/presets. Hyphenated names are OK here, for example `my-effect`.
|
|
- `name`: display name in the UI.
|
|
- `parameters`: array of exposed controls. Use `[]` if there are no user parameters.
|
|
|
|
Optional fields:
|
|
|
|
- `description`: display/help text for the shader library.
|
|
- `category`: UI grouping label.
|
|
- `entryPoint`: Slang function to call. Defaults to `shadeVideo`.
|
|
- `textures`: texture assets to load and expose as samplers.
|
|
- `fonts`: packaged font assets for live text parameters.
|
|
- `temporal`: history-buffer requirements.
|
|
|
|
Shader-visible identifiers must be valid Slang-style identifiers:
|
|
|
|
- `entryPoint`
|
|
- parameter `id`
|
|
- texture `id`
|
|
- font `id`
|
|
|
|
Use letters, numbers, and underscores only, and start with a letter or underscore. For example, `logoTexture` is valid; `logo-texture` is not valid as a shader-visible texture ID.
|
|
|
|
## Slang Entry Point
|
|
|
|
Your shader file must implement the manifest `entryPoint`.
|
|
|
|
Default:
|
|
|
|
```slang
|
|
float4 shadeVideo(ShaderContext context)
|
|
{
|
|
return context.sourceColor;
|
|
}
|
|
```
|
|
|
|
The runtime owns the real fragment shader entry point. Your function is called from the wrapper, and the runtime handles final bypass/mix behavior:
|
|
|
|
```slang
|
|
return lerp(context.sourceColor, effectedColor, mixValue);
|
|
```
|
|
|
|
That means:
|
|
|
|
- Return the fully effected color from your function.
|
|
- Respect alpha if your shader produces an overlay or sprite.
|
|
- The runtime will blend your result with the source according to `mixAmount` and bypass state.
|
|
|
|
## ShaderContext
|
|
|
|
Your entry point receives:
|
|
|
|
```slang
|
|
struct ShaderContext
|
|
{
|
|
float2 uv;
|
|
float4 sourceColor;
|
|
float2 inputResolution;
|
|
float2 outputResolution;
|
|
float time;
|
|
float utcTimeSeconds;
|
|
float utcOffsetSeconds;
|
|
float startupRandom;
|
|
float frameCount;
|
|
float mixAmount;
|
|
float bypass;
|
|
int sourceHistoryLength;
|
|
int temporalHistoryLength;
|
|
};
|
|
```
|
|
|
|
Fields:
|
|
|
|
- `uv`: normalized texture coordinates, usually `0..1`.
|
|
- `sourceColor`: decoded RGBA source video at `uv`.
|
|
- `inputResolution`: decoded input video resolution in pixels.
|
|
- `outputResolution`: shader render resolution in pixels. The current pipeline renders the shader stack at input resolution, then scales the final frame to the configured DeckLink output mode.
|
|
- `time`: elapsed runtime time in seconds.
|
|
- `utcTimeSeconds`: current UTC time of day from the host PC clock, expressed as seconds since UTC midnight.
|
|
- `utcOffsetSeconds`: host PC local UTC offset in seconds. Add this to `utcTimeSeconds` and wrap to `0..86400` to get local time of day.
|
|
- `startupRandom`: random `0..1` value generated once when the host process starts. It stays constant for the lifetime of the app and changes on the next launch.
|
|
- `frameCount`: incrementing frame counter.
|
|
- `mixAmount`: runtime mix amount.
|
|
- `bypass`: `1.0` when the layer is bypassed, otherwise `0.0`.
|
|
- `sourceHistoryLength`: number of usable source-history frames currently available.
|
|
- `temporalHistoryLength`: number of usable temporal frames currently available for this layer.
|
|
|
|
Color/precision notes:
|
|
|
|
- `context.sourceColor`, `sampleVideo()`, and temporal history samples are display-referred Rec.709-like RGB, not linear-light RGB.
|
|
- The host prefers 10-bit DeckLink YUV capture and output when the card/mode supports it, with automatic 8-bit fallback.
|
|
- Internal decoded, layer, composite, output, and temporal render targets are 16-bit floating point, so gradients and LUT work have more headroom than the packed DeckLink byte formats.
|
|
- Do not add extra Rec.709 or linear conversions unless the shader intentionally documents that behavior.
|
|
|
|
## Helper Functions
|
|
|
|
The wrapper provides:
|
|
|
|
```slang
|
|
float4 sampleVideo(float2 uv);
|
|
float4 sampleSourceHistory(int framesAgo, float2 uv);
|
|
float4 sampleTemporalHistory(int framesAgo, float2 uv);
|
|
```
|
|
|
|
`sampleVideo` samples the live decoded source video.
|
|
|
|
`sampleSourceHistory` samples previous decoded source frames. `framesAgo` is clamped into the available range. If no history is available, it falls back to `sampleVideo`.
|
|
|
|
`sampleTemporalHistory` samples previous pre-layer input frames for temporal shaders that request `preLayerInput` history. `framesAgo` is clamped into the available range. If no temporal history is available, it falls back to `sampleVideo`.
|
|
|
|
Example:
|
|
|
|
```slang
|
|
float4 shadeVideo(ShaderContext context)
|
|
{
|
|
float4 previous = sampleSourceHistory(1, context.uv);
|
|
return lerp(context.sourceColor, previous, 0.35);
|
|
}
|
|
```
|
|
|
|
## Parameters
|
|
|
|
Manifest parameters are exposed to Slang as global values with the same `id`.
|
|
|
|
Supported types:
|
|
|
|
| Manifest type | Slang type | JSON value |
|
|
| --- | --- | --- |
|
|
| `float` | `float` | number |
|
|
| `vec2` | `float2` | `[x, y]` |
|
|
| `color` | `float4` | `[r, g, b, a]` |
|
|
| `bool` | `bool` | `true` or `false` |
|
|
| `enum` | `int` | selected option index |
|
|
| `text` | generated texture/helper | string |
|
|
| `trigger` | `int <id>`, `float <id>Time` | pulse/count |
|
|
|
|
Float example:
|
|
|
|
```json
|
|
{
|
|
"id": "brightness",
|
|
"label": "Brightness",
|
|
"type": "float",
|
|
"default": 1.0,
|
|
"min": 0.0,
|
|
"max": 2.0,
|
|
"step": 0.01
|
|
}
|
|
```
|
|
|
|
```slang
|
|
color.rgb *= brightness;
|
|
```
|
|
|
|
Vector example:
|
|
|
|
```json
|
|
{
|
|
"id": "offset",
|
|
"label": "Offset",
|
|
"type": "vec2",
|
|
"default": [0.0, 0.0],
|
|
"min": [-0.2, -0.2],
|
|
"max": [0.2, 0.2],
|
|
"step": [0.001, 0.001]
|
|
}
|
|
```
|
|
|
|
```slang
|
|
float2 uv = clamp(context.uv + offset, float2(0.0), float2(1.0));
|
|
```
|
|
|
|
Color example:
|
|
|
|
```json
|
|
{
|
|
"id": "tint",
|
|
"label": "Tint",
|
|
"type": "color",
|
|
"default": [1.0, 1.0, 1.0, 1.0]
|
|
}
|
|
```
|
|
|
|
```slang
|
|
color *= tint;
|
|
```
|
|
|
|
Boolean example:
|
|
|
|
```json
|
|
{
|
|
"id": "invert",
|
|
"label": "Invert",
|
|
"type": "bool",
|
|
"default": false
|
|
}
|
|
```
|
|
|
|
```slang
|
|
if (invert)
|
|
color.rgb = 1.0 - color.rgb;
|
|
```
|
|
|
|
Enum example:
|
|
|
|
```json
|
|
{
|
|
"id": "mode",
|
|
"label": "Mode",
|
|
"type": "enum",
|
|
"default": "normal",
|
|
"options": [
|
|
{ "value": "normal", "label": "Normal" },
|
|
{ "value": "luma", "label": "Luma" },
|
|
{ "value": "posterize", "label": "Posterize" }
|
|
]
|
|
}
|
|
```
|
|
|
|
Enums are stored in presets/state by their string `value`, but exposed to Slang as a zero-based integer index in option order:
|
|
|
|
```slang
|
|
if (mode == 1)
|
|
{
|
|
float luma = dot(color.rgb, float3(0.2126, 0.7152, 0.0722));
|
|
color.rgb = float3(luma);
|
|
}
|
|
else if (mode == 2)
|
|
{
|
|
color.rgb = floor(color.rgb * 4.0) / 4.0;
|
|
}
|
|
```
|
|
|
|
Text example:
|
|
|
|
```json
|
|
{
|
|
"fonts": [
|
|
{ "id": "inter", "path": "fonts/Inter-Regular.ttf" }
|
|
],
|
|
"parameters": [
|
|
{
|
|
"id": "titleText",
|
|
"label": "Title",
|
|
"type": "text",
|
|
"default": "LIVE",
|
|
"font": "inter",
|
|
"maxLength": 64
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Text parameters are runtime-owned strings. They are not emitted as uniform values. Instead, the runtime renders the current string into a single-line SDF mask texture and the shader wrapper exposes helpers based on the parameter id:
|
|
|
|
```slang
|
|
float mask = sampleTitleText(textUv);
|
|
float4 premultipliedText = drawTitleText(textUv, float4(1.0, 1.0, 1.0, 1.0));
|
|
```
|
|
|
|
Text is currently limited to printable ASCII. `maxLength` defaults to `64` and is clamped to `1..256`. The optional `font` field references a packaged font declared in `fonts`; if no font is specified, the runtime uses its fallback sans-serif renderer.
|
|
|
|
Trigger example:
|
|
|
|
```json
|
|
{
|
|
"id": "flash",
|
|
"label": "Flash",
|
|
"type": "trigger"
|
|
}
|
|
```
|
|
|
|
A trigger appears as a button in the control UI. Pressing it increments the shader-visible integer `flash` and records the runtime time in `flashTime`:
|
|
|
|
```slang
|
|
float age = context.time - flashTime;
|
|
float intensity = flash > 0 ? exp(-age * 5.0) : 0.0;
|
|
color.rgb += intensity;
|
|
```
|
|
|
|
Triggers are useful for one-shot shader reactions such as flashes, ripples, cuts, or randomized looks. They do not execute arbitrary CPU code; they only update uniforms consumed by the shader.
|
|
|
|
Parameter validation:
|
|
|
|
- Float values are clamped to `min`/`max` if provided.
|
|
- `vec2` must have exactly 2 numbers.
|
|
- `color` must have exactly 4 numbers.
|
|
- Enum defaults must match one of the declared option values.
|
|
- Text defaults must be strings. Non-printable characters are dropped and values are clamped to `maxLength`.
|
|
- Trigger values are incremented by the host when triggered. The shader sees the trigger count and last trigger time.
|
|
- Non-finite numeric values are rejected.
|
|
|
|
## Texture Assets
|
|
|
|
Declare texture assets in the manifest:
|
|
|
|
```json
|
|
{
|
|
"textures": [
|
|
{
|
|
"id": "logoTexture",
|
|
"path": "logo.png"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Rules:
|
|
|
|
- `id` must be a valid shader identifier.
|
|
- `path` is relative to the shader package directory.
|
|
- The file must exist when the manifest is loaded.
|
|
- Texture asset changes trigger shader reload.
|
|
|
|
Texture IDs become `Sampler2D<float4>` globals:
|
|
|
|
```slang
|
|
float4 logo = logoTexture.Sample(logoUv);
|
|
```
|
|
|
|
For sprite or overlay shaders, return premultiplied-looking output if you want clean composition:
|
|
|
|
```slang
|
|
float alpha = logo.a;
|
|
return float4(logo.rgb * alpha, alpha);
|
|
```
|
|
|
|
See `shaders/dvd-bounce/` for a complete texture-driven example.
|
|
|
|
## Font Assets
|
|
|
|
Declare packaged font assets in the manifest:
|
|
|
|
```json
|
|
{
|
|
"fonts": [
|
|
{
|
|
"id": "inter",
|
|
"path": "fonts/Inter-Regular.ttf"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Rules:
|
|
|
|
- `id` must be a valid shader identifier.
|
|
- `path` is relative to the shader package directory.
|
|
- The file must exist when the manifest is loaded.
|
|
- Font asset changes trigger shader reload.
|
|
- V1 text layout is single-line; shaders position and scale the generated text texture themselves.
|
|
|
|
See `shaders/text-overlay/` for a complete live text example. The sample bundles Roboto Regular and includes its OFL license beside the font file.
|
|
|
|
## Temporal Shaders
|
|
|
|
Temporal shaders can request access to previous frames.
|
|
|
|
Manifest example:
|
|
|
|
```json
|
|
{
|
|
"temporal": {
|
|
"enabled": true,
|
|
"historySource": "preLayerInput",
|
|
"historyLength": 12
|
|
}
|
|
}
|
|
```
|
|
|
|
Supported `historySource` values:
|
|
|
|
- `source`: decoded source-video history from previous frames.
|
|
- `preLayerInput`: history of the input arriving at this layer before the shader runs.
|
|
|
|
`historyLength` is the requested frame count. The runtime clamps it by `maxTemporalHistoryFrames` in `config/runtime-host.json`.
|
|
|
|
Temporal history resets when:
|
|
|
|
- layers are added, removed, or reordered
|
|
- a layer bypass state changes
|
|
- a layer changes shader
|
|
- a shader is reloaded or recompiled
|
|
- render dimensions change
|
|
|
|
Use the available history lengths to avoid assuming history is ready on the first frame:
|
|
|
|
```slang
|
|
float4 shadeVideo(ShaderContext context)
|
|
{
|
|
if (context.temporalHistoryLength <= 0)
|
|
return context.sourceColor;
|
|
|
|
float4 oldFrame = sampleTemporalHistory(3, context.uv);
|
|
return lerp(context.sourceColor, oldFrame, 0.4);
|
|
}
|
|
```
|
|
|
|
See `shaders/temporal-ghost-trail/` and `shaders/temporal-low-fps/` for examples.
|
|
|
|
## Coordinate And Color Notes
|
|
|
|
- `uv` is normalized.
|
|
- Use `context.outputResolution` for pixel-sized effects.
|
|
- Use `context.inputResolution` when sampling source video by input pixel size.
|
|
- `sourceColor` and `sampleVideo` return RGBA values in normalized `0..1` range.
|
|
- Prefer `saturate(color)` or explicit `clamp` before returning if your math can overshoot.
|
|
- For generated calibration charts, test patterns, gradients, and exposure ramps, state whether patch values are linear-light, display-referred gamma encoded, Rec.709 encoded, or intentionally artistic.
|
|
- For one-stop exposure patches, each patch should normally be `baseLevel * 2^patchIndex` before any display/tone encoding.
|
|
- For Rec.709 OETF encoding, use:
|
|
|
|
```slang
|
|
float rec709Oetf(float linearLevel)
|
|
{
|
|
float value = saturate(linearLevel);
|
|
if (value < 0.018)
|
|
return 4.5 * value;
|
|
return 1.099 * pow(value, 0.45) - 0.099;
|
|
}
|
|
```
|
|
|
|
Pixel-size example:
|
|
|
|
```slang
|
|
float2 pixel = 1.0 / max(context.outputResolution, float2(1.0));
|
|
float4 right = sampleVideo(context.uv + float2(pixel.x, 0.0));
|
|
```
|
|
|
|
## Animation And Timing Notes
|
|
|
|
- `context.time` is elapsed runtime time in seconds and is the default animation source for generative shaders.
|
|
- `context.frameCount` increments once per rendered output frame and is useful when an effect must be frame-locked.
|
|
- Avoid expensive CPU-like timing logic in the shader; animation should usually be a simple function of `context.time`, `context.frameCount`, trigger uniforms, or parameters.
|
|
- If a shader appears to judder only while animated, first test whether freezing its time removes the issue. That usually separates animation cadence issues from rendering or transfer issues.
|
|
- Do not add custom timer uniforms to the wrapper. Use the fields already in `ShaderContext`.
|
|
|
|
## Performance Notes
|
|
|
|
The app has to meet a fixed video frame cadence, so avoid shader code that only looks good in unconstrained browser demos.
|
|
|
|
Guidelines:
|
|
|
|
- Keep loops bounded with compile-time constants where possible.
|
|
- Avoid very high per-pixel raymarch counts by default. If a heavy loop is needed, expose a quality/steps control with a safe default.
|
|
- Prefer early exits only when they are simple; highly divergent branches can be expensive across a full frame.
|
|
- Avoid repeated texture sampling in large loops unless the effect really needs it.
|
|
- Use `context.outputResolution` carefully. A 1080p frame is over 2 million fragments; a tiny extra loop can become expensive.
|
|
- The UI render time may measure CPU command submission rather than true GPU execution time, so visual frame issues can still be GPU-related even when reported render time is small.
|
|
- Do not write debug files, allocate resources, or assume CPU-side work can happen from `shader.slang`. Shader code is GPU-only.
|
|
|
|
## Reload And Generated Files
|
|
|
|
When a shader compiles, the runtime writes generated files under `runtime/shader_cache/`:
|
|
|
|
- `active_shader_wrapper.slang`
|
|
- `active_shader.raw.frag`
|
|
- `active_shader.frag`
|
|
|
|
These files are ignored by git and are useful for debugging compiler output. If a shader fails to compile, inspect the wrapper first; it shows the exact generated Slang code including your included shader.
|
|
|
|
## Common Pitfalls
|
|
|
|
- Do not use hyphens in parameter IDs, texture IDs, or entry point names.
|
|
- Do not declare your own `ShaderContext`, `GlobalParams`, `sampleVideo`, `sampleSourceHistory`, or `sampleTemporalHistory`.
|
|
- Do not write a `[shader("fragment")]` entry point in `shader.slang`; the runtime provides it.
|
|
- Remember enum globals are integer indexes, not strings.
|
|
- Declare every texture in `shader.json`; undeclared texture samplers will not be bound.
|
|
- Declare packaged fonts in `shader.json` when text parameters should use a specific font.
|
|
- Keep temporal history requests modest. They consume texture units and memory and are capped by runtime config.
|
|
- If a parameter appears in the UI but not in Slang, the shader may still compile, but the control has no effect.
|
|
- If a Slang name collides with a generated global, rename your parameter or local symbol.
|
|
|
|
## Minimal Package Checklist
|
|
|
|
Before committing a new shader package:
|
|
|
|
- `shader.json` is valid JSON.
|
|
- `id` is unique across `shaders/`.
|
|
- `entryPoint`, parameter IDs, and texture IDs are valid identifiers.
|
|
- `shader.slang` implements the configured entry point.
|
|
- Texture files referenced by `textures` exist.
|
|
- Font files referenced by `fonts` exist.
|
|
- Enum defaults are present in their `options`.
|
|
- Temporal shaders handle short or empty history gracefully.
|
|
- The app can reload and compile the shader without errors.
|