video-shader-toys/SHADER_CONTRACT.md

Shader Package Contract

This document explains how to create shaders for the Video Shader runtime.

Each shader is a small package under shaders/<id>/:

shaders/my-effect/
  shader.json
  shader.slang
  optional-texture.png

The runtime reads shader.json, generates a Slang wrapper from runtime/templates/shader_wrapper.slang.in, includes your shader.slang, compiles the result to GLSL, and exposes the shader in the local control UI.

Quick Start

Create a folder:

shaders/my-effect/

Add shader.json:

{
  "id": "my-effect",
  "name": "My Effect",
  "description": "A simple starter shader.",
  "category": "Custom",
  "entryPoint": "shadeVideo",
  "parameters": [
    {
      "id": "strength",
      "label": "Strength",
      "type": "float",
      "default": 0.5,
      "min": 0.0,
      "max": 1.0,
      "step": 0.01
    }
  ]
}

Add shader.slang:

float4 shadeVideo(ShaderContext context)
{
    float4 color = context.sourceColor;
    color.rgb = lerp(color.rgb, 1.0 - color.rgb, strength);
    return saturate(color);
}

With autoReload enabled in config/runtime-host.json, edits to shader source, manifests, and declared texture assets are picked up automatically.

Guidance For Shaders

When generating a new shader package, prefer matching the existing runtime contract over copying code verbatim from Shadertoy, GLSL sandbox sites, or WebGL demos.

Important rules:

• Generate a complete package: shaders/<id>/shader.json and shaders/<id>/shader.slang.
• Use float4 shadeVideo(ShaderContext context) unless the manifest explicitly sets a different entryPoint.
• Do not create mainImage, main, fragColor, iResolution, iTime, iChannel0, or a fragment shader attribute layout. The runtime wrapper provides the real fragment entry point.
• Replace Shadertoy fragCoord with context.uv * context.outputResolution.
• Replace iResolution.xy with context.outputResolution.
• Replace iTime with context.time.
• Replace iFrame with context.frameCount.
• Replace source-video iChannel0 sampling with sampleVideo(uv) or context.sourceColor.
• Use Slang/HLSL names and syntax: float2, float3, float4, float2x2, lerp, frac, saturate, and mul(matrix, vector).
• Do not use GLSL-only types/functions such as vec2, vec3, vec4, mat2, mix, fract, mod, texture, or mainImage.
• Keep parameter IDs, texture IDs, font IDs, and function entry points as valid shader identifiers: letters, numbers, and underscores only, starting with a letter or underscore.
• Add only controls that are actually used by the shader.
• Prefer a small number of clear controls with conservative defaults.
• Keep shaders deterministic unless randomness is an explicit feature. For stable process-level variation, use context.startupRandom; for per-pixel pseudo-randomness, hash from uv, pixel coordinates, frameCount, or trigger values.
• If adapting third-party code, include attribution and source URL in the manifest description when the license allows adaptation.
• If the source license is unclear or incompatible, do not add the shader package.

Before finishing, compile-check the shader through the runtime wrapper or launch the app and verify the shader appears without an error in the selector.

Manifest Fields

shader.json is the runtime-facing description of the shader.

Required fields:

• id: package ID used by state/presets. Hyphenated names are OK here, for example my-effect.
• name: display name in the UI.
• parameters: array of exposed controls. Use [] if there are no user parameters.

Optional fields:

• description: display/help text for the shader library.
• category: UI grouping label.
• entryPoint: Slang function to call. Defaults to shadeVideo.
• textures: texture assets to load and expose as samplers.
• fonts: packaged font assets for live text parameters.
• temporal: history-buffer requirements.

Shader-visible identifiers must be valid Slang-style identifiers:

• entryPoint
• parameter id
• texture id
• font id

Use letters, numbers, and underscores only, and start with a letter or underscore. For example, logoTexture is valid; logo-texture is not valid as a shader-visible texture ID.

Slang Entry Point

Your shader file must implement the manifest entryPoint.

Default:

float4 shadeVideo(ShaderContext context)
{
    return context.sourceColor;
}

The runtime owns the real fragment shader entry point. Your function is called from the wrapper, and the runtime handles final bypass/mix behavior:

return lerp(context.sourceColor, effectedColor, mixValue);

That means:

• Return the fully effected color from your function.
• Respect alpha if your shader produces an overlay or sprite.
• The runtime will blend your result with the source according to mixAmount and bypass state.

ShaderContext

Your entry point receives:

struct ShaderContext
{
    float2 uv;
    float4 sourceColor;
    float2 inputResolution;
    float2 outputResolution;
    float time;
    float utcTimeSeconds;
    float utcOffsetSeconds;
    float startupRandom;
    float frameCount;
    float mixAmount;
    float bypass;
    int sourceHistoryLength;
    int temporalHistoryLength;
};

Fields:

• uv: normalized texture coordinates, usually 0..1.
• sourceColor: decoded RGBA source video at uv.
• inputResolution: decoded input video resolution in pixels.
• outputResolution: shader render resolution in pixels. The current pipeline renders the shader stack at input resolution, then scales the final frame to the configured video I/O output mode.
• time: elapsed runtime time in seconds.
• utcTimeSeconds: current UTC time of day from the host PC clock, expressed as seconds since UTC midnight.
• utcOffsetSeconds: host PC local UTC offset in seconds. Add this to utcTimeSeconds and wrap to 0..86400 to get local time of day.
• startupRandom: random 0..1 value generated once when the host process starts. It stays constant for the lifetime of the app and changes on the next launch.
• frameCount: incrementing frame counter.
• mixAmount: runtime mix amount.
• bypass: 1.0 when the layer is bypassed, otherwise 0.0.
• sourceHistoryLength: number of usable source-history frames currently available.
• temporalHistoryLength: number of usable temporal frames currently available for this layer.

Color/precision notes:

• context.sourceColor, sampleVideo(), and temporal history samples are display-referred Rec.709-like RGB, not linear-light RGB.
• The current DeckLink backend prefers 10-bit YUV capture and output when the card/mode supports it, with automatic 8-bit fallback.
• Internal decoded, layer, composite, output, and temporal render targets are 16-bit floating point, so gradients and LUT work have more headroom than packed byte video I/O formats.
• Do not add extra Rec.709 or linear conversions unless the shader intentionally documents that behavior.

Helper Functions

The wrapper provides:

float4 sampleVideo(float2 uv);
float4 sampleSourceHistory(int framesAgo, float2 uv);
float4 sampleTemporalHistory(int framesAgo, float2 uv);

sampleVideo samples the live decoded source video.

sampleSourceHistory samples previous decoded source frames. framesAgo is clamped into the available range. If no history is available, it falls back to sampleVideo.

sampleTemporalHistory samples previous pre-layer input frames for temporal shaders that request preLayerInput history. framesAgo is clamped into the available range. If no temporal history is available, it falls back to sampleVideo.

Example:

float4 shadeVideo(ShaderContext context)
{
    float4 previous = sampleSourceHistory(1, context.uv);
    return lerp(context.sourceColor, previous, 0.35);
}

Parameters

Manifest parameters are exposed to Slang as global values with the same id.

Supported types:

Manifest type	Slang type	JSON value
float	float	number
vec2	float2	[x, y]
color	float4	[r, g, b, a]
bool	bool	true or false
enum	int	selected option index
text	generated texture/helper	string
trigger	int <id>, float <id>Time	pulse/count

Float example:

{
  "id": "brightness",
  "label": "Brightness",
  "type": "float",
  "default": 1.0,
  "min": 0.0,
  "max": 2.0,
  "step": 0.01
}

color.rgb *= brightness;

Vector example:

{
  "id": "offset",
  "label": "Offset",
  "type": "vec2",
  "default": [0.0, 0.0],
  "min": [-0.2, -0.2],
  "max": [0.2, 0.2],
  "step": [0.001, 0.001]
}

float2 uv = clamp(context.uv + offset, float2(0.0), float2(1.0));

Color example:

{
  "id": "tint",
  "label": "Tint",
  "type": "color",
  "default": [1.0, 1.0, 1.0, 1.0]
}

color *= tint;

Boolean example:

{
  "id": "invert",
  "label": "Invert",
  "type": "bool",
  "default": false
}

if (invert)
    color.rgb = 1.0 - color.rgb;

Enum example:

{
  "id": "mode",
  "label": "Mode",
  "type": "enum",
  "default": "normal",
  "options": [
    { "value": "normal", "label": "Normal" },
    { "value": "luma", "label": "Luma" },
    { "value": "posterize", "label": "Posterize" }
  ]
}

Enums are stored in presets/state by their string value, but exposed to Slang as a zero-based integer index in option order:

if (mode == 1)
{
    float luma = dot(color.rgb, float3(0.2126, 0.7152, 0.0722));
    color.rgb = float3(luma);
}
else if (mode == 2)
{
    color.rgb = floor(color.rgb * 4.0) / 4.0;
}

Text example:

{
  "fonts": [
    { "id": "inter", "path": "fonts/Inter-Regular.ttf" }
  ],
  "parameters": [
    {
      "id": "titleText",
      "label": "Title",
      "type": "text",
      "default": "LIVE",
      "font": "inter",
      "maxLength": 64
    }
  ]
}

Text parameters are runtime-owned strings. They are not emitted as uniform values. Instead, the runtime renders the current string into a single-line SDF mask texture and the shader wrapper exposes helpers based on the parameter id:

float mask = sampleTitleText(textUv);
float4 premultipliedText = drawTitleText(textUv, float4(1.0, 1.0, 1.0, 1.0));

Text is currently limited to printable ASCII. maxLength defaults to 64 and is clamped to 1..256. The optional font field references a packaged font declared in fonts; if no font is specified, the runtime uses its fallback sans-serif renderer.

Trigger example:

{
  "id": "flash",
  "label": "Flash",
  "type": "trigger"
}

A trigger appears as a button in the control UI. Pressing it increments the shader-visible integer flash and records the runtime time in flashTime:

float age = context.time - flashTime;
float intensity = flash > 0 ? exp(-age * 5.0) : 0.0;
color.rgb += intensity;

Triggers are useful for one-shot shader reactions such as flashes, ripples, cuts, or randomized looks. They do not execute arbitrary CPU code; they only update uniforms consumed by the shader.

Parameter validation:

• Float values are clamped to min/max if provided.
• vec2 must have exactly 2 numbers.
• color must have exactly 4 numbers.
• Enum defaults must match one of the declared option values.
• Text defaults must be strings. Non-printable characters are dropped and values are clamped to maxLength.
• Trigger values are incremented by the host when triggered. The shader sees the trigger count and last trigger time.
• Non-finite numeric values are rejected.

Texture Assets

Declare texture assets in the manifest:

{
  "textures": [
    {
      "id": "logoTexture",
      "path": "logo.png"
    }
  ]
}

Rules:

• id must be a valid shader identifier.
• path is relative to the shader package directory.
• The file must exist when the manifest is loaded.
• Texture asset changes trigger shader reload.

Texture IDs become Sampler2D<float4> globals:

float4 logo = logoTexture.Sample(logoUv);

For sprite or overlay shaders, return premultiplied-looking output if you want clean composition:

float alpha = logo.a;
return float4(logo.rgb * alpha, alpha);

See shaders/dvd-bounce/ for a complete texture-driven example.

Font Assets

Declare packaged font assets in the manifest:

{
  "fonts": [
    {
      "id": "inter",
      "path": "fonts/Inter-Regular.ttf"
    }
  ]
}

Rules:

• id must be a valid shader identifier.
• path is relative to the shader package directory.
• The file must exist when the manifest is loaded.
• Font asset changes trigger shader reload.
• V1 text layout is single-line; shaders position and scale the generated text texture themselves.

See shaders/text-overlay/ for a complete live text example. The sample bundles Roboto Regular and includes its OFL license beside the font file.

Temporal Shaders

Temporal shaders can request access to previous frames.

Manifest example:

{
  "temporal": {
    "enabled": true,
    "historySource": "preLayerInput",
    "historyLength": 12
  }
}

Supported historySource values:

• source: decoded source-video history from previous frames.
• preLayerInput: history of the input arriving at this layer before the shader runs.

historyLength is the requested frame count. The runtime clamps it by maxTemporalHistoryFrames in config/runtime-host.json.

Temporal history resets when:

• layers are added, removed, or reordered
• a layer bypass state changes
• a layer changes shader
• a shader is reloaded or recompiled
• render dimensions change

Use the available history lengths to avoid assuming history is ready on the first frame:

float4 shadeVideo(ShaderContext context)
{
    if (context.temporalHistoryLength <= 0)
        return context.sourceColor;

    float4 oldFrame = sampleTemporalHistory(3, context.uv);
    return lerp(context.sourceColor, oldFrame, 0.4);
}

See shaders/temporal-ghost-trail/ and shaders/temporal-low-fps/ for examples.

Coordinate And Color Notes

• uv is normalized.
• Use context.outputResolution for pixel-sized effects.
• Use context.inputResolution when sampling source video by input pixel size.
• sourceColor and sampleVideo return RGBA values in normalized 0..1 range.
• Prefer saturate(color) or explicit clamp before returning if your math can overshoot.
• For generated calibration charts, test patterns, gradients, and exposure ramps, state whether patch values are linear-light, display-referred gamma encoded, Rec.709 encoded, or intentionally artistic.
• For one-stop exposure patches, each patch should normally be baseLevel * 2^patchIndex before any display/tone encoding.
• For Rec.709 OETF encoding, use:

float rec709Oetf(float linearLevel)
{
    float value = saturate(linearLevel);
    if (value < 0.018)
        return 4.5 * value;
    return 1.099 * pow(value, 0.45) - 0.099;
}

Pixel-size example:

float2 pixel = 1.0 / max(context.outputResolution, float2(1.0));
float4 right = sampleVideo(context.uv + float2(pixel.x, 0.0));

Animation And Timing Notes

• context.time is elapsed runtime time in seconds and is the default animation source for generative shaders.
• context.frameCount increments once per rendered output frame and is useful when an effect must be frame-locked.
• Avoid expensive CPU-like timing logic in the shader; animation should usually be a simple function of context.time, context.frameCount, trigger uniforms, or parameters.
• If a shader appears to judder only while animated, first test whether freezing its time removes the issue. That usually separates animation cadence issues from rendering or transfer issues.
• Do not add custom timer uniforms to the wrapper. Use the fields already in ShaderContext.

Performance Notes

The app has to meet a fixed video frame cadence, so avoid shader code that only looks good in unconstrained browser demos.

Guidelines:

• Keep loops bounded with compile-time constants where possible.
• Avoid very high per-pixel raymarch counts by default. If a heavy loop is needed, expose a quality/steps control with a safe default.
• Prefer early exits only when they are simple; highly divergent branches can be expensive across a full frame.
• Avoid repeated texture sampling in large loops unless the effect really needs it.
• Use context.outputResolution carefully. A 1080p frame is over 2 million fragments; a tiny extra loop can become expensive.
• The UI render time may measure CPU command submission rather than true GPU execution time, so visual frame issues can still be GPU-related even when reported render time is small.
• Do not write debug files, allocate resources, or assume CPU-side work can happen from shader.slang. Shader code is GPU-only.

Reload And Generated Files

When a shader compiles, the runtime writes generated files under runtime/shader_cache/:

• active_shader_wrapper.slang
• active_shader.raw.frag
• active_shader.frag

These files are ignored by git and are useful for debugging compiler output. If a shader fails to compile, inspect the wrapper first; it shows the exact generated Slang code including your included shader.

Common Pitfalls

• Do not use hyphens in parameter IDs, texture IDs, or entry point names.
• Do not declare your own ShaderContext, GlobalParams, sampleVideo, sampleSourceHistory, or sampleTemporalHistory.
• Do not write a [shader("fragment")] entry point in shader.slang; the runtime provides it.
• Remember enum globals are integer indexes, not strings.
• Declare every texture in shader.json; undeclared texture samplers will not be bound.
• Declare packaged fonts in shader.json when text parameters should use a specific font.
• Keep temporal history requests modest. They consume texture units and memory and are capped by runtime config.
• If a parameter appears in the UI but not in Slang, the shader may still compile, but the control has no effect.
• If a Slang name collides with a generated global, rename your parameter or local symbol.

Minimal Package Checklist

Before committing a new shader package:

• shader.json is valid JSON.
• id is unique across shaders/.
• entryPoint, parameter IDs, and texture IDs are valid identifiers.
• shader.slang implements the configured entry point.
• Texture files referenced by textures exist.
• Font files referenced by fonts exist.
• Enum defaults are present in their options.
• Temporal shaders handle short or empty history gracefully.
• The app can reload and compile the shader without errors.