video-shader-toys/shaders/SHADER_CONTRACT.md

Shader Package Contract

This document explains how to create shaders for the Video Shader runtime.

Each shader is a small package under shaders/<id>/:

shaders/my-effect/
  shader.json
  shader.slang
  optional-texture.png

The runtime reads shader.json, generates a Slang wrapper from runtime/templates/shader_wrapper.slang.in, includes your shader.slang, compiles the result to GLSL, and exposes the shader in the local control UI.

Quick Start

Create a folder:

shaders/my-effect/

Add shader.json:

{
  "id": "my-effect",
  "name": "My Effect",
  "description": "A simple starter shader.",
  "category": "Custom",
  "entryPoint": "shadeVideo",
  "parameters": [
    {
      "id": "strength",
      "label": "Strength",
      "type": "float",
      "default": 0.5,
      "min": 0.0,
      "max": 1.0,
      "step": 0.01
    }
  ]
}

Add shader.slang:

float4 shadeVideo(ShaderContext context)
{
    float4 color = context.sourceColor;
    color.rgb = lerp(color.rgb, 1.0 - color.rgb, strength);
    return saturate(color);
}

With autoReload enabled in config/runtime-host.json, edits to shader source, manifests, and declared texture assets are picked up automatically. You can also use Reload shaders in the control UI to manually rescan the shader library.

Guidance For Shaders

When generating a new shader package, prefer matching the existing runtime contract over copying code verbatim from Shadertoy, GLSL sandbox sites, or WebGL demos.

Important rules:

• Generate a complete package: shaders/<id>/shader.json and shaders/<id>/shader.slang.
• Use float4 shadeVideo(ShaderContext context) unless the manifest explicitly sets a different entryPoint.
• Do not create mainImage, main, fragColor, iResolution, iTime, iChannel0, or a fragment shader attribute layout. The runtime wrapper provides the real fragment entry point.
• Replace Shadertoy fragCoord with context.uv * context.outputResolution.
• Replace iResolution.xy with context.outputResolution.
• Replace iTime with context.time.
• Replace iFrame with context.frameCount.
• Replace source-video iChannel0 sampling with sampleVideo(uv) or context.sourceColor.
• Use Slang/HLSL names and syntax: float2, float3, float4, float2x2, lerp, frac, saturate, and mul(matrix, vector).
• Do not use GLSL-only types/functions such as vec2, vec3, vec4, mat2, mix, fract, mod, texture, or mainImage.
• Keep parameter IDs, texture IDs, font IDs, and function entry points as valid shader identifiers: letters, numbers, and underscores only, starting with a letter or underscore.
• Add only controls that are actually used by the shader.
• Prefer a small number of clear controls with conservative defaults.
• Keep shaders deterministic unless randomness is an explicit feature. For stable process-level variation, use context.startupRandom; for per-pixel pseudo-randomness, hash from uv, pixel coordinates, frameCount, or trigger values.
• If adapting third-party code, include attribution and source URL in the manifest description when the license allows adaptation.
• If the source license is unclear or incompatible, do not add the shader package.

Before finishing, compile-check the shader through the runtime wrapper or launch the app and verify the shader appears without an error in the selector. CI also runs shader validation, so every available package in shaders/ should compile successfully. Intentionally broken examples should stay visibly marked as broken rather than pretending to be production shaders.

Manifest Fields

shader.json is the runtime-facing description of the shader.

Required fields:

• id: package ID used by state/presets. Hyphenated names are OK here, for example my-effect.
• name: display name in the UI.
• parameters: array of exposed controls. Use [] if there are no user parameters.

Optional fields:

• description: display/help text for the shader library.
• category: UI grouping label.
• entryPoint: Slang function to call. Defaults to shadeVideo.
• passes: advanced render-pass declarations. Omit this for normal single-pass shaders.
• textures: texture assets to load and expose as samplers.
• fonts: packaged font assets for live text parameters.
• temporal: history-buffer requirements.
• feedback: optional previous-frame shader-local feedback surface.

Parameter objects may also include an optional description string. The control UI displays it as one-line helper text with the full text available on hover, so use it for short operational guidance rather than long documentation.

Shader-visible identifiers must be valid Slang-style identifiers:

• entryPoint
• parameter id
• texture id
• font id

Use letters, numbers, and underscores only, and start with a letter or underscore. For example, logoTexture is valid; logo-texture is not valid as a shader-visible texture ID.

Render Passes

Most shaders should omit passes. The runtime then creates one implicit pass:

{
  "id": "main",
  "source": "shader.slang",
  "entryPoint": "shadeVideo",
  "output": "layerOutput"
}

Advanced shaders may declare explicit passes. All passes may live in one .slang file by using different entryPoint values, or they may be split across multiple source files:

{
  "passes": [
    {
      "id": "blurX",
      "source": "blur-x.slang",
      "entryPoint": "blurHorizontal",
      "inputs": ["layerInput"],
      "output": "blurredX"
    },
    {
      "id": "final",
      "source": "final.slang",
      "entryPoint": "finish",
      "inputs": ["blurredX"],
      "output": "layerOutput"
    }
  ]
}

Pass fields:

• id: required pass identifier. It must be a valid shader identifier and unique inside the package.
• source: required Slang source path relative to the package directory.
• entryPoint: optional Slang function for this pass. Defaults to the package-level entryPoint.
• inputs: optional list of named inputs. The first input is used as the pass input texture.
• output: optional output name. Use layerOutput for the final visible layer result.

Pass input names:

• layerInput: the input to this layer, before any of its passes run.
• previousPass: the previous pass output in this layer. If there is no previous pass, this falls back to layerInput.
• Any earlier pass id or output name from the same layer.

If inputs is omitted, the first pass samples layerInput and later passes sample previousPass.

Single-file multipass example:

{
  "passes": [
    {
      "id": "mask",
      "source": "shader.slang",
      "entryPoint": "makeMask",
      "output": "maskBuffer"
    },
    {
      "id": "final",
      "source": "shader.slang",
      "entryPoint": "finish",
      "inputs": ["maskBuffer"],
      "output": "layerOutput"
    }
  ]
}

Pass output names:

• layerOutput: the final visible output of this layer.
• Any other name creates an intermediate 16-bit float render target that later passes may sample.

If the final declared pass does not explicitly output layerOutput, the runtime still treats that final pass as the visible layer output. Existing single-pass shaders are unaffected.

Feedback Surface

Shaders may opt in to a persistent previous-frame feedback surface:

{
  "feedback": {
    "enabled": true,
    "writePass": "final"
  }
}

Fields:

• enabled: when true, the runtime allocates one persistent RGBA16F feedback surface for this shader at the current render resolution.
• writePass: optional pass id whose output should become next frame's feedback surface. If omitted, the runtime uses the final declared pass, or the implicit main pass for single-pass shaders.

Behavior:

• all passes may sample the same previous-frame feedback surface
• one designated pass writes the next feedback surface
• feedback is previous-frame state, not same-frame pass chaining

Guardrails:

• Feedback is best suited to image-like state such as trails, masks, luminance fields, decay maps, and shader-local analysis buffers.
• Feedback is not a precise long-term data store. The surface uses RGBA16F, so repeated accumulation, exact counters, and tightly packed metadata can drift or clamp over time.
• The feedback surface is currently filtered like an image, not configured as strict texel-addressed storage. If you reserve texels as data slots, sample them carefully and do not assume exact CPU-style array semantics.
• Each feedback-enabled layer allocates two full-resolution feedback textures for ping-pong state. This increases VRAM use and adds one extra full-frame feedback copy per rendered frame.
• In multipass shaders, feedback remains previous-frame state even when a pass also consumes same-frame pass outputs. Do not treat feedback as another same-frame intermediate buffer.

Single-pass example:

{
  "id": "feedback-glow",
  "name": "Feedback Glow",
  "feedback": {
    "enabled": true
  },
  "parameters": []
}

Multipass example:

{
  "passes": [
    {
      "id": "analysis",
      "source": "shader.slang",
      "entryPoint": "analyzeFrame",
      "output": "analysisBuffer"
    },
    {
      "id": "final",
      "source": "shader.slang",
      "entryPoint": "finishFrame",
      "inputs": ["analysisBuffer"],
      "output": "layerOutput"
    }
  ],
  "feedback": {
    "enabled": true,
    "writePass": "final"
  }
}

The wrapper exposes:

float4 sampleFeedback(float2 uv);

On the first frame, or after a reset, sampleFeedback returns transparent black.

Feedback resets when:

• layers are added, removed, or reordered
• a layer bypass state changes
• a layer changes shader
• a shader is reloaded or recompiled
• render dimensions change
• the app restarts

So feedback should be treated as live runtime state, not durable saved state.

Slang Entry Point

Your shader file must implement the manifest entryPoint.

Default:

float4 shadeVideo(ShaderContext context)
{
    return context.sourceColor;
}

The runtime owns the real fragment shader entry point. Your function is called from the wrapper, and the runtime handles final bypass/mix behavior:

return lerp(context.sourceColor, effectedColor, mixValue);

That means:

• Return the fully effected color from your function.
• Respect alpha if your shader produces an overlay or sprite.
• The runtime will blend your result with the source according to mixAmount and bypass state.

ShaderContext

Your entry point receives:

struct ShaderContext
{
    float2 uv;
    float4 sourceColor;
    float2 inputResolution;
    float2 outputResolution;
    float time;
    float utcTimeSeconds;
    float utcOffsetSeconds;
    float startupRandom;
    float frameCount;
    float mixAmount;
    float bypass;
    int sourceHistoryLength;
    int temporalHistoryLength;
    int feedbackAvailable;
};

Fields:

• uv: normalized texture coordinates, usually 0..1.
• sourceColor: decoded RGBA source video at uv.
• inputResolution: decoded input video resolution in pixels.
• outputResolution: shader render resolution in pixels. The current pipeline renders the shader stack at input resolution, then scales the final frame to the configured video I/O output mode.
• time: elapsed runtime time in seconds.
• utcTimeSeconds: current UTC time of day from the host PC clock, expressed as seconds since UTC midnight.
• utcOffsetSeconds: host PC local UTC offset in seconds. Add this to utcTimeSeconds and wrap to 0..86400 to get local time of day.
• startupRandom: random 0..1 value generated once when the host process starts. It stays constant for the lifetime of the app and changes on the next launch.
• frameCount: incrementing frame counter.
• mixAmount: runtime mix amount.
• bypass: 1.0 when the layer is bypassed, otherwise 0.0.
• sourceHistoryLength: number of usable source-history frames currently available.
• temporalHistoryLength: number of usable temporal frames currently available for this layer.
• feedbackAvailable: 1 when previous-frame feedback exists for this layer, otherwise 0.

Color/precision notes:

• context.sourceColor, sampleVideo(), and temporal history samples are display-referred Rec.709-like RGB, not linear-light RGB.
• The current DeckLink backend prefers 10-bit YUV capture and output when the card/mode supports it, with automatic 8-bit fallback. If external keying is enabled, output prefers 10-bit YUVA (Ay10) when supported so shader alpha can drive the key signal, then falls back to 8-bit BGRA.
• Internal decoded, layer, composite, output, and temporal render targets are 16-bit floating point, so gradients and LUT work have more headroom than packed byte video I/O formats.
• Do not add extra Rec.709 or linear conversions unless the shader intentionally documents that behavior.

Helper Functions

The wrapper provides:

float4 sampleLayerInput(float2 uv);
float4 sampleVideo(float2 uv);
float4 sampleSourceHistory(int framesAgo, float2 uv);
float4 sampleTemporalHistory(int framesAgo, float2 uv);
float4 sampleFeedback(float2 uv);

sampleLayerInput samples the input arriving at this shader layer before any of the layer's own passes run. If this layer follows another shader, it sees that previous shader's output. If this is the first shader layer, it sees the decoded source image.

sampleVideo samples the current pass input texture. In single-pass shaders this is usually the layer input. In multipass shaders it may instead be a named pass output or previousPass, depending on the manifest routing for that pass.

sampleSourceHistory samples previous decoded source frames. framesAgo is clamped into the available range. If no history is available, it falls back to sampleVideo.

sampleTemporalHistory samples previous pre-layer input frames for temporal shaders that request preLayerInput history. framesAgo is clamped into the available range. If no temporal history is available, it falls back to sampleVideo.

sampleFeedback samples the shader-local previous-frame feedback surface. If feedback has not been written yet, it returns transparent black.

Example:

float4 shadeVideo(ShaderContext context)
{
    float4 previous = sampleSourceHistory(1, context.uv);
    return lerp(context.sourceColor, previous, 0.35);
}

Layer-input example:

float4 finishPass(ShaderContext context)
{
    float3 baseColor = sampleLayerInput(context.uv).rgb;
    float3 passResult = context.sourceColor.rgb;
    return float4(baseColor + passResult * 0.25, 1.0);
}

Feedback example:

float4 shadeVideo(ShaderContext context)
{
    float4 previous = sampleFeedback(context.uv);
    float4 current = context.sourceColor;
    return lerp(current, previous, 0.2);
}

Multipass feedback example:

float4 analyzeFrame(ShaderContext context)
{
    float4 previous = sampleFeedback(context.uv);
    float luma = dot(context.sourceColor.rgb, float3(0.2126, 0.7152, 0.0722));
    return float4(lerp(previous.rgb, float3(luma), 0.1), 1.0);
}

float4 finishFrame(ShaderContext context)
{
    float4 analysis = context.sourceColor;
    return float4(analysis.rgb, 1.0);
}

In that multipass case:

• analyzeFrame reads last frame's feedback
• finishFrame receives the same-frame pass output through normal multipass routing
• the writePass decides which pass output becomes next frame's feedback

That means:

• use context.sourceColor or sampleVideo() when you want this pass's routed input
• use sampleLayerInput() when you want the pre-pass layer input
• use sampleFeedback() when you want previous-frame persistent shader-local state

Parameters

Manifest parameters are exposed to Slang as global values with the same id.

Supported types:

Manifest type	Slang type	JSON value
float	float	number
vec2	float2	[x, y]
color	float4	[r, g, b, a]
bool	bool	true or false
enum	int	selected option index
text	generated texture/helper	string
trigger	int <id>, float <id>Time	pulse/count

Float example:

{
  "id": "brightness",
  "label": "Brightness",
  "type": "float",
  "default": 1.0,
  "min": 0.0,
  "max": 2.0,
  "step": 0.01
}

color.rgb *= brightness;

Vector example:

{
  "id": "offset",
  "label": "Offset",
  "type": "vec2",
  "default": [0.0, 0.0],
  "min": [-0.2, -0.2],
  "max": [0.2, 0.2],
  "step": [0.001, 0.001]
}

float2 uv = clamp(context.uv + offset, float2(0.0), float2(1.0));

Color example:

{
  "id": "tint",
  "label": "Tint",
  "type": "color",
  "default": [1.0, 1.0, 1.0, 1.0]
}

color *= tint;

Boolean example:

{
  "id": "invert",
  "label": "Invert",
  "type": "bool",
  "default": false
}

if (invert)
    color.rgb = 1.0 - color.rgb;

Enum example:

{
  "id": "mode",
  "label": "Mode",
  "type": "enum",
  "default": "normal",
  "options": [
    { "value": "normal", "label": "Normal" },
    { "value": "luma", "label": "Luma" },
    { "value": "posterize", "label": "Posterize" }
  ]
}

Enums are stored in presets/state by their string value, but exposed to Slang as a zero-based integer index in option order:

if (mode == 1)
{
    float luma = dot(color.rgb, float3(0.2126, 0.7152, 0.0722));
    color.rgb = float3(luma);
}
else if (mode == 2)
{
    color.rgb = floor(color.rgb * 4.0) / 4.0;
}

Text example:

{
  "fonts": [
    { "id": "inter", "path": "fonts/Inter-Regular.ttf" }
  ],
  "parameters": [
    {
      "id": "titleText",
      "label": "Title",
      "type": "text",
      "default": "LIVE",
      "font": "inter",
      "maxLength": 64
    }
  ]
}

Text parameters are runtime-owned strings. They are not emitted as uniform values. Instead, the runtime renders the current string into a single-line SDF mask texture and the shader wrapper exposes helpers based on the parameter id:

float mask = sampleTitleText(textUv);
float4 premultipliedText = drawTitleText(textUv, float4(1.0, 1.0, 1.0, 1.0));

Text is currently limited to printable ASCII. maxLength defaults to 64 and is clamped to 1..256. The optional font field references a packaged font declared in fonts; if no font is specified, the runtime uses its fallback sans-serif renderer.

Trigger example:

{
  "id": "flash",
  "label": "Flash",
  "type": "trigger"
}

A trigger appears as a button in the control UI. Pressing it increments the shader-visible integer flash and records the runtime time in flashTime:

float age = context.time - flashTime;
float intensity = flash > 0 ? exp(-age * 5.0) : 0.0;
color.rgb += intensity;

Triggers are useful for one-shot shader reactions such as flashes, ripples, cuts, or randomized looks. They do not execute arbitrary CPU code; they only update uniforms consumed by the shader.

Parameter validation:

• Float values are clamped to min/max if provided.
• vec2 must have exactly 2 numbers.
• color must have exactly 4 numbers.
• Enum defaults must match one of the declared option values.
• Text defaults must be strings. Non-printable characters are dropped and values are clamped to maxLength.
• Trigger values are incremented by the host when triggered. The shader sees the trigger count and last trigger time.
• Non-finite numeric values are rejected.

Texture Assets

Declare texture assets in the manifest:

{
  "textures": [
    {
      "id": "logoTexture",
      "path": "logo.png"
    }
  ]
}

Rules:

• id must be a valid shader identifier.
• path is relative to the shader package directory.
• The file must exist when the manifest is loaded.
• Texture asset changes trigger shader reload.

Texture IDs become Sampler2D<float4> globals:

float4 logo = logoTexture.Sample(logoUv);

For sprite or overlay shaders, return premultiplied-looking output if you want clean composition:

float alpha = logo.a;
return float4(logo.rgb * alpha, alpha);

See shaders/dvd-bounce/ for a complete texture-driven example.

Font Assets

Declare packaged font assets in the manifest:

{
  "fonts": [
    {
      "id": "inter",
      "path": "fonts/Inter-Regular.ttf"
    }
  ]
}

Rules:

• id must be a valid shader identifier.
• path is relative to the shader package directory.
• The file must exist when the manifest is loaded.
• Font asset changes trigger shader reload.
• V1 text layout is single-line; shaders position and scale the generated text texture themselves.

See shaders/text-overlay/ for a complete live text example. The sample bundles Roboto Regular and includes its OFL license beside the font file.

Temporal Shaders

Temporal shaders can request access to previous frames.

Manifest example:

{
  "temporal": {
    "enabled": true,
    "historySource": "preLayerInput",
    "historyLength": 12
  }
}

Supported historySource values:

• source: decoded source-video history from previous frames.
• preLayerInput: history of the input arriving at this layer before the shader runs.

historyLength is the requested frame count. The runtime clamps it by maxTemporalHistoryFrames in config/runtime-host.json.

Temporal history resets when:

• layers are added, removed, or reordered
• a layer bypass state changes
• a layer changes shader
• a shader is reloaded or recompiled
• render dimensions change

Use the available history lengths to avoid assuming history is ready on the first frame:

float4 shadeVideo(ShaderContext context)
{
    if (context.temporalHistoryLength <= 0)
        return context.sourceColor;

    float4 oldFrame = sampleTemporalHistory(3, context.uv);
    return lerp(context.sourceColor, oldFrame, 0.4);
}

See shaders/temporal-ghost-trail/ and shaders/temporal-low-fps/ for examples.

Coordinate And Color Notes

• uv is normalized.
• Use context.outputResolution for pixel-sized effects.
• Use context.inputResolution when sampling source video by input pixel size.
• sourceColor and sampleVideo return RGBA values in normalized 0..1 range.
• Prefer saturate(color) or explicit clamp before returning if your math can overshoot.
• For generated calibration charts, test patterns, gradients, and exposure ramps, state whether patch values are linear-light, display-referred gamma encoded, Rec.709 encoded, or intentionally artistic.
• For one-stop exposure patches, each patch should normally be baseLevel * 2^patchIndex before any display/tone encoding.
• For Rec.709 OETF encoding, use:

float rec709Oetf(float linearLevel)
{
    float value = saturate(linearLevel);
    if (value < 0.018)
        return 4.5 * value;
    return 1.099 * pow(value, 0.45) - 0.099;
}

Pixel-size example:

float2 pixel = 1.0 / max(context.outputResolution, float2(1.0));
float4 right = sampleVideo(context.uv + float2(pixel.x, 0.0));

Animation And Timing Notes

• context.time is elapsed runtime time in seconds and is the default animation source for generative shaders.
• context.frameCount increments once per rendered output frame and is useful when an effect must be frame-locked.
• Avoid expensive CPU-like timing logic in the shader; animation should usually be a simple function of context.time, context.frameCount, trigger uniforms, or parameters.
• If a shader appears to judder only while animated, first test whether freezing its time removes the issue. That usually separates animation cadence issues from rendering or transfer issues.
• Do not add custom timer uniforms to the wrapper. Use the fields already in ShaderContext.

Performance Notes

The app has to meet a fixed video frame cadence, so avoid shader code that only looks good in unconstrained browser demos.

Guidelines:

• Keep loops bounded with compile-time constants where possible.
• Avoid very high per-pixel raymarch counts by default. If a heavy loop is needed, expose a quality/steps control with a safe default.
• Prefer early exits only when they are simple; highly divergent branches can be expensive across a full frame.
• Avoid repeated texture sampling in large loops unless the effect really needs it.
• Use context.outputResolution carefully. A 1080p frame is over 2 million fragments; a tiny extra loop can become expensive.
• The UI render time may measure CPU command submission rather than true GPU execution time, so visual frame issues can still be GPU-related even when reported render time is small.
• Do not write debug files, allocate resources, or assume CPU-side work can happen from shader.slang. Shader code is GPU-only.

Reload And Generated Files

When a shader compiles, the runtime writes generated files under runtime/shader_cache/:

• active_shader_wrapper.slang
• active_shader.raw.frag
• active_shader.frag

These files are ignored by git and are useful for debugging compiler output. If a shader fails to compile, inspect the wrapper first; it shows the exact generated Slang code including your included shader.

For multipass shaders, these files reflect the most recently compiled pass. If a package has several passes, the reported compile error and pass name are usually more useful than assuming the cache contains the first pass.

Common Pitfalls

• Do not use hyphens in parameter IDs, texture IDs, or entry point names.
• Do not declare your own ShaderContext, GlobalParams, sampleVideo, sampleSourceHistory, or sampleTemporalHistory.
• Do not write a [shader("fragment")] entry point in shader.slang; the runtime provides it.
• Remember enum globals are integer indexes, not strings.
• Declare every texture in shader.json; undeclared texture samplers will not be bound.
• Declare packaged fonts in shader.json when text parameters should use a specific font.
• Keep temporal history requests modest. They consume texture units and memory and are capped by runtime config.
• If a parameter appears in the UI but not in Slang, the shader may still compile, but the control has no effect.
• If a Slang name collides with a generated global, rename your parameter or local symbol.

Minimal Package Checklist

Before committing a new shader package:

• shader.json is valid JSON.
• id is unique across shaders/.
• entryPoint, parameter IDs, and texture IDs are valid identifiers.
• shader.slang implements the configured entry point.
• Texture files referenced by textures exist.
• Font files referenced by fonts exist.
• Enum defaults are present in their options.
• Temporal shaders handle short or empty history gracefully.
• The app can reload and compile the shader without errors.