video-shader-toys/docs/SHADER_FEEDBACK_TARGET_IDEA.md

Shader Feedback Target Idea

This note summarizes a possible feature where a shader can request a persistent render target for storing and reusing its own internal information across frames.

Goal

Allow a shader to keep shader-local state without writing arbitrary values back into host-owned parameters.

This is useful for cases like:

• storing sampled color information across frames
• storing luminance or mask values per pixel
• keeping running filtered values
• tracking simple analysis data
• reserving a small texel region as an array-like metadata store

Core Idea

A shader may opt in, via shader.json, to receive one persistent RGBA16F render target at client/output resolution.

The intended model is:

• the shader writes into the feedback target this frame
• the shader reads the previous frame’s feedback target on the next frame

This makes it a shader-local “previous frame state” surface.

Why This Makes Sense

This is a better fit than letting shaders push arbitrary values back into the host because:

• state stays inside the render domain
• shaders remain render-focused rather than becoming host-state mutators
• host-owned parameters, UI state, and persistence remain predictable
• timing is easier to reason about
• it fits naturally with multipass and temporal rendering patterns

Recommended Behavior

1. Make it opt-in

Shaders should explicitly request this capability in shader.json.

Reasons:

• most shaders will not need it
• it avoids unnecessary VRAM/bandwidth cost
• it keeps shader capabilities explicit
• it avoids silently changing the contract for every shader

The runtime should only allocate/bind the feedback surface for shaders that request it.

2. Start with previous-frame feedback only

The first version should expose only one frame of history:

• current frame writes
• next frame reads previous state

Reasons:

• simpler mental model
• lower memory cost
• easier to document
• enough for many practical use cases

If a shader wants longer memory, it can accumulate or encode that over time into the same persistent surface.

3. Keep it shader-local

The feedback target should be treated as internal shader state, not as host-visible parameter state.

That means:

• it does not automatically update exposed parameters
• it does not automatically show up in the UI
• it does not automatically persist to runtime state

4. Keep it separate from normal multipass chaining

This feedback target should not replace or blur the meaning of the existing multipass system.

The clean model is:

• normal multipass outputs are for same-frame chaining
• the feedback target is for previous-frame persistent state

In other words:

• pass A can write an output that pass B reads later in the same frame
• the feedback target written during frame N is read back during frame N + 1

That means the feedback target should be thought of as a separate cross-frame resource, not as “another pass output.”

Recommended behavior for multipass shaders that request feedback:

• all passes in the shader may read the same previous-frame feedback surface
• one designated pass should produce the next feedback surface for the following frame
• feedback writes should not be interpreted as same-frame pass-to-pass communication

This avoids ambiguity such as:

• whether pass 2 sees pass 1’s feedback writes from the same frame
• whether multiple passes are racing to write the persistent surface
• whether feedback is supposed to mean same-frame scratch space or next-frame state

The intended separation is:

• use named pass outputs and previousPass for same-frame chaining
• use the feedback target for persistent previous-frame state

What It Could Store

Because the target would be a full-resolution RGBA16F texture, a shader could use it in a few ways.

Full-frame per-pixel storage

Examples:

• luminance per pixel
• confidence/mask values
• filtered or decayed image information
• rolling per-pixel state used by a temporal effect

Small array-like metadata regions

A shader could reserve a few texels or a small block as a logical data region.

Example:

• pixel (0, 0) stores value 0
• pixel (1, 0) stores value 1
• pixel (2, 0) stores value 2

Because each texel is RGBA16F, one texel can hold up to four scalar values.

This makes it possible to emulate a small array-like structure inside the texture.

Important Caveats

This is not a true random-access structured buffer. It is still a texture-backed GPU surface.

That means:

• it is best suited to texel- or pixel-oriented storage
• per-pixel “write to your own location” patterns are natural
• many-to-one reductions or arbitrary scatter writes are harder
• precision is limited to half-float storage

So the main question is usually not “can the shader store this?” but “can the shader update it cleanly with fragment-style GPU access?”

Example Use Case

For a greenscreen workflow, a shader could:

• sample a small box region of the input
• compute an average or representative screen color
• store that color in reserved texels of the feedback target
• reuse that stored value next frame as its internal key color

This would let the shader maintain its own sampled screen color over time without mutating the exposed host-side screenColor parameter.

Multipass Interaction Summary

For a multipass shader, the most sensible mental model is:

• same-frame intermediate images still flow through the existing pass system
• previous-frame persistent state flows through the feedback target

So if a shader has multiple passes:

• pass outputs are still used for within-frame work
• the feedback target is read as last frame’s stored state
• the feedback target is written once for use on the next frame

This keeps the feature understandable and prevents the feedback surface from becoming a confusing second pass graph.

Recommended First Version

The simplest strong first version would be:

• opt-in via shader.json
• one persistent RGBA16F target
• full client/output resolution
• shader reads previous frame’s feedback
• shader writes current frame’s feedback
• no deeper history at first
• no automatic host writeback

Summary

This feature would give shaders a safe, GPU-native way to hold internal state across frames.

The recommended approach is:

• make it opt-in per shader
• keep it shader-local
• expose only previous-frame feedback initially
• treat it as a persistent render-state surface, not host parameter state

That keeps the design powerful without crossing the architectural boundary into shader-driven host mutation.