video-shader-toys/docs/subsystems/RuntimeStore.md

RuntimeStore Subsystem Design

This document expands the RuntimeStore portion of PHASE_1_SUBSYSTEM_BOUNDARIES_DESIGN.md into a subsystem-specific design note.

The purpose of RuntimeStore is to give the Phase 1 target architecture one clear home for durable runtime data. Before the Phase 1 runtime split, that responsibility was spread through RuntimeHost, where persistence, mutation entrypoints, render-state building, shader metadata access, and status reporting all shared the same object and lock domain. RuntimeStore is the design boundary that separates "what the app knows and saves" from "how the app decides to mutate it" and "how rendering consumes it."

Role In The Phase 1 Architecture

Within the Phase 1 subsystem model, RuntimeStore is the durable data authority.

It exists to answer questions like:

• what runtime configuration is currently loaded
• what the saved layer stack structure is
• what the saved parameter values are
• what stack presets exist and what they contain
• what package and manifest metadata is available for validation and snapshot building

It should not answer questions like:

• should this control mutation be allowed
• should this OSC value be treated as transient or persisted
• how should the render thread consume state
• when should output frames be scheduled
• what warnings should be shown to the operator

That policy belongs elsewhere:

• mutation policy: RuntimeCoordinator
• render-facing publication: RuntimeSnapshotProvider
• hardware timing: VideoBackend
• operational visibility: HealthTelemetry

Design Goals

RuntimeStore should optimize for:

• explicit ownership of durable runtime data
• predictable disk-backed load and save behavior
• minimal knowledge of GL, callbacks, or live playout timing
• stable read models for validation and snapshot building
• a clean seam for introducing debounced or asynchronous persistence later
• testability without GPU or DeckLink dependencies

Responsibilities

RuntimeStore owns persisted and operator-authored state.

Primary responsibilities:

• load runtime host configuration from disk
• load saved runtime state from disk
• save runtime state snapshots to disk
• own the stored layer stack model
• own persisted parameter values and bypass flags
• own stack preset serialization and deserialization
• own package/manifest metadata needed across renders and reloads
• expose query/read APIs over stored state
• expose write APIs for coordinator-approved durable mutations
• normalize or repair stored data at load boundaries when necessary

Secondary responsibilities that still fit here:

• path resolution for runtime state and preset files
• preset name normalization/file-stem safety
• compatibility handling for older saved-state schemas
• default seeding of initial persistent state when no saved runtime exists

Non-Responsibilities

RuntimeStore must not become a general convenience layer again.

It does not own:

• render-thread timing
• GL objects or resource lifetime
• shader compilation orchestration
• render-local transient state such as temporal history, feedback buffers, preview caches, or playout queues
• OSC smoothing, coalescing, or overlay application
• websocket broadcast policy
• REST or OSC ingress handling
• device callbacks, queue-depth policy, or preroll policy
• app-wide health aggregation

It also should not directly decide:

• whether a mutation is valid in policy terms
• whether a change should persist immediately, eventually, or not at all
• when a new render snapshot should be published
• whether a reload should be treated as config-only, package-only, or render-affecting

Those are coordinator concerns, not store concerns.

State Ownership

RuntimeStore should own the following state categories.

Phase 5 names this boundary in code through RuntimeStateLayerModel: persisted layer stack data, saved parameter values, and stack presets are classified as base persisted state. Operator/session values are owned by CommittedLiveState; their mutation policy is committed-live policy owned by the coordinator, not durable-store policy by default.

Phase 5 also adds CommittedLiveState as the physical owner of current session/operator layer state and CommittedLiveStateReadModel as the named read boundary for render snapshot publication. RuntimeStore still owns file IO, config, package metadata, preset persistence, and persistence requests, but it delegates current-session layer mutations to CommittedLiveState.

Runtime Configuration

Examples:

• server/control ports
• OSC bind address
• OSC smoothing defaults
• runtime paths and directory configuration
• any host-side configuration loaded from config/runtime-host.json

This data is durable, file-backed, and not inherently render-local.

Persistent Layer Stack State

Examples:

• ordered layer list
• stable layer ids
• selected shader id per layer
• bypass state
• persisted parameter values

This is the stored "official" layer model, not a render-thread working copy.

Stack Presets

Examples:

• preset names
• serialized saved layer stacks under runtime/stack_presets

Preset files are durable artifacts and should remain in the store domain even if later phases add async writing.

Shader/Package Metadata Needed As Durable Reference Data

Examples:

• discovered shader package manifests
• parameter definitions used for validation/default restoration
• manifest-level capability metadata such as temporal history and feedback declarations
• package ordering that should survive across reloads

Important distinction:

• manifest and package metadata belongs here
• render-ready compiled programs and GPU resources do not

Load-Time Compatibility/Repair State

Examples:

• schema version adaptation
• default value filling for missing parameters
• removal or migration of layers that reference missing packages
• preset compatibility cleanup

This should be treated as store hygiene during ingest, not runtime mutation policy.

Data Model Boundaries

RuntimeStore should present data in durable-model terms rather than live-render terms.

Core model groupings:

• RuntimeConfigModel
• PersistentLayerStackModel
• LayerStoredState
• StoredParameterValue
• StackPresetModel
• ShaderPackageCatalog or equivalent durable package registry view

The exact C++ types may differ from these names, but the boundary should hold:

• store models describe durable intent
• snapshot models describe render consumption

That means RuntimeStore should not expose render-optimized structures such as RuntimeRenderState directly as its primary interface.

Interface Shape

The Phase 1 architecture doc already sketches the high-level interface. This section expands it.

Load / Save Interface

Expected responsibilities:

• LoadConfig()
• LoadPersistentState()
• SavePersistentStateSnapshot(...)
• LoadStackPreset(...)
• SaveStackPreset(...)
• GetStackPresetNames()

Design notes:

• Load* operations should parse and normalize external file content into durable in-memory models.
• Save* operations should serialize durable models without needing render or control subsystem context.
• later debounce/background writing should wrap these operations, not redefine their ownership

Read Interface

Expected responsibilities:

• GetRuntimeConfig()
• GetStoredLayerStack()
• FindStoredLayer(...)
• GetShaderPackageCatalog()
• GetStackPresetNames()
• BuildPersistenceSnapshot() or equivalent stable serialization input

Design notes:

• read APIs should support coordinator validation and snapshot building
• read APIs should avoid exposing raw mutable internals across subsystem boundaries
• stable read snapshots from the store are fine; render snapshots are still the snapshot provider's job

Write Interface

Expected responsibilities:

• SetStoredLayerStack(...)
• ReplaceStoredLayer(...)
• SetStoredParameterValue(...)
• SetStoredBypassState(...)
• SetStoredShaderSelection(...)
• ReplaceShaderPackageCatalog(...)

Design notes:

• writes should assume the coordinator already decided the mutation is allowed
• store APIs may still enforce structural invariants and shape correctness
• writes should not contain ingress-specific policy like OSC smoothing or UI throttling

Normalization / Validation-Support Interface

Expected responsibilities:

• NormalizeLoadedState(...)
• EnsureStoredDefaults(...)
• MakeSafePresetFileStem(...)
• package lookup helpers for parameter-definition queries

Design notes:

• lightweight structure and schema validation belongs here
• policy validation belongs in the coordinator
• render compatibility translation belongs in the snapshot provider

Concurrency Expectations

RuntimeStore should be designed as a shared data authority, but not as the app's global lock for everything.

Phase 1 design expectations:

• coordinator-driven writes may still be synchronized internally
• read APIs should be safe for coordinator and snapshot-provider use
• render should not directly take a large mutable store lock in the target architecture

This implies:

• RuntimeStore may keep an internal mutex during migration
• that mutex should protect durable models only
• render-facing consumers should eventually read via RuntimeSnapshotProvider, not by reaching into the store

One of the main goals here is avoiding the old situation where runtime lock scope effectively mixed:

• persistent state
• status reporting
• render-state caches
• timing stats
• reload flags

RuntimeStore should sharply narrow that scope.

Dependency Rules

Per the Phase 1 subsystem design, RuntimeStore should sit low in the dependency graph.

Allowed inbound dependencies:

• RuntimeCoordinator -> RuntimeStore
• RenderSnapshotBuilder -> RuntimeStore
• temporary migration shims from ControlServices only where explicitly tolerated

Allowed outbound dependencies:

• file/serialization helpers
• package manifest parsing helpers
• pure utility types

Not allowed:

• RuntimeStore -> RenderEngine
• RuntimeStore -> VideoBackend
• RuntimeStore -> ControlServices
• RuntimeStore -> HealthTelemetry for behavior control

The store may emit errors or return result objects, but it should not coordinate the rest of the system directly.

Current Code Mapping

Before the Phase 1 runtime split, RuntimeHost contained many responsibilities that needed to move into RuntimeStore or adjacent runtime collaborators.

Previous code paths:

• config load:
  • RuntimeHost.cpp
• persistent state load:
  • RuntimeHost.cpp
• persistent state save:
  • RuntimeHost.cpp
• preset save/load:
  • RuntimeHost.cpp
  • RuntimeHost.cpp
• state serialization helpers:
  • RuntimeHost.cpp
  • RuntimeHost.cpp
  • RuntimeHost.cpp
• path and file helpers:
  • RuntimeHost.cpp
  • RuntimeHost.cpp
  • RuntimeHost.cpp

Durable-state mutation entrypoints that previously lived on RuntimeHost but conceptually split between coordinator and store:

• layer stack edits:
  • AddLayer
  • RemoveLayer
  • MoveLayer
  • MoveLayerToIndex
• committed-state edits:
  • SetLayerBypass
  • SetLayerShader
  • UpdateLayerParameter
  • ResetLayerParameters

The target split should be:

• validation/policy/orchestration -> RuntimeCoordinator
• durable state write application -> RuntimeStore

Methods that were intentionally not moved into RuntimeStore because they belong under other runtime subsystems:

• render-state building and caching:
  • GetLayerRenderStates
  • TryRefreshCachedLayerStates
  • BuildLayerRenderStatesLocked
• status/timing reporting:
  • SetSignalStatus
  • SetPerformanceStats
  • SetFramePacingStats
  • AdvanceFrame
• live reload flags/polling shell:
  • PollFileChanges
  • ManualReloadRequested
  • ClearReloadRequest

Those belong under other target subsystems.

Proposed Internal Subcomponents

RuntimeStore does not need to be one monolithic class forever. A practical internal shape would be:

• RuntimeConfigStore
  • runtime host config load and resolved paths

The current codebase has completed this part of the split: RuntimeConfigStore owns config parsing, path resolution, configured ports/formats, runtime roots, and shader compiler paths, while RuntimeStore exposes compatibility-shaped delegates for existing callers.

• CommittedLiveState
  • current committed/session layer stack and parameter values
  • layer CRUD/reorder and shader selection for the running session
  • committed-live read model for snapshot publication
• LayerStackStore
  • backing layer stack mechanics used by committed-live state
  • layer CRUD/reorder and shader selection helpers
  • stack preset value serialization/load helpers
• RuntimeStatePresenter / RuntimeStateJson
  • runtime-state JSON assembly
  • layer-stack presentation serialization
• RenderSnapshotBuilder
  • render-state assembly and parameter refresh
  • dynamic frame-field refresh and render snapshot version counters
• ShaderPackageCatalog
  • durable manifest/package metadata
  • shader package scanning, status/order/lookup, and asset/source change comparison
• PersistenceWriter helper
  • synchronous at first, async/debounced later

The current codebase has completed the committed-live split: CommittedLiveState owns current committed/session layer state using LayerStackStore backing mechanics. RuntimeStore keeps file IO, package metadata, persistence serialization, persistence requests, preset file access, and facade methods for existing callers.

The current codebase has completed the render snapshot split: RenderSnapshotBuilder owns render-state assembly, cached parameter refresh, dynamic frame-field refresh, and render snapshot versions. RuntimeSnapshotProvider depends on this builder rather than on RuntimeStore friendship.

The current codebase has also completed the presentation split: RuntimeStatePresenter owns top-level runtime-state JSON assembly, while RuntimeStateJson owns the layer-stack and parameter presentation shape used by runtime state clients.

The current codebase has also completed the package split: ShaderPackageCatalog owns package scanning and registry comparison, while RuntimeStore uses it to keep layer state valid and to build compatibility read models.

These can still be presented through one subsystem façade during migration.

Persistence Model

The store should treat persistence as durable snapshot management, not incremental side-effect spraying.

Target behavior:

• in-memory durable models are updated first
• serialization snapshots are built from those models
• save requests persist a coherent snapshot

This matters because the current code still calls SavePersistentState() directly from many mutation paths. That is one of the architectural pressure points already called out in ARCHITECTURE_RESILIENCE_REVIEW.md.

The Phase 1 design for RuntimeStore should therefore assume:

• store ownership of serialization remains
• immediate save-after-mutate is a migration detail, not the final behavioral contract

By Phase 6, a background snapshot writer may sit underneath or beside this subsystem, but the durable model still belongs here.

Migration Plan From Current Code

The safest migration path is to extract responsibilities by interface, not by big-bang rename.

Step 1: Introduce The RuntimeStore Name And Facade

Create a facade interface for the durable-data parts that used to live in RuntimeHost.

Initial likely contents:

• config load/save access
• persistent layer-stack get/set access
• preset load/save access
• package catalog read access

This stage is complete: RuntimeStore owns its durable/session backing fields directly rather than wrapping a RuntimeHost object.

Step 2: Move Pure Persistence Helpers First

Low-risk extractions:

• path resolution helpers
• file read/write helpers
• preset enumeration and serialization helpers
• persistent-state serialization/deserialization helpers

These have relatively low coupling to GL and backend timing.

Step 3: Split Durable Models From Render Cache/Status Fields

Move out or conceptually separate:

• mPersistentState
• runtime config fields
• preset roots and runtime roots
• package catalog/order metadata

From fields that should stay elsewhere:

• render-state dirty flags and caches
• status/timing counters
• reload flags

This is one of the most important separations in the whole program.

Step 4: Route Durable Mutations Through Coordinator-Owned Policy

Once the coordinator exists, RuntimeStore write calls should become lower-level and less policy-rich.

Examples:

• SetStoredParameterValue(...) rather than ApplyOscTargetByControlKey(...)
• ReplaceStoredLayerStack(...) rather than LoadStackPreset(...) directly mutating every downstream concern

Step 5: Keep Render Off The Store

As RuntimeSnapshotProvider arrives, render should stop reading store internals directly.

That is the moment where RuntimeStore becomes a proper durable authority instead of a shared mutable app center.

Risks

1. Recreating RuntimeHost Under A New Name

The biggest risk is calling something RuntimeStore while leaving policy, status, and render-cache behavior attached.

Guardrail:

• only durable data and store hygiene belong here

2. Letting Validation Drift Into Persistence

Store-level shape validation is appropriate. High-level mutation policy is not.

Risk examples:

• store decides whether OSC should persist
• store decides whether a layer reorder should trigger snapshot publication
• store decides whether a reload is render-only or package-affecting

Those are coordinator decisions.

3. Overexposing Mutable Internals

If callers keep direct mutable access to the underlying vectors/maps, the subsystem boundary will exist only on paper.

Guardrail:

• prefer controlled write methods and stable read models

4. Coupling Package Metadata Too Tightly To Compile Outputs

Package manifest and parameter-definition metadata belongs here. Compiled program state does not.

Guardrail:

• keep compile products and GPU artifacts out of the store

5. Using The Store Lock As A Global Synchronization Shortcut

This would recreate timing and contention issues in a new form.

Guardrail:

• store locking protects durable models only
• render synchronization must happen through snapshots, not by sharing the store lock

Open Questions

1. How Much Shader Package Data Should Live Here?

Clear yes:

• manifest metadata
• parameter definitions
• package discovery/order information

Still open:

• whether compile-ready transformed sources belong here or in a later build-focused subsystem

Current recommendation:

• keep only durable reference/package metadata here

2. Should Preset Application Be A Store Operation Or A Coordinator Operation?

The file load and preset parse clearly belong here.

The policy question of how a loaded preset affects live state, snapshot publication, overlays, and notifications belongs in the coordinator.

Current recommendation:

• RuntimeStore loads preset content
• RuntimeCoordinator decides how to apply it

3. How Early Should Async Persistence Land?

Phase 1 does not require it, but the store design should not block it.

Current recommendation:

• keep synchronous save semantics initially if needed
• shape the interfaces so a background writer can be introduced without changing subsystem ownership

Success Criteria For This Subsystem

RuntimeStore can be considered well-defined once the codebase can say, without ambiguity:

• all durable runtime config and saved layer data has one authoritative home
• stack presets are owned by that same durable-data subsystem
• render does not depend on store internals directly
• timing/status/reporting state is no longer mixed into the same subsystem
• persistence ownership is clear even before async persistence is introduced

Short Version

RuntimeStore is the subsystem that should answer:

• what durable runtime data exists
• what saved layer stack and parameters exist
• what presets and package metadata exist
• how that durable data is loaded and serialized

It should not answer:

• whether a mutation should happen
• how rendering should consume state
• how hardware pacing should work
• what health warnings should be emitted

If this boundary holds, later phases can continue without recreating the old coupling under a different class name.