docs

docs/subsystems/RuntimeStore.md

@@ -0,0 +1,573 @@
+# RuntimeStore Subsystem Design
+
+This document expands the `RuntimeStore` portion of [PHASE_1_SUBSYSTEM_BOUNDARIES_DESIGN.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/PHASE_1_SUBSYSTEM_BOUNDARIES_DESIGN.md) into a subsystem-specific design note.
+
+The purpose of `RuntimeStore` is to give the Phase 1 target architecture one clear home for durable runtime data. In the current codebase, that responsibility is spread through `RuntimeHost`, where persistence, mutation entrypoints, render-state building, shader metadata access, and status reporting all share the same object and lock domain. `RuntimeStore` is the design boundary that separates "what the app knows and saves" from "how the app decides to mutate it" and "how rendering consumes it."
+
+## Role In The Phase 1 Architecture
+
+Within the Phase 1 subsystem model, `RuntimeStore` is the durable data authority.
+
+It exists to answer questions like:
+
+- what runtime configuration is currently loaded
+- what the saved layer stack structure is
+- what the saved parameter values are
+- what stack presets exist and what they contain
+- what package and manifest metadata is available for validation and snapshot building
+
+It should not answer questions like:
+
+- should this control mutation be allowed
+- should this OSC value be treated as transient or persisted
+- how should the render thread consume state
+- when should output frames be scheduled
+- what warnings should be shown to the operator
+
+That policy belongs elsewhere:
+
+- mutation policy: `RuntimeCoordinator`
+- render-facing publication: `RuntimeSnapshotProvider`
+- hardware timing: `VideoBackend`
+- operational visibility: `HealthTelemetry`
+
+## Design Goals
+
+`RuntimeStore` should optimize for:
+
+- explicit ownership of durable runtime data
+- predictable disk-backed load and save behavior
+- minimal knowledge of GL, callbacks, or live playout timing
+- stable read models for validation and snapshot building
+- a clean seam for introducing debounced or asynchronous persistence later
+- testability without GPU or DeckLink dependencies
+
+## Responsibilities
+
+`RuntimeStore` owns persisted and operator-authored state.
+
+Primary responsibilities:
+
+- load runtime host configuration from disk
+- load saved runtime state from disk
+- save runtime state snapshots to disk
+- own the stored layer stack model
+- own persisted parameter values and bypass flags
+- own stack preset serialization and deserialization
+- own package/manifest metadata needed across renders and reloads
+- expose query/read APIs over stored state
+- expose write APIs for coordinator-approved durable mutations
+- normalize or repair stored data at load boundaries when necessary
+
+Secondary responsibilities that still fit here:
+
+- path resolution for runtime state and preset files
+- preset name normalization/file-stem safety
+- compatibility handling for older saved-state schemas
+- default seeding of initial persistent state when no saved runtime exists
+
+## Non-Responsibilities
+
+`RuntimeStore` must not become a general convenience layer again.
+
+It does not own:
+
+- render-thread timing
+- GL objects or resource lifetime
+- shader compilation orchestration
+- render-local transient state such as temporal history, feedback buffers, preview caches, or playout queues
+- OSC smoothing, coalescing, or overlay application
+- websocket broadcast policy
+- REST or OSC ingress handling
+- device callbacks, queue-depth policy, or preroll policy
+- app-wide health aggregation
+
+It also should not directly decide:
+
+- whether a mutation is valid in policy terms
+- whether a change should persist immediately, eventually, or not at all
+- when a new render snapshot should be published
+- whether a reload should be treated as config-only, package-only, or render-affecting
+
+Those are coordinator concerns, not store concerns.
+
+## State Ownership
+
+`RuntimeStore` should own the following state categories.
+
+### Runtime Configuration
+
+Examples:
+
+- server/control ports
+- OSC bind address
+- OSC smoothing defaults
+- runtime paths and directory configuration
+- any host-side configuration loaded from `config/runtime-host.json`
+
+This data is durable, file-backed, and not inherently render-local.
+
+### Persistent Layer Stack State
+
+Examples:
+
+- ordered layer list
+- stable layer ids
+- selected shader id per layer
+- bypass state
+- persisted parameter values
+
+This is the stored "official" layer model, not a render-thread working copy.
+
+### Stack Presets
+
+Examples:
+
+- preset names
+- serialized saved layer stacks under `runtime/stack_presets`
+
+Preset files are durable artifacts and should remain in the store domain even if later phases add async writing.
+
+### Shader/Package Metadata Needed As Durable Reference Data
+
+Examples:
+
+- discovered shader package manifests
+- parameter definitions used for validation/default restoration
+- manifest-level capability metadata such as temporal history and feedback declarations
+- package ordering that should survive across reloads
+
+Important distinction:
+
+- manifest and package metadata belongs here
+- render-ready compiled programs and GPU resources do not
+
+### Load-Time Compatibility/Repair State
+
+Examples:
+
+- schema version adaptation
+- default value filling for missing parameters
+- removal or migration of layers that reference missing packages
+- preset compatibility cleanup
+
+This should be treated as store hygiene during ingest, not runtime mutation policy.
+
+## Data Model Boundaries
+
+`RuntimeStore` should present data in durable-model terms rather than live-render terms.
+
+Core model groupings:
+
+- `RuntimeConfigModel`
+- `PersistentLayerStackModel`
+- `LayerStoredState`
+- `StoredParameterValue`
+- `StackPresetModel`
+- `ShaderPackageCatalog` or equivalent durable package registry view
+
+The exact C++ types may differ from these names, but the boundary should hold:
+
+- store models describe durable intent
+- snapshot models describe render consumption
+
+That means `RuntimeStore` should not expose render-optimized structures such as `RuntimeRenderState` directly as its primary interface.
+
+## Interface Shape
+
+The Phase 1 architecture doc already sketches the high-level interface. This section expands it.
+
+### Load / Save Interface
+
+Expected responsibilities:
+
+- `LoadConfig()`
+- `LoadPersistentState()`
+- `SavePersistentStateSnapshot(...)`
+- `LoadStackPreset(...)`
+- `SaveStackPreset(...)`
+- `GetStackPresetNames()`
+
+Design notes:
+
+- `Load*` operations should parse and normalize external file content into durable in-memory models.
+- `Save*` operations should serialize durable models without needing render or control subsystem context.
+- later debounce/background writing should wrap these operations, not redefine their ownership
+
+### Read Interface
+
+Expected responsibilities:
+
+- `GetRuntimeConfig()`
+- `GetStoredLayerStack()`
+- `FindStoredLayer(...)`
+- `GetShaderPackageCatalog()`
+- `GetStackPresetNames()`
+- `BuildPersistenceSnapshot()` or equivalent stable serialization input
+
+Design notes:
+
+- read APIs should support coordinator validation and snapshot building
+- read APIs should avoid exposing raw mutable internals across subsystem boundaries
+- stable read snapshots from the store are fine; render snapshots are still the snapshot provider's job
+
+### Write Interface
+
+Expected responsibilities:
+
+- `SetStoredLayerStack(...)`
+- `ReplaceStoredLayer(...)`
+- `SetStoredParameterValue(...)`
+- `SetStoredBypassState(...)`
+- `SetStoredShaderSelection(...)`
+- `ReplaceShaderPackageCatalog(...)`
+
+Design notes:
+
+- writes should assume the coordinator already decided the mutation is allowed
+- store APIs may still enforce structural invariants and shape correctness
+- writes should not contain ingress-specific policy like OSC smoothing or UI throttling
+
+### Normalization / Validation-Support Interface
+
+Expected responsibilities:
+
+- `NormalizeLoadedState(...)`
+- `EnsureStoredDefaults(...)`
+- `MakeSafePresetFileStem(...)`
+- package lookup helpers for parameter-definition queries
+
+Design notes:
+
+- lightweight structure and schema validation belongs here
+- policy validation belongs in the coordinator
+- render compatibility translation belongs in the snapshot provider
+
+## Concurrency Expectations
+
+`RuntimeStore` should be designed as a shared data authority, but not as the app's global lock for everything.
+
+Phase 1 design expectations:
+
+- coordinator-driven writes may still be synchronized internally
+- read APIs should be safe for coordinator and snapshot-provider use
+- render should not directly take a large mutable store lock in the target architecture
+
+This implies:
+
+- `RuntimeStore` may keep an internal mutex during migration
+- that mutex should protect durable models only
+- render-facing consumers should eventually read via `RuntimeSnapshotProvider`, not by reaching into the store
+
+One of the main goals here is reducing the current situation where `RuntimeHost` lock scope effectively mixes:
+
+- persistent state
+- status reporting
+- render-state caches
+- timing stats
+- reload flags
+
+`RuntimeStore` should sharply narrow that scope.
+
+## Dependency Rules
+
+Per the Phase 1 subsystem design, `RuntimeStore` should sit low in the dependency graph.
+
+Allowed inbound dependencies:
+
+- `RuntimeCoordinator -> RuntimeStore`
+- `RuntimeSnapshotProvider -> RuntimeStore`
+- temporary migration shims from `ControlServices` only where explicitly tolerated
+
+Allowed outbound dependencies:
+
+- file/serialization helpers
+- package manifest parsing helpers
+- pure utility types
+
+Not allowed:
+
+- `RuntimeStore -> RenderEngine`
+- `RuntimeStore -> VideoBackend`
+- `RuntimeStore -> ControlServices`
+- `RuntimeStore -> HealthTelemetry` for behavior control
+
+The store may emit errors or return result objects, but it should not coordinate the rest of the system directly.
+
+## Current Code Mapping
+
+Today, `RuntimeHost` contains most of the responsibilities that should move into `RuntimeStore`.
+
+Key current code paths:
+
+- config load:
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:1651)
+- persistent state load:
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:1748)
+- persistent state save:
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:1842)
+- preset save/load:
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:1286)
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:1304)
+- state serialization helpers:
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:2061)
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:2172)
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:2268)
+- path and file helpers:
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:1988)
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:2002)
+  - [RuntimeHost.cpp](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/apps/LoopThroughWithOpenGLCompositing/runtime/RuntimeHost.cpp:2034)
+
+Durable-state mutation entrypoints that currently live on `RuntimeHost` but conceptually split between coordinator and store:
+
+- layer stack edits:
+  - `AddLayer`
+  - `RemoveLayer`
+  - `MoveLayer`
+  - `MoveLayerToIndex`
+- committed-state edits:
+  - `SetLayerBypass`
+  - `SetLayerShader`
+  - `UpdateLayerParameter`
+  - `ResetLayerParameters`
+
+The target split should be:
+
+- validation/policy/orchestration -> `RuntimeCoordinator`
+- durable state write application -> `RuntimeStore`
+
+Methods that should not move into `RuntimeStore` even though they currently live on `RuntimeHost`:
+
+- render-state building and caching:
+  - `GetLayerRenderStates`
+  - `TryRefreshCachedLayerStates`
+  - `BuildLayerRenderStatesLocked`
+- status/timing reporting:
+  - `SetSignalStatus`
+  - `SetPerformanceStats`
+  - `SetFramePacingStats`
+  - `AdvanceFrame`
+- live reload flags/polling shell:
+  - `PollFileChanges`
+  - `ManualReloadRequested`
+  - `ClearReloadRequest`
+
+Those belong under other target subsystems.
+
+## Proposed Internal Subcomponents
+
+`RuntimeStore` does not need to be one monolithic class forever. A practical internal shape would be:
+
+- `RuntimeConfigStore`
+  - runtime host config load/save and resolved paths
+- `PersistentLayerStore`
+  - durable layer stack and parameter values
+- `StackPresetStore`
+  - preset enumeration/load/save
+- `ShaderPackageCatalogStore`
+  - durable manifest/package metadata
+- `PersistenceWriter` helper
+  - synchronous at first, async/debounced later
+
+These can still be presented through one subsystem façade during migration.
+
+## Persistence Model
+
+The store should treat persistence as durable snapshot management, not incremental side-effect spraying.
+
+Target behavior:
+
+- in-memory durable models are updated first
+- serialization snapshots are built from those models
+- save requests persist a coherent snapshot
+
+This matters because the current code still calls `SavePersistentState()` directly from many mutation paths. That is one of the architectural pressure points already called out in [ARCHITECTURE_RESILIENCE_REVIEW.md](/c:/Users/Aiden/Documents/GitHub/video-shader-toys/docs/ARCHITECTURE_RESILIENCE_REVIEW.md).
+
+The Phase 1 design for `RuntimeStore` should therefore assume:
+
+- store ownership of serialization remains
+- immediate save-after-mutate is a migration detail, not the final behavioral contract
+
+By Phase 6, a background snapshot writer may sit underneath or beside this subsystem, but the durable model still belongs here.
+
+## Migration Plan From Current Code
+
+The safest migration path is to extract responsibilities by interface, not by big-bang rename.
+
+### Step 1: Introduce The `RuntimeStore` Name And Facade
+
+Create a façade interface that wraps the durable-data parts of `RuntimeHost`.
+
+Initial likely contents:
+
+- config load/save access
+- persistent layer-stack get/set access
+- preset load/save access
+- package catalog read access
+
+At this stage, `RuntimeHost` may still be the implementation behind the façade.
+
+### Step 2: Move Pure Persistence Helpers First
+
+Low-risk extractions:
+
+- path resolution helpers
+- file read/write helpers
+- preset enumeration and serialization helpers
+- persistent-state serialization/deserialization helpers
+
+These have relatively low coupling to GL and backend timing.
+
+### Step 3: Split Durable Models From Render Cache/Status Fields
+
+Move out or conceptually separate:
+
+- `mPersistentState`
+- runtime config fields
+- preset roots and runtime roots
+- package catalog/order metadata
+
+From fields that should stay elsewhere:
+
+- render-state dirty flags and caches
+- status/timing counters
+- reload flags
+
+This is one of the most important separations in the whole program.
+
+### Step 4: Route Durable Mutations Through Coordinator-Owned Policy
+
+Once the coordinator exists, `RuntimeStore` write calls should become lower-level and less policy-rich.
+
+Examples:
+
+- `SetStoredParameterValue(...)` rather than `ApplyOscTargetByControlKey(...)`
+- `ReplaceStoredLayerStack(...)` rather than `LoadStackPreset(...)` directly mutating every downstream concern
+
+### Step 5: Keep Render Off The Store
+
+As `RuntimeSnapshotProvider` arrives, render should stop reading store internals directly.
+
+That is the moment where `RuntimeStore` becomes a proper durable authority instead of a shared mutable app center.
+
+## Risks
+
+### 1. Recreating `RuntimeHost` Under A New Name
+
+The biggest risk is calling something `RuntimeStore` while leaving policy, status, and render-cache behavior attached.
+
+Guardrail:
+
+- only durable data and store hygiene belong here
+
+### 2. Letting Validation Drift Into Persistence
+
+Store-level shape validation is appropriate. High-level mutation policy is not.
+
+Risk examples:
+
+- store decides whether OSC should persist
+- store decides whether a layer reorder should trigger snapshot publication
+- store decides whether a reload is render-only or package-affecting
+
+Those are coordinator decisions.
+
+### 3. Overexposing Mutable Internals
+
+If callers keep direct mutable access to the underlying vectors/maps, the subsystem boundary will exist only on paper.
+
+Guardrail:
+
+- prefer controlled write methods and stable read models
+
+### 4. Coupling Package Metadata Too Tightly To Compile Outputs
+
+Package manifest and parameter-definition metadata belongs here. Compiled program state does not.
+
+Guardrail:
+
+- keep compile products and GPU artifacts out of the store
+
+### 5. Using The Store Lock As A Global Synchronization Shortcut
+
+This would recreate timing and contention issues in a new form.
+
+Guardrail:
+
+- store locking protects durable models only
+- render synchronization must happen through snapshots, not by sharing the store lock
+
+## Open Questions
+
+### 1. How Much Shader Package Data Should Live Here?
+
+Clear yes:
+
+- manifest metadata
+- parameter definitions
+- package discovery/order information
+
+Still open:
+
+- whether compile-ready transformed sources belong here or in a later build-focused subsystem
+
+Current recommendation:
+
+- keep only durable reference/package metadata here
+
+### 2. Should Committed Live State Be Co-Located With Persisted State?
+
+The Phase 1 parent doc leaves open whether committed live state stays in the store or is split with a live companion model owned by the coordinator.
+
+For `RuntimeStore`, the important rule is:
+
+- if a piece of state is part of the durable truth model, the store should own it
+- if it is transient or session-only, it should not be forced into the store just for convenience
+
+### 3. Should Preset Application Be A Store Operation Or A Coordinator Operation?
+
+The file load and preset parse clearly belong here.
+
+The policy question of how a loaded preset affects live state, snapshot publication, overlays, and notifications belongs in the coordinator.
+
+Current recommendation:
+
+- `RuntimeStore` loads preset content
+- `RuntimeCoordinator` decides how to apply it
+
+### 4. How Early Should Async Persistence Land?
+
+Phase 1 does not require it, but the store design should not block it.
+
+Current recommendation:
+
+- keep synchronous save semantics initially if needed
+- shape the interfaces so a background writer can be introduced without changing subsystem ownership
+
+## Success Criteria For This Subsystem
+
+`RuntimeStore` can be considered well-defined once the codebase can say, without ambiguity:
+
+- all durable runtime config and saved layer data has one authoritative home
+- stack presets are owned by that same durable-data subsystem
+- render does not depend on store internals directly
+- timing/status/reporting state is no longer mixed into the same subsystem
+- persistence ownership is clear even before async persistence is introduced
+
+## Short Version
+
+`RuntimeStore` is the subsystem that should answer:
+
+- what durable runtime data exists
+- what saved layer stack and parameters exist
+- what presets and package metadata exist
+- how that durable data is loaded and serialized
+
+It should not answer:
+
+- whether a mutation should happen
+- how rendering should consume state
+- how hardware pacing should work
+- what health warnings should be emitted
+
+If this boundary holds, later phases can safely split `RuntimeHost` without just recreating the same coupling under a different class name.