video-shader-toys/docs/AUDIO_TEARING_INVESTIGATION.md

Audio / SDI Tearing Investigation

Date: 2026-05-05

Problem

After adding DeckLink audio pass-through, the SDI output intermittently shows a torn/corrupted frame. The preview window does not show the artifact.

Observed artifact:

• Bottom portion of the SDI image can show an offset mix of current/previous frame.
• Looks like a frame-buffer or output-transfer issue rather than shader rendering.
• Occurs even with all shaders bypassed.
• Main branch is known good with no tearing.

Later tests also showed audio tearing/stutter when non-silent audio was scheduled.

Known Good Baseline

• main branch has no SDI tearing.
• Current branch with audioEnabled: false ran for several minutes with no visible tearing.

This strongly suggests the issue is tied to DeckLink audio output/scheduling rather than the shader stack.

SDK References Checked

InputLoopThrough

Location:

3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/InputLoopThrough

Findings:

• This is the SDK loop-through sample that keeps audio.
• It preserves DeckLink audio packet timestamps using GetPacketTime(..., m_frameTimescale).
• It schedules audio packets with ScheduleAudioSamples(..., packetTime, m_frameTimescale, ...).
• It uses 16-channel 32-bit embedded audio by default.
• It has separate scheduling threads for video/audio.
• It waits for both video and audio preroll before StartScheduledPlayback.

LoopThroughWithOpenGLCompositing

Location:

3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/LoopThroughWithOpenGLCompositing

Findings:

• This sample is the base for this app.
• It ignores IDeckLinkAudioInputPacket.
• It does not demonstrate audio pass-through.

SignalGenerator

Location:

3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/SignalGenerator

Findings:

• Uses RenderAudioSamples() callback to top up audio when DeckLink requests samples.
• Uses GetBufferedAudioSampleFrameCount() and a water level before scheduling more audio.

Tests Tried And Results

1. Initial audio pass-through with FIFO and sample-time accumulator

Implementation:

• Copied incoming audio into a stereo FIFO.
• Scheduled audio with a generated mNextAudioSampleFrame clock in 48 kHz timescale.
• Matched delay to video preroll.

Result:

• Audio eventually worked.
• SDI video tearing appeared.

Conclusion:

• Basic audio output path triggered SDI instability.

2. Reworked audio toward SDK InputLoopThrough packet-timestamp model

Implementation:

• Preserved incoming packet time via GetPacketTime(..., mFrameTimescale).
• Queued timestamped audio packets.
• Scheduled packets with ScheduleAudioSamples(..., packet.streamTime, mFrameTimescale, ...).

Result:

• Tearing persisted.

Conclusion:

• Simply matching SDK timestamp domain did not fix the issue.

3. Restored video callback closer to main

Implementation:

• Removed extra glFinish() calls.
• Restored preview/readback ordering closer to main.
• Re-enabled fast transfer path after earlier tests disabled it.
• Removed audio texture upload from video playout callback.
• Removed audio analysis and audio locks from video playout callback.
• Removed DeckLink scheduling mutex around ScheduleVideoFrame.

Result:

• Tearing frequency seemed reduced at one point, but tearing persisted.

Conclusion:

• Extra work in the playout callback may have made timing worse, but was not the root cause.

4. Disabled audio completely

Config:

"audioEnabled": false

Result:

• Ran for several minutes with no visible tearing.

Conclusion:

• The tearing is tied to audio being enabled.

5. Enabled audio input/analysis but disabled DeckLink audio output

Config:

"audioEnabled": true,
"audioOutputEnabled": false

Result:

• No tearing appeared.

Conclusion:

• DeckLink audio input and CPU analysis are not the trigger.
• The problem is on the DeckLink audio output side.

6. Enabled DeckLink audio output but disabled scheduling

Config:

"audioEnabled": true,
"audioOutputEnabled": true,
"audioScheduleEnabled": false

Result:

• No video tearing.
• Slight stutter appeared.

Conclusion:

• EnableAudioOutput() alone did not produce the tearing.
• Stutter was likely from enabling an audio output stream without feeding it samples.

7. Enabled audio scheduling but skipped audio preroll

Config:

"audioEnabled": true,
"audioOutputEnabled": true,
"audioScheduleEnabled": true,
"audioPrerollEnabled": false

Result:

• Video tearing returned.
• Stutter also present.

Conclusion:

• BeginAudioPreroll() / EndAudioPreroll() are not required to trigger the tear.
• ScheduleAudioSamples() is strongly implicated.

8. Retained scheduled audio packet memory after ScheduleAudioSamples

Implementation:

• Kept scheduled packet buffers alive in a retain queue after scheduling.
• Avoided passing DeckLink pointers to vectors that immediately went out of scope.

Result:

• Video tearing and stutter persisted.

Conclusion:

• Buffer lifetime after ScheduleAudioSamples() was not the root cause.

9. Added audio water-level cap

Implementation:

• Restored SDK-style GetBufferedAudioSampleFrameCount() check.
• Only scheduled more audio if DeckLink buffer was below the target water level.

Result:

• Stutter was reduced.
• Video tearing persisted.

Conclusion:

• Overscheduling contributed to stutter/timing pressure.
• It did not explain the tearing.

10. Removed standalone audio scheduler thread

Implementation:

• Stopped starting the dedicated audio scheduler thread.
• Audio top-up occurred from input packet arrival and RenderAudioSamples() callback.

Result:

• No meaningful change.

Conclusion:

• The polling thread itself was not the cause.

11. Switched from timestamped audio output to continuous audio output

Implementation:

• Changed audio output to bmdAudioOutputStreamContinuous.
• Scheduled audio using a monotonic 48 kHz sample clock.

Result:

• Video tearing and stutter persisted.

Conclusion:

• The issue was not specific to timestamped output mode.

12. Rendered into the actual completedFrame

Implementation:

• Changed PlayoutFrameCompleted() to reuse the exact completedFrame passed by DeckLink rather than rotating an independent output-frame queue.

Result:

• No change.

Conclusion:

• The app was probably not overwriting a still-in-use frame from its output queue.

13. Scheduled generated silence instead of captured audio

Config:

"audioScheduleSilence": true

Result:

• Occasional stutter.
• No video tearing.

Conclusion:

• Scheduling audio buffers itself can be stable if the audio data is zero.
• Non-zero audio data appears to be important.

14. Flattened captured audio into PCM FIFO and scheduled fixed chunks

Implementation:

• Captured packets were flattened into a PCM FIFO.
• DeckLink received fixed 10 ms chunks rather than original packet boundaries.
• Missing audio was padded with silence.

Result:

• Video tearing returned.
• Audio stutter/tearing returned.

Conclusion:

• Packet boundaries/timestamps were not the whole cause.
• Non-zero captured audio data still triggered instability.

15. Scheduled generated 440 Hz tone

Config:

"audioScheduleTone": true

Result:

• Video tearing occurred.
• Tone/audio also tore.

Conclusion:

• The issue is not specific to captured input data.
• Non-zero scheduled audio, even generated tone, triggers the problem.

16. Changed DeckLink output to 16 embedded audio channels

Implementation:

• Enabled DeckLink audio output with 16 channels instead of 2.
• Mapped stereo to channels 1/2.
• Filled channels 3-16 with silence.

Result:

• Video tearing and audio tearing still occurred.

Conclusion:

• The issue is not simply caused by 2-channel embedded audio output.

17. Used DeckLink-owned output video frames with audio enabled

Implementation:

• When audio output is enabled:
  • disabled fast transfer path
  • created output frames with CreateVideoFrame()
  • avoided CreateVideoFrameWithBuffer() and the custom pinned playout allocator

Result:

• Video tearing and audio tearing still occurred.

Conclusion:

• The custom pinned output video buffers are likely not the root cause.

Current Strong Conclusions

• Shader stack is not the cause.
• Preview/render output is not showing the issue, so the artifact is SDI/output-side.
• DeckLink audio input is not the cause.
• DeckLink audio output enabled but unscheduled does not cause tearing.
• ScheduleAudioSamples() with zero/silent buffers does not cause tearing.
• ScheduleAudioSamples() with non-zero audio causes both video tearing and audio tearing.
• The problem persists across:
  • timestamped audio output
  • continuous audio output
  • captured audio
  • generated tone
  • 2-channel output
  • 16-channel embedded output
  • app-owned/pinned output video buffers
  • DeckLink-owned output video frames

Current Hypothesis

The issue appears to be a DeckLink output interaction where non-zero embedded audio samples disturb SDI video/audio output in this app’s scheduling model.

Since silence is stable but tone is not, the next likely areas to investigate are:

• Audio sample format/range/endian expectations.
• Whether DeckLink expects 32-bit audio samples to be in a different effective range than we are providing.
• Whether the scheduled audio buffer layout for the selected hardware/output mode differs from our assumptions.
• Whether the selected output mode/keyer/SDI configuration has constraints when non-zero embedded audio is present.
• Whether the SDK sample behaves correctly on the same hardware with a generated tone and same video mode.

Suggested Next Tests

1. Schedule very low amplitude non-zero audio, e.g. constant 1, then 256, then a very quiet sine.
2. Try 16-bit audio output instead of 32-bit if supported.
3. Try bmdAudioOutputStreamContinuousDontResample.
4. Disable external keying and test with non-zero audio.
5. Build/run the SDK SignalGenerator or InputLoopThrough sample on the same DeckLink device, video mode, and SDI output path with non-zero embedded audio.
6. Add instrumentation for DeckLink status/errors around scheduled video/audio completion.
7. Confirm Desktop Video setup panel audio/SDI settings for the selected output.

Current Config At Time Of Note

"audioEnabled": true,
"audioOutputEnabled": true,
"audioScheduleEnabled": true,
"audioPrerollEnabled": true,
"audioScheduleSilence": false,
"audioScheduleTone": false