407 lines
10 KiB
Markdown
407 lines
10 KiB
Markdown
# Audio / SDI Tearing Investigation
|
||
|
||
Date: 2026-05-05
|
||
|
||
## Problem
|
||
|
||
After adding DeckLink audio pass-through, the SDI output intermittently shows a torn/corrupted frame. The preview window does not show the artifact.
|
||
|
||
Observed artifact:
|
||
|
||
- Bottom portion of the SDI image can show an offset mix of current/previous frame.
|
||
- Looks like a frame-buffer or output-transfer issue rather than shader rendering.
|
||
- Occurs even with all shaders bypassed.
|
||
- Main branch is known good with no tearing.
|
||
|
||
Later tests also showed audio tearing/stutter when non-silent audio was scheduled.
|
||
|
||
## Known Good Baseline
|
||
|
||
- `main` branch has no SDI tearing.
|
||
- Current branch with `audioEnabled: false` ran for several minutes with no visible tearing.
|
||
|
||
This strongly suggests the issue is tied to DeckLink audio output/scheduling rather than the shader stack.
|
||
|
||
## SDK References Checked
|
||
|
||
### `InputLoopThrough`
|
||
|
||
Location:
|
||
|
||
`3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/InputLoopThrough`
|
||
|
||
Findings:
|
||
|
||
- This is the SDK loop-through sample that keeps audio.
|
||
- It preserves DeckLink audio packet timestamps using `GetPacketTime(..., m_frameTimescale)`.
|
||
- It schedules audio packets with `ScheduleAudioSamples(..., packetTime, m_frameTimescale, ...)`.
|
||
- It uses 16-channel 32-bit embedded audio by default.
|
||
- It has separate scheduling threads for video/audio.
|
||
- It waits for both video and audio preroll before `StartScheduledPlayback`.
|
||
|
||
### `LoopThroughWithOpenGLCompositing`
|
||
|
||
Location:
|
||
|
||
`3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/LoopThroughWithOpenGLCompositing`
|
||
|
||
Findings:
|
||
|
||
- This sample is the base for this app.
|
||
- It ignores `IDeckLinkAudioInputPacket`.
|
||
- It does not demonstrate audio pass-through.
|
||
|
||
### `SignalGenerator`
|
||
|
||
Location:
|
||
|
||
`3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/SignalGenerator`
|
||
|
||
Findings:
|
||
|
||
- Uses `RenderAudioSamples()` callback to top up audio when DeckLink requests samples.
|
||
- Uses `GetBufferedAudioSampleFrameCount()` and a water level before scheduling more audio.
|
||
|
||
## Tests Tried And Results
|
||
|
||
### 1. Initial audio pass-through with FIFO and sample-time accumulator
|
||
|
||
Implementation:
|
||
|
||
- Copied incoming audio into a stereo FIFO.
|
||
- Scheduled audio with a generated `mNextAudioSampleFrame` clock in 48 kHz timescale.
|
||
- Matched delay to video preroll.
|
||
|
||
Result:
|
||
|
||
- Audio eventually worked.
|
||
- SDI video tearing appeared.
|
||
|
||
Conclusion:
|
||
|
||
- Basic audio output path triggered SDI instability.
|
||
|
||
### 2. Reworked audio toward SDK `InputLoopThrough` packet-timestamp model
|
||
|
||
Implementation:
|
||
|
||
- Preserved incoming packet time via `GetPacketTime(..., mFrameTimescale)`.
|
||
- Queued timestamped audio packets.
|
||
- Scheduled packets with `ScheduleAudioSamples(..., packet.streamTime, mFrameTimescale, ...)`.
|
||
|
||
Result:
|
||
|
||
- Tearing persisted.
|
||
|
||
Conclusion:
|
||
|
||
- Simply matching SDK timestamp domain did not fix the issue.
|
||
|
||
### 3. Restored video callback closer to `main`
|
||
|
||
Implementation:
|
||
|
||
- Removed extra `glFinish()` calls.
|
||
- Restored preview/readback ordering closer to `main`.
|
||
- Re-enabled fast transfer path after earlier tests disabled it.
|
||
- Removed audio texture upload from video playout callback.
|
||
- Removed audio analysis and audio locks from video playout callback.
|
||
- Removed DeckLink scheduling mutex around `ScheduleVideoFrame`.
|
||
|
||
Result:
|
||
|
||
- Tearing frequency seemed reduced at one point, but tearing persisted.
|
||
|
||
Conclusion:
|
||
|
||
- Extra work in the playout callback may have made timing worse, but was not the root cause.
|
||
|
||
### 4. Disabled audio completely
|
||
|
||
Config:
|
||
|
||
```json
|
||
"audioEnabled": false
|
||
```
|
||
|
||
Result:
|
||
|
||
- Ran for several minutes with no visible tearing.
|
||
|
||
Conclusion:
|
||
|
||
- The tearing is tied to audio being enabled.
|
||
|
||
### 5. Enabled audio input/analysis but disabled DeckLink audio output
|
||
|
||
Config:
|
||
|
||
```json
|
||
"audioEnabled": true,
|
||
"audioOutputEnabled": false
|
||
```
|
||
|
||
Result:
|
||
|
||
- No tearing appeared.
|
||
|
||
Conclusion:
|
||
|
||
- DeckLink audio input and CPU analysis are not the trigger.
|
||
- The problem is on the DeckLink audio output side.
|
||
|
||
### 6. Enabled DeckLink audio output but disabled scheduling
|
||
|
||
Config:
|
||
|
||
```json
|
||
"audioEnabled": true,
|
||
"audioOutputEnabled": true,
|
||
"audioScheduleEnabled": false
|
||
```
|
||
|
||
Result:
|
||
|
||
- No video tearing.
|
||
- Slight stutter appeared.
|
||
|
||
Conclusion:
|
||
|
||
- `EnableAudioOutput()` alone did not produce the tearing.
|
||
- Stutter was likely from enabling an audio output stream without feeding it samples.
|
||
|
||
### 7. Enabled audio scheduling but skipped audio preroll
|
||
|
||
Config:
|
||
|
||
```json
|
||
"audioEnabled": true,
|
||
"audioOutputEnabled": true,
|
||
"audioScheduleEnabled": true,
|
||
"audioPrerollEnabled": false
|
||
```
|
||
|
||
Result:
|
||
|
||
- Video tearing returned.
|
||
- Stutter also present.
|
||
|
||
Conclusion:
|
||
|
||
- `BeginAudioPreroll()` / `EndAudioPreroll()` are not required to trigger the tear.
|
||
- `ScheduleAudioSamples()` is strongly implicated.
|
||
|
||
### 8. Retained scheduled audio packet memory after `ScheduleAudioSamples`
|
||
|
||
Implementation:
|
||
|
||
- Kept scheduled packet buffers alive in a retain queue after scheduling.
|
||
- Avoided passing DeckLink pointers to vectors that immediately went out of scope.
|
||
|
||
Result:
|
||
|
||
- Video tearing and stutter persisted.
|
||
|
||
Conclusion:
|
||
|
||
- Buffer lifetime after `ScheduleAudioSamples()` was not the root cause.
|
||
|
||
### 9. Added audio water-level cap
|
||
|
||
Implementation:
|
||
|
||
- Restored SDK-style `GetBufferedAudioSampleFrameCount()` check.
|
||
- Only scheduled more audio if DeckLink buffer was below the target water level.
|
||
|
||
Result:
|
||
|
||
- Stutter was reduced.
|
||
- Video tearing persisted.
|
||
|
||
Conclusion:
|
||
|
||
- Overscheduling contributed to stutter/timing pressure.
|
||
- It did not explain the tearing.
|
||
|
||
### 10. Removed standalone audio scheduler thread
|
||
|
||
Implementation:
|
||
|
||
- Stopped starting the dedicated audio scheduler thread.
|
||
- Audio top-up occurred from input packet arrival and `RenderAudioSamples()` callback.
|
||
|
||
Result:
|
||
|
||
- No meaningful change.
|
||
|
||
Conclusion:
|
||
|
||
- The polling thread itself was not the cause.
|
||
|
||
### 11. Switched from timestamped audio output to continuous audio output
|
||
|
||
Implementation:
|
||
|
||
- Changed audio output to `bmdAudioOutputStreamContinuous`.
|
||
- Scheduled audio using a monotonic 48 kHz sample clock.
|
||
|
||
Result:
|
||
|
||
- Video tearing and stutter persisted.
|
||
|
||
Conclusion:
|
||
|
||
- The issue was not specific to timestamped output mode.
|
||
|
||
### 12. Rendered into the actual `completedFrame`
|
||
|
||
Implementation:
|
||
|
||
- Changed `PlayoutFrameCompleted()` to reuse the exact `completedFrame` passed by DeckLink rather than rotating an independent output-frame queue.
|
||
|
||
Result:
|
||
|
||
- No change.
|
||
|
||
Conclusion:
|
||
|
||
- The app was probably not overwriting a still-in-use frame from its output queue.
|
||
|
||
### 13. Scheduled generated silence instead of captured audio
|
||
|
||
Config:
|
||
|
||
```json
|
||
"audioScheduleSilence": true
|
||
```
|
||
|
||
Result:
|
||
|
||
- Occasional stutter.
|
||
- No video tearing.
|
||
|
||
Conclusion:
|
||
|
||
- Scheduling audio buffers itself can be stable if the audio data is zero.
|
||
- Non-zero audio data appears to be important.
|
||
|
||
### 14. Flattened captured audio into PCM FIFO and scheduled fixed chunks
|
||
|
||
Implementation:
|
||
|
||
- Captured packets were flattened into a PCM FIFO.
|
||
- DeckLink received fixed 10 ms chunks rather than original packet boundaries.
|
||
- Missing audio was padded with silence.
|
||
|
||
Result:
|
||
|
||
- Video tearing returned.
|
||
- Audio stutter/tearing returned.
|
||
|
||
Conclusion:
|
||
|
||
- Packet boundaries/timestamps were not the whole cause.
|
||
- Non-zero captured audio data still triggered instability.
|
||
|
||
### 15. Scheduled generated 440 Hz tone
|
||
|
||
Config:
|
||
|
||
```json
|
||
"audioScheduleTone": true
|
||
```
|
||
|
||
Result:
|
||
|
||
- Video tearing occurred.
|
||
- Tone/audio also tore.
|
||
|
||
Conclusion:
|
||
|
||
- The issue is not specific to captured input data.
|
||
- Non-zero scheduled audio, even generated tone, triggers the problem.
|
||
|
||
### 16. Changed DeckLink output to 16 embedded audio channels
|
||
|
||
Implementation:
|
||
|
||
- Enabled DeckLink audio output with 16 channels instead of 2.
|
||
- Mapped stereo to channels 1/2.
|
||
- Filled channels 3-16 with silence.
|
||
|
||
Result:
|
||
|
||
- Video tearing and audio tearing still occurred.
|
||
|
||
Conclusion:
|
||
|
||
- The issue is not simply caused by 2-channel embedded audio output.
|
||
|
||
### 17. Used DeckLink-owned output video frames with audio enabled
|
||
|
||
Implementation:
|
||
|
||
- When audio output is enabled:
|
||
- disabled fast transfer path
|
||
- created output frames with `CreateVideoFrame()`
|
||
- avoided `CreateVideoFrameWithBuffer()` and the custom pinned playout allocator
|
||
|
||
Result:
|
||
|
||
- Video tearing and audio tearing still occurred.
|
||
|
||
Conclusion:
|
||
|
||
- The custom pinned output video buffers are likely not the root cause.
|
||
|
||
## Current Strong Conclusions
|
||
|
||
- Shader stack is not the cause.
|
||
- Preview/render output is not showing the issue, so the artifact is SDI/output-side.
|
||
- DeckLink audio input is not the cause.
|
||
- DeckLink audio output enabled but unscheduled does not cause tearing.
|
||
- `ScheduleAudioSamples()` with zero/silent buffers does not cause tearing.
|
||
- `ScheduleAudioSamples()` with non-zero audio causes both video tearing and audio tearing.
|
||
- The problem persists across:
|
||
- timestamped audio output
|
||
- continuous audio output
|
||
- captured audio
|
||
- generated tone
|
||
- 2-channel output
|
||
- 16-channel embedded output
|
||
- app-owned/pinned output video buffers
|
||
- DeckLink-owned output video frames
|
||
|
||
## Current Hypothesis
|
||
|
||
The issue appears to be a DeckLink output interaction where non-zero embedded audio samples disturb SDI video/audio output in this app’s scheduling model.
|
||
|
||
Since silence is stable but tone is not, the next likely areas to investigate are:
|
||
|
||
- Audio sample format/range/endian expectations.
|
||
- Whether DeckLink expects 32-bit audio samples to be in a different effective range than we are providing.
|
||
- Whether the scheduled audio buffer layout for the selected hardware/output mode differs from our assumptions.
|
||
- Whether the selected output mode/keyer/SDI configuration has constraints when non-zero embedded audio is present.
|
||
- Whether the SDK sample behaves correctly on the same hardware with a generated tone and same video mode.
|
||
|
||
## Suggested Next Tests
|
||
|
||
1. Schedule very low amplitude non-zero audio, e.g. constant `1`, then `256`, then a very quiet sine.
|
||
2. Try 16-bit audio output instead of 32-bit if supported.
|
||
3. Try `bmdAudioOutputStreamContinuousDontResample`.
|
||
4. Disable external keying and test with non-zero audio.
|
||
5. Build/run the SDK `SignalGenerator` or `InputLoopThrough` sample on the same DeckLink device, video mode, and SDI output path with non-zero embedded audio.
|
||
6. Add instrumentation for DeckLink status/errors around scheduled video/audio completion.
|
||
7. Confirm Desktop Video setup panel audio/SDI settings for the selected output.
|
||
|
||
## Current Config At Time Of Note
|
||
|
||
```json
|
||
"audioEnabled": true,
|
||
"audioOutputEnabled": true,
|
||
"audioScheduleEnabled": true,
|
||
"audioPrerollEnabled": true,
|
||
"audioScheduleSilence": false,
|
||
"audioScheduleTone": false
|
||
```
|