Audio experiments
This commit is contained in:
406
docs/AUDIO_TEARING_INVESTIGATION.md
Normal file
406
docs/AUDIO_TEARING_INVESTIGATION.md
Normal file
@@ -0,0 +1,406 @@
|
||||
# Audio / SDI Tearing Investigation
|
||||
|
||||
Date: 2026-05-05
|
||||
|
||||
## Problem
|
||||
|
||||
After adding DeckLink audio pass-through, the SDI output intermittently shows a torn/corrupted frame. The preview window does not show the artifact.
|
||||
|
||||
Observed artifact:
|
||||
|
||||
- Bottom portion of the SDI image can show an offset mix of current/previous frame.
|
||||
- Looks like a frame-buffer or output-transfer issue rather than shader rendering.
|
||||
- Occurs even with all shaders bypassed.
|
||||
- Main branch is known good with no tearing.
|
||||
|
||||
Later tests also showed audio tearing/stutter when non-silent audio was scheduled.
|
||||
|
||||
## Known Good Baseline
|
||||
|
||||
- `main` branch has no SDI tearing.
|
||||
- Current branch with `audioEnabled: false` ran for several minutes with no visible tearing.
|
||||
|
||||
This strongly suggests the issue is tied to DeckLink audio output/scheduling rather than the shader stack.
|
||||
|
||||
## SDK References Checked
|
||||
|
||||
### `InputLoopThrough`
|
||||
|
||||
Location:
|
||||
|
||||
`3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/InputLoopThrough`
|
||||
|
||||
Findings:
|
||||
|
||||
- This is the SDK loop-through sample that keeps audio.
|
||||
- It preserves DeckLink audio packet timestamps using `GetPacketTime(..., m_frameTimescale)`.
|
||||
- It schedules audio packets with `ScheduleAudioSamples(..., packetTime, m_frameTimescale, ...)`.
|
||||
- It uses 16-channel 32-bit embedded audio by default.
|
||||
- It has separate scheduling threads for video/audio.
|
||||
- It waits for both video and audio preroll before `StartScheduledPlayback`.
|
||||
|
||||
### `LoopThroughWithOpenGLCompositing`
|
||||
|
||||
Location:
|
||||
|
||||
`3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/LoopThroughWithOpenGLCompositing`
|
||||
|
||||
Findings:
|
||||
|
||||
- This sample is the base for this app.
|
||||
- It ignores `IDeckLinkAudioInputPacket`.
|
||||
- It does not demonstrate audio pass-through.
|
||||
|
||||
### `SignalGenerator`
|
||||
|
||||
Location:
|
||||
|
||||
`3rdParty/Blackmagic DeckLink SDK 16.0/Win/Samples/SignalGenerator`
|
||||
|
||||
Findings:
|
||||
|
||||
- Uses `RenderAudioSamples()` callback to top up audio when DeckLink requests samples.
|
||||
- Uses `GetBufferedAudioSampleFrameCount()` and a water level before scheduling more audio.
|
||||
|
||||
## Tests Tried And Results
|
||||
|
||||
### 1. Initial audio pass-through with FIFO and sample-time accumulator
|
||||
|
||||
Implementation:
|
||||
|
||||
- Copied incoming audio into a stereo FIFO.
|
||||
- Scheduled audio with a generated `mNextAudioSampleFrame` clock in 48 kHz timescale.
|
||||
- Matched delay to video preroll.
|
||||
|
||||
Result:
|
||||
|
||||
- Audio eventually worked.
|
||||
- SDI video tearing appeared.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- Basic audio output path triggered SDI instability.
|
||||
|
||||
### 2. Reworked audio toward SDK `InputLoopThrough` packet-timestamp model
|
||||
|
||||
Implementation:
|
||||
|
||||
- Preserved incoming packet time via `GetPacketTime(..., mFrameTimescale)`.
|
||||
- Queued timestamped audio packets.
|
||||
- Scheduled packets with `ScheduleAudioSamples(..., packet.streamTime, mFrameTimescale, ...)`.
|
||||
|
||||
Result:
|
||||
|
||||
- Tearing persisted.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- Simply matching SDK timestamp domain did not fix the issue.
|
||||
|
||||
### 3. Restored video callback closer to `main`
|
||||
|
||||
Implementation:
|
||||
|
||||
- Removed extra `glFinish()` calls.
|
||||
- Restored preview/readback ordering closer to `main`.
|
||||
- Re-enabled fast transfer path after earlier tests disabled it.
|
||||
- Removed audio texture upload from video playout callback.
|
||||
- Removed audio analysis and audio locks from video playout callback.
|
||||
- Removed DeckLink scheduling mutex around `ScheduleVideoFrame`.
|
||||
|
||||
Result:
|
||||
|
||||
- Tearing frequency seemed reduced at one point, but tearing persisted.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- Extra work in the playout callback may have made timing worse, but was not the root cause.
|
||||
|
||||
### 4. Disabled audio completely
|
||||
|
||||
Config:
|
||||
|
||||
```json
|
||||
"audioEnabled": false
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- Ran for several minutes with no visible tearing.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- The tearing is tied to audio being enabled.
|
||||
|
||||
### 5. Enabled audio input/analysis but disabled DeckLink audio output
|
||||
|
||||
Config:
|
||||
|
||||
```json
|
||||
"audioEnabled": true,
|
||||
"audioOutputEnabled": false
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- No tearing appeared.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- DeckLink audio input and CPU analysis are not the trigger.
|
||||
- The problem is on the DeckLink audio output side.
|
||||
|
||||
### 6. Enabled DeckLink audio output but disabled scheduling
|
||||
|
||||
Config:
|
||||
|
||||
```json
|
||||
"audioEnabled": true,
|
||||
"audioOutputEnabled": true,
|
||||
"audioScheduleEnabled": false
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- No video tearing.
|
||||
- Slight stutter appeared.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- `EnableAudioOutput()` alone did not produce the tearing.
|
||||
- Stutter was likely from enabling an audio output stream without feeding it samples.
|
||||
|
||||
### 7. Enabled audio scheduling but skipped audio preroll
|
||||
|
||||
Config:
|
||||
|
||||
```json
|
||||
"audioEnabled": true,
|
||||
"audioOutputEnabled": true,
|
||||
"audioScheduleEnabled": true,
|
||||
"audioPrerollEnabled": false
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- Video tearing returned.
|
||||
- Stutter also present.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- `BeginAudioPreroll()` / `EndAudioPreroll()` are not required to trigger the tear.
|
||||
- `ScheduleAudioSamples()` is strongly implicated.
|
||||
|
||||
### 8. Retained scheduled audio packet memory after `ScheduleAudioSamples`
|
||||
|
||||
Implementation:
|
||||
|
||||
- Kept scheduled packet buffers alive in a retain queue after scheduling.
|
||||
- Avoided passing DeckLink pointers to vectors that immediately went out of scope.
|
||||
|
||||
Result:
|
||||
|
||||
- Video tearing and stutter persisted.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- Buffer lifetime after `ScheduleAudioSamples()` was not the root cause.
|
||||
|
||||
### 9. Added audio water-level cap
|
||||
|
||||
Implementation:
|
||||
|
||||
- Restored SDK-style `GetBufferedAudioSampleFrameCount()` check.
|
||||
- Only scheduled more audio if DeckLink buffer was below the target water level.
|
||||
|
||||
Result:
|
||||
|
||||
- Stutter was reduced.
|
||||
- Video tearing persisted.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- Overscheduling contributed to stutter/timing pressure.
|
||||
- It did not explain the tearing.
|
||||
|
||||
### 10. Removed standalone audio scheduler thread
|
||||
|
||||
Implementation:
|
||||
|
||||
- Stopped starting the dedicated audio scheduler thread.
|
||||
- Audio top-up occurred from input packet arrival and `RenderAudioSamples()` callback.
|
||||
|
||||
Result:
|
||||
|
||||
- No meaningful change.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- The polling thread itself was not the cause.
|
||||
|
||||
### 11. Switched from timestamped audio output to continuous audio output
|
||||
|
||||
Implementation:
|
||||
|
||||
- Changed audio output to `bmdAudioOutputStreamContinuous`.
|
||||
- Scheduled audio using a monotonic 48 kHz sample clock.
|
||||
|
||||
Result:
|
||||
|
||||
- Video tearing and stutter persisted.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- The issue was not specific to timestamped output mode.
|
||||
|
||||
### 12. Rendered into the actual `completedFrame`
|
||||
|
||||
Implementation:
|
||||
|
||||
- Changed `PlayoutFrameCompleted()` to reuse the exact `completedFrame` passed by DeckLink rather than rotating an independent output-frame queue.
|
||||
|
||||
Result:
|
||||
|
||||
- No change.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- The app was probably not overwriting a still-in-use frame from its output queue.
|
||||
|
||||
### 13. Scheduled generated silence instead of captured audio
|
||||
|
||||
Config:
|
||||
|
||||
```json
|
||||
"audioScheduleSilence": true
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- Occasional stutter.
|
||||
- No video tearing.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- Scheduling audio buffers itself can be stable if the audio data is zero.
|
||||
- Non-zero audio data appears to be important.
|
||||
|
||||
### 14. Flattened captured audio into PCM FIFO and scheduled fixed chunks
|
||||
|
||||
Implementation:
|
||||
|
||||
- Captured packets were flattened into a PCM FIFO.
|
||||
- DeckLink received fixed 10 ms chunks rather than original packet boundaries.
|
||||
- Missing audio was padded with silence.
|
||||
|
||||
Result:
|
||||
|
||||
- Video tearing returned.
|
||||
- Audio stutter/tearing returned.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- Packet boundaries/timestamps were not the whole cause.
|
||||
- Non-zero captured audio data still triggered instability.
|
||||
|
||||
### 15. Scheduled generated 440 Hz tone
|
||||
|
||||
Config:
|
||||
|
||||
```json
|
||||
"audioScheduleTone": true
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- Video tearing occurred.
|
||||
- Tone/audio also tore.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- The issue is not specific to captured input data.
|
||||
- Non-zero scheduled audio, even generated tone, triggers the problem.
|
||||
|
||||
### 16. Changed DeckLink output to 16 embedded audio channels
|
||||
|
||||
Implementation:
|
||||
|
||||
- Enabled DeckLink audio output with 16 channels instead of 2.
|
||||
- Mapped stereo to channels 1/2.
|
||||
- Filled channels 3-16 with silence.
|
||||
|
||||
Result:
|
||||
|
||||
- Video tearing and audio tearing still occurred.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- The issue is not simply caused by 2-channel embedded audio output.
|
||||
|
||||
### 17. Used DeckLink-owned output video frames with audio enabled
|
||||
|
||||
Implementation:
|
||||
|
||||
- When audio output is enabled:
|
||||
- disabled fast transfer path
|
||||
- created output frames with `CreateVideoFrame()`
|
||||
- avoided `CreateVideoFrameWithBuffer()` and the custom pinned playout allocator
|
||||
|
||||
Result:
|
||||
|
||||
- Video tearing and audio tearing still occurred.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- The custom pinned output video buffers are likely not the root cause.
|
||||
|
||||
## Current Strong Conclusions
|
||||
|
||||
- Shader stack is not the cause.
|
||||
- Preview/render output is not showing the issue, so the artifact is SDI/output-side.
|
||||
- DeckLink audio input is not the cause.
|
||||
- DeckLink audio output enabled but unscheduled does not cause tearing.
|
||||
- `ScheduleAudioSamples()` with zero/silent buffers does not cause tearing.
|
||||
- `ScheduleAudioSamples()` with non-zero audio causes both video tearing and audio tearing.
|
||||
- The problem persists across:
|
||||
- timestamped audio output
|
||||
- continuous audio output
|
||||
- captured audio
|
||||
- generated tone
|
||||
- 2-channel output
|
||||
- 16-channel embedded output
|
||||
- app-owned/pinned output video buffers
|
||||
- DeckLink-owned output video frames
|
||||
|
||||
## Current Hypothesis
|
||||
|
||||
The issue appears to be a DeckLink output interaction where non-zero embedded audio samples disturb SDI video/audio output in this app’s scheduling model.
|
||||
|
||||
Since silence is stable but tone is not, the next likely areas to investigate are:
|
||||
|
||||
- Audio sample format/range/endian expectations.
|
||||
- Whether DeckLink expects 32-bit audio samples to be in a different effective range than we are providing.
|
||||
- Whether the scheduled audio buffer layout for the selected hardware/output mode differs from our assumptions.
|
||||
- Whether the selected output mode/keyer/SDI configuration has constraints when non-zero embedded audio is present.
|
||||
- Whether the SDK sample behaves correctly on the same hardware with a generated tone and same video mode.
|
||||
|
||||
## Suggested Next Tests
|
||||
|
||||
1. Schedule very low amplitude non-zero audio, e.g. constant `1`, then `256`, then a very quiet sine.
|
||||
2. Try 16-bit audio output instead of 32-bit if supported.
|
||||
3. Try `bmdAudioOutputStreamContinuousDontResample`.
|
||||
4. Disable external keying and test with non-zero audio.
|
||||
5. Build/run the SDK `SignalGenerator` or `InputLoopThrough` sample on the same DeckLink device, video mode, and SDI output path with non-zero embedded audio.
|
||||
6. Add instrumentation for DeckLink status/errors around scheduled video/audio completion.
|
||||
7. Confirm Desktop Video setup panel audio/SDI settings for the selected output.
|
||||
|
||||
## Current Config At Time Of Note
|
||||
|
||||
```json
|
||||
"audioEnabled": true,
|
||||
"audioOutputEnabled": true,
|
||||
"audioScheduleEnabled": true,
|
||||
"audioPrerollEnabled": true,
|
||||
"audioScheduleSilence": false,
|
||||
"audioScheduleTone": false
|
||||
```
|
||||
Reference in New Issue
Block a user