Audio/Video stutter & intermittent audio drop during live streaming

[noho501]

Hi,

First of all, thank you for this excellent library — it works very well overall.
However, I’ve been encountering two recurring issues when running the Example project directly from this repository:

1. Camera stutter
   When panning or moving the camera, both the local preview and the live stream show noticeable frame stuttering.
   This happens on both iPad and iPhone 16 Pro.
   I suspect it may be related to displayLink timing or frame processing threads, but I’m not entirely sure.
   I’ve also tested the latest main branch with GPU integration, and the issue still persists.

2. Audio issues
   Occasionally, the audio becomes desynchronized from the video or produces crackling sounds.
   In some sessions, the audio becomes completely muted for several minutes before recovering on its own.
   This issue is intermittent and difficult to reproduce consistently, but it happens often enough to be noticeable.
   It might be related to StreamRecorder.Error 6 when stopRecording after long live (90m) – video not saved (30–40% cases) #1812 — once this issue occurs, the video also fails to save properly.
   → I believe there should be a controller or mechanism to monitor A/V timestamps.
   If the audio and video timestamps drift out of sync, the stream could automatically restart or adjust timing to recover synchronization.
   Currently, I have to manually restart the app and begin a new live session for audio and video to realign.

I’d really appreciate it if you could take a look or suggest possible debugging or optimization approaches. We can sponsor a small contribution to support further development.

Thank you again for your great work on this project.

[shogo4405]

It’s difficult to give proper advice without knowing how to reproduce the issue.
Even if the resulting problem looks the same, the fix may differ depending on how it occurs.

First, please tell me how to reproduce it.
By “how to reproduce,” I don’t just mean the steps you take, but also details such as what kind of code you’re using.

For example:

• I’m aware that enabling VideoStabilizationMode can cause camera stutter, and I don’t think that’s something that can be fixed.
• If this issue occurs when using AVAudioEngine, then it’s a known problem.

Also, even if we both say “camera stutter,” our understanding of that term might not be exactly the same, so a video would be very helpful.

[noho501]

Hi,

1. I've attached a sample video showing that when the camera moves quickly, the live stream become choppy.
   => https://drive.google.com/file/d/1_3DI_FZIBkPs_RDwdAJRbpErXBA_Ik79/view?usp=sharing

This issue also appears in the Example project without any modification.
=> https://drive.google.com/file/d/1e2zy7RgYCydNsj7rDww3YbST5RJWZuMk/view

We’ve also shared these videos directly to your personal email for easier review.

2. I’ve attached another video demonstrating the sound crackling issue.
   We’re not using AVAudioEngine — only the built-in microphone — but the audio sometimes becomes distorted or crackly during live streaming (YouTube )
   https://www.youtube.com/live/8lHiCdPdRY0?si=XF5Ali88EovzEEWD&t=1309
   https://www.youtube.com/live/8lHiCdPdRY0?si=yvDvDTJ8O5Tpi0SE&t=4505

3. Audio muted or out of sync.
   The audio becomes muted or goes out of sync with the video, especially after switching the camera or muting/unmuting the microphone for about a minute. This issue is intermittent and difficult to reproduce.

Below is our setup code:

// MARK: - 1. Audio Monitoring
let isHeadphoneConnected = DeviceUtil.isHeadphoneConnected()
await mixer.setMonitoringEnabled(isHeadphoneConnected)

// MARK: - 2. Mixer Setup
await mixer.setSessionPreset(.hd1920x1080)

var videoMixerSettings = await mixer.videoMixerSettings
videoMixerSettings.mode = .offscreen
await mixer.setVideoMixerSettings(videoMixerSettings)

let _ = await mixer.screen.registerVideoEffect(videoEffect)
await multipleStreamManager.setPreferences()
setCamerePreviewFullscreenIfNeed()

// MARK: - 3. Codec Settings
let audioConfig = 128000
let videoBitrate = 6000 * 1000

let audioSetting = AudioCodecSettings(
    bitRate: audioConfig.bitrate,
    sampleRate: audioConfig.sampleRate
)

var videoSetting = VideoCodecSettings(
    videoSize: CGSize(width: 1920, height: 1080),
    bitRate: videoBitrate,
    scalingMode: .letterbox,
    bitRateMode: .average,
    maxKeyFrameIntervalDuration: 2
)

if #available(iOS 16.0, *) {
    videoSetting.bitRateMode = .constant
}
videoSetting.profileLevel = kVTProfileLevel_H264_Main_AutoLevel as String

// MARK: - 4. Setup Streams
let stream = await streamManager.getStream()

try? await stream.setAudioSettings(audioSetting)
try? await stream.setVideoSettings(videoSetting)
await mixer.addOutput(stream)
await stream.addOutput(cameraScreenPreview)
await stream.addOutput(recorder)

// MARK: - 5. Start Mixer
await mixer.startRunning()

// MARK: - 6. Frame Rate Configuration
do {
    // Input frame rate
    try? await mixer.configuration(video: 0) { video in
        do {
            try video.setFrameRate(60)
        } catch {
            logger.error(error)
        }
    }

    // Output frame rate
    try await mixer.setFrameRate(60)
} catch {

}

Thanks & best regards,

[noho501]

Thank you for the confirmation. I hope this can be improved or that audio/video desynchronization can be prevented in a future release.

[shogo4405]

When the audio was muted, values were being unnecessarily updated in the AVAudioConverter.
If your issue was caused by this behavior, it may now be improved. #1819

[noho501]

I will try it. Thank you so much

[noho501]

Hi,

1. What I mean is that when I move the camera even slightly fast, the live stream video becomes noticeably less smooth — it looks like dropped or uneven frames.
   Below is a reference video I recorded using the PRISM Live app, just to show the kind of smoothness I’m referring to:
   👉 https://drive.google.com/file/d/1O55F4uLZXb4UaNPGJEYg9OE9aOivorD4/view?usp=sharing
   Hopefully, this helps illustrate the issue more clearly.
2. Here’s how we currently toggle audio on and off:

Task {
    var audioSetting = await mixer.audioMixerSettings
    if let firstTrack = audioSetting.tracks[0] {
        isMuted = !firstTrack.isMuted
        audioSetting.tracks[0]?.isMuted = isMuted
    }
    await mixer.setAudioMixerSettings(audioSetting)
}

Thanks & best regards,

Hi,

About issue 1, do you have any plan to improve it so the video doesn’t lag when moving the camera? I’m currently using an iPhone 16 Pro.

If I switch the video source to passthrough mode, the problem disappears — but in my case I need to use video effects to display a watermark during live streaming.

Thank you for your hard work on this project.

[shogo4405]

I’d like to improve it, but I can’t think of a good technical solution.
It may take some time to resolve, or it might not be possible to fix.

[shogo4405]

Audio issue

I tested both with and without the StreamRecorder feature enabled, but I wasn’t able to reproduce the issue on my end.

My hypothesis is that:
The microphone capture might be interrupted due to a system-level interruption request. For example,

• when receiving a phone call
• an alarm goes off, a notification with sound plays
• Siri is triggered

These events can cause an interruption request, which leads to AVAssetWriter pausing its write operation and microphone capture being suspended.

I recommend adding logging as shown in the following documentation to check whether any interruption requests are occurring:
https://developer.apple.com/documentation/avfaudio/handling-audio-interruptions

[noho501]

I recommend adding logging as shown in the following documentation to check whether any interruption requests are occurring:
https://developer.apple.com/documentation/avfaudio/handling-audio-interruptions

Yes, I implemented AVAudioSession.interruptionNotification in our project and tested it by playing music and videos from YouTube.
I can confirm that the interruption notification is triggered correctly, and your code handles this case well — the audio remains stable without delay.

However, in some cases (though I’m not yet sure how to reproduce them), at least five users have reported that over time, the audio and video gradually drift out of sync.
According to their feedback, this usually happens after switching the camera or changing the microphone during a live stream.

I’ll continue monitoring and investigating this issue to determine the root cause.

2. We’re not using AVAudioEngine — only the built-in microphone — but the audio sometimes becomes distorted or crackly during live streaming (YouTube )
https://www.youtube.com/live/8lHiCdPdRY0?si=XF5Ali88EovzEEWD&t=1309
https://www.youtube.com/live/8lHiCdPdRY0?si=yvDvDTJ8O5Tpi0SE&t=4505

Do you know what might cause this? It happens in all of my live streams.

I also tested it using your Example project, and the issue still occurs, although less frequently.

Here’s my current setup:

• Resolution: 1920×1080
• Frame rate: 30 fps
• Bitrate: 4000 × 1000
• StreamRecorder feature: enabled
• StreamBitRateStrategy: enabled
• bitRateMode: .constant
• captureSessionMode: .single
• multiTrackAudioMixingEnabled: false
• Input: built-in microphone

Best regards,

[shogo4405]

I’d like you to fork the repository and make the changes there.
There are several different parts related to the timestamp.

[noho501]

Yes. Could you please point me to the key places in the code where the audio and video frames are sent to the server, so I can add some logs and check the timestamps?

[shogo4405]

Here you are:

• HaishinKit.swift/RTMPHaishinKit/Sources/RTMP/RTMPMessage.swift

  Line 511 in 11519f1

  self.timestamp = timestamp

• HaishinKit.swift/RTMPHaishinKit/Sources/RTMP/RTMPMessage.swift

  Line 417 in 11519f1

  self.timestamp = timestamp

[shogo4405]

I investigated the ConstantBitrate issue and found the following results:

• iPhone 17 Pro Max + iOS 26.0.1 → OK
• iPhone SE2 + iOS 16.0 → NG

Since environments using iOS 16 are quite rare, I think it should be fine.

[noho501]

Thank you very much for your kind support.
However, I’m not using iOS 16 — my devices are running on iOS 26.0.1, including iPhone 16 Pro, iPad Air, and iPhone 11, all of which are experiencing audio issues.

Previously, I used average bitrate, but the issue still occurred, so I switched to constant bitrate. That’s why I don’t think the problem is related to the bitrate mode. The audio crackling happens especially when speaking loudly.

I’m still debugging the problem but haven’t found the cause yet. I’ll continue testing and share the results.