AVCaptureMultiCamSession sometimes blocks the main thread during app startup on iOS18.0, iOS18.1...

[levs42]

Describe the bug

During the migration to 2.0.0 our main thread started hanging for ~10s during startup. The issue is similar to #1581. I've found that the block occurs during session.commitConfiguration() call and is related only to attachAudio:

1. app calls mixer.attachOutput(stream)
2. MediaMixer calls session.startRunning()
3. app calls mixer.attachAudio(device)
4. MediaMixer calls session.commitConfiguration()

The block doesn't occur in two cases:

• attachAudio is called before session.startRunning()
• attachAudio is called a few seconds after session.startRunning()

I tried to reproduce this issue with the iOS example and realized that Task in viewDidLoad takes longer to perform than Task in viewWillAppear so attachAudio executes before session.startRunning(). Modifying the viewDidLoad Task reproduces the issue:

IngestViewController

override func viewDidLoad() {
    logger.info("viewDidLoad")
    super.viewDidLoad()
    Task {
        if let orientation = DeviceUtil.videoOrientation(by: UIApplication.shared.statusBarOrientation) {
            await mixer.setVideoOrientation(orientation)
        }
        await mixer.setMonitoringEnabled(DeviceUtil.isHeadphoneConnected())
        var videoMixerSettings = await mixer.videoMixerSettings
        videoMixerSettings.mode = .offscreen
        await mixer.setVideoMixerSettings(videoMixerSettings)
        await netStreamSwitcher.setPreference(Preference.default)
        if let stream = await netStreamSwitcher.stream {
            await mixer.addOutput(stream)
            if let view = view as? (any HKStreamOutput) {
                await stream.addOutput(view)
            }
        }
        // code from viewWillAppear:
        let back = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: currentPosition)
        try? await mixer.attachVideo(back, track: 0)
        try? await mixer.attachAudio(AVCaptureDevice.default(for: .audio))
        let front = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front)
        try? await mixer.attachVideo(front, track: 1) { videoUnit in
            videoUnit.isVideoMirrored = true
        }
    }
    // ...

I wanted to get your opinion about a fix where attachAudio is called before addOutput during the setup:

Example

func viewDidLoad() {
  await attach()
  await mixer.addOutput(stream)
}

func viewWillAppear()  {
  await attach()
}

func attach() async {
  await mixer.attachAudio(AVCaptureDevice.default(for: .audio), track: 0)
}

This fix might be beneficial for the iOS example to avoid potential race condition between tasks inviewDidLoad and viewWillAppear. I also wanted to ask if you know why commitConfiguration takes so long to execute. Thanks.

To Reproduce

Provided in the description

Expected behavior

To avoid main thread blocking

Version

2.0.0-rc.2

Smartphone info.

18.1, 15 Pro Max

Additional context

No response

Screenshots

No response

Relevant log output

2024-11-08 5:44:20 PM +0000 before session.commitConfiguration()
2024-11-08 5:44:29 PM +0000 after session.commitConfiguration()

[shogo4405]

• I recognize that the slowdown has become noticeable on iOS 18.0 and later, though I'm not sure of the reason.
• Considering the lifecycle of tab-based apps like the Example app, I believe the current setup locations are appropriate.
  • Registering addOutput in viewDidLoad is appropriate.
  • Performing attach/detach during tab transitions (viewWillAppear/viewWillDisappear) is appropriate.

A strong option at this point would be to stop HK’s automatic calls and have developers call startCapturing/stopCapturing in alignment with the lifecycle. However, this may be confusing as it differs from the previous ease of use.

[shogo4405]

When launching the Example app from Xcode, it takes a long time to start up, which is bothersome. However, during a normal launch, it doesn’t seem to lock the main thread, and it starts up smoothly, so I don’t think there’s an issue.

Is this specifically about launching from Xcode, or are you referring to starting up the app in general?

[levs42]

Thank you for the response! I would prefer manual startCapturing call, but I understand that it might be less obvious. This report is more to save time in case it surfaces in the future.

during a normal launch, it doesn’t seem to lock the main thread

The example that I provided for IngestViewController blocks the thread with a general app launch (built with XCode 16.1, iOS 18.1)

[shogo4405]

This issue occurs in both HaishinKit 1.9.x and 2.0.x. It has been confirmed on iOS 18.0 and iOS 18.1.
A simple way to reproduce the issue, independent of HaishinKit, is as follows:

1. Use AVCaptureMultiCamSession.
2. Call session.startRunning().
3. Call session.addOutput().
4. Perform these steps right after the session starts running.

At step 4, a 10-second freeze occurs. Since the freeze duration is consistently around 10 seconds, it seems to be related to some process on the OS side.

To avoid this issue safely, I believe it’s better to manage session.startRunning() consciously on the user side instead of triggering it automatically.

[shogo4405]

Here is the PR that allows manual handling:

• Add an option to start capture manually. #1642