updateSpeaker in react client is not changing the speakers correctly

[itsikbelson-spotlight]

After I use updateSpeaker from usePipecatClientMediaDevices to switch between audio out devices, the audio output is flickering between these audio devices (e.g. microphone and built-in speakers).
I am running on MacOS and tried it both on Safari and Chrome, and it happened on both.
I tried it both with bot audio and with an audio file I played, and it happened on bot.
Also tried different ways to setSinkId myself, but the issue persisted.

When I created my own version of audio output change (by using enumerateDevices and setSinkId) the issue didn't happen.

Here's some sample code of what I tried.

import React, { FC, useEffect, useState } from 'react';
import { PipecatClientProvider } from '@pipecat-ai/client-react';
import { PipecatClient } from '@pipecat-ai/client-js';
import { DailyTransport } from '@pipecat-ai/daily-transport';
import { FcLoadingScreen } from '@root/shared/components/LoadingScreen/LoadingScreen';
import { FcPipecatAudioTest } from './PipecatAudioTest';

const FcPipecatAudioPlayground: FC = () => {
   const [isInitialized, setIsInitialized] = useState(false);
   const client = new PipecatClient({
      transport: new DailyTransport(),
      callbacks: {
         onSpeakerUpdated: (speaker: MediaDeviceInfo) => {
            console.log(
               'xxx. Speaker updated (Pipecat client):',
               speaker.label,
            );
         },
      },
   });
   useEffect(() => {
      if (!client) return;
      client.initDevices().then(() => {
         setIsInitialized(true);
      });
   }, [client]);
   return (
      <PipecatClientProvider client={client}>
         {isInitialized ? <FcPipecatAudioTest /> : <FcLoadingScreen />}
      </PipecatClientProvider>
   );
};

export { FcPipecatAudioPlayground };

import React, { FC, useCallback, useRef, useState } from 'react';
import {
   usePipecatClientMediaDevices,
   useRTVIClientEvent,
} from '@pipecat-ai/client-react';
import { FcButton } from '@shared/components/Buttons/Button/Button';
import { RTVIEvent } from '@pipecat-ai/client-js';
import gongAudio from '@assets/sounds/gong-end.mp3';

export const FcPipecatAudioTest: FC = () => {
   const audioRef = useRef<HTMLAudioElement>(null);
   const [isPlaying, setIsPlaying] = useState(false);
   const { selectedSpeaker, updateSpeaker, availableSpeakers } =
      usePipecatClientMediaDevices();

   const selectedSpeakerId = selectedSpeaker?.deviceId;

   const handleStartAudio = () => {
      if (audioRef.current) {
         audioRef.current.play();
         setIsPlaying(true);
      }
   };

   const handleStopAudio = () => {
      if (audioRef.current) {
         audioRef.current.pause();
         audioRef.current.currentTime = 0;
         setIsPlaying(false);
      }
   };

   useRTVIClientEvent(
      RTVIEvent.SpeakerUpdated,
      useCallback((speaker: MediaDeviceInfo) => {
         console.log('xxx. Speaker updated (RTVI):', speaker.label);
         if (!audioRef.current) return;
         if (typeof audioRef.current.setSinkId !== 'function') return;
         audioRef.current.setSinkId(speaker.deviceId);
      }, []),
   );

   const handleSpeakerChange = (e: React.ChangeEvent<HTMLSelectElement>) => {
      const deviceId = e.target.value;
      const deviceLabel = availableSpeakers.find(
         (speaker) => speaker.deviceId === deviceId,
      )?.label;
      console.log(
         'xxx. User changed speaker (handleSpeakerChange):',
         deviceLabel,
      );
      updateSpeaker(deviceId);
   };

   return (
      <div className="flex min-h-screen items-center justify-center bg-gray-100">
         <audio ref={audioRef} src={gongAudio} loop />
         <div className="flex w-full max-w-md flex-col gap-6 rounded-lg border border-gray-300 bg-white p-6 shadow-lg">
            <h1 className="text-2xl font-bold text-gray-800">
               Pipecat Audio Test
            </h1>

            <div className="flex flex-col gap-3">
               <label className="text-sm font-semibold text-gray-700">
                  Audio Controls
               </label>
               <div className="flex gap-3">
                  <FcButton onClick={handleStartAudio} disabled={isPlaying}>
                     Start Audio
                  </FcButton>
                  <FcButton onClick={handleStopAudio} disabled={!isPlaying}>
                     Stop Audio
                  </FcButton>
               </div>
            </div>

            <div className="flex flex-col gap-3">
               <label
                  htmlFor="speaker-select"
                  className="text-sm font-semibold text-gray-700"
               >
                  Select Speaker
               </label>
               <select
                  id="speaker-select"
                  value={selectedSpeakerId || ''}
                  onChange={handleSpeakerChange}
                  className="rounded border border-gray-300 bg-white px-3 py-2 text-gray-800 focus:border-blue-500 focus:ring-2 focus:ring-blue-200 focus:outline-none"
               >
                  <option value="">-- Select a speaker --</option>
                  {availableSpeakers.map((speaker) => (
                     <option key={speaker.deviceId} value={speaker.deviceId}>
                        {speaker.label || `Speaker (${speaker.deviceId})`}
                     </option>
                  ))}
               </select>
            </div>

            <div className="border-t border-gray-200 pt-4">
               <div className="flex flex-col gap-4">
                  <div className="flex flex-col gap-2">
                     <p className="text-sm text-gray-600">
                        <span className="font-semibold">Current Speaker:</span>
                     </p>
                     <div className="rounded bg-gray-50 p-3 text-sm text-gray-700">
                        {selectedSpeaker?.label ||
                           selectedSpeakerId ||
                           'No speaker selected'}
                     </div>
                  </div>
               </div>
            </div>
         </div>
      </div>
   );
};

[itsikbelson-spotlight]

I found out that the problematic code is the initDevices I call directly on the client.
I guess there's something duplicate there.
However, when I remove the initDevices, the selectedSpeaker is not updated after I call updateSpeaker, which basically mean this didn't have any effect.

[itsikbelson-spotlight]

Okay. So I've found the issue/s.
It turns out initDevices in the client calls startCamera in the transport layer which initiates the daily call machine, and allows the speakerUpdated event to shoot.
This is why calling initDevices on the client is mandatory.
The above is issue #1, which should at least be documented in my opinion (see https://docs.pipecat.ai/client/react/hooks#usepipecatclientmediadevices).

Now, after I added initDevices, as part of the initialization process, the daily call machine starts a sync process (called via startPollingToSyncPreferredAudioOutputToDOM), who's purpose is to update sinkId af all existing audio elements in the DOM (run under setInterval).
I guess that the reason for it is in order to make sure all elements are playing to the selected audio output devices.
Here is where issue #2 comes in, if we don't perform cleanup when component unmounts, the sync process keeps running, switching back to the original audio output device (before we changed it using updateSpeaker. This happens when running with StrictMode which unmounts and mounts the component. Unfortunately there's no dedicated cleanup method available on the client. There is a disconnect method which works if added to the cleanup (stop the synching process triggered by the unmounted component). Calling disconnect is a bit assymetric, as we didn't call connect, only initDevices.
Issue #3 is that if we want to have another client (e.g. in our case we open a settings panel in which the user can change settings, but they will apply only when clicking apply), the sync process will run on both clients, causing flickering between the speakers. Not sure if there is a solution for this issue though.