Strange audio track selection behaviour with different device profiles

[rasko-dev]

Describe the bug

There appear to be some quirks with the device profiles I'm encountering specifically with TrueHD audio tracks. Some profiles hide all tracks other than the TrueHD tracks from being selected, and there are some issues where the server claims to be transcoding the audio, but Swiftfin still doesn't play any audio.

Read update in comment for potential cause

Here are my observations with the different profiles:
Automatic profile:

• IF the playback starts with TrueHD selected, then TrueHD is the only availible track in the player, despite other compatible tracks being availible in the video file. If another track has been selected (which is possible in the direct profile, or probably if another is marked as default) then all tracks are shown and can be chosen. Selecting TrueHD and then quitting playback and starting it again will again lock it to only displaying TrueHD.
• The Jellyfin server seems to be "Direct streaming", a.k.a. transcoding the audio, however no audio is being played.

Most compatible:

• TrueHD again is the only availible to select audio, and the server claims to transcode yet no audio is played.

Direct profile:

• Now I can finally choose the other availible tracks, although TrueHD is still selected as default (this might be intentional though as this profile is not supposed to transcode at all?), however choosing one of the other compatible tracks by default would make sense IMO, even though the TrueHD is selected as default in the actual video file, due to Swiftfin not actually being able to directly play it.

Overall it seems like the selected track is remembered across changing profiles, even though it isn't quite compatible, and also there are some issues where Swiftfin doesn't properly use the transcoded audio from Jellyfin server and instead doesn't play any audio at all. Tracks are also hidden from the playback selection in the Auto and Most compatible mode, specifically if a TrueHD track is selected. There might also be some nessecary consideration for whether or not a TrueHD track should be selected by default across different profiles

Application version

1.4(13)

Where did you install the app from?

TestFlight

Device information

iPad Air M2

OS version

iPadOS 26

Jellyfin server version

10.10.7

[rasko-dev]

Update to this:
I've done some debugging and it appears this may be due to the AudioActionButton.swift getting its availible streams from the MediaPlayerItem.audioStreams, which will only include the first track for transcoded streams due to the following:

Swiftfin/Shared/Objects/MediaPlayerManager/MediaPlayerItem/MediaPlayerItem.swift

Line 82 in 9d4f11d

let adjustedMediaStreams = mediaSource.mediaStreams?.adjustedTrackIndexes(

Swiftfin/Shared/Extensions/JellyfinAPI/MediaStream.swift

Lines 219 to 240 in 9d4f11d

	func adjustedTrackIndexes(for playMethod: PlayMethod, selectedAudioStreamIndex: Int) -> [MediaStream] {
	let internalTracks = self.filter { !($0.isExternal ?? false) }
	let externalTracks = self.filter { $0.isExternal ?? false }
	var orderedInternal: [MediaStream] = []
	let subtitleInternal = internalTracks.filter { $0.type == .subtitle }
	if playMethod == .transcode {
	// Only include the first video and first audio track for transcode.
	let videoInternal = internalTracks.filter { $0.type == .video }
	let audioInternal = internalTracks.filter { $0.type == .audio }
	if let firstVideo = videoInternal.first {
	orderedInternal.append(firstVideo)
	}
	if let selectedAudio = audioInternal.first(where: { $0.index == selectedAudioStreamIndex }) {
	orderedInternal.append(selectedAudio)
	}
	orderedInternal += subtitleInternal
	} else {

And because the stream only includes the first audio track, this means if it is transcoding then the Audio selector will only see one track and be unable to change it to any other. This is why it only happens on the Auto and Compatible profiles, as this is where transcoding is required for the TrueHD format.

I suspect changing this code to return all tracks instead of only the first track will probably have some unintended consequences, and it is the way it is for a reason. (although it seems to kinda work in the simulator, but the simulator seems very slow/buggy for playback anyways Shows the extra tracks but wont play any audio when selecting them, presumably a deeper change is necessary)
I guess @LePips is the one with the most knowledge about how these systems interact with each other.
I'm not sure what the best way of solving this is, but if I could have some pointers I could have a go at solving it (if it's not too complicated, although it may take some time, i'm neither familiar with the codebase or proficient with swift at all)

[samglt]

And because the stream only includes the first audio track, this means if it is transcoding then the Audio selector will only see one track and be unable to change it to any other

I believe this is because when you start playing something that needs to be transcoded, Jellyfin will only provide the audio track the app requests, so changing tracks live while playing would do nothing. I think #1581 makes it easier to enable switching tracks in transcoded files while playing (see this comment)

As for which track is selected by default, as of #1581 Swiftfin now seems to use the defaultAudioStreamIndex that Jellyfin provides (see your first link) and in my experience that's basically the last track you selected on any Jellyfin client, so if you played something on another device with TrueHD audio selected and then play it in Swiftfin, Jellyfin would select that audio track by default.

I can't speak to the transcoding of TrueHD audio though.

[rasko-dev]

Looking into it some more and testing some more I'm starting to wonder if this could be a result of audio transcoding not being supported fully yet (or bugged), because I'm noticing transcoding audio doesn't work at all, even for lower quality codecs. Essentially when the Compatible device profile is selected I can't get audio to play for any media. Maybe as a result of this the stream gets borked when starting out using transcoded audio. I guess it could also be separate, but to me it looks like changing the audio stream mid-play should already be implemented. But this is definitively above my expertise level, so I'm not gonna spit out any more probably incorrect theories 😅.

[LePips]

In a comment on #1581 (4,5) I explain the possible source of this issue.

When we build a media player item, we currently search for the MediaSourceInfo to use. That media source is then used to populate the available media streams and build the stream URL.

If it's a transcoded media source then most likely only a subset of media streams are available to select from. Now we need to actually pass along the full media source for the UI to use, implement logic in MediaPlayerItem to determine if the current media source used for playback doesn't support streams from the full media source, and make a new stream if selecting a stream that the current media source doesn't support.