Custom Subtitles Renderer for AVPlayerLayer [Swift library already available]

[DrHurt]

This feature request respects the following points:

• This feature has not already been requested on GitHub.
• I agree to follow Jellyfin's Code of Conduct.

Platform

tvOS, iOS

Describe the feature you'd like to see

@JPKribs @LePips

As you already know, subtitles support in AVPlayer is limited & management via HLS is a nightmare.

Motivation:

The Jellyfin web client on Safari (AVPlayer) uses a custom renderer for subtitles that draws on top of AVPlayer.

Server already extracts and converts all subtitles ahead of time. Web client has separate controls drawn over AVPlayer to enable/disable & change subtitle tracks seamlesslessly. No need to mess with HLS or restart video playback.

The next chapter of Swiftfin?

Swift ASS Renderer is a free, open source library that renders ASS subtitles seamlessly on top of AVPlayerLayer and will be the perfect addition to Swiftfin.

This will initially provide ASS subtitles support for Swiftfin but can be expanded to support other formats.

A quick client-side text subtitles to ASS conversion (using ffmpeg) can also be employed allowing use of that library nearly unaltered with SRT (and many others) as well.

https://github.com/mihai8804858/swift-ass-renderer

[sjorge]

That would be awesome, most of my media should direct play with AVPlayer except for the ASS subs. Not having to use libvlc which seems to impact battery more would be great.

[DrHurt]

That would be awesome, most of my media should direct play with AVPlayer except for the ASS subs. Not having to use libvlc which seems to impact battery more would be great.

I don't know why your comment got hidden but I agree.

I avoid Swiftfin on iPhone/iPad in favour of using the web client in Safari for that exact same reason.

VLC is not just energy inefficient, it's also not good at dealing with HDR/DV (for video purists like me), it doesn't support Atmos, and it doesn't play nice with AirPlay speakers (like EarPods and HomePods).

Having AVPlayer with a custom subtitles renderer would be a game changer.

[lavavex]

As a stopgap, i would much rather have an option in the Native video player settings to force burn in subtitles when using Native player...

[JPKribs]

As a stopgap, i would much rather have an option in the Native video player settings to force burn in subtitles when using Native player...

AVPlayer can natively use some subtitles. There are some other steps that we need to take to make them accessible via the UI that we have documented here: #1853

I have a branch where I have started looking at #1853 with subtitle and audio switching. AVPlayer has limited subtitle support but the supported ones play without issue and the unsupported ones are burned-in appropriately. I have a demo of this in action here: #1882 (comment). You can ignore the audio in that video.... I clipped that at an unfortunate moment in the movie...

This issue would be specifically for custom ASS rendering which isn't feasible to look at prior to #1853. My branch working on this isn't usable without #1882 so that's the first domino to fall since that handles some of the HLS track switching that we need for AVPlayer. Finally, we'd need to do this after the tvOS UI is complete so #1902 would need to be completed first.

In order:

1. HLS Recreation Logic - Index/Track Fixes #1882
2. tvOS UI - [tvOS] Media Player #1902
3. Migrate to AVMediaPlayerProxy - Replace Native Player with AVMediaPlayerProxy #1853 <- No Public PR at this time
4. This Issue.

[lavavex]

As a stopgap, i would much rather have an option in the Native video player settings to force burn in subtitles when using Native player...

AVPlayer can natively use some subtitles. There are some other steps that we need to take to make them accessible via the UI that we have documented here: #1853

I have a branch where I have started looking at #1853 with subtitle and audio switching. AVPlayer has limited subtitle support but the supported ones play without issue and the unsupported ones are burned-in appropriately. I have a demo of this in action here: #1882 (comment). You can ignore the audio in that video.... I clipped that at an unfortunate moment in the movie...

This issue would be specifically for custom ASS rendering which isn't feasible to look at prior to #1853. My branch working on this isn't usable without #1882 so that's the first domino to fall since that handles some of the HLS track switching that we need for AVPlayer. Finally, we'd need to do this after the tvOS UI is complete so #1902 would need to be completed first.

In order:

1. HLS Recreation Logic - [Index/Track Fixes #1882](https://github.com/jellyfin/Swiftfin/pull/1882)

2. tvOS UI - [[tvOS] Media Player #1902](https://github.com/jellyfin/Swiftfin/pull/1902)

3. Migrate to `AVMediaPlayerProxy`  - [Replace `Native` Player with `AVMediaPlayerProxy` #1853](https://github.com/jellyfin/Swiftfin/issues/1853) <- No Public PR at this time

4. **This Issue.**

just check that branch out to play with on my appletv, luckily for me it seems it is auto burning in the subs due to unsupported codec anyways so it works fine.
Im forced to Native player cause i use homepod stereo pair so i can finally stop using my android tv just for jellyfin lol

[JPKribs]

just check that branch out to play with on my appletv

My branch working on AVPlayerProxy doesn't build for tvOS at this time. The tvOS UI is needed first. A final version of this is on the list. There's just a lot of other items first.

[lavavex]

just check that branch out to play with on my appletv

My branch working on AVPlayerProxy doesn't build for tvOS at this time. The tvOS UI is needed first. A final version of this is on the list. There's just a lot of other items first.

ah, my dumb butt built on wrong branch, oops