mixed encrypted/clear fairplay audio content fails on append

[gkatsev]

Have you read the FAQ and checked for duplicate open issues?
Yes

If the problem is related to FairPlay, have you read the tutorial?

Yes

What version of Shaka Player are you using?

4.10

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from main?
Yes

Are you using the demo app or your own custom app?
Both

If custom app, can you reproduce the issue using our demo app?
Yes

What browser and OS are you using?
Safari 18.3

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

What are the manifest and license server URIs?

Using the public fairplay source and another public hls source, I spliced the two together, using a disco tag, for test content.
https://friendly-creponne-991405.netlify.app/mixed-fairplay.m3u8 - encrypted followed by clear
https://friendly-creponne-991405.netlify.app/fairplay-mixed.m3u8 - clear followed by encrypted

https://fps.ezdrm.com/api/licenses/b99ed9e5-c641-49d1-bfa8-43692b686ddb - license server url

https://fps.ezdrm.com/demo/video/eleisure.cer - certificate server url

What configuration are you using? What is the output of player.getNonDefaultConfiguration()?

{
  "streaming": {
    "useNativeHlsForFairPlay": false
  },
}

Adding manifest.disableAudio makes playback work.

{
  "manifest": {
    "disableAudio": true
  },
  "streaming": {
    "useNativeHlsForFairPlay": false
  },
}

What did you do?

Play one of the above streams and wait for shaka to start appending the opposite type of what is currently playing.

What did you expect to happen?
The append should succeed and playback should continue.

What actually happened?

Shaka throws a 3016 error on append, saying media failed to decode.
If I disable the audio stream, then the append succeeds and playback continues across the discontinuity.

Are you planning to send a PR to fix it?
I'll continue looking at it, but any help would be greatly appreciated as I haven't been able to find anything obviously wrong here.

[joeyparrish]

A decoder error is usually something we don't have direct control over. Either the content is malformed or incompatible with the decoder in some way, but we don't control either content encoding or decoding. Some exceptions to that guidelines could be:

• We transmuxed the content and broke the container or codec data in the process
• We fetched the wrong bytes, and appended a malformed or partial media or init segment because of it
• We constructed the wrong URL and fetched bytes from the wrong segment, again resulting in a malformed or partial media or init segment

I don't see MPEG-2 TS here, so I don't think we're transmuxing. I see byte ranges on the encrypted portions of these streams, but not on the clear portions. And I think if we were fetching the wrong byte ranges, we would have heard feedback on that by now.

So I'm assuming for the moment that it's not exactly our fault in player land. I'm guessing it's a bug in FairPlay or Safari, because that is the other detail that sticks out to me. Can FairPlay handle mixed clear/encrypted content without tearing down MediaSource in some other player? (Excluding a player that tears down the stack on discontinuities, avoiding the transition.) If not, I think Apple needs to hear about this.

If the stack is torn down on discontinuities, would playback succeed? (It's implied that it would, because you only get a failure on that transition.) This could be a workaround, and we have some infrastructure in the player to tear down on codec changes. We could maybe artificially signal discontinuities as codec changes and then force tear-down as a proof-of-concept.

[gkatsev]

Yeah, it definitely seems like this may be a Safari issue here. The weird thing is that if you disable audio, it all works fine. Plus, the codec, sample rate, and audio channels for the two audio discos match, according to mediastream validator. Native Safari is able to play it without any issues.

For comparison, hls.js (demo with the above content) also has issues crossing the boundary, but if you seek directly to the encrypted portion it seems to work.

The interesting thing is that the license isn't being downloaded. On hls.js the license doesn't get downloaded on regular play through, but it does get downloaded on the seek. In video-only mode, we do get a license request.

The cross boundary strategy doesn't seem to do it, but I think it is missing something for hls. Also, while it'll enable playback, it's probably not ideal user behavior for it to reset mse on every discontinuity.

[absidue]

Sounds like this PR would have been useful for you although it was reverted almost instantly: #8156

About it working for native playback, I'm not really surprised that FairPlay works differently for native vs MSE playback in Safari, considering that internally there are probably quite a lot of differences.

[avelad]

Sounds like this PR would have been useful for you although it was reverted almost instantly: #8156

This is wrong, it has not been reverted! :) It is included in the latest release.

[gkatsev]

Looks like this is a known issue on hls.js: video-dev/hls.js#4230

In a thread on video-dev, Rob mentioned that the clear segments need to get patched so that the encrypted pathway is used by Safari. Perhaps tenc or enca/v boxes