Low latency mode enabled should not fallback to normal latency mode

[truong-hua]

What version of Hls.js are you using?

#8e77d46

What browser (including version) are you using?

Chrome 132.0.6834.83 (Official build) x64

What OS (including version) are you using?

Window 10

Test stream

No response

Configuration

{
  "debug": true,
  "enableWorker": true,
  "lowLatencyMode": true,
  "backBufferLength": 90
}

Additional player setup steps

Play a low latency stream which a segment part is corrupted. In reality we got some race condition that cause the latest part to not fully flushed and available to the HTTP service while the manifest is a little bit faster.

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
• The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Play a low latency stream with at least a corrupted part (not segment)

Expected behaviour

The player can retry the whole state or report error and stop rather than automatically fallback to normal-latency mode while the config lowLatencyMode is true.

My question is not sure it's a right way that the player should do because in my perspective, when I upgraded my stream to use the low-latency mode which mean the normal-latency stream does not qualify our needs. Automatically falling back to use normal-latency mode will cause my end-users to angry with our business even that the issue may be come from another side like corrupted parts. But my users are willing to see a stream reload rather than a high-latency one.

If the current behavior is right, so I think there should be a flag to prevent the player to do the fallback, ex: lowLatencyEnforced = true

What actually happened?

HLS.js fallback to play the low-latency stream as normal-latency stream and never turn it back. The falling to normal-latency happened here

hls.js/src/controller/base-stream-controller.ts

Line 908 in 8e77d46

`LL-Part loading OFF after next part miss @${targetBufferTime.toFixed(

, when the handling of low latency parts were not success.

Console output

Error event: {type: 'mediaError', details: 'bufferAppendingError', sourceBufferName: 'video', error: Error: video SourceBuffer error. MediaSource readyState: ended
    at BufferController.onSBUpdateEr…, fatal: false, …}
(anonymous) @ hls-demo.js:24786
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
onSBUpdateError @ hls.js:19487
error
addBufferListener @ hls.js:19651
trackSourceBuffer @ hls.js:19447
createSourceBuffers @ hls.js:19415
checkPendingTracks @ hls.js:19339
onBufferCodecs @ hls.js:18777
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
_bufferInitSegment @ hls.js:33307
_handleTransmuxComplete @ hls.js:33109
handleTransmuxComplete @ hls.js:16684
push @ hls.js:16617
_handleFragmentLoadProgress @ hls.js:32791
progressCallback @ hls.js:9092
onSuccess @ hls.js:6155
readystatechange @ hls.js:30130
XMLHttpRequest.send
openAndSendXhr @ hls.js:30078
loadInternal @ hls.js:30052
load @ hls.js:30016
(anonymous) @ hls.js:6145
loadPart @ hls.js:6122
loadPart @ hls.js:9441
(anonymous) @ hls.js:9457
doFragPartsLoad @ hls.js:9436
_doFragLoad @ hls.js:9367
_loadFragForPlayback @ hls.js:9094
loadFragment @ hls.js:9080
loadFragment @ hls.js:32485
doTickIdle @ hls.js:32473
doTick @ hls.js:32360
tick @ hls.js:6413
setInterval
setInterval @ hls.js:6375
startLoad @ hls.js:32285
startLoad @ hls.js:34596
startLoadingPrimaryAt @ hls.js:24596
attachPrimary @ hls.js:24591
resumePrimary @ hls.js:24532
advanceSchedule @ hls.js:24505
setSchedulePosition @ hls.js:24415
onMediaAttached @ hls.js:24186
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
BufferController._this2._onMediaSourceOpen @ hls.js:18307
 [warn] > [buffer-controller]: Failed 1/3 times to append segment in "video" sourceBuffer
onError @ hls.js:18999
onSBUpdateError @ hls.js:19497
error
addBufferListener @ hls.js:19651
trackSourceBuffer @ hls.js:19447
createSourceBuffers @ hls.js:19415
checkPendingTracks @ hls.js:19339
onBufferCodecs @ hls.js:18777
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
_bufferInitSegment @ hls.js:33307
_handleTransmuxComplete @ hls.js:33109
handleTransmuxComplete @ hls.js:16684
push @ hls.js:16617
_handleFragmentLoadProgress @ hls.js:32791
progressCallback @ hls.js:9092
onSuccess @ hls.js:6155
readystatechange @ hls.js:30130
XMLHttpRequest.send
openAndSendXhr @ hls.js:30078
loadInternal @ hls.js:30052
load @ hls.js:30016
(anonymous) @ hls.js:6145
loadPart @ hls.js:6122
loadPart @ hls.js:9441
(anonymous) @ hls.js:9457
doFragPartsLoad @ hls.js:9436
_doFragLoad @ hls.js:9367
_loadFragForPlayback @ hls.js:9094
loadFragment @ hls.js:9080
loadFragment @ hls.js:32485
doTickIdle @ hls.js:32473
doTick @ hls.js:32360
tick @ hls.js:6413
setInterval
setInterval @ hls.js:6375
startLoad @ hls.js:32285
startLoad @ hls.js:34596
startLoadingPrimaryAt @ hls.js:24596
attachPrimary @ hls.js:24591
resumePrimary @ hls.js:24532
advanceSchedule @ hls.js:24505
setSchedulePosition @ hls.js:24415
onMediaAttached @ hls.js:24186
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
BufferController._this2._onMediaSourceOpen @ hls.js:18307
 [warn] > [content-steering]: Could not resolve bufferAppendError ("video SourceBuffer error. MediaSource readyState: ended") with content-steering for Pathway: . levels: 1 priorities: ["."] penalized: {".":63540418.30000001}
onError @ hls.js:21442
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
onError @ hls.js:19004
onSBUpdateError @ hls.js:19497
error
addBufferListener @ hls.js:19651
trackSourceBuffer @ hls.js:19447
createSourceBuffers @ hls.js:19415
checkPendingTracks @ hls.js:19339
onBufferCodecs @ hls.js:18777
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
_bufferInitSegment @ hls.js:33307
_handleTransmuxComplete @ hls.js:33109
handleTransmuxComplete @ hls.js:16684
push @ hls.js:16617
_handleFragmentLoadProgress @ hls.js:32791
progressCallback @ hls.js:9092
onSuccess @ hls.js:6155
readystatechange @ hls.js:30130
XMLHttpRequest.send
openAndSendXhr @ hls.js:30078
loadInternal @ hls.js:30052
load @ hls.js:30016
(anonymous) @ hls.js:6145
loadPart @ hls.js:6122
loadPart @ hls.js:9441
(anonymous) @ hls.js:9457
doFragPartsLoad @ hls.js:9436
_doFragLoad @ hls.js:9367
_loadFragForPlayback @ hls.js:9094
loadFragment @ hls.js:9080
loadFragment @ hls.js:32485
doTickIdle @ hls.js:32473
doTick @ hls.js:32360
tick @ hls.js:6413
setInterval
setInterval @ hls.js:6375
startLoad @ hls.js:32285
startLoad @ hls.js:34596
startLoadingPrimaryAt @ hls.js:24596
attachPrimary @ hls.js:24591
resumePrimary @ hls.js:24532
advanceSchedule @ hls.js:24505
setSchedulePosition @ hls.js:24415
onMediaAttached @ hls.js:24186
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
BufferController._this2._onMediaSourceOpen @ hls.js:18307
 [log] > [stream-controller]: Reset loading state
 [log] > [stream-controller]: PARSED->IDLE
 [log] > stopLoad
 [log] > [stream-controller]: IDLE->STOPPED
 [log] > [subtitle-stream-controller]: IDLE->STOPPED
 Error event: {type: 'mediaError', parent: 'main', details: 'bufferAppendError', sourceBufferName: 'video', frag: Fragment, …}
(anonymous) @ hls-demo.js:24786
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
onError @ hls.js:19004
onSBUpdateError @ hls.js:19497
error
addBufferListener @ hls.js:19651
trackSourceBuffer @ hls.js:19447
createSourceBuffers @ hls.js:19415
checkPendingTracks @ hls.js:19339
onBufferCodecs @ hls.js:18777
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
_bufferInitSegment @ hls.js:33307
_handleTransmuxComplete @ hls.js:33109
handleTransmuxComplete @ hls.js:16684
push @ hls.js:16617
_handleFragmentLoadProgress @ hls.js:32791
progressCallback @ hls.js:9092
onSuccess @ hls.js:6155
readystatechange @ hls.js:30130
XMLHttpRequest.send
openAndSendXhr @ hls.js:30078
loadInternal @ hls.js:30052
load @ hls.js:30016
(anonymous) @ hls.js:6145
loadPart @ hls.js:6122
loadPart @ hls.js:9441
(anonymous) @ hls.js:9457
doFragPartsLoad @ hls.js:9436
_doFragLoad @ hls.js:9367
_loadFragForPlayback @ hls.js:9094
loadFragment @ hls.js:9080
loadFragment @ hls.js:32485
doTickIdle @ hls.js:32473
doTick @ hls.js:32360
tick @ hls.js:6413
setInterval
setInterval @ hls.js:6375
startLoad @ hls.js:32285
startLoad @ hls.js:34596
startLoadingPrimaryAt @ hls.js:24596
attachPrimary @ hls.js:24591
resumePrimary @ hls.js:24532
advanceSchedule @ hls.js:24505
setSchedulePosition @ hls.js:24415
onMediaAttached @ hls.js:24186
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
BufferController._this2._onMediaSourceOpen @ hls.js:18307
 Fatal error : bufferAppendError
(anonymous) @ hls-demo.js:24861
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
onError @ hls.js:19004
onSBUpdateError @ hls.js:19497
error
addBufferListener @ hls.js:19651
trackSourceBuffer @ hls.js:19447
createSourceBuffers @ hls.js:19415
checkPendingTracks @ hls.js:19339
onBufferCodecs @ hls.js:18777
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
_bufferInitSegment @ hls.js:33307
_handleTransmuxComplete @ hls.js:33109
handleTransmuxComplete @ hls.js:16684
push @ hls.js:16617
_handleFragmentLoadProgress @ hls.js:32791
progressCallback @ hls.js:9092
onSuccess @ hls.js:6155
readystatechange @ hls.js:30130
XMLHttpRequest.send
openAndSendXhr @ hls.js:30078
loadInternal @ hls.js:30052
load @ hls.js:30016
(anonymous) @ hls.js:6145
loadPart @ hls.js:6122
loadPart @ hls.js:9441
(anonymous) @ hls.js:9457
doFragPartsLoad @ hls.js:9436
_doFragLoad @ hls.js:9367
_loadFragForPlayback @ hls.js:9094
loadFragment @ hls.js:9080
loadFragment @ hls.js:32485
doTickIdle @ hls.js:32473
doTick @ hls.js:32360
tick @ hls.js:6413
setInterval
setInterval @ hls.js:6375
startLoad @ hls.js:32285
startLoad @ hls.js:34596
startLoadingPrimaryAt @ hls.js:24596
attachPrimary @ hls.js:24591
resumePrimary @ hls.js:24532
advanceSchedule @ hls.js:24505
setSchedulePosition @ hls.js:24415
onMediaAttached @ hls.js:24186
emit @ hls.js:396
emit @ hls.js:34449
trigger @ hls.js:34453
BufferController._this2._onMediaSourceOpen @ hls.js:18307
 [log] > recoverMediaError
 [log] > detachMedia
 [log] > [buffer-controller]: media source detaching
 [log] > attachMedia
 [log] > [buffer-controller]: created media source: MediaSource
 [log] > startLoad(60.125249)
 [log] > [level-controller]: Loading level index 0 age 34.5 http://s2.enc.sctvonline.vn/sctv2/video_hd/playlist.m3u8
 [log] > [stream-controller]: STOPPED->IDLE
 [log] > [subtitle-stream-controller]: STOPPED->IDLE
segment_392.m4s.2:1  Failed to load resource: the server responded with a status of 404 (Not Found)
 [log] > [buffer-controller]: Media source opened
 [log] > [buffer-controller]: Updating MediaSource duration to 67.840
 [log] > [interstitials]: setSchedulePosition 0, undefined
 [log] > [interstitials]: buffered to boundary [primary: 0.00-Infinity]
 [log] > [interstitials]: resuming [primary: 0.00-Infinity]
 [log] > startLoad(59.76509)
 [log] > [level-controller]: Loading level index 0 age 34.6 http://s2.enc.sctvonline.vn/sctv2/video_hd/playlist.m3u8
 [log] > [playlist-loader]: ignore http://s2.enc.sctvonline.vn/sctv2/video_hd/playlist.m3u8 ongoing request
 [log] > [stream-controller]: IDLE->WAITING_LEVEL
 [log] > [stream-controller]: WAITING_LEVEL->STOPPED
 [log] > [stream-controller]: STOPPED->IDLE
 [log] > [subtitle-stream-controller]: IDLE->STOPPED
 [log] > [subtitle-stream-controller]: STOPPED->IDLE
 [log] > [buffer-controller]: checkPendingTracks (pending: 0 codec events expected: 1) {}
 [log] > [level-controller]: live playlist 0 REFRESHED 401-1
 [log] > [level-controller]: Loading level index 0 age 34.6 http://s2.enc.sctvonline.vn/sctv2/video_hd/playlist.m3u8
 [log] > [stream-controller]: IDLE->WAITING_LEVEL
 [log] > [stream-controller]: Level 0 loaded [393,400][part-401-1], cc [0, 0] duration:33.28
 [log] > [buffer-controller]: Updating MediaSource duration to 102.400
 [log] > [stream-controller]: WAITING_LEVEL->IDLE
 [log] > [stream-controller]: LL-Part loading ON loading sn undefined->400
 [log] > [stream-controller]: Loading main sn: 400 part: 2 (8/10) of level 0 (part:[98.560-99.840]INDEPENDENT=YES) cc: 0 [393-400], target: 98.569
 [log] > [stream-controller]: IDLE->FRAG_LOADING
 [log] > [level-controller]: live playlist 0 MISSED
 [log] > [level-controller]: reload live playlist 0bps in 585 ms
 [log] > [stream-controller]: Level 0 loaded [393,400][part-401-1], cc [0, 0] duration:33.28
 [log] > [transmuxer-interface]: Starting new transmux session for main sn: 400 part: 2 level: 0 id: 1
        discontinuity: true
        trackSwitch: true
        contiguous: false
        accurateTimeOffset: false
        timeOffset: 98.56000000001222
        initSegmentChange: true
 [log] > [stream-controller]: FRAG_LOADING->PARSING
 [log] > [stream-controller]: Init video buffer, container:video/mp4, codecs[level/parsed]=[/hvc1.1.40000000.L120.90}]
 [log] > [buffer-controller]: BUFFER_CODECS: "video" (current SB count 0)
 [log] > [buffer-controller]: switching codec undefined to hvc1.1.40000000.L120.90
 [log] > [buffer-controller]: checkPendingTracks (pending: 1 codec events expected: 1) {"video":{"listeners":[],"codec":"hvc1.1.40000000.L120.90","container":"video/mp4","id":"main"}}
 [log] > [buffer-controller]: creating sourceBuffer(video/mp4;codecs=hvc1.1.40000000.L120.90) Queued {"listeners":[],"codec":"hvc1.1.40000000.L120.90","container":"video/mp4","id":"main"}
 [log] > [buffer-controller]: SourceBuffers created. Running queue: 
video: (SourceBuffer) change-type=video/mp4;codecs=hvc1.1.40000000.L120.90
audio: (none) 
audiovideo: (none) }
 [log] > [buffer-controller]: changing video sourceBuffer type to video/mp4;codecs=hvc1.1.40000000.L120.90
 [log] > [transmuxer.ts]: Flushed main sn: 400 part: 2 of level 0
 [log] > [stream-controller]: PARSING->PARSED
 [log] > [stream-controller]: Parsed main sn: 400 part: 2 of level 0 (part:[98.560-99.840]INDEPENDENT=YES)
 [log] > [stream-controller]: Buffered main sn: 400 part: 2 of level 0 (part:[98.560-99.840]INDEPENDENT=YES > buffer:[98.560-99.840])
 [log] > [stream-controller]: PARSED->IDLE
 [log] > [stream-controller]: seek to target start position 59.76509 from current time 0 buffer start 98.56
 [log] > [stream-controller]: Live playlist, switching playlist, load frag with same PDT: 1737886768102
 [log] > [stream-controller]: Live playlist, switching playlist, load frag with same CC: 396
 [log] > [stream-controller]: Setting startPosition to 98.73810009766848 to match initial live edge. mainStart: 59.76509 liveSyncPosition: 98.73810009766848 frag.start: 80.64000000001222
 [log] > [stream-controller]: LL-Part loading OFF loading sn 400->396
 [log] > [stream-controller]: Loading main sn: 396 of level 0 (frag:[80.640-84.480]) cc: 0 [393-400], target: 80.64
 [log] > [stream-controller]: IDLE->FRAG_LOADING
 [log] > [stream-controller]: media seeking to 59.765, state: FRAG_LOADING
 [log] > [audio-stream-controller]: media seeking to 59.765, state: STOPPED
 [log] > [subtitle-stream-controller]: media seeking to 59.765, state: IDLE
 [log] > [transmuxer-interface]: Starting new transmux session for main sn: 396 level: 0 id: 1
        discontinuity: false
        trackSwitch: false
        contiguous: false
        accurateTimeOffset: false
        timeOffset: 80.64000000001222
        initSegmentChange: false
 [log] > [stream-controller]: FRAG_LOADING->PARSING

Chrome media internals output

[robwalch]

Latency should fallback to standard playback when the playlist server control does not meet the requirements of the HLS spec Appendix B.1 Low-Latency Server Configuration Profile. I left a comment in PR #6975:

#6975 (comment)

[robwalch]

If you like we can add logging to make this behavior clearer while verifying that playback is indeed starting at standard latency because of missing or disabled server-control blocking-reload attribute.

[truong-hua]

Thank @robwalch, but this issue is just a question and is not related to my pull request. I will check if this behaviour is still happened after the blocking reload is activated. But let's assume that the manifest is correctly implemented blocking reload, is it true that the buffer corruption should not fallback to regular-latency but instead trying to recover the stream?

[robwalch]

But let's assume that the manifest is correctly implemented blocking reload, is it true that the buffer corruption should not fallback to regular-latency but instead trying to recover the stream?

Buffer corruption is usually fatal and recovery is non-trivial. The client shouldn't be expected to play assets with corrupted parts at all. If it manages to recover, you should expect playback to be delayed by the amount of time it took to recover.