Low Latency playback - recovering after stalls

[vk342]

What do you want to do with Hls.js?

I am trying to support ultra low-latency HLS stream using 6 second segments and 0.5 second parts with liveSyncDuration of 1.4 seconds.

Hls.js version is 1.5.7.

My config looks as following:

{
    "debug": false,
    "enableWorker": true,
    "lowLatencyMode": true,
    "backBufferLength": 90,
    "liveSyncDuration": 1.4,
    "liveMaxLatencyDuration": 5,
    "maxLiveSyncPlaybackRate": 1.2,
    "liveDurationInfinity": true,
    "startFragPrefetch": true
}

Periodically the player encounters a stall, and after each stall LatencyController target latency is increased by 1 second:
this.stallCount * liveSyncOnStallIncrease, see targetLatency below:

hls.js/src/controller/latency-controller.ts

Lines 42 to 71 in 0a49731

	get targetLatency(): number | null {
	const { levelDetails } = this;
	if (levelDetails === null) {
	return null;
	}
	const { holdBack, partHoldBack, targetduration } = levelDetails;
	const { liveSyncDuration, liveSyncDurationCount, lowLatencyMode } =
	this.config;
	const userConfig = this.hls.userConfig;
	let targetLatency = lowLatencyMode ? partHoldBack || holdBack : holdBack;
	if (
	userConfig.liveSyncDuration ||
	userConfig.liveSyncDurationCount ||
	targetLatency === 0
	) {
	targetLatency =
	liveSyncDuration !== undefined
	? liveSyncDuration
	: liveSyncDurationCount * targetduration;
	}
	const maxLiveSyncOnStallIncrease = targetduration;
	const liveSyncOnStallIncrease = 1.0;
	return (
	targetLatency +
	Math.min(
	this.stallCount * liveSyncOnStallIncrease,
	maxLiveSyncOnStallIncrease,
	)
	);
	}

So if the stream is played for an hour and encountered 3 stalls our target latency drifts from 1.4 to 4.4.

I might be missing something, but I don't seem to see a mechanism to recover from this condition.

I do not see the API to programmatically change targetLatency after player has been created, so the only way to return to lower latency is to reset the player.

I am looking for a more graceful way to return to lower targetLatency after a period of successful playback.

What have you tried so far?

I have tried to programmatically set LatencyController.stallCount to a lower value after X seconds of playback without a stall.

hls.js/src/controller/latency-controller.ts

Line 19 in 0a49731

private stallCount: number = 0;

This did reduce player.targetLatency, but did not affect the actual playback latency, despite the fact that maxLiveSyncPlaybackRate is set to 1.2 and I was hoping that the player would try to catch up to targetLatency.

I have also tried to change liveSyncOnStallIncrease from 1.0 to 0.1 second.

hls.js/src/controller/latency-controller.ts

Line 63 in 0a49731

const liveSyncOnStallIncrease = 1.0;

This did cause targetLatency to drift slower, however I still observed that the actual playback latency remained higher then the target latency and the player did not attempt to catch up.

Note: whenever the drift happened, I was able to start another playback in the same stream in a different window using the same version of hls.js and the second player would start and play the stream at my initial target latency of 1.4. For example, my first player would have targetLatency = 1.4, but actual latency of 3.4 sec, while my second player would have both target and actual latency of 1.4 sec.

This indicates that the content has been available on the server.

I would be very grateful for the guidance on how to improve the playback latency in this scenario.

[robwalch]

While stalling might be inevitable, support for preload hints will help reduce stalls. Ask @iamboorrito to rebase and open a PR for https://github.com/video-dev/hls.js/tree/feature/preload-hint (resolves #3988).

• Implement support for LL-HLS #EXT-X-PRELOAD-HINT part loading #3988

[robwalch]

Not a Contribution

You can find Issues and PRs for Low-Latency on this project board https://github.com/orgs/video-dev/projects/3. You might consider commenting on #4681 as these issues are somewhat related:

• maxLiveSyncPlaybackRate does not seem to enable "catch up" #4681

[robwalch]

My config looks as following:

I would consider (and test) disabling/dropping "enableWorker": true and "startFragPrefetch": true. There is likely no efficiency gained by using a worker with such small parts (you are adding async messaging for short processes that in this case probably slow down buffering of loaded parts). startFragPrefetch is really intended for VOD. Maybe we can make sure it only applies to init segments for live - that would be a good enhancement request to file if you notice a difference. The problem with using it for live is that you might preload a segment that is never used.

Periodically the player encounters a stall, and after each stall LatencyController target latency is increased by 1 second:
this.stallCount * liveSyncOnStallIncrease, see targetLatency below:
...
So if the stream is played for an hour and encountered 3 stalls our target latency drifts from 1.4 to 4.4.
I might be missing something, but I don't seem to see a mechanism to recover from this condition.

Correct.

I am looking for a more graceful way to return to lower targetLatency after a period of successful playback.

There is no mechanism to reset or reduce the stallCount and its impact on "target latency" over time. Would you consider contributing one?

I have tried to programmatically set LatencyController.stallCount to a lower value after X seconds of playback without a stall.

You are on the right track. Being able to recover or reset target latency would be the first step.

I have also tried to change liveSyncOnStallIncrease from 1.0 to 0.1 second.

liveSyncOnStallIncrease could be a configuration option. We'd accept such a contribution if you would be generous enough to make one.

[robwalch]

Not a Contribution

This did reduce player.targetLatency, but did not affect the actual playback latency

In general HLS.js tries to avoid any direct control of playback - it never calls play() or pause() and only sets currentTime when necessary to start or recover playback. The point is that control of playback (and latency) can be performed using the media element provided to HLS.js. If you think HLS.js should handle this, please submit a contribution that improves or adds more configuration options to the current behavior.

[vk342]

Thank you so much @robwalch, for the thorough and thoughtful reply, and I apology for such a long delay in reacting.

I have tried the following:

• tried Implement support for LL-HLS #EXT-X-PRELOAD-HINT part loading #3988. The PR code worked well with my LL stream (see Implement support for LL-HLS #EXT-X-PRELOAD-HINT part loading #3988 (comment)), but I still had the same issue with the player recovering after stalls.

• tried disabling/dropping "enableWorker": true and "startFragPrefetch": true. - This did not improve the stalling behaviour, but did not have any negative effects either.

• tried to change liveSyncOnStallIncrease from 1.0 to 0.1 second. This did not change the stall behaviour, but, as @robwalch suggested, I opened a PR to make liveSyncOnStallIncrease configurable: Add liveSyncOnStallIncrease config option #6455

• tried to programmatically set LatencyController.stallCount. This did not change the stall behaviour, but, as @robwalch suggested, I plan to open a PR to make targetLatency settable.

As a next step, I will try to see if some suggestions for #4681 could be implemented (by a newbie) and improve LL stall behaviour.