Optimizing delivery of LL-HLS with NGINX

[mauricioabreu]

My latest question here was about how to use NGINX along with OvenMediaEngine.

Today, I’m sharing my configuration here. It might help others bootstrap their own configurations.

worker_processes  auto;

pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    proxy_cache_path /opt/nginx_cache levels=1:2 keys_zone=cache_zone_one:10m;
    log_format main 'time="$time_iso8601" remote_addr="$remote_addr" req="$request" status="$status" '
                  'req_time="$request_time" bytes_sent="$body_bytes_sent" up_cache_status="$upstream_cache_status" '
                  'up_status="$upstream_status" up_addr"$upstream_addr" up_resp_time="$upstream_response_time" '
                  'ref="$http_referer" ua="$http_user_agent" scheme="$scheme"';
    access_log /dev/stdout main;

    upstream origin {
        server origin:3333;
        keepalive 64;
    }

    server {
        listen       80;
        server_name  localhost;
        proxy_ignore_headers "vary";
        proxy_hide_header "access-control-allow-origin";
        proxy_hide_header "access-control-allow-credentials";
        proxy_hide_header "access-control-allow-methods";
        proxy_hide_header "access-control-allow-headers";

        location ~ ^/.*.m3u8 {
            proxy_cache_key $request_uri;
            proxy_set_header Connection "";
            proxy_http_version 1.1;
            proxy_cache cache_zone_one;
            proxy_cache_lock on;
            add_header X-Cache-Status $upstream_cache_status;

            proxy_pass http://origin;
        }

        location ~ ^/.*.m4s {
            proxy_cache_key $request_uri;
            proxy_set_header Connection "";
            proxy_http_version 1.1;
            proxy_cache cache_zone_one;
            proxy_cache_lock on;
            add_header X-Cache-Status $upstream_cache_status;

            proxy_pass http://origin;
        }
    }
}

This configuration should have http2, but enabling locally is not straightforward - use mkcert

For those who are already familiar with LL-HLS and NGINX, what other settings or approaches would you recommend?

[naanlizard]

Are you certain you want to have your upstream connect to port 3333? That's typically the WebRTC port, not LLHLS

Can you explain your thinking with some comments inline?

I'd also say your logging seems excessive, but perhaps that's what you want for debugging

[maruf-cs]

I noticed that the session (Yellow) appears to be unique for each user. By using "proxy_cache_key $request_uri;" the cache hit ratio decreases significantly. I'm trying to understand more about this session behavior—what exactly is it, and how is it managed for each user? If I skip the session in the proxy server, the stream doesn’t play smoothly. Could anyone provide some guidance on how to handle this efficiently?

[naanlizard]

I'm also curious about how the session key works.

I did find this guide to ignoring the session number but it sounds like you already tried that, how did you implement it?

How are you tracking your cache hit rate?

[getroot]

When there is an edge behind the origin, set LLHLS OriginMode to true. The session_key will be deleted, so it can be cached on the edge.

OvenMediaEngine/misc/conf_examples/Origin.xml

Line 359 in cdc0b62

<OriginMode>true</OriginMode>

<LLHLS>
	<OriginMode>true</OriginMode>

[maruf-cs]

I'm also curious about how the session key works.

I did find this guide to ignoring the session number but it sounds like you already tried that, how did you implement it?

How are you tracking your cache hit rate?

I tried manually parsing and ignoring the session parameter for the cache key, but it wasn’t the right approach. As for tracking the cache hit rate, I'm not actually tracking it directly at the moment. Instead, I'm monitoring the bandwidth IN and OUT ratio of the server as a rough indicator of how well the cache is performing.

[naanlizard]

Another topic here

Recently, we tried to deploy a change that would serve thumbs from OME, rather than our bespoke thumbnailing process. We use nginx as a reverse proxy, and we had ome set as an upstream like so:

upstream ome {
        server internalip:800;
        keepalive 32; # Note keepalive
    }

Some other relevant config:

proxy_set_header Connection $connection_upgrade; and proxy_http_version 1.1; in our proxy.conf (this is largely the default docker-swag config)

    location /live {

        include /config/nginx/proxy.conf;
        include /config/nginx/securityheaders.conf;

        proxy_cache ome_cache;
        proxy_cache_valid 200 302 1h;
        proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
        proxy_no_cache $arg_nocache;
        add_header X-Cache-Status $upstream_cache_status;
        proxy_cache_key $scheme$proxy_host$request_uri$http_origin;
        proxy_cache_lock on;
        proxy_cache_lock_timeout 2s;

        # Don't verify OME's ssl cert, it wouldn't pass as the hostname is incorrect
        proxy_ssl_verify off;

        proxy_pass https://ome/live;

Note, at no point do we add the following line, which I believe is a mistake, since the nginx docs say for keepalive connections you want to clear the Connection header

proxy_set_header Connection "";

The keepalive is intended to improve performance by reusing connections rather than making/closing connections every time

Before the attempted deploy, the only thing that uses this upstream was the playback endpoint, which worked great.

We deployed an update that added another location that proxy passed requests for images to OME, for the thumbnails

This eventually caused issues for stream playback and resulted in 404s for stream playlists (m3u8 files), which were a huge pain in the ass to track down. My guess is that either OME or nginx don't like requesting different content types over the same connection, or something along those lines, or perhaps a header issue (one is served from a location like hostname.tld/ome-screenshots, the other from a subdomain).

To fix this, we've made a separate upstream definition, added the line to clear the connection header to the playback location (but not the image upstream, which I suppose might cause issues, now that I'm thinking about it?) and more recently I've disabled keepalive entirely for the images upstream (since I assume the performance gains there are minimal)

Note that with 32 workers, and a keepalive of 32, that means we'll have up to 32ˆ2 connections to OME, which I'm not sure is ideal.

How have the rest of you sorted nginx proxying? I'm sure we can improve on this, but I'm not fantastic with nginx. Do we even care about keepalives between nginx and OME on the same physical server (and different docker containers?)

Related, how does OME determine a client is a different client or not? It would be nice to have accurate client counts, but filtered through nginx, I'm not sure we could

For reference, here's our current upstream set

    upstream ome-images {
	    server internalip:800;
    }

    upstream ome-playback {
	    server internalip:800;
	    keepalive 32;
    }

[naanlizard]

Following up with this, it seems to have continued even after this config change, and ultimately we had to disable the image endpoint we had created - after that, HLS playback returned to normal. I'll try keepalives again in a while

For the moment, I'm using a simple bash script to copy the thumbnails to disk and then serve them like usual with nginx