Add support for H265 encoded streams

[Budovi]

Originally based on #117 , I went through the hassle and tuned the parameters to make it work. I've tested the software and NVidia encoder paths. The update to 3.1.0 made it even easier, with updated dependencies.

Note that H265 decoding is poorly supported in Linux browsers, but works fine on Android / Chrome or Mac. Part of the journey was trying to make it work in my desktop browser, but I've failed with my setup. That's also the reason I won't be using this myself at the moment, but maybe in near future? It is possible that with AMD GPU and Chrome it can be made functional. Anyway, I thought since it was already done, someone may appreciate this.

While trying to update the legacy pipelines with the H265 variants, I've discovered that the used VAAPI plugin was merged into GStreamer (plugins-bad) and is deprecated. I've tried to make it so that the result uses the new plugin and is mostly consistent with the rest of the H26x pipelines, but since I own an NVidia GPU, I can't try the VAAPI encoders (only VAAPI decoding driver is available for NVidia GPUs).

Possible resolutions:

1. Merge as suggested, ideally after someone with a different GPU confirms it is working (current branch)
2. Drop the changes to the H264 pipeline and leave the H265 as is until someone provides fixes
3. Drop the changes to the H264 pipeline and use the vaapih265enc for H265 instead (still untested from my side)
4. Drop the changes to the H264 pipeline and drop the support for VAAPI backed H265

[Budovi]

Thanks to elFarto/nvidia-vaapi-driver#427 I've managed to test the VAAPI for both new H.264 and H.265 legacy pipelines on my NVidia GPU.

Fixes pushed, from my PoV this is ready to be merged.

[m1k1o]

@jameskitt616 would you have time to test out this new h264 configuration if that works with your compose on your hardware?

This should be enough, no need to even create a new neko version as the pipeline is configurable:

- vaapih264enc rate-control=vbr bitrate=6144 keyframe-period=180 quality-level=7
+ vah264enc rate-control=cbr bitrate=6144 key-int-max=60 target-usage=7

[jameskitt616]

      NEKO_CAPTURE_VIDEO_PIPELINE: |
        ximagesrc display-name={display} show-pointer=false use-damage=false !
        videoconvert !
        videoscale !
        video/x-raw,width=1920,height=1080,framerate=30/1 !
        vah264enc rate-control=cbr bitrate=6144 key-int-max=60 target-usage=7 !
        h264parse !
        queue !

      NEKO_CAPTURE_BROADCAST_PIPELINE: >
        ximagesrc display-name={display} show-pointer=true use-damage=false !
        queue max-size-buffers=2 leaky=downstream !
        videoconvert !
        videorate !
        videoscale !
        video/x-raw,width=1920,height=1080,framerate=30/1 !
        vah264enc rate-control=cbr bitrate=6144 key-int-max=60 target-usage=7 !
        h264parse !
        queue !
        mpegtsmux name=mux alignment=7 !
        udpsink host=rtsp_server port=1234 sync=false buffer-size=524288

        pulsesrc device=audio_output.monitor !
        queue !
        audioconvert !
        audioresample !
        audio/x-raw,channels=2 !
        voaacenc bitrate=320000 !
        aacparse !
        queue !
        mux.

Any idea why it works for the normal pipeline but it does not for the broadcast pipeline?
I get this error
neko-v3 | 7:41PM ERR request failed (500): Internal Server Error error="(pipeline error) no element \"vah264enc\"" module=http req={"agent":"Go-http-client/1.1","id":"63edd608da2d/0swfCSBWBH-000008","method":"POST","proto":"HTTP/1.1","remote":"127.0.0.1:51234","scheme":"http","uri":"http://:8080/api/room/broadcast/start"} res={"bytes":47,"elapsed":2.321587,"status":500,"time":"Tue, 26 May 2026 17:41:45 UTC"} session_id=james-jV1eO

EDIT:
Ah i think it's falling back to vaapih264enc for my main pipeline for some reason. would need to investigate but no time today anymore.

[Budovi]

@jameskitt616 the error message you're seeing may mean a lot of things, because from my testing, the plugins "appear" as non-existent when they fail to initialize.

Things that come to my mind:

• Based on the n.eko version you use, you may not have the recent-enough gstreamer (1.20+). But I'm quite sure I've checked the trixie image (I'm running nvidia ubuntu based one myself, since I need the CUDA libs, even to test out VAAPI via nvidia-vaapi-driver). That wouldn't explain why the main pipeline "alias" to the old implementation, but the broadcast one doesn't.
• I haven't checked the handling for the broadcast pipeline, but it is definitely weird if it'd work for the capture one, but not for the broadcast. I can imagine it fail to initialize a plugin the second time due to some bug in VA, or mix-matching the plugins...
• I suppose resource issue (exhausted encoding pipelines) would exert differently (vaCreateContext: resource allocation failed).

Which brings me to a bit of an unrelated topic, but even for the broadcast setup, since your encoding pipeline effectively doesn't differ, I think it is a waste of resources to encode twice. It would be nice to just use "tee" and some "shmem handoff" plugin to the other pipeline. But I suppose that could bring other issues (i.e. with neko's pipeline management). Maybe idea for the future? Maybe already captured somewhere?

[jameskitt616]

Yeah about the double encoding issue, i already had a DM talk with @m1k1o that they also think this would make more sense than wasting the resources twice.

Regarding my neko setup: i use the ghcr.io/m1k1o/neko/intel-firefox image (with an intel integrated iGPU or dedicated Intel ARC GPU), but i am currently not sure if i used the latest version or v3.0.11