[Feature Request] Add support for video streaming over Websockets

[p3javier]

The current implementation only support WebRTC.
While this is a very good protocol sometimes has the inconvenience of being blocked by some firewalls or corporate proxy configurations.
Having the possibility of using Websockets for video transmission as fallback would increase the scenarios where Neko can be used as Websockets is a battle tested protocol and works over HTTP which is rarely blocked by any proxy or firewall.

https://github.com/pikvm/ustreamer/blob/1f32e875c3c4b5b7f245d5753f5e69cbf7198778/docs/h264.md

janus seems to support websocket with webcodec

[p3javier]

@danya0123 thanks for the recommendation but there are several points I don't get:

1. Janus is another completely different WebRTC server although as I saw on the Pion's WebRTC project on github there is some kind of interoperability between them but I don't understand what because there is little reference on the Pion WebRTC project
2. There is this project on Github that seems is the one with WebSockets support but again, I don't know how it relates to Pion WebRTC https://github.com/meetecho/janus-gateway

I would be grateful if you can provide me more insights into this. I'm completely new to these libraries (well Janus is not a library but a server on itself)

I think you don't really need any of these.

https://github.com/pikvm/kvmd/blob/87f78990a5d75ea04129a07adce6d25cc28d95dd/web/share/js/kvm/stream_media.js#L165

You simply send h264 via websocket and use the new webcodec to decode it. That's all.

[p3javier]

Sorry @danya0123 I'm new to this project. How do you suggest I can route the video stream on neko over WebSockets instead of WebRTC?
The client part I think I can figure out on my own but the server side changes is what I don't get.

I also don't know how this can be done in golang.

@m1k1o do you have any idea?

[Technetium1]

Seems relevant to mention #457 and #371

[m1k1o]

For example gortc implements MSE stream (fMP4 over WebSocket) nicely. That is something that would be relevant to this usecase. Maybe we could use isofmp4mux for it on server side?

In neko v3 there is already a full fallback mechanism available for controlling over websockets, but for video its only JPEG images loaded maybe couple of times in a second. And no audio, so very limited, just to stay up-to-date what is happening on the screen if you loose WebRTC connection.

bi-directional audio may also be supported by websocket with webcodec too

[p3javier]

For example gortc implements MSE stream (fMP4 over WebSocket) nicely. That is something that would be relevant to this usecase. Maybe we could use isofmp4mux for it on server side?

In neko v3 there is already a full fallback mechanism available for controlling over websockets, but for video its only JPEG images loaded maybe couple of times in a second. And no audio, so very limited, just to stay up-to-date what is happening on the screen if you loose WebRTC connection.

Great, I think the gortc is the way to go.
@m1k1o why do you mention isofmp4mux ? I think we could just copy the way gortc does this already.

[m1k1o]

why do you mention isofmp4mux

Because we use gstreamer to do screen capture, and we need to feed somehow the data to gortc. So we need to find the best compatible format, and since the payload itself is fMP4, we could get directly that from gstreamer, send that over websockets and just use some JavaScript client to interpret the data as a video.

does webcodec use fMP4?@m1k1o

[p3javier]

@m1k1o By the way setting the development environment on bare metal mac is impossible, at least for me. The furthest I've gotten is a black screen and basically block the rest of the system.
I have an older Ubuntu laptop here and I will try to implement the solution there.
Finally, the content on master is the new v3.0 or the old v2.9? I think master is 3.0 but just to confirm that I'm working on the right version.

[m1k1o]

@p3javier Do you have M3? I was considering getting one to accomodate the development environment for it. But now i cannot help you as I have not access to any such device.

content on master is the new v3.0 or the old v2.9?

Its v3, you are wothing on the right version.

[p3javier]

@m1k1o No, it is M2 MacBook Pro. The setup was very difficult many libraries missing and I've even needed to modify the code of the server. But honestly I won't bother to try to make the development environment compatible with mac. If someone doesn't have a Linux machine around there always the alternative of using a virtual machine right?