video: add video compression support to tcpserversink sample

[hfruchet-st]

This PR adds support of camera streaming with compression of camera frames in video format such as H264 instead of sending big raw uncompressed frames, reducing drastically the network bandwidth needed for such use-case.

This video compression mode is enabled when a video encoder is found in device-tree, such as the VENC video encoder driver of the STM32N6 platform [1].

On :zephyr:board:stm32n6570_dk, the MB1854 IMX335 camera module must be plugged in the CSI-2 camera connector. A RJ45 ethernet cable must be plugged in the ethernet CN6 connector.
For an optimal image experience, it is adviced to embed the STM32 image signal processing middleware [2].

To build sample: (flash using ST-Link and boot with FSBL [3])

zephyr$> west build -p -S video-stm32-venc -b stm32n6570_dk//fsbl --shield st_b_cams_imx_mb1854 samples/drivers/video/tcpserversink

The default configuration allows to capture and stream 1080p camera content from STM32N6 which can be received, decoded and displayed by an host PC using a GStreamer command line:

host$> gst-launch-1.0 tcpclientsrc host=192.0.2.1 port=5000 ! queue ! decodebin ! queue ! fpsdisplaysink sync=false

[1] #92884
[2] https://github.com/stm32-hotspot/zephyr-stm32-mw-isp
[3] https://docs.zephyrproject.org/latest/boards/st/stm32n6570_dk/doc/index.html

[josuah]

Good idea to split dependencies out of #92884 to help review.

Is the current PR expected to be reviewed now? I am asking just because I did not see '264 mentioned anywhere.
If not, there is a "convert to draft" link below the reviewers list.
If yes, this means these there are missing definitions for H.264 format?

You may be interested in this PR which introduces H.264 into the video-sw-generator, which permits to test H.264 support in the tcpserversink sample end-to-end before VENC is complete. This could help merging VENC faster to have all the compression support done out of the VENC PR.

• video: H.264 video format support #95407

Thank you for bringing this forward!

[ngphibang]

Thank you for this interesting support for the 1st Zephyr H.264 encoding sample !
The important thing to me is the estimated encoded buffer size which should be done in a more robust way, I think.

[hfruchet-st]

All comments addressed so far...

[avolmat-st]

In order to avoid having this app always use the VENC in case of usage on the N6, I tried to have snippets in order to select the video encoder.

As an example I pushed a commit here avolmat-st@993aea3 as well a commit which removes unnecessary bits from the board specific conf in tcpserversink sample:
avolmat-st@6a76fac

I also made a similar snippet for the jpegenc which I've pushed in the stm32-jpeg PR (96678) in this commit 078a3a0

[mbolivar]

No objections from my side on the new chosen

[ngphibang]

In order to avoid having this app always use the VENC in case of usage on the N6, I tried to have snippets in order to select the video encoder.

As an example I pushed a commit here avolmat-st@993aea3 as well a commit which removes unnecessary bits from the board specific conf in tcpserversink sample:
avolmat-st@6a76fac

Sounds good. The snippet also helps to reduce the code duplication (#95862 (comment)). Could we put other configs into the snippet's .conf file or they are board-specific and cannot be generalized @avolmat-st @hfruchet-st ?

# Video encoder
CONFIG_MAIN_STACK_SIZE=4096

# Network buffers
CONFIG_NET_BUF_RX_COUNT=4
CONFIG_NET_BUF_TX_COUNT=8

[avolmat-st]

In order to avoid having this app always use the VENC in case of usage on the N6, I tried to have snippets in order to select the video encoder.
As an example I pushed a commit here avolmat-st@993aea3 as well a commit which removes unnecessary bits from the board specific conf in tcpserversink sample:
avolmat-st@6a76fac

Sounds good. The snippet also helps to reduce the code duplication (#95862 (comment)). Could we put other configs into the snippet's .conf file or they are board-specific and cannot be generalized @avolmat-st @hfruchet-st ?

# Video encoder
CONFIG_MAIN_STACK_SIZE=4096

# Network buffers
CONFIG_NET_BUF_RX_COUNT=4
CONFIG_NET_BUF_TX_COUNT=8

I think the snippet should be normally as much as possible application unrelated, so for example I don't think the NET_BUF part should be in there since it seems really related to the tcpserversink. Is that right @hfruchet-st ? For the MAIN_STACK_SIZE, don't know, I feel as well that this is also quite application related part.

I actually have an issue with the snippet which is that this set the ENCODED_FRAME_FORMAT config in it, which is a config only defined in the tcpserversink sample app and this leads to CI warnings.

[hfruchet-st]

New version pushed with sizeimage => size for video_format, new snippet to allow selection of venc H264 video encoder but also futur JPEG encoder, cleanup in configs due to use of camera shield and snippet.

[hfruchet-st]

Rebased

[josuah]

Thanks for one more round of editing. Here is one more round of review with minor details proposed.

[avolmat-st]

Looks good to me. Thanks a lot !! Tested ok also with the JPEG encoder PR on the N6.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
19.0% Duplication on New Code

See analysis details on SonarQube Cloud

[josuah]

I just noticed this already landed on this PR, but #97197 and/or #92884 are dependencies that introduce .size in struct video_format.

Sorry about this inconsistency.

Nothing is broken as as @avolmat-st pointed me, it is not enabled by any board, so CI is unaffected.

Any introduction of zephyr,videoenc chosen node will both reveal that bug and fix it as size is required for video encoder support.

Next time I will need to add a DNM flag after +1

[josuah]

P.S. This was not something the release team could have caught as very specific to the discussions on other PRs.