stm32: addition of the STM32 JPEG Codec (NV12 -> JPEG encoding) #96678
Conversation
zephyrbot
added
area: Video
zephyrbot
requested review from
FRASTM,
GeorgeCGV,
djiatsaf-st,
erwango,
etienne-lms,
gautierg-st,
josuah,
loicpoulain,
marwaiehm-st,
mathieuchopstm and
ngphibang
avolmat-st
force-pushed
the
pr-stm32-jpeg
branch
from
63ec263 to
76d0e06
Compare
avolmat-st
force-pushed
the
pr-stm32-jpeg
branch
from
76d0e06 to
5b55717
Compare
drivers/video/video_stm32_jpeg.c
Outdated
| /* Check that input / output formats are correct */ | ||
| if ((data->m2m.in.fmt.pixelformat == VIDEO_PIX_FMT_JPEG && | ||
| data->m2m.out.fmt.pixelformat == VIDEO_PIX_FMT_JPEG) || | ||
| (data->m2m.in.fmt.pixelformat != VIDEO_PIX_FMT_JPEG && | ||
| data->m2m.out.fmt.pixelformat != VIDEO_PIX_FMT_JPEG)) { | ||
| LOG_ERR("One of input or output format must be JPEG"); | ||
| ret = -EINVAL; | ||
| goto out; | ||
| } | ||
|
|
||
| /* FIXME - temporary until the decoder support get added */ | ||
| if (data->m2m.in.fmt.pixelformat == VIDEO_PIX_FMT_JPEG) { | ||
| LOG_ERR("Decoder not yet implemented"); | ||
| ret = -EIO; | ||
| goto out; | ||
| } |
There was a problem hiding this comment.
Nit: These two checks operate on global data rather than common so maybe they can be done somewhere else.
There was a problem hiding this comment.
Usually we do those checks at the set_stream time right before starting the device. I could have also block (the decoder) at set_fmt time, but in such case I have need to distinguish input / output in the supported format table so I found it less intrusive to put this FIXME (and the check) here, considering that this JPEG check is going to disappear when the decoder will be implemented
avolmat-st
force-pushed
the
pr-stm32-jpeg
branch
2 times, most recently
from
0946ea7 to
078a3a0
Compare
avolmat-st
force-pushed
the
pr-stm32-jpeg
branch
from
078a3a0 to
3097dac
Compare
|
Rebased on HEAD to fix conflict in stm32n6.dtsi and be able to use latest video apis. |
avolmat-st
force-pushed
the
pr-stm32-jpeg
branch
from
07c1e55 to
9bab893
Compare
avolmat-st
force-pushed
the
pr-stm32-jpeg
branch
from
9bab893 to
6285e4e
Compare
avolmat-st
force-pushed
the
pr-stm32-jpeg
branch
from
6285e4e to
e0db8c2
Compare
| .width_min = 16, | ||
| .width_max = 65520, | ||
| .height_min = 16, | ||
| .height_max = 65520, | ||
| .width_step = 16, | ||
| .height_step = 16, |
There was a problem hiding this comment.
May I know why width / height (min, max, step) of input and output are not the same ?
There was a problem hiding this comment.
That's an error. Since we work in 4 8x8 (in square shape), we need multiple of 16x16 images.
| data->current_in = k_fifo_get(in_fifo_in, K_NO_WAIT); | ||
| data->current_out = k_fifo_get(out_fifo_in, K_NO_WAIT); | ||
|
|
||
| if (data->m2m.in.fmt.pixelformat != VIDEO_PIX_FMT_JPEG) { |
There was a problem hiding this comment.
Why we check only input format ? I think the check should be already done in set_format().
There was a problem hiding this comment.
Indeed there is already a check before that so we are already sure that format in / out are already correct. This one is to detect if it is decoder or encoder.
Btw, I changed the set_format function as well to make it automatically set the output format, check values etc while, for the time being, forcing input = NV12 and output = JPEG (which is the only part done for the moment). Once more direction / format will be supported, one if statement within the set_format can be dropped.
| int ret = 0; | ||
|
|
||
| /* Validate the settings */ | ||
| conf = stm32_jpeg_get_conf(fmt->pixelformat); |
There was a problem hiding this comment.
Should we valid the set format here for input and output ?
drivers/video/video_stm32_jpeg.c
Outdated
| if (!((in_fmt == VIDEO_PIX_FMT_JPEG && out_fmt != VIDEO_PIX_FMT_JPEG) || | ||
| (in_fmt != VIDEO_PIX_FMT_JPEG && out_fmt == VIDEO_PIX_FMT_JPEG))) { | ||
| LOG_ERR("Exactly one of input/output format must be JPEG"); |
There was a problem hiding this comment.
Should this be already checked in set_format() ?
| return ret; | ||
| } | ||
|
|
||
| static int stm32_jpeg_set_stream(const struct device *dev, bool enable, enum video_buf_type type) |
There was a problem hiding this comment.
It seems set_stream() does nothing (related to the HW). Should we merge stm32_jpeg_start_codec() into this ?
There was a problem hiding this comment.
No, actually start_codec, (maybe badly named) is processing frames, so it is called from enqueue (as well), however, indeed, within the set_stream, we might also call start_codec is there are already buffers queued into both input & output. Adding this.
drivers/video/video_stm32_jpeg.c
Outdated
| enum stm32_jpeg_state { | ||
| STM32_JPEG_STOPPED = 0, | ||
| STM32_JPEG_WAIT_FOR_BUFFER, | ||
| STM32_JPEG_RUNNING, | ||
| }; |
There was a problem hiding this comment.
It seems STM32_JPEG_STOPPED is not used ? Could we simplify the state by using only a bool is_running ?
| { | ||
| if (caps->type == VIDEO_BUF_TYPE_OUTPUT) { | ||
| caps->format_caps = stm32_jpeg_out_fmts; | ||
| } else { |
There was a problem hiding this comment.
If interested in supporting other VIDEO_BUF_TYPE_* in the future, this could be an else if.
| { | ||
| struct stm32_jpeg_data *data = | ||
| CONTAINER_OF(hjpeg, struct stm32_jpeg_data, hjpeg); | ||
| ARG_UNUSED(nb_encoded_data); |
There was a problem hiding this comment.
Need an empty line above.
Ditto in HAL_JPEG_DataReadyCallback().
| ret = stm32_jpeg_start_codec(data->dev); | ||
| if (ret) { | ||
| LOG_ERR("Failed to start codec, err: %d", ret); | ||
| goto out; |
There was a problem hiding this comment.
Is the goto really needed.? You expect later additions?
Ditto in stm32_jpeg_enqueue().
There was a problem hiding this comment.
Currently, no, I don't expect. However I find it less error prone to put a goto here, aka keep the usual error handling pattern so that if somehow a new statement is put right after, it still won't be executed because there were no goto put in this one.
|
|
||
| /* JPEG Encoding */ | ||
| jpeg_conf.ColorSpace = JPEG_YCBCR_COLORSPACE; | ||
| jpeg_conf.ChromaSubsampling = conf->subsampling; |
There was a problem hiding this comment.
May conf be NULL at this point? Maybe it has already been validated by stm32_jpeg_set_fmt(p).
Would __ASSERT_NO_MSG(conf != NULL); make sense here?
There was a problem hiding this comment.
The set would prevent that. The only reason I can think of is if the default value of the driver get changed during a dev / fix and this is not supported, leading to getting a conf == NULL if the appli decide not to do a set_format (which it could). Hence ok, I add the __ASSERT_NO_MSG for that case.
In case of a M2M device, it is expected that input is set prior to the output since configuration of the input might affect the output format such as resolution or pixelformat. Upon setting the input, the application can tune the output format such as pixelformat (if several are applicable) or even resolution in case of M2M device embedding scalers. Signed-off-by: Alain Volmat <[email protected]>
Add description of the ST JPEG codec IP node. Signed-off-by: Alain Volmat <[email protected]>
Initial version of the support for the STM32 JPEG codec, currently supporting only NV12 to JPEG without DMA support and using SW based conversion from NV12 to MCU required for the JPEG codec. Signed-off-by: Alain Volmat <[email protected]>
Add the node describing the JPEG codec within the stm32n6 Signed-off-by: Alain Volmat <[email protected]>
Enable the JPEG codec on the stm32n6570_dk board. Signed-off-by: Alain Volmat <[email protected]>
Add conf / overlay files in order to enable the jpeg encoder on the stm32n6570_dk. Signed-off-by: Alain Volmat <[email protected]>
avolmat-st
force-pushed
the
pr-stm32-jpeg
branch
from
e0db8c2 to
d04dd19
Compare
|
I also added a new commit at the beginning to change the input/output setting order in the encoder. Indeed, I feel it make sense to ensure that the input is set PRIOR to setting the output since the driver might adapt the output settings after having received the input settings. Does that make sense ? (VENC still work with this modification, and this also already the way I coded it into the UVC encoder sample) |
|
| The STM32 JPEG codec can decode JPEG compressed frames | ||
| and encode uncompressed frames to the JPEG format. |
There was a problem hiding this comment.
As I'm not familiar with video conventions: Is it worth specifying that this is an HW accelerator ie that we're not talking about a SW Codec ?
| select USE_STM32_HAL_DMA | ||
| select USE_STM32_HAL_DMA_EX |
7 participants
The PR adds an initial version of the STM32 JPEG Codec, available on several STM32. This PR adds it within the N6.
While the HW can perform both encoding & decoding, for the time being only encoding is provided.
The HW needs data in MCU format in order to encore into JPEG. Formating of the data into MCU is provided only for the NV12 for the time being (for that purpose probably only NV12 should be kept in this driver in order to avoid confusion).
DMA can also be used in order to inject data into the codec but this is not implemented in this version.
Main purpose of this PR is actually to provide material in order to discuss about a possible abstraction for the m2m buffers.
The driver already provide several structures which embed k_fifo but it also make it clear that m2m helper functions could be possible in order to handle in a generic way checking about in & out buffer, and have the helper call a new m2m codec api which would be in charge to perform the conversion only.
As shown in this code, the queue / dequeue function do not have anything HW specific and also take care of pushing / pulling into / from the right fifo and triggering a codec run if all condition are matched.
I hope this could generate discussion about this and based on that I can provide some helpers to hide the non HW specific code (buffer handling).
This has been tested using the tcpclientsink application modified within the PR #95862.
I however had to perform some more changes into the application, especially in order to properly set the in / out formats of the encoded video device. I provide those modifications for reference within this commit (avolmat-st@86249bf) part of my git repo.
This has been tested successfully on the STM32N6 connected over network with laptop into which following command is ran in order to grab JPEG data: