APIs for direct image item payload access

[programingKing]

Hello,

Our team runs an image processing system that handles transcoding and operations across multiple image formats. For HEIC, we use the libheif C API with NVIDIA hardware (NVDEC/NVENC).
libheif's codec plugin mechanism doesn't fit our system well — we're facing issues similar to #1793.
As a workaround, I wrote a passthrough plugin that just forwards the bytes to our external decoder. It works, but it feels hacky.
A more natural fit for our case would be direct APIs to:

1. Extract an image item's HEVC bitstream.
2. Build an image item from a pre-encoded HEVC bitstream.

I think the codec plugin might be an intentional abstraction. That said, the plugin contract makes integration difficult for us. Would you be open to adding a public API for accessing the codec-level data, something like below? I'd be happy to contribute if possible.

// 1. Get the coded bitstream (config NALs + slice NALs) of an image item,
//    either length-prefixed or in Annex B form.
heif_image_handle_get_compressed_data(...)

// 2. Create an image item from pre-encoded HEVC bytes
//    (VPS/SPS/PPS + slice NALs), building hvcC + iloc from the input.
heif_context_add_compressed_image(...)

// 3. Link an image item as the alpha auxiliary of another image item
//    (sets up auxC property + auxl iref).
heif_image_handle_assign_as_alpha_of(...)

[farindk]

Hello,

yes, an API to extract the compressed data or build images from pre-compressed data could be added. It was already discussed earlier, but postponed.

When designing such an API, it will be important that it covers all use cases, like tiled images. Maybe even image sequences. Concerning heif_image_handle_assign_as_alpha_of, I would prefer making it general enough that it also works for other auxiliary image references (e.g., depth).
It will also require work to extract parameters like CICP from the bitstream and fill the respective HEIF colr box with it, similar pixi, ispe, and others that I do not foresee yet.

I imagine that this will involve some major refactoring of the core. Your contributions are welcome, but be prepared that this will also need considerable time for review and also coding on my side that I need to be able to allocate for this.

[bradh]

While I agree that extraction is a complex task, generating the wrapping for an existing bitstream could well be handled with a small amount of user side code, so that use case seems less important to add to libheif.