Skip to content

Support for Progressive AVIF encoding #761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 18 commits into
base: main
Choose a base branch
from
Draft

Support for Progressive AVIF encoding #761

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
29189e4
Support for Progressive AVIF encoding
tongyuantongyu Sep 21, 2021
7e82a52
Accept arbitrary scale ratio in layer config
tongyuantongyu Sep 23, 2021
50a7ae9
Allow different input for different layers (experimental, mostly reus…
tongyuantongyu Oct 5, 2021
7857530
pull main branch
tongyuantongyu Jun 7, 2022
2016e2e
address review changes
tongyuantongyu Jun 7, 2022
b69f8c9
Merge remote-tracking branch 'upstream/main' into progressive_encoding
tongyuantongyu Jul 1, 2022
c9005bc
address review changes
tongyuantongyu Jul 1, 2022
017e736
Merge remote-tracking branch 'upstream/main' into progressive_encoding
tongyuantongyu Jul 8, 2022
c1383ee
address review changes
tongyuantongyu Jul 8, 2022
4a7690f
remove `layerCount` parameter
tongyuantongyu Jul 8, 2022
5bdb27c
Add a basic test for progressive encoding.
tongyuantongyu Jul 8, 2022
cf05850
Revert changes to avifenc
tongyuantongyu Jul 8, 2022
953aee4
apply clang-format
tongyuantongyu Jul 9, 2022
8eb3a8d
Merge remote-tracking branch 'upstream/main' into progressive_encoding
tongyuantongyu Jul 16, 2022
1f41e17
address review changes
tongyuantongyu Jul 16, 2022
b8219ad
not setting AVIF_ADD_IMAGE_FLAG_SINGLE for layer image
tongyuantongyu Jul 16, 2022
ae6a5b5
fix compile error
tongyuantongyu Jul 16, 2022
bc71300
fix MSVC compile error
tongyuantongyu Jul 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions include/avif/avif.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1043,8 +1043,9 @@ typedef struct avifEncoder
int keyframeInterval; // How many frames between automatic forced keyframes; 0 to disable (default).
uint64_t timescale; // timescale of the media (Hz)

int extraLayerCount; // Extra layers for color sub image; 0 to disable layer image (default).
int extraLayerCountAlpha; // Extra layers for alpha sub image; 0 to disable layer image (default).
// Layers (used by progressive rendering)
int extraLayerCount; // Extra color layers; 0 for regular single-layer color image (default).
int extraLayerCountAlpha; // Extra alpha layers; 0 for regular single-layer alpha image (default).
avifLayerConfig layers[MAX_AV1_LAYER_COUNT];
avifLayerConfig layersAlpha[MAX_AV1_LAYER_COUNT];

Expand Down
12 changes: 5 additions & 7 deletions src/codec_aom.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -786,8 +786,8 @@ static avifResult aomCodecEncodeImage(avifCodec * codec,

#if defined(AVIF_AOM_LAYER_CONFIG_PREFER_SVC_PARAMS)
Copy link
Contributor

@y-guyon y-guyon May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should hardcode this choice if it is a detail or expose it in CMakeLists.txt if it is important. I suggest the former for simplicity.

Copy link
Contributor Author

@tongyuantongyu tongyuantongyu Jun 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm experimenting here, and actually I haven't figure out how to make AV1E_SET_SVC_PARAMS method working.

Are you familiar, or can invite someone who is familiar with aom's API, to review my usage here?

Copy link
Contributor

@y-guyon y-guyon Jun 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you share more details? Is aom_codec_control(AV1E_SET_SVC_PARAMS) returning an error or is the encoding broken later on?

@jzern may have some insight on this or know who to ping.

Copy link
Contributor Author

@tongyuantongyu tongyuantongyu Jul 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AV1E_SET_SVC_PARAMS produces valid bitstream, but scaling_factor_num and scaling_factor_den values in avifLayerConfig are not honored.

Copy link
Contributor

@y-guyon y-guyon Jul 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have a short snippet to reproduce the issue, I suggest filing a bug.

avifBool useSvcParams = AVIF_TRUE;
for (uint8_t configIndex = 0; configIndex < layerCount; ++configIndex) {
avifLayerConfig * layer = &layers[configIndex];
for (int configIndex = 0; configIndex < layerCount; ++configIndex) {
const avifLayerConfig * layer = &layers[configIndex];
if (layer->horizontalMode.numerator != layer->verticalMode.numerator ||
layer->horizontalMode.denominator != layer->verticalMode.denominator) {
useSvcParams = AVIF_FALSE;
Expand Down Expand Up @@ -1076,11 +1076,9 @@ static avifResult aomCodecEncodeImage(avifCodec * codec,
if (addImageFlags & AVIF_ADD_IMAGE_FLAG_FORCE_KEYFRAME) {
encodeFlags |= AOM_EFLAG_FORCE_KF;
}
if (extraLayerCount > 0) {
if (layerIndex > 0) {
encodeFlags |= AOM_EFLAG_NO_REF_GF | AOM_EFLAG_NO_REF_ARF | AOM_EFLAG_NO_REF_BWD | AOM_EFLAG_NO_REF_ARF2 |
AOM_EFLAG_NO_UPD_GF | AOM_EFLAG_NO_UPD_ARF;
}
if ((extraLayerCount > 0) && (layerIndex > 0)) {
encodeFlags |= AOM_EFLAG_NO_REF_GF | AOM_EFLAG_NO_REF_ARF | AOM_EFLAG_NO_REF_BWD | AOM_EFLAG_NO_REF_ARF2 |
AOM_EFLAG_NO_UPD_GF | AOM_EFLAG_NO_UPD_ARF;
}

aom_codec_err_t encodeErr = aom_codec_encode(&codec->internal->encoder, &aomImage, 0, 1, encodeFlags);
Expand Down
59 changes: 24 additions & 35 deletions src/write.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -630,11 +630,16 @@ static avifResult avifEncoderAddImageInternal(avifEncoder * encoder,
// -----------------------------------------------------------------------
// Validate images

if ((layerCount == 0) || (layerCount > MAX_AV1_LAYER_COUNT)) {
return AVIF_RESULT_INVALID_LAYERS;
}

if ((gridCols == 0) || (gridCols > 256) || (gridRows == 0) || (gridRows > 256)) {
return AVIF_RESULT_INVALID_IMAGE_GRID;
}

const uint32_t cellCount = gridCols * gridRows;
const uint32_t imageCount = cellCount * layerCount;
if (cellCount == 0) {
return AVIF_RESULT_INVALID_ARGUMENT;
}

const avifImage * firstCell = cellImages[0];
if ((firstCell->depth != 8) && (firstCell->depth != 10) && (firstCell->depth != 12)) {
Expand Down Expand Up @@ -844,7 +849,7 @@ static avifResult avifEncoderAddImageInternal(avifEncoder * encoder,
if (item->codec) {
item->extraLayerCount = item->alpha ? encoder->extraLayerCountAlpha : encoder->extraLayerCount;
for (uint32_t layerIndex = 0; layerIndex < item->extraLayerCount + 1; ++layerIndex) {
const uint32_t index = (layerIndex > (layerCount - 1)) ? (layerCount - 1) : layerIndex;
const uint32_t index = AVIF_MIN(layerIndex, layerCount - 1);
const avifImage * layerImage = cellImages[item->cellIndex * layerCount + index];
avifResult encodeResult =
item->codec->encodeImage(item->codec, encoder, layerImage, item->alpha, layerIndex, addImageFlags, item->encodeOutput);
Expand Down Expand Up @@ -876,9 +881,6 @@ avifResult avifEncoderAddImageGrid(avifEncoder * encoder,
avifAddImageFlags addImageFlags)
{
avifDiagnosticsClearError(&encoder->diag);
if ((gridCols == 0) || (gridCols > 256) || (gridRows == 0) || (gridRows > 256)) {
return AVIF_RESULT_INVALID_IMAGE_GRID;
}
return avifEncoderAddImageInternal(encoder, gridCols, gridRows, 1, cellImages, 1, addImageFlags | AVIF_ADD_IMAGE_FLAG_SINGLE); // only single image grids are supported
}

Expand All @@ -889,10 +891,7 @@ avifResult avifEncoderAddImageProgressive(avifEncoder * encoder,
avifAddImageFlags addImageFlags)
{
avifDiagnosticsClearError(&encoder->diag);
if ((layerCount == 0) || (layerCount > MAX_AV1_LAYER_COUNT)) {
return AVIF_RESULT_INVALID_LAYERS;
}
return avifEncoderAddImageInternal(encoder, 1, 1, layerCount, layerImages, 1, addImageFlags | AVIF_ADD_IMAGE_FLAG_SINGLE); // only single image grids are supported
return avifEncoderAddImageInternal(encoder, 1, 1, layerCount, layerImages, 1, addImageFlags | AVIF_ADD_IMAGE_FLAG_SINGLE); // only single frame progressive images are supported
}

avifResult avifEncoderAddImageProgressiveGrid(avifEncoder * encoder,
Expand All @@ -902,13 +901,7 @@ avifResult avifEncoderAddImageProgressiveGrid(avifEncoder * encoder,
const avifImage * const * layerImages,
avifAddImageFlags addImageFlags) {
avifDiagnosticsClearError(&encoder->diag);
if ((layerCount == 0) || (layerCount > MAX_AV1_LAYER_COUNT)) {
return AVIF_RESULT_INVALID_LAYERS;
}
if ((gridCols == 0) || (gridCols > 256) || (gridRows == 0) || (gridRows > 256)) {
return AVIF_RESULT_INVALID_IMAGE_GRID;
}
return avifEncoderAddImageInternal(encoder, gridCols, gridRows, layerCount, layerImages, 1, addImageFlags);
return avifEncoderAddImageInternal(encoder, gridCols, gridRows, layerCount, layerImages, 1, addImageFlags | AVIF_ADD_IMAGE_FLAG_SINGLE);
}

static size_t avifEncoderFindExistingChunk(avifRWStream * s, size_t mdatStartOffset, const uint8_t * data, size_t size)
Expand Down Expand Up @@ -1540,10 +1533,11 @@ avifResult avifEncoderFinish(avifEncoder * encoder, avifRWData * output)
avifArrayCreate(&layeredColorItems, sizeof(avifEncoderItemReference), 1);
avifBool useInterleave = (encoder->extraLayerCount > 0) || (encoder->extraLayerCountAlpha > 0);

for (uint32_t itemPasses = 0; itemPasses < 2; ++itemPasses) {
for (uint32_t itemPasses = 0; itemPasses < 3; ++itemPasses) {
// Use multiple passes to pack in the following order:
// * Pass 0: metadata (Exif/XMP)
// * Pass 1: item data (AV1 color & alpha)
// * Pass 1: alpha (AV1)
// * Pass 2: all other item data (AV1 color)
//
// See here for the discussion on alpha coming before color:
// https://github.com/AOMediaCodec/libavif/issues/287
Expand All @@ -1553,6 +1547,7 @@ avifResult avifEncoderFinish(avifEncoder * encoder, avifRWData * output)
// and ignoreExif are enabled.
//
const avifBool metadataPass = (itemPasses == 0);
const avifBool alphaPass = (itemPasses == 1);

for (uint32_t itemIndex = 0; itemIndex < encoder->data->items.count; ++itemIndex) {
avifEncoderItem * item = &encoder->data->items.item[itemIndex];
Expand All @@ -1565,11 +1560,15 @@ avifResult avifEncoderFinish(avifEncoder * encoder, avifRWData * output)
// only process metadata (XMP/Exif) payloads when metadataPass is true
continue;
}
if (alphaPass != item->alpha) {
// only process alpha payloads when alphaPass is true
continue;
}

size_t chunkOffset = 0;

// Interleave - Pick out and record layered image items, interleave them later.
// Expect layer image item has same number of samples and fixups.
// Layer image items have same number of samples and fixups.
if (useInterleave && item->encodeOutput->samples.count > 0 &&
item->encodeOutput->samples.count == item->mdatFixups.count) {

Expand Down Expand Up @@ -1620,33 +1619,23 @@ avifResult avifEncoderFinish(avifEncoder * encoder, avifRWData * output)
}
}

// Interleave each sample (layer) of each item.
//
// - for each layer
// - for each grid cell
// - write alpha of this layer of this grid cell
// - write color of this layer of this grid cell
//
// See here for the discussion on alpha coming before color:
// https://github.com/AOMediaCodec/libavif/issues/287

if (useInterleave) {
avifBool hasMoreSample;
uint32_t layerIndex = 0;
do {
hasMoreSample = AVIF_FALSE;
// Assume color and alpha having same number of items (both single image, or both grid of same dimension)
for (uint32_t itemIndex = 0; itemIndex < layeredColorItems.count; ++itemIndex) {
for (uint32_t itemIndex = 0; itemIndex < AVIF_MAX(layeredColorItems.count, layeredAlphaItems.count); ++itemIndex) {
for (int samplePass = 0; samplePass < 2; ++samplePass) {
// Alpha coming before color
avifEncoderItemReferenceArray * currentItems = (samplePass == 0) ? &layeredAlphaItems : &layeredColorItems;
if (itemIndex >= currentItems->count) {
break;
continue;
}

// TODO: Offer the ability for a user to specify which grid cell should be written first.
avifEncoderItem * item = currentItems->ref[itemIndex].item;
if (item->encodeOutput->samples.count <= layerIndex) {
// We've already written all samples from this item
// We've already written all samples of this item
continue;
} else if (item->encodeOutput->samples.count > layerIndex + 1) {
hasMoreSample = AVIF_TRUE;
Expand Down