AbstractJackson2MessageConverter: GH-3038 fix incomplete - encoding parameter prevents byte[] path for binary formats

[ugurberkecan]

In what version(s) of Spring AMQP are you seeing this issue?

3.2.6 (and later versions with GH-3038 backport)

Describe the bug

The fix for GH-3038 is incomplete. While supportedCTCharset is now properly set when charset is provided in the MimeType constructor, the byte[] path in convertBytesToObject is only reached when encoding == null, which rarely happens in practice.

Root cause

In convertBytesToObject:

private Object convertBytesToObject(byte[] body, @Nullable String encoding, JavaType targetJavaType) {
    String encodingToUse = encoding;

    if (encodingToUse == null) { // encoding is usually "UTF-8", not null
        if (this.charsetIsUtf8 & this.supportedCTCharset == null) {
            return this.objectMapper.readValue(body, targetJavaType);
        }
        encodingToUse = getDefaultCharset();
    }

    // Always reaches here when encoding is set
    String contentAsString = new String(body, encodingToUse); 
    return this.objectMapper.readValue(contentAsString, targetJavaType); // CBOR fails
}



The encoding parameter comes from determineEncoding() which always returns a non-null value (typically "UTF-8"). Therefore, even with supportedCTCharset set, the code never reaches the byte[] path and always converts to String, breaking CBOR/Smile/Avro binary formats.

To Reproduce

@Bean
public MessageConverter cborConverter(CBORMapper cborMapper) {
    return new AbstractJackson2MessageConverter(
        cborMapper,
        new MimeType("application", "json", Charset.forName("UTF-8"))
    ) {};
}

// Producer sends JSON-serialized message with content-type: application/json
// Consumer tries to deserialize
// Result: UnsupportedOperationException: Can not create parser for non-byte-based source


Expected behavior

When supportedCTCharset is set, binary-format ObjectMappers (CBOR, Smile, Avro) should receive byte[] directly, not String.


Proposed fix

private Object convertBytesToObject(byte[] body, @Nullable String encoding, JavaType targetJavaType)
        throws IOException {

    if (this.supportedCTCharset != null) {
        return this.objectMapper.readValue(body, targetJavaType);
    }

    if (encoding == null) {
        if (this.charsetIsUtf8) {
            return this.objectMapper.readValue(body, targetJavaType);
        }
        encoding = getDefaultCharset();
    }

    String contentAsString = new String(body, encoding);
    return this.objectMapper.readValue(contentAsString, targetJavaType);
}



This ensures binary formats always get byte[] when charset is explicitly configured, matching the documentation: "If this contains a charset parameter, the raw bytes are passed to Jackson."

[artembilan]

I don't think your analysis is correct.
When we set a supportedCTCharset, there is no guarantee that it is something what is supported by Jackson.
Your proposal:

    if (this.supportedCTCharset != null) {
        return this.objectMapper.readValue(body, targetJavaType);
    }

eliminates the purpose of that provided charset.
It is here just out of use and that does not mean the data reflects the provided charset.
I think that Javadoc is a bit wrong and it has to reflect the current logic.
And it is like:

1. If charset is not provided, the byte[] is propagated to Jackson
2. If it is there, then we convert to String and then give it to Jackson.

You custom converter must come with a bit different configuration to make it working:

@Bean
public MessageConverter cborConverter(CBORMapper cborMapper) {
    return new AbstractJackson2MessageConverter(
        cborMapper,
        new MimeType("application", "cbor"))
    ) {};
}

Makes sense?

[ugurberkecan]

@Bean
public MessageConverter cborConverter(CBORMapper cborMapper) {
    return new AbstractJackson2MessageConverter(
        cborMapper,
        new MimeType("application", "cbor")
    ) {};
}

Changing the message converter type does not take effect because the content type is determined from the message itself. To address this, I tried explicitly changing the message content type as shown below:

newProperties.setContentType("application/cbor");

The content type is showing as and i assume it is correct:

application/x-java-serialized-object

I am also unable to add the SPRING_AMQP_DESERIALIZATION_TRUST_ALL property via environment variables.

Previously, we were sending messages as JSON, and it was working fine. The following code path was used:

if (this.supportedCTCharset != null) { // Jackson will determine encoding
    return this.objectMapper.readValue(
        body,
        this.objectMapper.constructType(targetClass)
    );
}

Now, because the encoding is null, this check is blocked by the current logic. As a result, it falls back to string conversion, which eventually fails.

Ideally, we should always determine the message type from the message payload itself, rather than relying solely on the contentType header. What do you think ?

One more question:
Do you know why the spring.amqp.deserialization.trust.all property is no longer available as an environment setting?

[artembilan]

Do you know why the spring.amqp.deserialization.trust.all property is no longer available as an environment setting?

See SerializationUtils:

	private static final String TRUST_ALL_ENV = "SPRING_AMQP_DESERIALIZATION_TRUST_ALL";

	private static final String TRUST_ALL_PROP = "spring.amqp.deserialization.trust.all";

	private static final boolean TRUST_ALL;

	static {
		TRUST_ALL = Boolean.parseBoolean(System.getenv(TRUST_ALL_ENV))
				|| Boolean.parseBoolean(System.getProperty(TRUST_ALL_PROP));
	}

So, env var is SPRING_AMQP_DESERIALIZATION_TRUST_ALL, while spring.amqp.deserialization.trust.all is sys prop.
Either way, that SerializationUtils is used only from the AllowedListDeserializingMessageConverter impls, where Jackson-based is not one of them.
So, please, don't mix concerns in one issue.

does not take effect because the content type is determined from the message itself.

I think I see what you mean:

	private Object convertBytesToObject(byte[] body, @Nullable String encoding, JavaType targetJavaType)
			throws IOException {

		String encodingToUse = encoding;

		if (encodingToUse == null) {
			if (this.charsetIsUtf8 & this.supportedCTCharset == null) {
				return this.objectMapper.readValue(body, targetJavaType);
			}
			encodingToUse = getDefaultCharset();
		}

		String contentAsString = new String(body, encodingToUse);
		return this.objectMapper.readValue(contentAsString, targetJavaType);
	}

So, currently we go to the readValue(byte[]) only of no encoding determined from the incoming message.
That is possible only in case of producer does not set a Content-Encoding property on the message to publish.
Or if a Content-Type property does not come with a charset param.

always determine the message type from the message payload itself

I'm not sure how to do that, but I guess we can go with some mitigation logic where we check that incoming encoding is UTF-8 if you claim that is almost never null.

[artembilan]

I think I will restrain from any fixes for now until we see the whole picture of the problem from your side.
There is the way to make it working now via not setting charset this or other way into message you publish.
And that is probably should be a true story for that CBOR format: no any charset exchanges and CBORMapper knows what to do on deserialization.

[ugurberkecan]

Previously, we were not changing the content type. It was set to JSON, and we were sending a byte[], so decoding worked correctly.

However, when we set:

newProperties.setContentType("application/cbor")

we can no longer use convertBytesToObject because it uses SimpleMessageConverter instead of the Jackson-based converter.

This leads to several additional problems, such as:

Trust dependency issues

Extra configuration requirements

The issue occurs because of the following logic in the converter:

@Override
public Object fromMessage(Message message) throws MessageConversionException {
    Object content = null;
    MessageProperties properties = message.getMessageProperties();
    if (properties != null) {
        String contentType = properties.getContentType();
        if (contentType != null && contentType.startsWith("text")) {
            String encoding = properties.getContentEncoding();
            if (encoding == null) {
                encoding = this.defaultCharset;
            }
            try {
                content = new String(message.getBody(), encoding);
            }
            catch (UnsupportedEncodingException e) {
                throw new MessageConversionException("failed to convert text-based Message content", e);
            }
        }
        else if (contentType != null &&
                contentType.equals(MessageProperties.CONTENT_TYPE_SERIALIZED_OBJECT)) {
            try {
                content = SerializationUtils.deserialize(
                        createObjectInputStream(new ByteArrayInputStream(message.getBody())));
            }
            catch (IOException | IllegalArgumentException | IllegalStateException e) {
                throw new MessageConversionException("failed to convert serialized Message content", e);
            }
        }
    }
    if (content == null) {
        content = message.getBody();
    }
    return content;
}

The problematic part is here:

content = SerializationUtils.deserialize(
    createObjectInputStream(new ByteArrayInputStream(message.getBody())));

Because of this logic, SimpleMessageConverter is used instead of the Jackson converter.

Even after configuring trusted packages, we still cannot add the required trusted libraries to Jackson because:

The object is generated internally by the SimpleMessageConverter

We cannot explicitly configure trusted libraries for it

The payload is treated as a generic serialized object

This behavior worked correctly before, but the additional content-type check breaks the existing logic

[artembilan]

How does it go to the SimpleMessageConverter if you can explicitly configure your custom cborConverter based on the application/cbor content type?

[ugurberkecan]

Because of this condition in SimpleMessageConverter:

if (contentType != null &&
        contentType.equals(MessageProperties.CONTENT_TYPE_SERIALIZED_OBJECT)) {
    try {
        content = SerializationUtils.deserialize(
                createObjectInputStream(new ByteArrayInputStream(message.getBody())));
    }
    catch (IOException | IllegalArgumentException | IllegalStateException e) {
        throw new MessageConversionException("failed to convert serialized Message content", e);
    }
}

this means that even if you configure a converter with application/json (or another content type), it does not matter in this case.

For example:

ContentTypeDelegatingMessageConverter messageConverter =
        new ContentTypeDelegatingMessageConverter();

Jackson2JsonMessageConverter jackson2JsonMessageConverter =
        new Jackson2JsonMessageConverter(cborMapper);

jackson2JsonMessageConverter.setSupportedContentType(
        new MimeType("application", "cbor"));

messageConverter.addDelegate("application/cbor", jackson2JsonMessageConverter);

The code above is useless, because the message is handled by SimpleMessageConverter because of this condition

contentType.equals(MessageProperties.CONTENT_TYPE_SERIALIZED_OBJECT)

SerializationUtils.deserialize(...) is invoked by the simple converter when the content type is
application/x-java-serialized-object, so the message is deserialized before the
ContentTypeDelegatingMessageConverter has any chance to process it.

Previously, when I did not explicitly define the content type (or set it to JSON), execution did not reach this branch.

[artembilan]

OK. You have not mentioned before a ContentTypeDelegatingMessageConverter.
That was really confusing part.
Thank you for clarification!

Now tell us, please, how that happens we end up with a CONTENT_TYPE_SERIALIZED_OBJECT if we talk about application/cbor?
Why don't you send a proper CBOR message from the producer?
How that happens that application/cbor is not populated to the MessageProperties.setContentType()?

Previously, when I did not explicitly define the content type

The default content-type is application/octet-stream. Not sure how that worked for your before with that ContentTypeDelegatingMessageConverter.
It would never reach an AbstractJackson2MessageConverter anyway.