Handling of x-dotnet-pub-seq-no header from rabbitmq-dotnet-client

[zlepper]

I have started a discussion in the rabbitmq-dotnet-client repo about some strange header that that library seems to be emitting. I thought you might want a chance to weight in from the amqprs perspective: rabbitmq/rabbitmq-dotnet-client#1805

In an attempt to keep all information in one place i have not copied the entire main question here for now. I will do that if requested :)

[michaelklishin]

@zlepper a client library can decide to use an x- header for its own needs. There's nothing weird about it, it is an attempt to simplify the use of publisher confirms.

Other client libraries are not forced to use it in any way, so I'm not sure what kind of opinion you are looking for.

[zlepper]

It's more that amqprs crashes the moment it attempts to even pop the message from the queue, I assume based on some sort of embedded type information. So even if I don't want to do anything with the header, I don't have the opportunity as the message never gets based back to me.

So I guess the error kinda is on amqprs for attempting to parse any and all headers immediately, rather than keeping the headers as their raw byte sequences and deferring the parsing?

[michaelklishin]

Every client parses the headers immediately. It's just that this client does not handle a .NET-encoded UInt64 for a currently unknown reason. It should.

[michaelklishin]

Looks like the issue is that this client does not handle a header value serialized as i, that is, an unsigned long.

The .NET client writes it as a UInt64 and this client fails to deserialize it.

Since the way to reproduce is known and @zlepper even has a libpcap/Wireshark traffic capture, it's reasonable to just file an issue.

In every discussion of AMQP 0-9-1 data types, headers and tables (it's a data type in AMQP 0-9-1) in general we must mention this AMQP 0-9-1 errata document.

[michaelklishin]

Despite rabbitmq/rabbitmq-dotnet-client#1806, the scenarios where a header value is set to an unsigned 32 or 64-bit integer should be handled by this library.

Neither of those are common scenarios, and in some programming languages you cannot really express an unsigned integer, so it's not something urgent but still.

Here are both Java and .NET client implementations for reference, plus the AMQP 0-9-1 errata document from the RabbitMQ core team.

@gftea I hope this summarizes the problem for you. I can file an issue if you'd like me to

[lukebakken]

rabbitmq/rabbitmq-dotnet-client#1806

There isn't anything strictly necessary for this library to do. I'm treating the S data type, long string, as an opaque byte array. Since strings can be encoded many different ways, the safest behavior is that S values are not decoded using any specific string encoding, but remain as byte arrays for the application to decode.

Having said that, the RabbitMQ Java client encodes and decodes strings in UTF-8. The .NET client has never done so, and of course, the AMQP 0.9.1 spec says nothing about how strings should be encoded (link).

[gftea]

Hi, this client does assume utf8 for str encode/decode.
@michaelklishin @lukebakken , what is the agreement on protocol side?

[michaelklishin]

@gftea AMQP 0-9-1 does not require values to be valid UTF-8 in most if not all contexts

[gftea]

Ok
, but isn't almost everyone think of utf8 is de facto standard for string inter-operability?

[michaelklishin]

@gftea a lot of binary protocols allow non-UTF-8 and non-Unicode data to be passed around.

I don't think non-Unicode payloads are all that common in the case of the protocols RabbitMQ supports but sometimes that's the case, for one reason or another.

[gftea]

It would be breaking changes due to we have provide api for LongStr.
AsRef no longer work if we use raw bytes in LongStr, and From will be lossy

impl AsRef<String> for LongStr {
    fn as_ref(&self) -> &String {
        &self.1
    }
}
impl From<LongStr> for String {
    fn from(s: LongStr) -> Self {
        s.1
    }
}

[michaelklishin]

This library can requiring the headers to contain UTF-8 characters only. I'm just saying that for headers specifically (unlike certain other fields of protocol methods), RabbitMQ itself would accept a non-UTF-8 header key (or value).