How to serialize datetime type?

[SunFulong]

https://wamp-proto.org/wamp_latest_ietf.html#section-2.2-2

I'm trying to serialize datetime type with cbor, server side using cbor2 (python) library, and client side using cbor-web (javascript), they got different value.

So, should we define the datetime type serialize way?

[KSDaemon]

Well, I don't think it is a problem with the WAMP protocol itself. Are you sure that both implementations are aligned with the CBOR SPEC at 100%?

Datetime values may be stored differently in different languages/ecosystems. But in any case, they fall into NUMBER or STRING representation. We state that any used serializer must support both of these core types.

Also, what do you mean under define the datetime type serialize way? How this should be expressed in a way that every seriazilzer will understand it? IDK. In the end it still will be transfered/encoded/decoded by the rules of used serializer.

[SunFulong]

Oh, I mean, should the spec first define we should format datetime to RFC3339 format or convert to unixtimestamp integer, then serialize?

[oberstet]

should the spec first define we should format datetime to RFC3339 format or convert to unixtimestamp integer, then serialize?

no, I agree with @KSDaemon

that being said, over the years, I found that uint64 with UTC epoch time in ns is pretty good .. but that's totally for the app to decide

[KSDaemon]

I don't think this should be defined in protocol. WAMP internally is not using datetimes (yeah there are META RPCs that somehow utilizes datetime type, but the format is defined in related RPC descriptions). So it is client side payload. So it is up to client side to define and follow the formats.

[SunFulong]

spec said

A message serialization format is assumed that (at least) provides the following types

But the datetime is not list on it, we don't provide detail serialize way, how to make different vendor implementation exchange the data? if one convert it to string, and another convert to integer, they cannot exchange the right value.

as a spec, I think we need to define the common datatype serialize way to the type we list on.

[KSDaemon]

Have you looked at CBOR RFC? :)
At least here: https://www.rfc-editor.org/rfc/rfc8949#section-3.4

Once again: Are you sure that both your cbor serializers are correctly following the CBOR spec?

[oberstet]

But the datetime is not list on it

yes. my Customer data type is also not in the list

how to make different vendor implementation exchange the data

by requiring a specific format in your app

[SunFulong]

Are you sure that both your cbor serializers are correctly following the CBOR spec?

OK, my fault here, sorry.

But I want to say, spec should give a guide to serialize common datatype to basic type.

[KSDaemon]

Well... Where is the list and who decides what is a common type? You said datetime in RFC3389 — is a common type. What if for me datetime in ISO 8601 format is a common type? (btw, these formats are not compatible in general view)

[oberstet]

But I want to say, spec should give a guide to serialize common datatype to basic type.

we could add such a "convenience / best practice" section ... but I'm not sure whether the spec is the right place. maybe we want a "user guide" and "implementor guide" as new pages on the web site?

also, it would be a tedious, controversial effort.

eg take "temperature (physical)".

• Fahrenheid or Celsius?
• Integer or Floats?
• If Floats, what kind of (so that serializer/platform round-tripping works)?
• If integers, how to scale?
• ...

this sounds like a boring discussion;) just chose what you need in your app ..

[oberstet]

let me underline that "serializer/platform round-tripping" only works for the types here jere https://wamp-proto.org/wamp_latest_ietf.html#name-serializers

floats in particular are not in this list. this is even more fundamental than "missing datetime format" ...

[SunFulong]

@oberstet you are right, float is more fundamental, why didn't the spec include it?

[oberstet]

why didn't the spec include it?

because there is no portable standard for binary or decimal floats as even a fully compliant IEEE754 isn't cross platform at the hardware level

ethereum solves this like this: all numbers are always integers, and all integers are uint256 little endian

even WASM hasn't deterministic floats

for a dicussion of the problems a blockchain based on WASM (not using only integers) would encounter, check out

openethereum/parity-ethereum#6511
https://medium.com/haderech-dev/determinism-wasm-40e0a03a9b45

---

the WASM spec is funny as well

https://webassembly.github.io/spec/core/syntax/types.html#number-types

it doesn't even mention that "as defined by the IEEE 754 standard" means: details are implementation defined - that is "implementation" == IEEE754 hardware in use

which means: floats in WASM cannot be serialized into bits in a portable way.

[oberstet]

fwiw, https://github.com/wasmerio/wasmer has added an option NaN canonicalization (nan_canonicalization) to fix above, making floats deterministic

wasmerio/wasmer#1269 (comment)
wasmerio/wasmer@5d26d92
https://github.com/bytecodealliance/wasmtime/blob/main/cranelift/codegen/src/nan_canonicalization.rs
WebAssembly/interface-types#134
WebAssembly/design#619

a full list of non-determinism in WASM:

https://github.com/WebAssembly/design/blob/main/Nondeterminism.md

the EVM in contrast has perfect deterministic operation - which include perfect deterministic representation of all EVM data types (of which there is only 1;)

https://ethereum.stackexchange.com/questions/30812/how-does-the-evm-ensure-perfect-determinism-and-what-does-that-mean
https://ethereum.stackexchange.com/questions/142/what-was-the-reason-to-invent-the-evm/3437#3437

[ecorm]

But I want to say, spec should give a guide to serialize common datatype to basic type.

Disagreed. The WAMP protocol should remain agnostic of application payloads. It should not be "opinionated" about application-level things like which time format to use (there are many). It should not be a general guide on how to write distributed apps or how to use serialization codecs.

Only where WAMP uses date-times internally should it specify time formats. Only two cases come to mind:

• CALL timeouts, which are specified as integer milliseonds
• Event history, which is specified as RFC3339 strings.

Here are the various time formats I'm aware of:

• RFC3339 strings
• ISO 8601 strings
• Split seconds and microseconds/nanoseconds since epoch as per POSIX gettimeofday function
• Seconds since epoch as per POSIX time function
• Floating-point seconds since the epoch
• 100-nanosecond intervals since January 1, 1601 in Windows
• Milliseconds since the epoch in Javascript
• The CBOR extended time format tag which is quite elaborate and not trivial to implement.

Those who use JSON will tend to favor RFC3339/ISO8601 as they are human-readable and can be serialized/deserialized natively in Javascript. Those who use CBOR will tend to favor more compact numeric formats that don't require parsing. How can we choose for everyone?

The great thing about standards is that there are so many to choose from! 😁

Something that WAMP may want to have an opinion about is how time formats are transcoded from one format to another. Same thing goes for maps with non-string keys. Also, CBOR bignums are lossy if you strip away their tags while naively transcoding to JSON. There are other proposed CBOR tags that are similarly lossy because their authors had the (IMO) misguided idea to encode part of the information in the tags to save a few previous bytes. Transcoding these "special types" has been a major pain in my butt in a C++ serialization library I've been working on.

[oberstet]

transcoded from one format to another

fwiw, AutobahnPython includes quite a bunch of tests for "cross-tripping" between serializers

https://github.com/crossbario/autobahn-python/blob/01165cf27ada0c246cc23a1e5aac07a6b6839bdc/autobahn/wamp/test/test_wamp_serializer.py#L384
https://github.com/crossbario/autobahn-python/blob/01165cf27ada0c246cc23a1e5aac07a6b6839bdc/autobahn/wamp/test/test_wamp_serializer.py#L335

however, the cross-tripping is only between different serializers on the same platform. what actually would need to be tested is cross-tripping between all serializers on all platforms (different CPUs)

[SunFulong]

I read the spec over and over again. The five basic data types are the minimum requirement, and all implementations must support them. Other data types are extended types, and there is no guarantee that the support of each implementation is consistent. Right?