Improve json serialization

[Ariana-B]

The jsonify_document function is used in several places (including other odc packages) to convert certain value types to strings - notably, it is part of the json serializer function passed to create_engine for both the postgres and postgis drivers. Per the comment in _to_json, this seems to be due to postgres<=9.5 not supporting nan and infinite values, and so should no longer be necessary as part of indexing.
However, removing these string conversions will create an issue with the use of orjson, which serialises nan and +/-inf as null and does not provide any option to alter this behaviour, so it will need to be replaced with a different json library such as rapidjson. This would be slower than orjson, but it may very well be that any performance benefits provided by orjson are currently being offset by the performance hit of jsonify_document anyway.
It is also worth noting that jsonify_document is used to compare existing metadata type/product/dataset definition/metadata documents with new ones, so we would need to update the affected entries or implement some extra comparison logic.

[pjonsson]

I'm not sure I understand the last sentence, is there a desire to use jsonify_document and then compare the results even if we have other ways to compare the results, or is that how it happens to be implemented today?

Doing a deep-copy in Python by walking the entire structure is bound to be slow, so it shouldn't be hard to come up with something faster. If we want to compare JSON documents for some reason, it still seems there are fast(er) diff libraries available that we might be able to use.

Since I don't know which parts of the code we're talking about, I'll mention that I enabled some additional format checking in #2310 so we've had a couple of releases with that so far. But our schemas are quite forgiving so it probably doesn't make a big difference in practice.

On the whole, I'm definitely positive to the change suggested by this issue.

[Ariana-B]

My understanding is that jsonify_document is currently applied to all documents inserted into the database, so when we want to compare a 'new' json document with an existing one taken from the database, we have to apply jsonify_document to the new doc, see for example:

datacube-core/datacube/index/postgis/_datasets.py

Line 354 in 61c864d

existing.metadata_doc, jsonify_document(dataset.metadata_doc)

. This is separate from schema validation.
If we change the engine logic to no longer apply jsonify_document, we would no longer need to use jsonify_document in the comparison logic for documents after the change, however for documents added previously the comparison may raise false positives without jsonify_document. So either those entries would need to be updated or we need to add logic to the comparison functions to account for the mismatches.

[pjonsson]

I see. My immediate feeling is that it would be preferable to convert the existing database to the "new" format so there would be a one-time cost and then insertions would be fast rather than having to pay for the extra comparison logic on each insertion. Personally I could just start from a clean slate so I don't even need a conversion command in datacube, but I realize larger installations can't necessarily do that.