`UnmarshalJSON` is limited to the default Tree

[Wondertan]

Problem

When we deserialize the EDS we hardcode the default tree. This is problematic because computed Row and Col roots of the deserialized eds will be utterly different if another custom tree is used.

Solutions

1. Have a global tree constructor var/registry that UnmarshalJSON would use instead
   • Users would then register their trees by name
     • The name would be serialized in the json and deserialization would then get the proper tree by the name
   • This is exactly what happens with the codec.
2. UnmarshalEDS func that accepts the tree.
   • This works in theory, but https://github.com/filecoin-project/go-jsonrpc/blob/17a35772b24f59ceee25cc8d8177a2b255a659ac/client.go#L614 blocks it
   • If we contribute an option that custom Unmarshall, then we would be able to inject the one for EDS

[Wondertan]

The first solution is bulletproof and requires minimum work outside of rsmt2d, as well the same approach is used for Codec, so we should go with it, imo

[Wondertan]

We reencountered this issue while debugging a different bug. The UnmarshalJSON was not using the NMTWrapper, leading to a different data root in tests, which became a major time wasting redherring for the original bug. Besides, this issue means JSON serialization doesn't work.

The original problem was solved, but then reverted by #295. However, I don't understand the motivation behind the revert, @rootulp. You say it leaked implementation detail, but it's unclear what exactly was leaking there to warrant a revert without a follow-up fix.

[rootulp]

You say it leaked implementation detail, but it's unclear what exactly was leaking there to warrant a revert without a follow-up fix.

From #295 I said it leaks this responsibility to celestia-app: "Celestia-app needs to be careful to register all the appropriate trees once (and only once) before they are used (via Compute or Import)." That PR was a long time ago but if I recall correctly, it wasn't easy to bump to the breaking release of rsmt2d.

I haven't explored this deeply but instead of registering tree names in consuming apps, can MarshallJSON serialize the options that define the custom tree (e.g. square size, NMT options) and then UnmarshalJSON creates the tree using those options? This way, the JSON is self-describing and doesn't need an extra registry in consuming apps.

[Wondertan]

can MarshallJSON serialize the options that define the custom tree (e.g. square size, NMT options) and then UnmarshalJSON creates the tree using those options?

@rootulp, it can be self-describing, and the original solution was too, by introducing tree field on the marshalled struct. It's just that the rsmt2d allowed users to register their own tree types, but they can be self-describing either way. The alternative is to make the tree optionality permissioned and allow only what's preregistered, just like codecs already on rsmt2d. I don't have a strong opinion on which way to go here, but if we make it permissioned rsmt2d would need to depend on nmt, which may not be a bad thing in the end.

[rootulp]

I don't understand how the original solution was self-describing if celestia-app needs a registry of tree names to tree options.

I'm in favor of a solution that doesn't need the registry in celestia-app because it was difficult to initialize the registry correctly. IIRC multiple packages in celestia-app were registering the same tree name multiple times.

[Wondertan]

I don't understand how the original solution was self-describing if celestia-app needs a registry of tree names to tree options.

Well, self-describing does not imply a lack of registry. Having this field makes it self-describing, as it clearly describes the type of tree it is. The opposite would be making this fixed to a particular tree type, making it implicit rather than explicit and self-describing.