[#5202] feat(client-python): Support Gravitino Type Serdes - serializer

[tsungchih]

What changes were proposed in this pull request?

This is the second part (totally 4 planned) of implementation to the following classes from Java to support Column and its default value, including:

• JsonUtils
• TypeSerializer

The TypeSerializer will be used in the incoming ColumnDTO implementation to serialize data_type field.

Why are the changes needed?

We need to support Column and its default value in python client.

#5202

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit tests

[unknowntpo]

@tsungchih Could you explain why you need to implement a json_serdes util ? See what MetalakeDTO did, simply inherit fromDataClassJsonMixin in dataclasses_json library.

from dataclasses_json import DataClassJsonMixin, config

...
@dataclass
class MetalakeDTO(Metalake, DataClassJsonMixin):
     ...

Reference:

• Docs from dataclasses-json

[tsungchih]

@tsungchih Could you explain why you need to implement a json_serdes util ? See what MetalakeDTO did, simply inherit fromDataClassJsonMixin in dataclasses_json library.

from dataclasses_json import DataClassJsonMixin, config

...
@dataclass
class MetalakeDTO(Metalake, DataClassJsonMixin):
     ...

Reference:

• Docs from dataclasses-json

@unknowntpo Thanks for your comments.

That is because all of the current Gravitino Type defined in gravitino.api.types.types.Types, such as NullType, BooleanType, etc., do not support dataclass-json. Namely they do not inherit from DataClassJsonMixin in dataclasses_json library.

IMHO, here comes with two solutions to serialize/deserialize Type listed as follows.

1. make all of the Types support for dataclass-json; (this will be a significant change to the interface and I'm not sure if it is acceptable)
2. remain Types unchanged and define serializer/deserializer for them (this is the current PR)

I'm going to elaborate more by taking ShortType as an example. The current class definition is

class Types:
    class ShortType(IntegralType):
        _instance: Types.ShortType = None
        _unsigned_instance: Types.ShortType = None
        ...

We may have it to support for dataclass_json listed as follows.

class Types:
    @dataclass
    class ShortType(IntegralType, DataClassJsonMixin):
        _instance: Types.ShortType = None
        _unsigned_instance: Types.ShortType = None
        ...

Note that, however, the ShortType has two forms, signed and unsigned. We need to choose different serialize/deserialize methods for the instance of ShortType based on which form it is. I haven't figured out how to fulfill this case by simply inherit fromDataClassJsonMixin in dataclasses_json library from the documents.

Regards

[tsungchih]

@xunliu @unknowntpo Based on the comments, I made the following changes:

• moved json_serdes into gravitino.api.types;
• combined GenericJsonSerializer and GenericJsonDeserializer into one JasonSerializable interface;
• rename TypeSerializer as TypeSerdes to implement the JsonSerializable interface;

In the end, the TypeSerdes will be used by ColumnDTO in the future such as

from gravitino.api.types.json_serdes.type_serdes import TypeSerdes

class ColumnDTO(Column):
    _data_type: Type = field(
        metadata=config(
            field_name="data_type",
            encoder=TypeSerdes.serialize,
            decoder=TypeSerdes.deserialize,
        )
    )

Please take a look at it when you're available. Thanks!

[unknowntpo]

@tsungchih Can we use Json to add some constraints to GravitinoTypeT and DeserializedTypeT ?

I found that in config.encoder, and config.decoder, it has some comments about the type of Callable, it says B is bound as a JSON type, and in dataclasses_json.core, it has Json type.
So I'm wondering we can use it to add constraints to GravitinoTypeT and DeserializedTypeT.

# in dataclasses-json cfg.py
def config(metadata: Optional[dict] = None, *,
           # TODO: these can be typed more precisely
           # Specifically, a Callable[A, B], where `B` is bound as a JSON type
           encoder: Optional[Callable] = None,
           decoder: Optional[Callable] = None,

[tsungchih]

@unknowntpo Sure, it would be better if we could have appropriate type bound. I didn’t know the Json. Allow me to investigate the type you mentioned. Thanks for your prompt comments.

[tsungchih]

@unknowntpo I submitted a commit to add for DeserializedTypeT to be bound as Json type and for GravitinoTypeT to be bound as Type. Thanks for the comments.

[unknowntpo]

@tsungchih Ok, but I found another problem,

def serialize(cls, data_type: GravitinoTypeT) -> DeserializedTypeT:

This is for serializing data_type to Json type, so I think DeserializedTypeT should be named as SerializedTypeT. Or we can simply use Json type.

def serialize(cls, data_type: GravitinoTypeT) -> Json:

[tsungchih]

@unknowntpo Yes, you're right. I didn't notice that. I changed it to use Json instead. Thanks for the comments.

[unknowntpo]

LGTM.

[xunliu]

hi @tsungchih Thank you for your contributions.
I read the PR for this and it mainly talks about how to do json serialization and deserialization, right?
But the title of the PR is column handling. Is it possible that the title of the PR should be changed?

[tsungchih]

hi @tsungchih Thank you for your contributions. I read the PR for this and it mainly talks about how to do json serialization and deserialization, right? But the title of the PR is column handling. Is it possible that the title of the PR should be changed?

The title has been changed. Thanks for the suggestion.

[unknowntpo]

@xunliu LGTM.

[jerryshao]

LGTM, thanks @tsungchih for your work.