Generic Type Problem

[Esatyilmaz0]

• Dataclass Wizard version:
• Python version:
• Operating System:

Description

Creating and using a data class to generate Generic classes work for me, dataclass-wizard fromdict not work . What is the best way for this?

What I Did

@DataClass
class User:
username: str
email:str

@DataClass
class Post:
title:str
slug:str

T = TypeVar("T")

@DataClass
class Collection(Generic[T]):
results:List[T]
count: int

@DataClass
class PostCollection(Collection[Post]):
pass

@DataClass
class UserCollection(Collection[User]):
pass

a = fromdict(UserCollection, {"count":35, "results":[{"username":"esat", "email":"esat3515@gmail.com"}, {"username":"test1", "email":"test3535@gmail.com"}]})

results is a dict not User Object List

UserCollection(results=[{'username': 'esat', 'email': 'esat3515@gmail.com'}, {'username': 'test1', 'email': 'test3535@gmail.com'}], count=35)

[rnag]

Hi @Esatyilmaz0, first of all thanks for opening this issue! I personally knew I'd have to deal with Generic type classes at some point, so in any case I'm glad as this is the first issue that actually addresses that. I'll have to dig a big deeper into typing.Generic to understand exactly how it works, as I'm not too familiar with it myself.

The main points that are worth asking here would probably be something along the lines of:

• How does one identify that a class or a dataclass is a typing.Generic type
• What is the best approach to get the Generic type argument out of a class

Notes on the above: I have a feeling that the builtin typing.get_type_hints might be able to resolve the type of fields given a dataclass that extends from Generic[T], but haven't looked too closely into it yet.

---

In the meantime, I would certainly encourage you to check out the Contributing docs if you are interested in adding your own changes. As mentioned, I'm going to dig a bit deeper to understand how typing.Generic works exactly, which honestly might end up taking me some time.

[tahouse]

I believe I'm running into the same issue using a Generic to define a JSONWizard dataclass. I've created a simple case with a User that can take a generic type to used for a user's name.

The first two examples work fine when the main class is a generic dataclass. To json and from json both work as expected. The problem seems to happen when I try to use the working dataclass as a type in another dataclass. Serialization works but deserialization doesn't.

Works..

from dataclasses import dataclass
from typing import Generic, TypeVar

from dataclass_wizard import JSONWizard  # type:ignore

T = TypeVar("T")


@dataclass
class User(Generic[T], JSONWizard):
    name: T


@dataclass
class Name:
    first: str
    last: str

# User.name is a Name
user_one = User[Name](Name("billy", "joel"))
# user_one -> User(name=Name(first='billy', last='joel'))
with open("temp_one.json", "w") as f:
    f.write(user_one.to_json())
# temp_one.json -> {"name": {"first": "billy", "last": "joel"}}
with open("temp_one.json", "r") as f:
    imported_user_one = User.from_json(f.read())
# imported_user_one -> User(name={'first': 'billy', 'last': 'joel'})

# User.name is now a string
user_two = User[str]("frank")
# user_two -> User(name='frank')
with open("temp_two.json", "w") as f:
    f.write(user_two.to_json())
# temp_two.json -> {"name": "frank"}

with open("temp_two.json", "r") as f:
    imported_user_two = User.from_json(f.read())
# imported_user_two -> User(name='frank')

The following does not work though:

@dataclass
class NamedUser(JSONWizard):
    user: User[Name]


user_three = NamedUser(User(Name("frank", "zappa")))
with open("temp_three.json", "w") as f:
    f.write(user_three.to_json())
# temp_three.json -> {"user": {"name": {"first": "frank", "last": "zappa"}}}
with open("temp_three.json", "r") as f:
    imported_user_three = NamedUser.from_json(f.read())
# ParseError: Failure parsing field `None` in class `None`. Expected a type User, got NoneType.
#   value: None
#   error: Provided type is not currently supported.
#   unsupported_type: <class '__main__.User'>

Full stack trace:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/anaconda3/lib/python3.8/site-packages/dataclass_wizard/loaders.py in fromdict(cls, d)
    531     try:
--> 532         load = _CLASS_TO_LOAD_FUNC[cls]
    533     except KeyError:

KeyError: <class '__main__.NamedUser'>

During handling of the above exception, another exception occurred:

ParseError                                Traceback (most recent call last)
~/git/py-tutorial/common/__init__.py in <module>
     12 # temp_three.json -> {"user": {"name": {"first": "frank", "last": "zappa"}}}
     13 with open("temp_two.json", "r") as f:
---> 14     imported_user_three = NamedUser.from_json(f.read())
     15 imported_user_three

~/anaconda3/lib/python3.8/site-packages/dataclass_wizard/serial_json.py in from_json(cls, string, decoder, **decoder_kwargs)
     46         o = decoder(string, **decoder_kwargs)
     47 
---> 48         return fromdict(cls, o) if isinstance(o, dict) else fromlist(cls, o)
     49 
     50     @classmethod

~/anaconda3/lib/python3.8/site-packages/dataclass_wizard/loaders.py in fromdict(cls, d)
    532         load = _CLASS_TO_LOAD_FUNC[cls]
    533     except KeyError:
--> 534         load = load_func_for_dataclass(cls)
    535 
    536     return load(d)

~/anaconda3/lib/python3.8/site-packages/dataclass_wizard/loaders.py in load_func_for_dataclass(cls, is_main_class, config)
    579     # This contains a mapping of the original field name to the parser for its
    580     # annotated type; the item lookup *can* be case-insensitive.
--> 581     field_to_parser = dataclass_field_to_load_parser(cls_loader, cls, config)
    582 
    583     # A cached mapping of each key in a JSON or dictionary object to the

~/anaconda3/lib/python3.8/site-packages/dataclass_wizard/class_helper.py in dataclass_field_to_load_parser(cls_loader, cls, config, save)
    118     """
    119     if cls not in _FIELD_NAME_TO_LOAD_PARSER:
--> 120         return _setup_load_config_for_cls(cls_loader, cls, config, save)
    121 
    122     return _FIELD_NAME_TO_LOAD_PARSER[cls]

~/anaconda3/lib/python3.8/site-packages/dataclass_wizard/class_helper.py in _setup_load_config_for_cls(cls_loader, cls, config, save)
    187         # Lookup the Parser (dispatcher) for each field based on its annotated
    188         # type, and then cache it so we don't need to lookup each time.
--> 189         name_to_parser[f.name] = cls_loader.get_parser_for_annotation(
    190             field_type, cls, field_extras
    191         )

~/anaconda3/lib/python3.8/site-packages/dataclass_wizard/loaders.py in get_parser_for_annotation(cls, ann_type, base_cls, extras)
    435                 # No matching hook is found for the type.
    436                 err = TypeError('Provided type is not currently supported.')
--> 437                 raise ParseError(
    438                     err, None, base_type,
    439                     unsupported_type=base_type

ParseError: Failure parsing field `None` in class `None`. Expected a type User, got NoneType.
  value: None
  error: Provided type is not currently supported.
  unsupported_type: <class '__main__.User'>

I've also tried:

NamedUserType = TypeVar("NamedUserType")


@dataclass
class NamedUser(Generic[NamedUserType], JSONWizard):
    user: User[NamedUserType]


user_three = NamedUser[Name](User(Name("frank", "zappa")))
with open("temp_three.json", "w") as f:
    f.write(user_three.to_json())
# temp_three.json -> {"user": {"name": {"first": "frank", "last": "zappa"}}}
with open("temp_two.json", "r") as f:
    imported_user_three = NamedUser[Name].from_json(f.read())

But this gives the same error.

Last, this does work, but I cannot define the Type should be restricted to Name (it accepts a string too)

class NamedUser(JSONWizard):
    user: User


user_three = NamedUser(User("frank"))
with open("temp_three.json", "w") as f:
    f.write(user_three.to_json())
# temp_three.json ->{"user": {"name": "frank"}}
with open("temp_three.json", "r") as f:
    imported_user_three = NamedUser.from_json(f.read())
# imported_user_three -> NamedUser(user=User(name='frank'))

[tahouse]

By the way, I've looked at adding support for custom/new types, but I'm unclear where to begin. Is this something that can happen in user code or does it need to live in the library? For example, I've been able to implement on own dump logic using DumpMeta in my user code. Works well!

class JSONSnakeWizard(JSONWizard, JSONFileWizard):
    """Helper for JSONWizard that ensures dumping to JSON puts keys in snake_case"""

    def __init_subclass__(cls) -> None:
        """Method for binding child class to DumpMeta"""
        DumpMeta(key_transform="SNAKE").bind_to(cls)

But I could not follow the logic on how to support additional types, or to say, avoid dumping specific types (just leaving as is) and making do with a to_dict

[rnag]

@tahouse Just noting that it is possible to achieve this through user code, only the process is somewhat convoluted at the moment, and I haven't gotten around to documenting that at present. I have added a section on Type hooks in the docs that explains how to manipulate the current load/dump functions for existing types, but I understand that's not too useful in this case in particular.

While there is not ideal support for adding custom/new types, it is possible to achieve this by subclassing from the Mixin classes LoadMixin, or DumpMixin for the serialization process. Currently most of the logic for de-serializing supported types lives in the get_parser_for_annotation implementation, so the trick here is actually to override this method when subclassing, to check for custom or user-defined types, and then to forward to the base implementation otherwise.

To illustrate this, here's a simple example I came up with for handling a dataclass field annotated as a pathlib.Path type, which I believe is not a currently supported type in the default implementation:

from dataclasses import dataclass
from pathlib import Path
from typing import Type

from dataclass_wizard import JSONWizard, LoadMixin
from dataclass_wizard.abstractions import AbstractParser
from dataclass_wizard.models import Extras
from dataclass_wizard.parsers import SingleArgParser
from dataclass_wizard.type_def import T
from dataclass_wizard.utils.typing_compat import eval_forward_ref_if_needed


@dataclass
class MyClass(JSONWizard, LoadMixin):
    name: str
    some_path: Path

    @classmethod
    def get_parser_for_annotation(cls, ann_type: Type[T],
                                  base_cls: Type = None,
                                  extras: Extras = None) -> AbstractParser:
        # evaluate any forward-declared annotations (i.e. strings) as needed
        ann_type = eval_forward_ref_if_needed(ann_type, base_cls)

        if issubclass(ann_type, Path):
            base_type = ann_type
            # alias: base_type(o)
            return SingleArgParser(base_cls, extras, base_type, ann_type)

        # else, forward to the default `LoadMixin.get_parser_for_annotation()` implementation
        return super().get_parser_for_annotation(ann_type, base_cls, extras)


c = MyClass.from_dict({'some_path': 'a/b/c', 'name': 'Jane Doe'})
print(f'object: {c!r}')

assert isinstance(c.name, str)
assert isinstance(c.some_path, Path)
assert c.some_path == Path('a/b/c')

[tahouse]

@rnag Thanks! I will give this method a try when I get a chance. BTW, will be filing a new ticket for mypy static type check support (didn't want to overload this particular issue)

[rnag]

Also just adding this for awareness, but I'll be on vacation until May 16th. This is mostly due to Pycon, which I'm attending this year - it's actually my first Pycon. However, will be back then to take a closer look at this issue. In the meantime, if anyone is able to look into it and implement a solution - or least a work-in-progress - I'd be more than happy to review, once I get back.

[rnag]

So I've made a bit of progress, and just wanted to share what I was able to put together so far. I've only tested this on Python 3.10, but hopefully this works on Python 3.7+ at least. FWIW I'm planning on dropping support for 3.6 in a future release (since 3.6 reached end-of-life a while ago), so I feel that maintaining support for 3.7+ is a solid goal moving forward.

To preface this, my initial goal here was to resolve the definitions of TypeVar variables to their concrete types, as declared in user code. This way we can maintain a running cache and its easier to resolve the type when we see a dataclass field like name: T for example.

I'm wondering if there's an easier or more straightforward way - I haven't looked too much into helper functions that the typing module provides, but I was curious if it provides an alternate way we can use to determine the generic/concrete type relations - but below is my working code that I managed to put together so far, at least on Python 3.10.

from dataclasses import dataclass
from typing import TypeVar, Generic, Dict, List


T1 = TypeVar('T1')
T2 = TypeVar('T2')
T3 = TypeVar('T3')

KT = TypeVar('KT')  # Key type.
VT = TypeVar('VT')  # Value type.


@dataclass
class MyGenericClass(Generic[T1, T2]):
    results: List[T2]


class MyGenericDict(Dict[KT, VT]):
    ...


@dataclass
class BaseClass:
    my_str: str


@dataclass
class MyTestClass(BaseClass, MyGenericClass[str, int], MyGenericDict[float, bool]):
    value: VT


@dataclass
class MyOtherTestClass(MyGenericClass[float, bool], Generic[T2, T3]):
    my_field: T2


def is_generic_cls(cls: type) -> bool:
    """TODO: Stub function to check if `cls` is a `Generic` type"""
    return hasattr(cls, '__orig_bases__')


def resolve_generics(cls: type):
    """Resolve generic to concrete types for a class `cls` which inherits from typing Generics"""

    # this will be the case for a dataclass `MyClass` that inherits from
    # `Generic[T]` and is then passed in as `cls` like `MyClass[str]`
    cls_origin = getattr(cls, '__origin__', None)

    if cls_origin is not None:
        # save the passed in args - which will be `(str, )` in above example
        cls_args = getattr(cls, '__args__', None)
        cls = cls_origin
    else:
        cls_args = None

    if is_generic_cls(cls):
        type_var_to_concrete_type = {}

        for base_cls in cls.__orig_bases__:
            cls_origin = getattr(base_cls, '__origin__', base_cls)

            if is_generic_cls(cls_origin):
                concrete_args = base_cls.__args__
                generic_args = cls_origin.__parameters__
                # it is possible that each class, `MyGenericClass` for example can inherit
                # from more than one "Generic" class, but for simplicity's sake we just
                # assume it inherits from one.
                generic_cls = cls_origin.__orig_bases__[0]

                print(cls_origin.__qualname__)
                print('  Generic base:  ', generic_cls)
                print('  Generic args:  ', generic_args)
                print('  Concrete args: ', concrete_args)
                print()

                type_var_to_concrete_type.update(zip(generic_args, concrete_args))

        # again, this is the case for a `cls` argument like `MyClass[str]`
        if cls_args:
            cls_params = cls.__parameters__
            type_var_to_concrete_type.update(zip(cls_params, cls_args))

        print('---')
        print('TypeVar to Concrete (user) type:')
        print(' ', type_var_to_concrete_type)


resolve_generics(MyTestClass)
# resolve_generics(MyOtherTestClass[str, bytes])

Result:

MyGenericClass
  Generic base:   typing.Generic[~T1, ~T2]
  Generic args:   (~T1, ~T2)
  Concrete args:  (<class 'str'>, <class 'int'>)

MyGenericDict
  Generic base:   typing.Dict[~KT, ~VT]
  Generic args:   (~KT, ~VT)
  Concrete args:  (<class 'float'>, <class 'bool'>)

---
TypeVar to Concrete (user) type:
  {~T1: <class 'str'>, ~T2: <class 'int'>, ~KT: <class 'float'>, ~VT: <class 'bool'>}

[rnag]

By the way, I've looked at adding support for custom/new types, but I'm unclear where to begin. Is this something that can happen in user code or does it need to live in the library? For example, I've been able to implement on own dump logic using DumpMeta in my user code. Works well!

class JSONSnakeWizard(JSONWizard, JSONFileWizard):
    """Helper for JSONWizard that ensures dumping to JSON puts keys in snake_case"""

    def __init_subclass__(cls) -> None:
        """Method for binding child class to DumpMeta"""
        DumpMeta(key_transform="SNAKE").bind_to(cls)

But I could not follow the logic on how to support additional types, or to say, avoid dumping specific types (just leaving as is) and making do with a to_dict

I know it's been a while but it's 2024 and lot of changes have been made, and on the roadmap for V1 is to ensure no key transform in dump process.

Accordingly, I've added a Mixin class JSONPyWizard that does exactly this, and also added a note that this will be the default behavior in V1.

Feel free to follow my announcement on #153 to keep up-to-date on what's expected in V1. Thanks!