[BUG] The new release of dataclasses-json broke flytekit stock installations

[eapolinario]

Describe the bug

The dataclasses-json package released a new version (0.5.12) which dropped the dependency on marshmallow-enum. Unfortunately, flytekit didn't list marshmallow-enum as a dependency, which ended up breaking stock flytekit installations.

Expected behavior

Stock flytekit installations (i.e. from coming pip install flytekit) should work.

Additional context to reproduce

> pip install flytekit
...
> pyflyte init test-sandbox
Traceback (most recent call last):
  File "/private/var/folders/p4/qvgny9x95p5_vhdc7gxh2hsc0000gn/T/tmp.CpjR9RUs/venv/bin/pyflyte", line 5, in <module>
    from flytekit.clis.sdk_in_container.pyflyte import main
  File "/private/var/folders/p4/qvgny9x95p5_vhdc7gxh2hsc0000gn/T/tmp.CpjR9RUs/venv/lib/python3.8/site-packages/flytekit/__init__.py", line 208, in <module>
    from flytekit.core.base_sql_task import SQLTask
  File "/private/var/folders/p4/qvgny9x95p5_vhdc7gxh2hsc0000gn/T/tmp.CpjR9RUs/venv/lib/python3.8/site-packages/flytekit/core/base_sql_task.py", line 4, in <module>
    from flytekit.core.base_task import PythonTask, TaskMetadata
  File "/private/var/folders/p4/qvgny9x95p5_vhdc7gxh2hsc0000gn/T/tmp.CpjR9RUs/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 34, in <module>
    from flytekit.core.interface import Interface, transform_interface_to_typed_interface
  File "/private/var/folders/p4/qvgny9x95p5_vhdc7gxh2hsc0000gn/T/tmp.CpjR9RUs/venv/lib/python3.8/site-packages/flytekit/core/interface.py", line 14, in <module>
    from flytekit.core.type_engine import TypeEngine
  File "/private/var/folders/p4/qvgny9x95p5_vhdc7gxh2hsc0000gn/T/tmp.CpjR9RUs/venv/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 24, in <module>
    from marshmallow_enum import EnumField, LoadDumpOptions
ModuleNotFoundError: No module named 'marshmallow_enum'

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[eapolinario]

The workaround is to pin dataclasses-json<0.5.12.

[fellhorn]

As we wanted to update dataclasses-json, I looked at the current state of this issue and got stuck:

lidatong/dataclasses-json#384 switches to native Enum instead of EnumField from marshmallow-enum. marshmallow-jsonschema though still only supports the deprecated marshmallow-enum, which has been archived since October 2022.

There is an open PR to add native Enum support to marshmallow-jsonschema but the maintainer seems to have quasi abandoned the project and there have not been any updates.

I did test the native Enum support from the PR mentioned above and it looks like we could make it work in flytekit. How should we deal with the quasi unmaintained marshmallow-jsonschema though?