Skip to content

Use arrow datasets for intermediates #221

@shreyashankar

Description

@shreyashankar

JSON is a bit bulky. It's easy to view, but for intermediates, especially when using the UI, it does not make sense to store and query the intermediates as JSON.

We should have an option in a docetl config to store the intermediates as arrow datasets. We will then have to query them in the UI accordingly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions