-
Notifications
You must be signed in to change notification settings - Fork 316
Open
Labels
efficiencyMaking docetl operations run fasterMaking docetl operations run fastergood first engineering issueEngineering-focused issue for newcomersEngineering-focused issue for newcomers
Description
JSON is a bit bulky. It's easy to view, but for intermediates, especially when using the UI, it does not make sense to store and query the intermediates as JSON.
We should have an option in a docetl config to store the intermediates as arrow datasets. We will then have to query them in the UI accordingly.
plpycoin
Metadata
Metadata
Assignees
Labels
efficiencyMaking docetl operations run fasterMaking docetl operations run fastergood first engineering issueEngineering-focused issue for newcomersEngineering-focused issue for newcomers