Skip to content

Create a JSON IO Manager #5282

@zaneselvans

Description

@zaneselvans

Overview

  • Currently we generate a number of datapackage.json files as well as some xbrl_taxonomy.json outputs.
  • Writing and reading these files is all managed manually, rather than running it through Dagster, so we're constantly sneaking around on the filesystem and indepednently need to make sure that the files are ready by depending on some other asset that isn't actually what we want.
  • Instead we could create a pretty simple JSON IO Manager that makes Dagster aware of how to write/read these assets and lets us centralize and standardize how we handle them.
  • E.g. some of the "enrichments" that happen at run-time in the new pudl_datapackage asset could just be a standard part of any datapackage.json descriptor that annotates a directory full of parquet files.

Metadata

Metadata

Assignees

No one assigned

    Labels

    dagsterIssues related to our use of the Dagster orchestratormetadataAnything having to do with the content, formatting, or storage of metadata. Mostly datapackages.
    No fields configured for Feature.

    Projects

    Status
    New

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions