Skip to content

Define schema generation strategy for schemaless datasources #235

Description

@GregorioBlazquez

After #176

Schema-based features (e.g. DA validation, schema exposure through the API, metadata persistence) require a well-defined FDA schema.

While this is straightforward for relational datasources, schemaless datasources such as MongoDB require additional design decisions.

This issue aims to define how the FDA schema should be generated and maintained for these datasources, including cases such as:

  • Nested JSON documents.
  • Arrays and complex objects.
  • Fields with heterogeneous types across documents.
  • Values that cannot be represented consistently in a tabular format.

The expected behaviour should be documented and covered by integration tests to ensure that every cached FDA can be materialized consistently as Parquet.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions