After #176
Schema-based features (e.g. DA validation, schema exposure through the API, metadata persistence) require a well-defined FDA schema.
While this is straightforward for relational datasources, schemaless datasources such as MongoDB require additional design decisions.
This issue aims to define how the FDA schema should be generated and maintained for these datasources, including cases such as:
- Nested JSON documents.
- Arrays and complex objects.
- Fields with heterogeneous types across documents.
- Values that cannot be represented consistently in a tabular format.
The expected behaviour should be documented and covered by integration tests to ensure that every cached FDA can be materialized consistently as Parquet.
After #176
Schema-based features (e.g. DA validation, schema exposure through the API, metadata persistence) require a well-defined FDA schema.
While this is straightforward for relational datasources, schemaless datasources such as MongoDB require additional design decisions.
This issue aims to define how the FDA schema should be generated and maintained for these datasources, including cases such as:
The expected behaviour should be documented and covered by integration tests to ensure that every cached FDA can be materialized consistently as Parquet.