Persist derived FDA metadata in MongoDB

[GregorioBlazquez]

fiware-data-access/src/lib/utils/mongo.js

Line 356 in 1c0fb00

await fdasCollection.insertOne({

Currently, the FDA document only stores its configuration and execution status.

It would be useful to persist metadata derived from the FDA itself (e.g. execution or schema information) in MongoDB so it can be reused by other features without recomputing it.

The goal of this issue is to evaluate which metadata should be stored, how it should be maintained after each refresh, and how it can simplify future (and actual) functionality.

This Issue includes #223

[GregorioBlazquez]

One example of useful metadata would be the resulting FDA schema.

Instead of deriving it every time, we could persist the output schema (column names and types) after the FDA is created or refreshed.

For example:

{
  "fdaId": "animals",
  "schema": [
    {
      "name": "id",
      "type": "INTEGER"
    },
    {
      "name": "species",
      "type": "VARCHAR"
    },
    {
      "name": "created_at",
      "type": "TIMESTAMP"
    }
  ]
}

Having this information readily available would enable features such as:

• Validating DA queries against the FDA schema before execution.
• Exposing schema information through the API.
• Supporting future metadata-driven functionality without inspecting Parquet files or re-executing the FDA query.

[fgalan]

What about other DS different from PG?

In particular, with DS MongoDB we don't have a predefined schema (columns names and types) but a schemalless JSON document.

Maybe the solution is to make schema optional so only FDAs with schema-based DS would have it.

[GregorioBlazquez]

What about other DS different from PG?

In particular, with DS MongoDB we don't have a predefined schema (columns names and types) but a schemalless JSON document.

Maybe the solution is to make schema optional so only FDAs with schema-based DS would have it.

For MongoDB, it would be built differently, but the result should be the same. Because, at the end, every cached FDA must end up as a Parquet file in MinIO, which implies that it must be based on a schema.

The same applies to FDAs from CSVs, when there are multiple sheets we need to merge them in one parquet file (already done in #218).

[GregorioBlazquez]

After PR #236 this Issue can be closed, discussion about schema can continue in #235 and #237