Skip to content

Better way to send resource fields information #22

Open
@hannelita

Description

At this time they are sent via array in the DAG params (e.g. "schema_fields_array": "['field1', 'field2']") and everything is treated as text type.
What are good alternatives?
Questions:

  • Modify this array to become a dictionary of names and types. What is a good way to pass it? I believe doing it in plain text can be tedious. What about adding another node on the DAG to fetch the header of the CSV and automatically create a dictionary of fields, hard-coding everything to text?
  • Should we start considering different types?
  • If yes, define how to treat errors

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestquestionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions