[FEATURE]: Automatic data curation of quarantined data using LLMs

[mwojtyczka]

Problem statement

DQX can quarantine the data but cannot curate it.

Proposed Solution

Certain curation tasks would be possible with LLMs like adjusting timestamp values to the correct format, autofilling missing values based on the data statistics etc.

[mwojtyczka]

Automated solution would be difficult to implement. This should rather be a manual task

[Palkers76]

it may work for date, phone. maybe instead of calling LLM for every value, could take top values formats and generate code to normalize values instead, that normalization functions could also have yaml config ( just an idea )
source_format
target_format

[mwojtyczka]

Yes, not every case will require LLM.