Salesforce has a separate API to the one used for ingestion to provide information on the Data Dictionary.
I believe that before data is ingested for each Salesforce table that a call could be made to the Salesforce Data Dictionary URL to dump as Salesforce data dictionary JSON response. This could be parsed to inform what a schema should look like for the Salesforce Table. I believe the landed data could be enriched as much as possible. It would be nice to include field descriptions so that there is information about the column in the landed table on the target side as well - Note: I haven't examined whether the Singer Spec can supply column and table comments (but that would be super cool).
It is relevant to refer to this issue as well #42 as it would be good exclude compound fields when looking at this issue.
To find out more about the Salesforce Data Dictionary and dumping the Data Dictionary look at this github repo https://github.com/s7clarke10/get-salesforce-data-dictionary.
Salesforce has a separate API to the one used for ingestion to provide information on the Data Dictionary.
I believe that before data is ingested for each Salesforce table that a call could be made to the Salesforce Data Dictionary URL to dump as Salesforce data dictionary JSON response. This could be parsed to inform what a schema should look like for the Salesforce Table. I believe the landed data could be enriched as much as possible. It would be nice to include field descriptions so that there is information about the column in the landed table on the target side as well - Note: I haven't examined whether the Singer Spec can supply column and table comments (but that would be super cool).
It is relevant to refer to this issue as well #42 as it would be good exclude compound fields when looking at this issue.
To find out more about the Salesforce Data Dictionary and dumping the Data Dictionary look at this github repo https://github.com/s7clarke10/get-salesforce-data-dictionary.