With the ongoing Iceberg implementation #252 #258 , we will need a function that exports the finalized dataframe into parquet files so they can be appended to the iceberg table.
I am wondering however upon normal circumstances going forward i.e. (at the stage where the tables have already been backfilled) would it be useful to have a flag or a function that stops the generation of the downstream files - i.e., .csv, .pdf etc? Because during the backfilling process I will not be doing all the downstream file creation because that process is quite slow, especially pdf generation.
@knikolla @QuanMPhm
With the ongoing Iceberg implementation #252 #258 , we will need a function that exports the finalized dataframe into parquet files so they can be appended to the iceberg table.
I am wondering however upon normal circumstances going forward i.e. (at the stage where the tables have already been backfilled) would it be useful to have a flag or a function that stops the generation of the downstream files - i.e.,
.csv,.pdfetc? Because during the backfilling process I will not be doing all the downstream file creation because that process is quite slow, especially pdf generation.@knikolla @QuanMPhm