Remove the limit of the incremental ingestion of files so that files can arrive out of order #522
Description
Background
Currently, incremental ingestion requires a monotonically increasing field. But for file-based ingestion we can be smarter and ingest new files by comparing file names in metastore for that day with files at the landing area.
Feature
Remove the limit of the incremental ingestion of files so that files can arrive out of order.
Example [Optional]
--
Proposed Solution [Optional]
--