Summary
FileIngestor.ingest_directory() catches per-file exceptions and continues
when fail_fast=False (the default), but provides no indication to the caller
that one or more files failed.
This can cause downstream consumers to process an incomplete set of files
while appearing to have completed successfully.
Current behavior
When a file fails during directory ingestion:
- the exception is logged
- the failed file is omitted from the returned list
- ingestion continues with remaining files
- the caller receives no failure information
The existing fail_fast=True behavior remains separate and raises immediately.
Scope
- make partial directory-ingestion failures visible to callers
- preserve successful file results
- preserve existing
fail_fast behavior
- add regression tests for mixed-success ingestion
- avoid unnecessary breaking API changes
Related
The same error-handling pattern may also exist in cloud ingestion, but this
issue is specifically scoped to FileIngestor.ingest_directory().
Summary
FileIngestor.ingest_directory()catches per-file exceptions and continueswhen
fail_fast=False(the default), but provides no indication to the callerthat one or more files failed.
This can cause downstream consumers to process an incomplete set of files
while appearing to have completed successfully.
Current behavior
When a file fails during directory ingestion:
The existing
fail_fast=Truebehavior remains separate and raises immediately.Scope
fail_fastbehaviorRelated
The same error-handling pattern may also exist in cloud ingestion, but this
issue is specifically scoped to
FileIngestor.ingest_directory().