Skip to content

Fix silent partial results in FileIngestor.ingest_directory #1732

Description

@Sameer6305

Summary

FileIngestor.ingest_directory() catches per-file exceptions and continues
when fail_fast=False (the default), but provides no indication to the caller
that one or more files failed.

This can cause downstream consumers to process an incomplete set of files
while appearing to have completed successfully.

Current behavior

When a file fails during directory ingestion:

  • the exception is logged
  • the failed file is omitted from the returned list
  • ingestion continues with remaining files
  • the caller receives no failure information

The existing fail_fast=True behavior remains separate and raises immediately.

Scope

  • make partial directory-ingestion failures visible to callers
  • preserve successful file results
  • preserve existing fail_fast behavior
  • add regression tests for mixed-success ingestion
  • avoid unnecessary breaking API changes

Related

The same error-handling pattern may also exist in cloud ingestion, but this
issue is specifically scoped to FileIngestor.ingest_directory().

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions