Skip to content

fix: preserve suffix filtering in _path2list when path is a list of S3 paths - #3434

Open
hsusul wants to merge 1 commit into
aws:mainfrom
hsusul:fix/s3-path2list-suffix-list-of-paths
Open

fix: preserve suffix filtering in _path2list when path is a list of S3 paths#3434
hsusul wants to merge 1 commit into
aws:mainfrom
hsusul:fix/s3-path2list-suffix-list-of-paths

Conversation

@hsusul

@hsusul hsusul commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes a bug in _path2list (awswrangler/s3/_list.py) where passing a list of S3 object paths (path=[...]) with a suffix filter (e.g., suffix=".parquet") ignored the suffix filtering whenever ignore_suffix was None (the default).

Root Cause

In _path2list, line 57 filters the input path list according to _suffix into paths.
Line 58 then assigned paths = path if _ignore_suffix is None else .... When _ignore_suffix is None, it assigned paths = path (the raw unfiltered list), discarding the filtered list produced on line 57.

Solution

Change paths = path if _ignore_suffix is None ... to paths = paths if _ignore_suffix is None ... so the result of suffix filtering is preserved.

Tests Added

Added test_path2list_suffix_with_list_of_paths in tests/unit/test_moto.py covering:

  1. suffix filtering on a list of S3 paths with ignore_suffix=None.
  2. Multiple suffixes as a list with a list of S3 paths.
  3. Combination of suffix and ignore_suffix with a list of S3 paths.

Validation Results

  • uv run pytest tests/unit/test_moto.py: 47 passed.
  • uv run ruff format --check: Passed.
  • uv run ruff check: Passed.
  • git diff --check: Passed.
  • Live AWS integration tests were skipped; tests were verified locally using moto.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant