Skip to content

An object storage prefix is listed as written, so a folder named "Case #12" syncs its files - #1421

Open
rudycelekli wants to merge 2 commits into
deeplethe:devfrom
rudycelekli:object-storage-prefix-raw
Open

rudycelekli wants to merge 2 commits into
deeplethe:devfrom
rudycelekli:object-storage-prefix-raw

Conversation

@rudycelekli

@rudycelekli rudycelekli commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Why

An S3, Azure Blob or GCS source can be limited to a prefix. fetch turns that prefix into an object_store path with Path::from. That constructor percent-encodes every character object_store considers unsafe in a path segment, including #, [, ], %, ~, {, }, |, ? and *.

The bucket's keys are not encoded. A folder named Case #12 is listed as Case %2312/, which matches nothing. The sync then reports success with zero files, which is the hardest failure for a user to notice. Listed object locations are already read raw (Path::parse on each key), so only the prefix is affected.

What changes

The prefix is read with Path::parse, which keeps the characters as written. It still strips leading and trailing slashes. If parsing fails, for example on an empty segment like a//b, the code falls back to Path::from, so such prefixes behave exactly as before.

How it was checked

A new test, a_prefix_with_symbols_lists_its_objects, puts three objects into an in-memory store and syncs two prefixes:

  • Case #12 lists s3://b/Case #12/brief.txt.
  • Q1 [final]/ lists s3://b/Q1 [final]/plan.md.

Results:

  • cargo test -p utopia-server --bins object_storage: 7 passed.
  • With the fix reverted, the test fails on Case #12 with left: [], right: ["s3://b/Case #12/brief.txt"].
  • cargo fmt --all --check and cargo clippy -p utopia-server --all-targets -- -D warnings pass.

Before review

  • Every commit is signed off (git commit -s)
  • cargo fmt --all --check, cargo clippy and cargo test pass for the touched crate

This also fixes non-ASCII prefixes. Path::from percent-encodes every non-ASCII byte, so on dev a Chinese folder prefix such as 合同/2024 lists no objects either. The test now covers that case.

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is right, and it fixes more than the title says: Path::from percent-encodes every non-ASCII byte, so a Chinese folder prefix lists nothing on dev either. Please add a case like 合同/2024 to the test and mention it in the description. Then it's ready.

…e #12" syncs its files

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
…rom also escaped non-ASCII prefixes

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
@rudycelekli
rudycelekli force-pushed the object-storage-prefix-raw branch from 0dad50d to 8ef820f Compare November 5, 2026 03:10
@rudycelekli

Copy link
Copy Markdown
Contributor Author

Added a 合同/2024 case to a_prefix_with_symbols_lists_its_objects (it lists nothing with Path::from and passes with the fix), and rebased on dev.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants