-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[core]Python: fix blob write when blob_as_descriptor is true #6404
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Please add blob-as-descriptor for read as well. |
021c7ec
to
2de55e4
Compare
f74b879
to
1b6dc5b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Fix blob write/read paths when using blob-as-descriptor in Python.
- Propagate blob-as-descriptor option through writer and reader.
- Introduce UriReader abstraction and factory with caching; refactor BlobRef to read via UriReader.
- Update tests to cover descriptor-based blob IO and add UriReaderFactory tests.
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.
Show a summary per file
File | Description |
---|---|
paimon-python/pypaimon/write/writer/data_writer.py | Pass blob_as_descriptor from options to FileIO.write_blob to control descriptor handling during writes. |
paimon-python/pypaimon/common/core_options.py | Add FILE_BLOB_AS_DESCRIPTOR config key to toggle descriptor mode. |
paimon-python/pypaimon/common/config.py | Add default cache size for UriReaderFactory. |
paimon-python/pypaimon/common/uri_reader.py | New UriReader/Factory with HTTP/file readers and cached construction. |
paimon-python/pypaimon/common/file_io.py | Add UriReaderFactory usage and change write_blob to support descriptor input; construct Blob via descriptors when needed. |
paimon-python/pypaimon/read/split_read.py | Thread blob_as_descriptor option through to FormatBlobReader. |
paimon-python/pypaimon/read/reader/format_blob_reader.py | Support returning descriptors or data based on flag; adapt iterator to construct Blob via FileIO. |
paimon-python/pypaimon/table/row/blob.py | Refactor Blob* APIs to work with UriReader; BlobRef now reads via UriReader. |
paimon-python/pypaimon/tests/blob_test.py | Update tests for new APIs and add end-to-end descriptor test; adjust calls to write_blob and readers. |
paimon-python/pypaimon/tests/uri_reader_factory_test.py | New tests for UriReaderFactory behavior, caching, and basic IO. |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
Purpose
Python: fix blob write when blob_as_descriptor is true
Tests
API and Format
Documentation