Skip to content

Conversation

jerry-024
Copy link
Contributor

@jerry-024 jerry-024 commented Oct 15, 2025

Purpose

Python: fix blob write when blob_as_descriptor is true

Tests

  • blob_test.BlobEndToEndTest.test_blob_end_to_end_with_descriptor

API and Format

Documentation

@jerry-024 jerry-024 changed the title [core]python: fix blob write when blob_as_descriptor is true [core]Python: fix blob write when blob_as_descriptor is true Oct 15, 2025
@leaves12138
Copy link
Contributor

Please add blob-as-descriptor for read as well.

@jerry-024 jerry-024 force-pushed the python_fix_blob_write branch from 021c7ec to 2de55e4 Compare October 16, 2025 03:21
@jerry-024 jerry-024 force-pushed the python_fix_blob_write branch from f74b879 to 1b6dc5b Compare October 16, 2025 06:03
@jerry-024 jerry-024 requested a review from Copilot October 16, 2025 06:12
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Fix blob write/read paths when using blob-as-descriptor in Python.

  • Propagate blob-as-descriptor option through writer and reader.
  • Introduce UriReader abstraction and factory with caching; refactor BlobRef to read via UriReader.
  • Update tests to cover descriptor-based blob IO and add UriReaderFactory tests.

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
paimon-python/pypaimon/write/writer/data_writer.py Pass blob_as_descriptor from options to FileIO.write_blob to control descriptor handling during writes.
paimon-python/pypaimon/common/core_options.py Add FILE_BLOB_AS_DESCRIPTOR config key to toggle descriptor mode.
paimon-python/pypaimon/common/config.py Add default cache size for UriReaderFactory.
paimon-python/pypaimon/common/uri_reader.py New UriReader/Factory with HTTP/file readers and cached construction.
paimon-python/pypaimon/common/file_io.py Add UriReaderFactory usage and change write_blob to support descriptor input; construct Blob via descriptors when needed.
paimon-python/pypaimon/read/split_read.py Thread blob_as_descriptor option through to FormatBlobReader.
paimon-python/pypaimon/read/reader/format_blob_reader.py Support returning descriptors or data based on flag; adapt iterator to construct Blob via FileIO.
paimon-python/pypaimon/table/row/blob.py Refactor Blob* APIs to work with UriReader; BlobRef now reads via UriReader.
paimon-python/pypaimon/tests/blob_test.py Update tests for new APIs and add end-to-end descriptor test; adjust calls to write_blob and readers.
paimon-python/pypaimon/tests/uri_reader_factory_test.py New tests for UriReaderFactory behavior, caching, and basic IO.

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@jerry-024 jerry-024 requested a review from Copilot October 16, 2025 06:30
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@jerry-024 jerry-024 requested a review from Copilot October 16, 2025 06:56
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Copy link
Contributor

@leaves12138 leaves12138 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@leaves12138 leaves12138 merged commit 87bdda4 into apache:master Oct 17, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants