Draft
Conversation
Add wrapper types for task streams, individual tasks, and ArrowReader context. Update scan structs to store file_io/batch_size/concurrency for use by create_reader. Refactor macros to field-agnostic pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds plan_files, create_reader, next_append/delete_task, and read_append/delete_task. Each read clones the ArrowReader to share the CachingDeleteFileLoader cache across concurrent consumers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds three synchronous FFI functions (iceberg_file_scan_task_data_file_path, iceberg_append_task_data_file_path, iceberg_delete_task_data_file_path) that return newly allocated C strings. Julia API exposes task_data_file_path() with multiple dispatch on FileScanTaskHandle, AppendTaskHandle, and DeleteTaskHandle. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds FFI functions to get file path from FileScanTask, AppendTask, and
DeleteTask. Julia functions use distinct names since handle types are
Ptr{Cvoid} aliases and can't dispatch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace type aliases (const X = Ptr{Cvoid}) with proper wrapper structs
for all FFI handle types (ArrowReaderContext, FileScanTaskStream,
FileScanTaskHandle, etc.). This enables Julia multiple dispatch for
functions like task_data_file_path, free_task, and free_task_stream.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add next_tasks/read_tasks (full scan), next_append_tasks/read_append_tasks, and next_delete_tasks/read_delete_tasks (incremental) batch APIs. These allow pulling multiple tasks at once (count=0 drains all) and reading them into a single Arrow stream, reducing FFI round-trips. Rust: IcebergTaskBatchResponse (generic for all task types), iceberg_free_task_ptr_array, plus 6 new FFI functions. Julia: TaskBatchResponse struct, free_task_ptr_array, plus 6 new functions with proper GC.@preserve on pointer arrays. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This reverts commit 4857d08.
Add record_count accessors for FileScanTaskHandle, AppendTaskHandle,
and DeleteTaskHandle. Returns Union{Int64, Nothing} — nothing when
the record count is unavailable (e.g. partial file reads).
Rust: 3 new FFI functions returning i64 (-1 sentinel for None).
Julia: task_record_count with multiple dispatch on wrapper structs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
vustef
commented
Mar 23, 2026
|
|
||
| let mut builder = ArrowReaderBuilder::new(file_io.clone()) | ||
| .with_row_group_filtering_enabled(true) | ||
| .with_row_selection_enabled(false); |
Collaborator
Author
There was a problem hiding this comment.
Why false? Should we have different paths for incremental vs full scan here?
Collaborator
Author
There was a problem hiding this comment.
Actually there's iceberg_create_incremental_reader
vustef
commented
Mar 25, 2026
Collaborator
Author
There was a problem hiding this comment.
Claude also mentioned this:
let op = Operator::new(builder)?.finish();
// Use HTTP/1.1 instead of HTTP/2 for better throughput with many concurrent streams.
// HTTP/2 multiplexes all requests over a single TCP connection, which caps bandwidth.
// HTTP/1.1 opens separate TCP connections per request, allowing full NIC utilization.
op.update_http_client(|_| {
let client = reqwest::ClientBuilder::new()
.http1_only()
.build()
.expect("failed to build http1-only reqwest client");
opendal::raw::HttpClient::with(client)
});
in iceberg-rust repo in s3.rs. I haven't noticed this helps though
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.