Commit cf8f12a
authored
Bumps ballista-core / ballista-executor / ballista-scheduler to spiceai-52.5
tip (07be66a8), which carries:
1. (#42) Wrap ObjectStoreShuffleStorage in object_store::prefix::PrefixStore
so the URL path is reattached to every key — without this, writers
uploaded to s3://bucket/<job>/... while readers looked under
s3://bucket/<prefix>/<job>/... and got NotFound on every reduce stage.
2. (#43) Dispatch s3:// partition paths inside BallistaClient::fetch_partition
to the existing object-store reader. Before this the gRPC FetchPartition
handler called tokio::fs::File::open("s3://...") and failed every
single-batch query (q1) and reduce-stage fetch (q2+).
3. (#43) Replace per-batch serialize_batch_to_ipc_bytes with a long-lived
StreamingMultipartIpcUploader: one StreamWriter per output partition
means the IPC stream has one header and one EOS marker instead of one
stream per batch concatenated together. Fixes the
ArrowError(IpcError("Unexpected EOS")) we saw on multi-batch hash-
repartition queries.
1 parent d01ea81 commit cf8f12a
2 files changed
Lines changed: 22 additions & 22 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
429 | 429 | | |
430 | 430 | | |
431 | 431 | | |
432 | | - | |
433 | | - | |
434 | | - | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
435 | 435 | | |
436 | 436 | | |
437 | 437 | | |
| |||
0 commit comments