Skip to content

Commit 06087c4

Browse files
committed
fix: AWS-CORS-safe byteLength discovery in asyncBufferFromStore
AWS S3 (and many other backends) strip `Content-Range` from CORS responses by default. The old 1-byte range trick then fell back to the partial `Content-Length` of 1, so hyparquet saw `byteLength = 1`, tried to read the 8-byte footer, and tripped `RangeError: Offset is outside the bounds of the DataView` on any public-S3-backed parquet. Two changes: - `asyncBufferFromStore`: when `store.getUrl` is defined, do an HTTP HEAD first — its `Content-Length` is the full object size (200, not 206) and survives CORS. Fall back to the 1-byte trick for stores without `getUrl` (native R2 binding, HTTP proxies that expose `Content-Range`). - `S3Store.get`: don't trust `Content-Length` as `totalSize` on a 206 response. Only set `totalSize` from `Content-Length` for 200s, or from a present `Content-Range`. Avoids the same misread bleeding into callers that don't use `asyncBufferFromStore`. Reproed on `ctbk/stations/ids_201306:202209.parquet` (88 KB, 3257 rows); now renders the full table.
1 parent 29f7829 commit 06087c4

2 files changed

Lines changed: 29 additions & 5 deletions

File tree

src/react/asyncBuffer.ts

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,28 @@ export interface AsyncBuffer {
1818
}
1919

2020
export async function asyncBufferFromStore(store: Store, path: string): Promise<AsyncBuffer> {
21-
// Need `byteLength` upfront. A 1-byte range read returns `totalSize`
22-
// from Content-Range (cheaper than fetching the whole object).
23-
const head = await store.get(path, { offset: 0, length: 1 })
24-
const byteLength = head.totalSize ?? head.bytes.byteLength
21+
// Prefer an HTTP HEAD via `store.getUrl(path)` when available. AWS S3
22+
// (and many other backends) strip `Content-Range` from CORS responses
23+
// by default, which makes the 1-byte range trick fall back to a
24+
// partial `Content-Length` of 1 — wrong, and hyparquet later trips
25+
// `RangeError: Offset is outside the bounds of the DataView`.
26+
// HEAD returns a 200 whose `Content-Length` is the full object size.
27+
let byteLength: number | undefined
28+
if (typeof store.getUrl === 'function') {
29+
try {
30+
const r = await fetch(store.getUrl(path), { method: 'HEAD' })
31+
if (r.ok) {
32+
const cl = parseInt(r.headers.get('Content-Length') ?? '', 10)
33+
if (Number.isFinite(cl) && cl > 0) byteLength = cl
34+
}
35+
} catch { /* HEAD blocked / unsupported — fall through */ }
36+
}
37+
if (byteLength === undefined) {
38+
// Fallback: 1-byte range read. Works for native bindings (R2Store)
39+
// and HTTP proxies that expose `Content-Range`.
40+
const head = await store.get(path, { offset: 0, length: 1 })
41+
byteLength = head.totalSize ?? head.bytes.byteLength
42+
}
2543
return {
2644
byteLength,
2745
async slice(start: number, end?: number): Promise<ArrayBuffer> {

src/stores/s3.ts

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -216,9 +216,15 @@ export function S3Store(opts: S3StoreOptions): Store {
216216
throw new Error(`S3 get ${path}: ${res.status} ${await res.text()}`)
217217
}
218218
const cr = res.headers.get('Content-Range')
219+
// 206 + no `Content-Range` (common: AWS S3 default CORS strips
220+
// it) means we can't know the total size from this response —
221+
// `Content-Length` is the partial length, not the full file's.
222+
// Only trust `Content-Length` for 200 responses.
219223
const totalSize = cr
220224
? parseInt(cr.split('/')[1], 10)
221-
: parseInt(res.headers.get('Content-Length') ?? '', 10)
225+
: res.status === 200
226+
? parseInt(res.headers.get('Content-Length') ?? '', 10)
227+
: NaN
222228
const bytes = new Uint8Array(await res.arrayBuffer())
223229
const out: GetResult = { bytes }
224230
if (Number.isFinite(totalSize)) out.totalSize = totalSize

0 commit comments

Comments
 (0)