Add custom fetch option to FetchStore (#388)

Closes #325, #288.

`FetchStore` previously only exposed an `overrides` option for setting
default `RequestInit` properties, which made it difficult to intercept
requests for things like URL presigning or status code remapping.

These changes add a `fetch` option that receives a standard
WinterTC-style fetch handler (`Request` in, `Response` out), similar to
Cloudflare Workers, Deno.serve, and Bun.serve. Internally the store now
builds Request objects and routes them through the custom handler (or
global fetch).

```ts
const store = new FetchStore("https://my-bucket.s3.amazonaws.com/data.zarr", {
  async fetch(request) {
    const newUrl = await presign(request.url);
    return fetch(new Request(newUrl, request));
  },
});
```

The `overrides` option is deprecated but still works and is merged into
the Request before it reaches the handler. Docs call out that sharding
and partial reads depend on Range headers, so transforming a request
should use `new Request(newUrl, original)` to preserve headers, abort
signals, and other options from `store.get(key, init)`.

Author: manzt

.changeset/fetch-store-custom-fetch.md

@@ -0,0 +1,60 @@
+---
+"@zarrita/storage": minor
+---
+
+Add custom `fetch` option to `FetchStore`. Accepts a WinterTC-style fetch handler ([`Request`](https://developer.mozilla.org/en-US/docs/Web/API/Request) in, [`Response`](https://developer.mozilla.org/en-US/docs/Web/API/Response) out) to cover the long tail of things users need at the fetch level. Deprecates `overrides`.
+
+Presigning a URL:
+
+```ts
+const store = new FetchStore("https://my-bucket.s3.amazonaws.com/data.zarr", {
+  async fetch(request) {
+    const newUrl = await presign(request.url);
+    return fetch(new Request(newUrl, request));
+  },
+});
+```
+
+Remapping response status codes:
+
+```ts
+const store = new FetchStore("https://my-bucket.s3.amazonaws.com/data.zarr", {
+  async fetch(request) {
+    const response = await fetch(request);
+    if (response.status === 403) {
+      return new Response(null, { status: 404 });
+    }
+    return response;
+  },
+});
+```
+
+#### Migrating from `overrides`
+
+`overrides` only supported static [`RequestInit`](https://developer.mozilla.org/en-US/docs/Web/API/RequestInit) properties. For anything dynamic (like refreshing a token), you had to thread it through every call site:
+
+```ts
+const store = new FetchStore("https://example.com/data.zarr");
+const arr = await zarr.open(store);
+
+// token logic leaks into every get call
+let chunk = await zarr.get(arr, null, {
+  opts: { headers: { Authorization: `Bearer ${await getAccessToken()}` } },
+});
+```
+
+With `fetch`, auth is configured once on the store and every request picks it up:
+
+```ts
+const store = new FetchStore("https://example.com/data.zarr", {
+  async fetch(request) {
+    const token = await getAccessToken();
+    request.headers.set("Authorization", `Bearer ${token}`);
+    return fetch(request);
+  },
+});
+const arr = await zarr.open(store);
+
+// call sites don't need to know about auth
+let chunk = await zarr.get(arr);
+```

docs/packages/storage.md

@@ -61,16 +61,80 @@ import { FetchStore } from "@zarrita/storage";
 const store = new FetchStore("http://localhost:8080/data.zarr");
 ```
 
-#### Default Fetch Options
+#### Response handling
 
-You can specify default fetch options using the `overrides` parameter when
-initializing the `FetchStore`. These act as base configurations for every fetch
-request:
+The store interprets HTTP responses as follows:
+
+| Status        | Meaning                                    |
+| ------------- | ------------------------------------------ |
+| **404**       | Missing key — returns `undefined`          |
+| **200 / 206** | Success — body is read as `Uint8Array`     |
+| **Any other** | Throws an error                            |
+
+If your backend returns different status codes for missing keys (e.g., S3
+returns **403** for missing keys on private buckets), you can remap them with a
+custom `fetch` (see below).
+
+#### Custom `fetch`
+
+You can provide a custom `fetch` function to intercept every request made by the
+store. It receives a standard
+[WinterTC fetch handler](https://github.com/nicolo-ribaudo/tc55-proposal-functions-api),
+similar to Cloudflare Workers, Deno.serve, and Bun.serve — a
+[`Request`](https://developer.mozilla.org/en-US/docs/Web/API/Request) in, a
+[`Response`](https://developer.mozilla.org/en-US/docs/Web/API/Response) out.
+
+This is the recommended way to add authentication, presign URLs, or transform
+requests.
+
+::: warning Important
+Sharding and partial reads rely on `Range` headers in the request. When
+transforming a request (e.g., changing the URL), always use
+`new Request(newUrl, originalRequest)` to preserve headers, abort signals, and
+other options passed via `store.get(key, init)`.
+:::
 
 ```javascript
-const fetchOptions = { headers: { Authorization: "XXXXX" } };
-const store = new FetchStore("http://localhost:8080/data.zarr", {
-	overrides: fetchOptions,
+const store = new FetchStore("https://my-bucket.s3.amazonaws.com/data.zarr", {
+	async fetch(request) {
+		const newUrl = await presign(request.url);
+		// Preserves headers, abort signal, and other options from store.get(key, init)
+		return fetch(new Request(newUrl, request));
+	},
+});
+```
+
+##### Adding authentication headers
+
+You can set headers on the request directly, which replaces the need for the
+deprecated `overrides` option. Since the handler runs on every request, you can
+also dynamically refresh credentials:
+
+```javascript
+const store = new FetchStore("https://example.com/data.zarr", {
+	async fetch(request) {
+		const token = await getAccessToken();
+		request.headers.set("Authorization", `Bearer ${token}`);
+		return fetch(request);
+	},
+});
+```
+
+##### Handling S3 403 responses
+
+S3 returns **403** (not 404) for missing keys on private buckets, which causes
+the store to throw. If you know that 403 means "not found" in your setup, remap
+it:
+
+```javascript
+const store = new FetchStore("https://my-bucket.s3.amazonaws.com/data.zarr", {
+	async fetch(request) {
+		const response = await fetch(request);
+		if (response.status === 403) {
+			return new Response(null, { status: 404 });
+		}
+		return response;
+	},
 });
 ```
 
@@ -88,8 +152,10 @@ const bytes = await store.get("/zarr.json", {
 });
 ```
 
-These options override the defaults for the store, except headers are merged
-(with the latter taking precedent).
+These per-request options are merged into the `Request` passed to your custom
+`fetch` (or the global `fetch` if none is provided). Headers are merged, with
+per-request headers taking precedence.
+
 
 ### FileSystemStore <Badge type="tip" text="Readable" /> <Badge type="tip" text="Writable" />
 

packages/@zarrita-storage/__tests__/fetch.test.ts

@@ -1,4 +1,4 @@
-import { afterEach, describe, expect, it, vi } from "vitest";
+import { afterEach, assert, describe, expect, it, vi } from "vitest";
 
 import FetchStore from "../src/fetch.js";
 
@@ -117,15 +117,21 @@ describe("FetchStore", () => {
 		let store = new FetchStore(href);
 		let spy = vi.spyOn(globalThis, "fetch");
 		await store.get("/zarr.json", { headers });
-		expect(spy).toHaveBeenCalledWith(`${href}/zarr.json`, { headers });
+		let request = spy.mock.calls[0][0];
+		assert(request instanceof Request);
+		expect(request.url).toBe(`${href}/zarr.json`);
+		expect(request.headers.get("x-test")).toBe("test");
 	});
 
 	it("forwards request options to fetch when configured globally", async () => {
 		let headers = { "x-test": "test" };
 		let store = new FetchStore(href, { overrides: { headers } });
 		let spy = vi.spyOn(globalThis, "fetch");
 		await store.get("/zarr.json");
-		expect(spy).toHaveBeenCalledWith(`${href}/zarr.json`, { headers });
+		let request = spy.mock.calls[0][0];
+		assert(request instanceof Request);
+		expect(request.url).toBe(`${href}/zarr.json`);
+		expect(request.headers.get("x-test")).toBe("test");
 	});
 
 	it("merges request options", async () => {
@@ -136,10 +142,12 @@ describe("FetchStore", () => {
 		let store = new FetchStore(href, { overrides });
 		let spy = vi.spyOn(globalThis, "fetch");
 		await store.get("/zarr.json", { headers: { "x-test": "override" } });
-		expect(spy).toHaveBeenCalledWith(`${href}/zarr.json`, {
-			headers: { "x-test": "override", "x-test2": "root" },
-			cache: "no-cache",
-		});
+		let request = spy.mock.calls[0][0];
+		assert(request instanceof Request);
+		expect(request.url).toBe(`${href}/zarr.json`);
+		expect(request.headers.get("x-test")).toBe("override");
+		expect(request.headers.get("x-test2")).toBe("root");
+		expect(request.cache).toBe("no-cache");
 	});
 
 	it("reads partial - suffixLength", async () => {
@@ -165,4 +173,108 @@ describe("FetchStore", () => {
 		`,
 		);
 	});
+
+	describe("custom fetch", () => {
+		it("uses custom fetch for get requests", async () => {
+			let customFetch = vi.fn((request: Request) => fetch(request));
+			let store = new FetchStore(href, { fetch: customFetch });
+			let bytes = await store.get("/zarr.json");
+			expect(bytes).toBeInstanceOf(Uint8Array);
+			expect(customFetch).toHaveBeenCalledOnce();
+			let request = customFetch.mock.calls[0][0];
+			expect(request).toBeInstanceOf(Request);
+			expect(request.url).toBe(`${href}/zarr.json`);
+		});
+
+		it("uses custom fetch for getRange requests", async () => {
+			let customFetch = vi.fn((request: Request) => fetch(request));
+			let store = new FetchStore(href, { fetch: customFetch });
+			let bytes = await store.getRange("/zarr.json", {
+				offset: 4,
+				length: 50,
+			});
+			expect(bytes).toBeInstanceOf(Uint8Array);
+			expect(customFetch).toHaveBeenCalledOnce();
+			let request = customFetch.mock.calls[0][0];
+			expect(request.headers.get("Range")).toBe("bytes=4-53");
+		});
+
+		it("allows intercepting and modifying the request", async () => {
+			let store = new FetchStore(href, {
+				async fetch(request) {
+					// Add a custom header before sending
+					let modified = new Request(request, {
+						headers: {
+							...Object.fromEntries(request.headers),
+							"x-custom": "test",
+						},
+					});
+					return fetch(modified);
+				},
+			});
+			let bytes = await store.get("/zarr.json");
+			expect(bytes).toBeInstanceOf(Uint8Array);
+		});
+
+		it("allows remapping status codes", async () => {
+			let store = new FetchStore("http://localhost:51204/does-not-exist", {
+				async fetch(request) {
+					let response = await fetch(request);
+					// Remap any error to 404 (simulates S3 403 → 404 pattern)
+					if (!response.ok && response.status !== 404) {
+						return new Response(null, { status: 404 });
+					}
+					return response;
+				},
+			});
+			let bytes = await store.get("/zarr.json");
+			expect(bytes).toBeUndefined();
+		});
+
+		it("receives merged overrides in the request", async () => {
+			let customFetch = vi.fn((request: Request) => fetch(request));
+			let store = new FetchStore(href, {
+				fetch: customFetch,
+				overrides: { headers: { "x-base": "base" } },
+			});
+			await store.get("/zarr.json", {
+				headers: { "x-request": "request" },
+			});
+			let request = customFetch.mock.calls[0][0];
+			expect(request.headers.get("x-base")).toBe("base");
+			expect(request.headers.get("x-request")).toBe("request");
+		});
+
+		it("allows setting headers on the request directly", async () => {
+			let spy = vi.spyOn(globalThis, "fetch");
+			let store = new FetchStore(href, {
+				async fetch(request) {
+					request.headers.set("Authorization", "Bearer test-token");
+					return fetch(request);
+				},
+			});
+			await store.get("/zarr.json");
+			let request = spy.mock.calls[0][0];
+			assert(request instanceof Request);
+			expect(request.headers.get("Authorization")).toBe("Bearer test-token");
+		});
+
+		it("abort signal survives request transformation", async () => {
+			let controller = new AbortController();
+			let store = new FetchStore(href, {
+				async fetch(request) {
+					// Simulate presigning: clone request with a new URL
+					let transformed = new Request(request.url, request);
+					expect(transformed.signal).toBe(request.signal);
+					expect(transformed.signal.aborted).toBe(false);
+					controller.abort();
+					expect(transformed.signal.aborted).toBe(true);
+					throw transformed.signal.reason;
+				},
+			});
+			await expect(
+				store.get("/zarr.json", { signal: controller.signal }),
+			).rejects.toThrow();
+		});
+	});
 });