Skip to content

Commit 47362c5

Browse files
authored
feat: Redirect Plug and Mirror Service (#24)
1 parent 9cf467b commit 47362c5

8 files changed

Lines changed: 1011 additions & 8 deletions

File tree

README.md

Lines changed: 44 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -276,9 +276,51 @@ AshStorage ships with:
276276
- `AshStorage.Service.Test` — In-memory storage for tests
277277
- `AshStorage.Service.S3` — S3-compatible storage (requires [`req_s3`](https://hex.pm/packages/req_s3))
278278
- `AshStorage.Service.AzureBlob` — Azure Blob Storage (requires [`req`](https://hex.pm/packages/req))
279+
- `AshStorage.Service.Mirror` — Composite service that fans uploads/deletes out across multiple child services for redundancy
279280

280281
Implement the `AshStorage.Service` behaviour to add custom backends.
281282

283+
### Mirroring across multiple backends
284+
285+
`AshStorage.Service.Mirror` wraps an ordered list of child services. Writes (`upload`, `delete`) fan out sequentially across every child; reads (`download`, `exists?`) consult the primary first and fall through to secondaries on `:not_found`; `url/2` and `direct_upload/2` always go through the primary.
286+
287+
```elixir
288+
storage do
289+
service {AshStorage.Service.Mirror,
290+
services: [
291+
{AshStorage.Service.S3, bucket: "primary"},
292+
{AshStorage.Service.S3, bucket: "backup", region: "eu-west-1"}
293+
]}
294+
end
295+
```
296+
297+
Failures are strict and fail-fast: if any child fails to upload or delete, the operation halts immediately without rolling back work done on earlier children. Orphan cleanup (a separate roadmap item) is responsible for reaping leftovers.
298+
299+
As a shorthand, you can decorate any service tuple with a `:mirrors` option and it expands into a Mirror automatically:
300+
301+
```elixir
302+
storage do
303+
service {AshStorage.Service.S3, [bucket: "primary", mirrors: [
304+
{AshStorage.Service.S3, bucket: "backup", region: "eu-west-1"}
305+
]]}
306+
end
307+
```
308+
309+
is equivalent to writing the Mirror tuple yourself:
310+
311+
```elixir
312+
storage do
313+
service {AshStorage.Service.Mirror, services: [
314+
{AshStorage.Service.S3, bucket: "primary"},
315+
{AshStorage.Service.S3, bucket: "backup", region: "eu-west-1"}
316+
]}
317+
end
318+
```
319+
320+
The expansion happens once at service-resolution time, so the `:mirrors` form works the same way at the resource level, per-attachment (`has_one_attached :avatar, service: {…, mirrors: […]}`), per-attachment for `has_many_attached`, or via app config.
321+
322+
Mirror is configured at runtime via the resource's `storage` DSL or app config; the child services are *not* persisted on the blob row. Synchronous attach/upload/url/download/delete work normally because the live config is in scope. Async paths that rebuild a context purely from `blob.parsed_service_opts` (e.g. AshOban purge jobs) need to re-resolve the Mirror config from app config before invoking the service — calling Mirror with no `:services` raises a clear error.
323+
282324
### Live service integration tests
283325

284326
External service integration tests are excluded from normal `mix test` runs:
@@ -296,8 +338,8 @@ The S3 suite starts MinIO with Docker. The Azure suite starts Azurite with Docke
296338
- ~~**Variants**~~ ✅ — File transformations: image resizing/conversion, PDF-to-thumbnail, video thumbnails, and any custom transform. Subsumes the previewer concept — a PDF thumbnail is just a variant. Three generation modes: `:on_demand` (default, generated inline on first URL request), `:eager` (during attach), `:oban` (background job via AshOban). Variant blobs are self-referential on the blob resource with digest-based cache invalidation. Named variants declared in DSL via `variant :name, {Module, opts}`. Custom transformers implement `AshStorage.Variant` behaviour.
297339
- **Per-variant oban jobs** — Currently all pending variants for a blob run in a single oban job. Refactor so each variant gets its own job lifecycle, enabling parallel generation and independent retries.
298340
- **Checksum verification (partial)**~~ ✅ — Server-side uploads send `Content-MD5` so S3/Azure reject corrupted bodies at the edge; Azure also persists the MD5 via `x-ms-blob-content-md5`. Direct uploads are auto-confirmed by `AttachBlob` against `Service.head/2` before linking. Downloads verified via `Operations.download/2`. Multipart/block-based verification is documented in `documentation/topics/checksum-verification.md` and ships when multipart upload itself does.
299-
- **Redirect handler** — A plug that redirects to the storage service URL instead of proxying
300-
- **Mirroring**Mirror service that replicates uploads across multiple backends for redundancy
341+
- ~~**Redirect handler**~~ ✅ — `AshStorage.Plug.Redirect` issues an HTTP redirect (default 302) to the underlying service's `url/2` instead of streaming bytes. Useful when you want app-level auth/signature checks but don't want to proxy bytes through the application. Supports the same `?token=&expires=` HMAC verification as `AshStorage.Plug.Proxy`, and forwards `?disposition=&filename=` query params into service opts so backends like S3/Azure can encode them into presigned URLs.
342+
- ~~**Mirroring**~~ ✅ — `AshStorage.Service.Mirror` fans `upload`/`delete` out across an ordered list of child services for redundancy. Reads consult the primary first and fall through to secondaries on `:not_found`. `url/2` and `direct_upload/2` go through the primary. Strict, sequential, fail-fast.
301343
- **Orphan cleanup** — Periodic cleanup of blobs without files or files without blobs. With AshOban: scheduled job. Without: manual invocation via `AshStorage.Operations.cleanup_orphans/1`.
302344

303345
### Azure Blob Storage follow-ups

lib/ash_storage/info.ex

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,12 +43,18 @@ defmodule AshStorage.Info do
4343
:many -> :has_many_attached
4444
end
4545

46-
with :error <- fetch_attachment_config(resource, entity_type, attachment.name, :service),
47-
nil <- attachment.service do
48-
Spark.Dsl.Extension.fetch_opt(resource, [:storage], :service, true)
49-
else
50-
{:ok, value} -> {:ok, value}
51-
{mod, opts} when is_atom(mod) -> {:ok, {mod, opts}}
46+
result =
47+
with :error <- fetch_attachment_config(resource, entity_type, attachment.name, :service),
48+
nil <- attachment.service do
49+
Spark.Dsl.Extension.fetch_opt(resource, [:storage], :service, true)
50+
else
51+
{:ok, value} -> {:ok, value}
52+
{mod, opts} when is_atom(mod) -> {:ok, {mod, opts}}
53+
end
54+
55+
case result do
56+
{:ok, tuple} -> {:ok, AshStorage.Service.Mirror.expand_sugar(tuple)}
57+
other -> other
5258
end
5359
end
5460

lib/ash_storage/plug/redirect.ex

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
defmodule AshStorage.Plug.Redirect do
2+
@moduledoc """
3+
A Plug that redirects to the storage service's URL instead of proxying file bytes.
4+
5+
Useful when you want to:
6+
- Apply application-level auth/signature checks before granting access, but
7+
- Avoid the cost of streaming bytes through the application (presigned S3/Azure
8+
URLs let the client fetch directly from the storage service).
9+
10+
Compared to `AshStorage.Plug.Proxy`, this plug never calls `download/2` on
11+
the underlying service. It calls `url/2` and issues an HTTP redirect.
12+
13+
## Usage
14+
15+
In your router:
16+
17+
forward "/storage", AshStorage.Plug.Redirect,
18+
service: {AshStorage.Service.S3, bucket: "my-bucket", region: "us-east-1"}
19+
20+
With signed URL verification (matches `AshStorage.Plug.Proxy`):
21+
22+
forward "/storage", AshStorage.Plug.Redirect,
23+
service: {AshStorage.Service.S3, bucket: "my-bucket"},
24+
secret: "a-long-secret-key"
25+
26+
## Disposition / filename forwarding
27+
28+
When the inbound request includes `?disposition=attachment&filename=foo.pdf`,
29+
those values are merged into the service options before `url/2` is called,
30+
so backends that support response-header overrides (S3
31+
`response-content-disposition`, Azure SAS `rscd`) can encode them into the
32+
generated URL.
33+
34+
## Options
35+
36+
- `:service` - (required) the `{module, opts}` tuple for the storage service.
37+
- `:secret` - secret key for verifying signed app-level URLs. When set,
38+
requests without a valid `token`/`expires` query pair are rejected with 403.
39+
- `:status` - HTTP status to use for the redirect (default: `302`). Use `307`
40+
if method preservation matters for clients/tools that consume the URL.
41+
"""
42+
43+
@behaviour Plug
44+
45+
@impl true
46+
def init(opts) do
47+
{service_mod, service_opts} = Keyword.fetch!(opts, :service)
48+
49+
%{
50+
service_mod: service_mod,
51+
service_opts: service_opts,
52+
secret: Keyword.get(opts, :secret),
53+
status: Keyword.get(opts, :status, 302)
54+
}
55+
end
56+
57+
@impl true
58+
def call(conn, opts) do
59+
key = conn.path_info |> Enum.join("/")
60+
61+
if key == "" do
62+
conn |> Plug.Conn.send_resp(404, "Not Found") |> Plug.Conn.halt()
63+
else
64+
case verify_signature(conn, opts) do
65+
:ok ->
66+
ctx = AshStorage.Service.Context.new(merged_service_opts(conn, opts))
67+
url = opts.service_mod.url(key, ctx)
68+
69+
conn
70+
|> Plug.Conn.put_resp_header("location", url)
71+
|> Plug.Conn.put_resp_header("cache-control", "no-store, private")
72+
|> Plug.Conn.send_resp(opts.status, "")
73+
|> Plug.Conn.halt()
74+
75+
{:error, :forbidden} ->
76+
conn |> Plug.Conn.send_resp(403, "Forbidden") |> Plug.Conn.halt()
77+
end
78+
end
79+
end
80+
81+
defp merged_service_opts(conn, opts) do
82+
params = Plug.Conn.fetch_query_params(conn).query_params
83+
84+
opts.service_opts
85+
|> maybe_put(:disposition, params["disposition"])
86+
|> maybe_put(:filename, params["filename"])
87+
end
88+
89+
defp maybe_put(keyword, _key, nil), do: keyword
90+
defp maybe_put(keyword, _key, ""), do: keyword
91+
defp maybe_put(keyword, key, value), do: Keyword.put(keyword, key, value)
92+
93+
defp verify_signature(_conn, %{secret: nil}), do: :ok
94+
95+
defp verify_signature(conn, %{secret: secret}) do
96+
params = Plug.Conn.fetch_query_params(conn).query_params
97+
98+
with token when is_binary(token) <- params["token"],
99+
expires when is_binary(expires) <- params["expires"],
100+
{expires_at, ""} <- Integer.parse(expires),
101+
true <- expires_at > System.system_time(:second) do
102+
key = conn.path_info |> Enum.join("/")
103+
expected = AshStorage.Token.sign(secret, key, expires_at)
104+
105+
if Plug.Crypto.secure_compare(token, expected) do
106+
:ok
107+
else
108+
{:error, :forbidden}
109+
end
110+
else
111+
_ -> {:error, :forbidden}
112+
end
113+
end
114+
end

0 commit comments

Comments
 (0)