GIPHY Bandwidth Saver

GIPHY Bandwidth Saver is an MIT-licensed reference implementation for a revalidating GIPHY media cache. It includes a local website/API proxy so you can demo the behavior against the live GIPHY website, plus conformance harnesses that other implementations can adapt to prove request accounting, revalidation, replacement, deletion, and range behavior.

This repository is intentionally not a drop-in production proxy, not a login proxy, not a crawler, and not an offline mirror. The reusable idea is narrower: keep GIPHY in the request path for freshness and removal semantics while avoiding redundant media-byte downloads when the same GIF rendition has already been observed.

Motivation

High-volume GIF integrations can show the same popular GIF rendition to thousands or tens of thousands of users in a short window. Those users do not need the same image bytes retransmitted ten thousand times in a minute, but GIPHY still needs request traffic that can prove whether the asset is current, allow fast takedown, and provide signal about which GIFs should appear for specific searches and shares.

This is where a revalidating cache is different from a mirror. The cache can store the bytes under a durable media key, but the proxy still sends GIPHY the current media URL path and query string when it revalidates. That revalidation URL includes the URL fingerprint, GIF ID, rendition name such as giphy, 200, or 200w, file type such as gif, webp, or mp4, and any query-string context that came with the load. GIPHY can return 200, 304, 404, or 410 based on that current request, and the proxy applies the result to the normalized cache key for the underlying bytes.

This project demonstrates a corporate-friendly middle ground:

• The browser still requests GIPHY web and media URLs through a local proxy.
• Each previously seen media object is revalidated against the current upstream media URL using its stored ETag and Last-Modified metadata.
• A 304 Not Modified response from GIPHY lets the proxy serve the local disk copy and count the avoided transfer.
• 404 and 410 responses mark local items as removed without serving stale bytes to that request; recently cached bytes are lazily deleted after a short transition window.
• GIF IDs, renditions, and media types are normalized for operational reporting without creating a user identity layer.

The result complies with the spirit of origin-controlled media delivery: GIPHY continues to receive validation traffic with enough request context to inform popularity-driven ranking and removal decisions, while consumers avoid unnecessary bandwidth spend and preserve user anonymity.

Demo Quick Start

This project uses pnpm and Node 24.

nvm use
pnpm install
pnpm tls:mkcert
pnpm dev

Then open:

https://web.localhost:8080

This starts the local website proxy for giphy.com, rewrites page/API/media URLs to local hostnames, and routes media requests through the revalidating media cache. Scroll the proxied GIPHY homepage to load GIFs, then scroll back through already-seen media while watching the TUI. Repeated media should still be revalidated upstream, but upstream 304 responses should let the proxy serve local bytes and count avoided downloads.

The server listens on 127.0.0.1 and uses local HTTPS with HTTP/2 by default. That default matters for the demo: without HTTP/2, the browser-to-proxy side no longer behaves like the CDN path and media loading looks much slower than the cache design actually is. You can override runtime paths with:

PORT=8081 CACHE_DIR=cache pnpm dev

To reset media files and SQLite metadata without regenerating the local HTTPS certificate:

pnpm cache:clear

Stop the proxy before clearing the cache so SQLite and in-flight disk writes are not active.

For non-interactive smoke testing or supervised processes, disable the TUI:

TUI=0 pnpm dev

The first run generates a short-lived self-signed certificate under cache/tls/ with SANs for web.localhost, api.localhost, media.localhost, media0.localhost, media1.localhost, media2.localhost, media3.localhost, media4.localhost, localhost, and configured web-host aliases. The browser may require accepting the certificate risk before it can load the demo. In the default HTTP/2 mode, rewritten API and media URLs keep their local hostnames, such as https://api.localhost:8080/... and https://media2.localhost:8080/.... That makes the demo closer to GIPHY's real multi-host shape while still allowing browser-to-proxy HTTP/2 for each local origin once the certificate is trusted.

For browser cache behavior that matches production HTTPS, use a locally trusted certificate. With mkcert installed:

pnpm tls:mkcert
pnpm dev

The helper installs the local mkcert CA if needed and writes cache/tls/mkcert.cert.pem plus cache/tls/mkcert.key.pem. HTTP/2 mode automatically prefers those files. You can also provide your own trusted certificate:

TLS_CERT_FILE=/path/cert.pem TLS_KEY_FILE=/path/key.pem pnpm dev

Browsers may refuse to put media responses into the HTTP cache when the page is loaded through an untrusted certificate, even if the interstitial warning was accepted. If the address bar still shows a certificate error, expect the proxy's disk cache to work while the browser's own disk cache remains mostly unused.

If your browser prefers the machine .local name, make that the primary rewritten origin when creating the certificate and starting the proxy:

WEB_HOST=<your-machine>.local pnpm tls:mkcert
WEB_HOST=<your-machine>.local pnpm dev

The proxy also accepts web.localhost and the machine .local names as web-host aliases, but the primary WEB_HOST is what rewritten API and media URLs will use.

Use HTTP/1.1 only for debugging fallback behavior:

HTTP2=0 pnpm dev

Then open http://web.localhost:8080. This mode is not the recommended demo path because the browser loses local HTTP/2 multiplexing and will overstate page-load slowness.

There is also an optional same-host tunnel mode:

COLLAPSE_PROXY_HOSTS=1 pnpm dev

That rewrites API and media URLs onto https://web.localhost:8080/__giphy_api/... and https://web.localhost:8080/__giphy_media/[media-host]/.... It is useful when debugging one-origin certificate behavior, but it is not the default because it hides the separate API and media hostnames behind path prefixes.

Local Hostnames

The proxy uses .localhost hostnames so the demo can run without editing /etc/hosts on modern browsers and operating systems.

Local host	Upstream host
web.localhost	giphy.com
api.localhost	api.giphy.com
media.localhost	media.giphy.com
media0.localhost	media0.giphy.com
media1.localhost	media1.giphy.com
media2.localhost	media2.giphy.com
media3.localhost	media3.giphy.com
media4.localhost	media4.giphy.com

The web and API proxies rewrite GIPHY host strings based on response content type for HTML, JSON, JavaScript, and React Server Component (text/x-component) responses. That includes bare host references such as api.giphy.com, full trending and search API URLs such as /v1/gifs/trending and /v1/gifs/search, and escaped URL strings embedded in serialized script payloads. JavaScript and RSC rewriting are included because the current GIPHY web app hydrates into client-side and streamed component payloads that can construct media and API URLs after the initial HTML load. Rewritten responses are sent back with gzip compression and cache-control: no-store so browser caches do not keep stale, unrewritten chunks during demos.

Code Boundaries

The demo-only website/API rewriting code is intentionally isolated from the media cache. This keeps the reusable bandwidth-saving reference path obvious.

Demo proxy files:

• src/demo-proxy.ts proxies web.localhost and api.localhost.
• src/demo-rewriter.ts rewrites HTML, JSON, JavaScript, and RSC host strings for the browser demo.

Media cache files:

• src/media-cache-proxy.ts handles media host requests, conditional revalidation, disk writes, deletion handling, and cached responses.
• src/media-cache-key.ts normalizes media and avatar URLs into cache keys.
• src/cache-db.ts owns the SQLite metadata store.

Shared infrastructure:

• src/proxy.ts only routes local hosts to the demo proxy or media cache proxy.
• src/http-proxy-utils.ts contains neutral HTTP proxy helpers used by both paths, including the explicit upstream HTTPS keep-alive agent.
• src/transfer-size.ts contains display formatting only.

Architecture tests assert that media cache modules do not import demo rewrite helpers and demo rewrite modules do not import cache storage.

For another implementation, the media-cache contract is the part to study first. The website/API proxy is scaffolding for making the reference implementation easy to run in a browser.

Upstream Networking

All upstream GIPHY calls use an explicit HTTPS keep-alive agent instead of relying on Node's global defaults. The agent keeps HTTP/1.1 connections reusable across requests, caps socket fan-out, and keeps a bounded pool of idle sockets:

• keepAlive: true
• maxSockets: 64 per upstream origin
• maxTotalSockets: 256 across all upstream origins
• maxFreeSockets: 16 per origin
• timeout: 30s

The local demo speaks HTTPS with HTTP/2 ALPN by default, with HTTP/1 fallback via Node's http2.createSecureServer({ allowHTTP1: true }). This gets browser-to-proxy multiplexing back for the demo, subject to using a browser-accepted local certificate. By default, API and media URLs are rewritten to their corresponding local hostnames (api.localhost, media.localhost, media0.localhost through media4.localhost). Upstream GIPHY calls still use HTTP/1.1 keep-alive. HTTP2=0 starts the older HTTP/1.1-only fallback mode for debugging, and COLLAPSE_PROXY_HOSTS=1 starts the optional same-host tunnel mode; neither is representative of the default browser experience.

Uncached media 200 responses are streamed to the browser while they are written to a temporary cache file. SQLite metadata is updated only after the temp file is complete and atomically moved into place.

Media Cache Design

The media proxy focuses on URLs shaped like:

https://media2.giphy.com/media/v1.Y2lkPTc5MGI3NjExM3UxbGI4dTUzaGxvdWk4aXB6MHp4cGFkMDNoMHhrNjB2MHYwdzhkZCZlcD12MV9naWZzX3RyZW5kaW5nJmN0PWc/WRQBXSCnEFJIuxktnw/200.webp

That request is normalized to this cache key:

/media/WRQBXSCnEFJIuxktnw/200.webp

The URL fingerprint and source media host are intentionally excluded from the disk cache key because they are not part of the byte identity. The GIF ID, rendition, and extension are the durable identity. Extensions are normalized to lowercase, so 200.GIF, 200.gif, and 200.Gif are treated as the same rendition/type.

The fingerprint still matters for the upstream request. A GIF loaded from a search result, a channel page, or a share can carry different path/query context even when the bytes are identical. The proxy should pass the full observed upstream media path and query string through to GIPHY during revalidation, then apply the resulting 200, 304, 404, or 410 to the normalized cache key. Some GIPHY API keys return modern path fingerprints like /media/v1.../<gif>/<rendition>, while others return older /media/<gif>/<rendition>?cid=...&ep=...&rid=...&ct=... URLs; the proxy preserves both the path and query string because either can carry upstream context.

The cache parser also supports hydrated client media URLs without the fingerprint segment, such as:

/media/WRQBXSCnEFJIuxktnw/200.webp

The TUI reports incoming media requests with path fingerprints as good and incoming media requests without path fingerprints as bad. A bad count means the browser is loading older hydrated media URL shapes; those requests still work, and any query context is preserved upstream, but they are less explicit than the current path-fingerprint URLs.

Avatar media URLs are cacheable as well:

https://media3.giphy.com/avatars/sarah-weber/kAP2xBeRrhQp/80h.gif

That request is normalized to:

/avatars/sarah-weber/kAP2xBeRrhQp/80h.gif

Disk files are stored under:

cache/[2-character-gif-prefix]/[GIFID]--[rendition].[type]

Metadata is stored in:

cache/metadata.sqlite

The metadata store is SQLite, using Node 24's built-in node:sqlite module.

The database tracks:

• GIF ID, rendition, type, normalized cache key, and disk path
• status: active or deleted
• ETag, Last-Modified, cache-related response headers, content type, and byte size
• positive representation variants and negative 404 / 410 tombstones when validators are available
• lazy-delete deadlines for assets that are in a CDN transition window
• avoided download count and saved bytes
• replacement count after upstream 200
• deleted-probe count and wasted bytes when a deleted item unexpectedly returns payload bytes

Smart Revalidating Cache

This project demonstrates a smart revalidating media cache rather than a blind offline cache. The proxy stores media bytes locally, but it keeps GIPHY in the validation path before reusing those bytes. That preserves origin freshness, removal semantics, and search/share signal while eliminating repeated media transfers when the origin says a rendition has not changed.

The proxy also separates two cache layers:

• Browser HTTP cache: Chrome may reuse an exact rewritten media URL directly from its own disk cache, or revalidate it with If-None-Match / If-Modified-Since.
• Proxy disk cache: the proxy normalizes GIPHY media URLs by GIF ID, rendition, and lowercase extension, so equivalent media bytes from different media hosts, URL fingerprints, or query variants share one local file.

The cache key should not include volatile URL fingerprints or query strings, but the upstream revalidation request should still use the current full media path and query string. The media proxy preserves upstream cache headers such as Cache-Control, ETag, Last-Modified, Date, Age, Expires, and Vary so Chrome can make normal HTTP cache decisions.

Request Flow

For a new media key:

1. The proxy validates the current full path and query string with the matching upstream media host.
2. On 200, concurrent callers for the same representation coalesce behind one owner GET; the proxy stores the whole response body on disk and records metadata in SQLite.
3. On 404 or 410, it records a deleted placeholder and negative tombstone metadata when validators are present.

For a known active media key:

1. The proxy validates the current full path and query string from the current upstream media host with HEAD when it only needs representation metadata.
2. It always revalidates with the origin before reusing local bytes when it has metadata in SQLite. If the browser supplied If-None-Match, the proxy records that signal, but still uses the proxy's saved ETag / Last-Modified validators for the upstream check.
3. On upstream 304, it records an avoided download. If the browser's If-None-Match matches the saved ETag, or its If-Modified-Since covers the saved Last-Modified, the proxy returns 304 Not Modified to the browser. Otherwise, it returns 200 from the proxy disk cache with the saved validators so the browser can revalidate later.
4. On upstream 200, concurrent callers for the same new representation coalesce behind one owner GET; the proxy stores the new variant, updates SQLite metadata and cached response headers, increments the replacement count, and returns the updated bytes.
5. On upstream 404 or 410, it does not serve the stale local file to that request. It records a negative tombstone and marks the item for lazy deletion, keeping bytes only for later requests whose own upstream result still justifies them.

For range requests:

1. The proxy caches whole media representations, not partial fragments.
2. A satisfiable single Range request is served from the cached whole file with 206 Partial Content, Content-Range, Accept-Ranges, and the exact requested bytes.
3. When a missing item is requested by overlapping ranges, the cache population path still fetches the whole upstream body once for the shared representation, then slices each client response locally.
4. Unsupported multipart ranges fall back to the full response path; unsatisfiable ranges return 416 without changing cache metadata.

For a known deleted media key:

1. The proxy still checks the current upstream URL.
2. A repeated 404 or 410 keeps the item deleted.
3. Any unexpected payload bytes are flagged in the TUI as wasted bytes.
4. A later 200 restores the item and records a replacement.

Corner Cases

The media cache is built around request provenance: every browser request that reaches the proxy must produce exactly one validation request to the target server, and any owner download must be attributable to the same request fingerprint. The E2E harness logs both client responses and target receipts so tests can fail on missing fingerprints, duplicate target receipts, unexpected target receipts, or response codes that do not match the final upstream response for that fingerprint.

Important edge-case contracts:

• HEAD-first validation: cached or known-deleted media is probed with HEAD so replacement storms do not download the same bytes thousands of times. When a GET client receives a non-cacheable HEAD result such as 500, the proxy follows with a real upstream GET and returns that GET response body to the client. Client HEAD requests remain bodyless.
• Stampede coalescing: when upstream HEAD returns 200 with an ETag or Last-Modified, concurrent callers for the same media key and representation wait behind one owner GET. Late callers recheck SQLite after their HEAD; if another request already stored the representation, they serve that variant instead of starting a second download.
• Unvalidated representations: if upstream HEAD returns 200 without an ETag or Last-Modified, the proxy does not coalesce the download on byte size alone. The eventual disk variant key is derived from the GET response body hash so two same-sized but different bodies cannot overwrite or wait on each other incorrectly.
• Validator identity: variant and tombstone keys use digests of the raw validator values. ETags such as "a/b" and "a_b" stay distinct instead of being collapsed by filename sanitization.
• Multiple positive variants: replacements do not destroy old positive variants immediately. A browser that still has an older ETag can be revalidated against that older cached file, while the current row points at the newest known representation.
• Lazy deletes: upstream 404 or 410 does not serve stale bytes to that request. For recently active media, the proxy records a negative tombstone and keeps positive bytes only during the lazy-delete window so later requests can survive eventually consistent CDN behavior when their own upstream result is still 304 or transient 5xx. Expired lazy deletes are swept on startup and also enforced before request selection, deleting every disk variant for the cache key.
• Deleted probes with bodies: known-deleted GET probes that produce 404 or 410 are confirmed with GET so unexpected payload bytes are counted as wasted bytes. HEAD-only deletion probes record tombstones but do not invent payload bytes.
• HEAD plus Range: a client HEAD request with a satisfiable Range receives the same 206, Content-Range, Accept-Ranges, and ranged Content-Length headers that a GET would receive, without a body.
• If-Range: range serving follows strong validator rules. A strong matching ETag or fresh date can receive 206; weak ETags and * do not authorize range slicing and fall back to the full cached response.
• Range population: cache population always fetches whole objects. Concurrent overlapping range requests for a missing item share the owner GET when upstream exposes validators, then each client receives its own sliced range response from the stored body.

Test Harness

The E2E tests double as validation harnesses for the reference implementation. They use a simulated target media server and a conformance client:

• The target server accepts a response program per asset: repeated 304s, replacement 200s, deletion 404 / 410 steps, weighted transition windows, delays, and payload bodies.
• The client sends unique fingerprints on every request and can vary method, range headers, If-Range, and browser validators.
• Assertions compare planned fingerprints, client results, target receipts, final cache state, and expected receipt counts. Tests that intentionally coalesce downloads expect one HEAD receipt per client request plus exactly the owner GET receipts.

Run the normal proxy, harness, and unit coverage with:

pnpm test

Run the high-volume fingerprint stressors with:

CONFORMANCE_STRESS=1 CONFORMANCE_STRESS_COUNT=30000 CONFORMANCE_STRESS_CONCURRENCY=128 pnpm test

The stress scenarios cover tens of thousands of unique fingerprints across shared media GIF IDs, renditions (giphy, 200, 200w, 100), and file types: GIFs as .gif, .mp4, and .webp; Clips as .mp4; Stickers as .gif and .webp.

To validate your own media proxy, adapt the harness model documented in docs/media-proxy-conformance-harness.md: send each client request with a unique x-giphy-conformance-fingerprint, preserve that header on upstream validation/download requests, record target-server receipts, and compare planned fingerprints, client responses, upstream receipts, and final cache state. The lowest-friction path is to reuse the target server, synthetic client, scenario definitions, and assertion ledger under test/support/, then replace this repo's proxy startup with your proxy under test.

Use these helper scripts when shaping or debugging another implementation:

• pnpm cache:summary [path/to/metadata.sqlite] summarizes a local cache database into safe aggregate fixture shape data.
• pnpm media:characterize <media-url> [more URLs] probes live media URL semantics such as HEAD, validators, negative responses, and range behavior without storing payload bytes.

Browser Cache Validation

To validate the browser cache layer in Chrome or Edge:

1. Run pnpm tls:mkcert if Chrome or Edge does not already trust the local certificate.
2. Start the proxy with pnpm dev.
3. Open https://web.localhost:8080/.
4. Open DevTools and select the Network tab.
5. Make sure Disable cache is not checked.
6. Filter to media/image requests if helpful.
7. Scroll all the way down the web.localhost homepage so many GIFs load.
8. Scroll back up and watch the media rows.

Expected browser behavior:

• Some rows show 200 with Fulfilled by set to disk cache.
• Some rows show 304 when the browser revalidates with If-None-Match or If-Modified-Since.
• Request headers for revalidated rows include If-None-Match and/or If-Modified-Since.

If you do not see browser disk-cache hits or browser-facing 304 responses, first confirm Disable cache is off. If it is off, the most likely cause is a certificate error preventing Chrome from caching and reusing images for that local HTTPS origin. In that case, the proxy disk cache is still working and will still avoid upstream media downloads after origin revalidation, but browser-to-proxy transfers will continue and the TUI will overstate end-user network bandwidth saved. Use pnpm tls:mkcert or TLS_CERT_FILE / TLS_KEY_FILE to run with a locally trusted certificate when validating browser cache behavior.

Exercises Left To The Reader

This reference implementation keeps several production concerns explicit instead of hiding them behind demo code:

• Production deployment: packaging this behind a real CDN, load balancer, container image, process supervisor, or multi-region topology is out of scope.
• Multi-node cache sharing: the demo uses local SQLite plus disk files. A production implementation needs its own answer for shared storage, eviction, backpressure, and node-local hot caches.
• Policy integration: authentication, authorization, rate limiting, tenant isolation, abuse controls, and audit logging are not implemented here.
• Observability: the TUI is useful for local demos, but production use needs metrics, tracing, alerting, and structured logs.
• Standalone harness packaging: the conformance harness is written so another proxy can adapt it, but it is not currently published as a separate npm package or CLI.
• Browser automation: browser-cache validation is documented as a manual Chrome/Edge workflow. A production-quality validation suite could add Playwright coverage for DevTools-visible disk-cache and 304 behavior.

Terminal UI

The TUI starts with the proxy and shows:

• total GIFs and renditions observed
• total avoided downloads
• total MB/GB saved
• total cached bytes
• replacements caused by upstream 200
• deleted items
• red wasted-byte indicators for deleted probes that still returned payload bytes

Each GIF ID expands into child rendition/type rows such as:

WRQBXSCnEFJIuxktnw  avoided 3x  saved 1.84 MB
  - 200.webp  active  avoided 2x  saved 820.0 KB  size 410.0 KB
  - 200.gif   active  avoided 1x  saved 1.04 MB   size 1.04 MB

Press q, Esc, or Ctrl-C to exit.

Checks

pnpm typecheck
pnpm test
pnpm build