Skip to content

API_AND_SCRAPING.md

Latest commit

 

History

History
100 lines (78 loc) · 4.48 KB

API_AND_SCRAPING.md

File metadata and controls

100 lines (78 loc) · 4.48 KB
Raw

Stash — Metadata Scraper & Proxy Engine

This document details how Stash extracts Open Graph metadata, rich media embeds, and thumbnails from arbitrary web links.

1. Architecture Overview

Stash uses a resilient multi-tier metadata extraction engine:

                      ┌────────────────────────┐
                      │    Incoming Link URL   │
                      └───────────┬────────────┘
                                  │
                  ┌───────────────┴───────────────┐
                  ▼                               ▼
       ┌──────────────────────┐        ┌──────────────────────┐
       │   Instagram Parser   │        │    YouTube Parser    │
       │  oEmbed API + ogs    │        │   Direct HD Poster   │
       └──────────┬───────────┘        └──────────┬───────────┘
                  │                               │
                  └───────────────┬───────────────┘
                                  ▼
                      ┌────────────────────────┐
                      │  Generic Open Graph    │
                      │  open-graph-scraper    │
                      └───────────┬────────────┘
                                  │
                                  ▼
                      ┌────────────────────────┐
                      │ Domain Fallback Engine │
                      │ Hostname & Favicon     │
                      └────────────────────────┘

2. Extraction Endpoints

POST /api/scrape

Scrapes Open Graph metadata from any URL.

Request Body

{
  "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
}

Successful Response

{
  "type": "video",
  "ogTitle": "Rick Astley - Never Gonna Give You Up (Official Video)",
  "ogDescription": "The official video for Never Gonna Give You Up by Rick Astley...",
  "ogImage": "https://img.youtube.com/vi/dQw4w9WgXcQ/hqdefault.jpg",
  "ogSiteName": "YouTube"
}

Specialized Handlers:

  1. Instagram (isInstagramUrl):

    • Fetches official oEmbed markup from https://api.instagram.com/oembed (for Reels and carousels).
    • Extracts shortcodes from URLs (/p/, /reel/, /tv/).
    • Uses open-graph-scraper with desktop User-Agent headers to retrieve high-res cover photos and captions.

  2. YouTube (isYouTubeUrl):

    • Parses video IDs (v=... or youtu.be/...).
    • Generates direct high-definition thumbnail links: https://img.youtube.com/vi/<videoId>/hqdefault.jpg.

  3. Articles & Generic Websites:

    • Reads standard Open Graph tags (og:title, og:description, og:image, og:site_name).
    • Reads Twitter card tags (twitter:title, twitter:image, twitter:description) when OG tags are missing.
    • Automatically decodes HTML entities (e.g. &amp; $\rightarrow$ &, &#8217; $\rightarrow$ ').

3. Streaming Image Proxy (/api/proxy-image)

To avoid CORS restrictions and hotlinking blockers on external image servers, Stash includes a lightweight streaming proxy:

GET /api/proxy-image?url=https://example.com/photo.jpg
  • Headers: Spoofs standard browser User-Agent and Accept headers to bypass referrer checks.
  • Caching: Sets Cache-Control: public, max-age=86400, s-maxage=86400 so images load instantaneously from browser cache on repeat visits.
  • Content-Type: Transparently preserves original image formats (image/avif, image/webp, image/jpeg, image/png).

4. Dual-Mode Operation (Server vs Static Export)

  • During Web Dev / Production Server (bun run dev / bun run start): Both /api/scrape and /api/proxy-image run as live Node.js server routes with full HTTP scraping capabilities.
  • During Static APK Compilation (NEXT_EXPORT=true / bun run cap:sync): Both routes statically compile with export const dynamic = 'force-dynamic' and mock fallbacks, enabling Next.js static HTML export for Capacitor.