Skip to content

Commit 8fb2cce

Browse files
committed
feat(enrichment): add screenshot pipeline with browser fetch and storage
- Add screenshot pipeline service for capturing page screenshots via headless browser - Add browser fetch service with Puppeteer-based rendering - Add screenshot storage service with local filesystem support - Add screenshot repository with DB schema migration (0011) - Add enrichment config schema fields for screenshot settings - Update open-graph provider to support screenshot capture - Enhance safe-fetch with configurable retries and user agent - Add comprehensive test suite for all new services - Update Dockerfile with Chromium and Puppeteer dependencies - Add design docs for screenshot and knowledge base features
1 parent 40f7514 commit 8fb2cce

28 files changed

Lines changed: 9393 additions & 33 deletions

apps/core/src/constants/cache.constant.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,9 @@ export enum RedisKeys {
3232
AnalyzeAggregate = 'analyze_aggregate',
3333
AnalyzeTrafficSource = 'analyze_traffic_source',
3434
AnalyzeDeviceDistribution = 'analyze_device_distribution',
35+
36+
/** Enrichment 截图 LRU touchAccess 节流 NX 锁 */
37+
EnrichmentScreenshotTouch = 'enrichment_screenshot_touch',
3538
}
3639
export const API_CACHE_PREFIX = 'mx-api-cache:'
3740
export enum CacheKeys {
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
-- migration-lint:allow=no-bare-create-index reason=enrichment_screenshots is a brand-new table, empty at deploy time
2+
CREATE TABLE IF NOT EXISTS "enrichment_screenshots" (
3+
"enrichment_id" text PRIMARY KEY NOT NULL,
4+
"object_key" text NOT NULL,
5+
"bytes" integer NOT NULL,
6+
"width" integer NOT NULL,
7+
"height" integer NOT NULL,
8+
"blurhash" text,
9+
"palette" jsonb,
10+
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
11+
"last_accessed_at" timestamp with time zone DEFAULT now() NOT NULL,
12+
CONSTRAINT "enrichment_screenshots_enrichment_id_enrichment_cache_id_fk" FOREIGN KEY ("enrichment_id") REFERENCES "public"."enrichment_cache"("id") ON DELETE cascade ON UPDATE no action
13+
);
14+
--> statement-breakpoint
15+
CREATE INDEX IF NOT EXISTS "enrichment_screenshots_lru_idx" ON "enrichment_screenshots" USING btree ("last_accessed_at");

0 commit comments

Comments
 (0)