curate

A personal information curation tool that automatically collects and organizes content from the web for your own use.

Overview

curate is designed around two axes of information gathering:

• Active collection (pick) - Manually triggered collection of specific content
• Passive collection - Automated periodic polling (planned)

Currently supports collecting X (Twitter) bookmarks with automatic media downloading, thread fetching, and Markdown export.

Features

• Fetches your X bookmarks via the official X API v2
• Downloads media (photos, videos, GIFs) with streaming support for large files
• Captures conversation threads automatically
• Exports each bookmark as a structured Markdown file with YAML frontmatter
• Tracks processed bookmarks in SQLite to avoid duplicates
• Automatic token refresh (OAuth 2.0 PKCE with offline access)
• Exponential backoff with jitter for API retries
• Rate limit detection and graceful handling
• Atomic file writes to prevent data corruption
• Structured JSON Lines logging with rotation
• macOS launchd integration for automated hourly polling

Prerequisites

• Deno v2.0+
• An X Developer Account with a project/app configured:
  • OAuth 2.0 enabled (Confidential Client)
  • Callback URL set to http://localhost:8739/callback
  • Scopes: tweet.read, bookmark.read, users.read, offline.access

Setup

1. Clone and configure

git clone https://github.com/thedomainai/curate.git
cd curate
cp .env.example .env

Edit .env and fill in your X API credentials:

X_CLIENT_ID=     # Your OAuth 2.0 Client ID
X_CLIENT_SECRET=  # Your OAuth 2.0 Client Secret

2. Authenticate

cd x
deno task auth

This opens a browser window for OAuth authorization. After granting access, tokens are saved locally in storage/credentials/.

3. Fetch bookmarks

deno task poll

Bookmarks are saved as Markdown files in storage/output/x-bookmarks/.

4. (Optional) Automated polling with launchd

deno task install    # Register with launchd (polls every hour)
deno task status     # Check service status
deno task uninstall  # Remove from launchd

Project Structure

curate/
├── .env.example              # Environment variables template
├── x/                        # X (Twitter) module
│   ├── config/
│   │   ├── default.json      # Default configuration
│   │   └── *.plist           # launchd template
│   ├── scripts/              # launchd management scripts
│   ├── src/
│   │   ├── cli/
│   │   │   └── auth.ts       # OAuth authentication CLI
│   │   ├── pick/x-bookmarks/
│   │   │   ├── api/          # X API client, media downloader, types
│   │   │   ├── auth/         # OAuth 2.0 PKCE, token management
│   │   │   ├── storage/      # SQLite store, Markdown generator, file writer
│   │   │   ├── index.ts      # Entry point
│   │   │   └── poller.ts     # Bookmark polling orchestrator
│   │   └── shared/           # Logger, config, retry, shared types
│   └── tests/                # Unit tests
└── storage/                  # Runtime data (gitignored)
    ├── credentials/          # OAuth tokens
    ├── state/                # SQLite database
    └── output/               # Generated Markdown & media

Configuration

Edit x/config/default.json to customize:

Key	Default	Description
polling.max_results_per_poll	100	Max bookmarks per poll
storage.output_dir	./storage/output	Output directory
logging.level	info	Log level (debug, info, warn, error)
logging.max_file_size_bytes	10485760	Log rotation threshold (10MB)
logging.retention_days	7	Log retention period
retry.max_attempts	3	Max retry attempts

Environment variable CURATE_OUTPUT_DIR can override the output directory.

Output Format

Each bookmark is saved as a Markdown file named {date}_{username}_{tweet_id}.md:

---
source: "x-bookmark"
tweet_id: "123456789"
author:
  username: "example_user"
  name: "Example User"
url: "https://twitter.com/example_user/status/123456789"
created_at: "2026-01-01T00:00:00.000Z"
metrics:
  likes: 42
  retweets: 10
  replies: 3
---

# [Example User](https://twitter.com/example_user) (@example_user)

> The tweet content goes here.

## Media

![image1](./media/123456789_0.jpg)

---

**Collected at**: 2026/1/1 12:00:00
**Original URL**: https://twitter.com/example_user/status/123456789

Testing

cd x
deno task test

License

MIT