๐บ๐ธ English ๏ฝ ๐ฏ๐ต ๆฅๆฌ่ช ๏ฝ ๐จ๐ณ ็ฎไฝไธญๆ ๏ฝ ๐น๐ญ เนเธเธข
Free, open-source software you run on your own computer or server โ
it turns scattered AI and tech updates into one daily newspaper and one searchable feed.
What it does ๏ฝ What you need ๏ฝ Get started ๏ฝ Why you can trust it ๏ฝ Learn more
Useful updates are scattered across social networks, specialist sites, and project pages,
so keeping up means checking many places every day โ and still missing the important context.
X Collector gathers updates from the sources you choose, organizes and connects them,
then delivers the same organized information to both people and AI agents.
Stop making the rounds โ your news arrives already organized.
๐ง Engineering guide ๏ฝ ๐ Reference
generation: a07270e (2026-08-29T18:28:40Z) ยท verify: API HEAD ยท status.json
If any of these ring a bell, X Collector was built for you.
- You follow AI news on X, Reddit, GitHub, and a dozen sites โ and still miss big announcements
- Checking every source takes an hour a day, and much of it is the same story repeated
- You can't tell which of the accounts you follow are still worth following
- Your AI assistant can't answer "what happened this week?" from the sources you trust
The common cause is simple: the news never arrives in one organized place. X Collector takes over that gathering-and-sorting work.
Four steps, always in this order.
flowchart LR
A["โ Collect<br/>from 8 source types"] --> B["โก Organize<br/>classify and connect"]
B --> C["โข Publish<br/>one daily newspaper"]
C --> D["โฃ Serve<br/>people and AI agents"]
-
๐ฅ Collect
Gathers updates from X (Twitter), Instagram, Facebook, Reddit, Qiita, GitHub, and RSS/YouTube feeds. You choose every source.
-
๐๏ธ Organize
Sorts every item into categories, links duplicates and follow-up stories, and scores how trustworthy each source has been lately.
-
๐ฐ Publish
Lays the selected stories out as a 13-section newspaper, every day, automatically.
-
๐ค Serve people and AI together
You read the newspaper and the searchable feed on the web; your AI agents read exactly the same data through an API and an MCP server (a standard way for AI tools to connect).
-
โ Keep humans in charge
It can suggest promising new sources it discovered, but nothing joins your collection without your explicit approval.
Three things. The full compatibility table is in the engineering guide.
- A place to run it โ your own computer or a server, with Node.js 20 or newer
- A PostgreSQL database โ where the collected items are stored
- API keys, only for the features you use โ see the table below
| What you want to do | What it needs |
|---|---|
| Sign in to the web interface | Google OAuth credentials (free) |
| Collect from X, Instagram, Facebook, Reddit | A ScrapeCreators key |
| Classify with AI and compose the newspaper | An OpenRouter key |
| Add YouTube transcripts to stories | A TranscriptAPI key (optional) |
| Collect from Qiita, GitHub, RSS | No key required |
About money: Google OAuth is required for signing in, but it is free โ it is a login method, not a paid API. ScrapeCreators and OpenRouter are paid, pay-as-you-go services; check their sites for current pricing. Without any paid key, the app starts, you can sign in, and keyless sources can be collected โ the AI steps (classification and the daily newspaper) stay off until you add an OpenRouter key. You can add keys later, one at a time.
If you use an AI coding agent (Claude Code, Codex CLI, and similar tools), the fastest path is to hand it the repository:
Set up https://github.com/caty-ai/x-collector on this machine.
Use .env.example to walk me through the settings I need.
The agent clones, installs, and asks you only for the values it cannot decide for you, such as the database address and sign-in keys. If you cannot answer one of those questions, just say so โ preparing the PostgreSQL database and setting up Google OAuth are also things the agent can walk through with you.
Step 1 โ download and install:
git clone https://github.com/caty-ai/x-collector.git
cd x-collector
npm install
cp .env.example .envStep 2 โ open .env in any text editor and fill in the required values. The first five are for the database and sign-in; the last two make the feed and newspaper screens work:
DATABASE_URL=postgresql://user:password@localhost:5432/x_collector
AUTH_SECRET=replace_with_a_long_random_secret
AUTH_GOOGLE_ID=your_google_oauth_client_id
AUTH_GOOGLE_SECRET=your_google_oauth_client_secret
NEXTAUTH_URL=http://localhost:3000
# For a single local install, point the app at itself
# and make up your own long random key
RAILWAY_API_BASE_URL=http://localhost:3000
FEED_API_KEY=any_long_random_string_you_issue_yourselfStep 3 โ start it:
npm run migrate
npm run devOpen http://localhost:3000, sign in, and register your sources under /settings. To try it right away with sample sources instead, run npm run seed once. When your collector keys are ready, open a second terminal window and run a collection with npm run collect.
If something goes wrong
command not found: npm
Node.js is not installed yet. Download it from nodejs.org (version 20 or newer), then reopen your terminal and try again.
The database connection fails
Make sure PostgreSQL is running and that the DATABASE_URL user, password, and database name actually exist. Creating a database named x_collector first is the most common missing step.
Google sign-in shows an error
Google OAuth credentials are created for free in the Google Cloud Console (APIs & Services โ Credentials โ Create credentials โ OAuth client ID). Check that NEXTAUTH_URL matches the address you opened in the browser, and that the OAuth redirect URI registered on Google Cloud is http://localhost:3000/api/auth/callback/google.
X Collector is designed so that automation never quietly takes over.
- You approve every new source โ discovered candidates are scored and presented, but only a person can promote them
- Manually added sources are never auto-stopped โ automatic retirement only ever applies to sources the system itself discovered, and only after two consecutive weekly checks
- Source quality is scored every day โ trust scores shape the newspaper's ranking, and stories from low-trust or unverified sources carry a warning badge instead of being silently treated as reliable
- Agent access is read-only โ the MCP server can search and read, never change anything
- Your data stays yours โ it runs on your own server and your own database, under the MIT license
- CI: the badge above is live โ vitest + TypeScript checks on every pull request and every push to main
- Verified environments: push-to-main CI pins Node.js 20 on Ubuntu and macOS; the pull-request gate uses the runner default Node.js (currently 22+)
- Maturity: core pipeline in daily production use; actively maintained
- Known constraints: collectors that talk to external platforms need your own API credentials and are not exercised by CI
Run the checks yourself: make test / make lint (wraps npm test and the TypeScript checks โ see CONTRIBUTING).
Community source catalog and contribution guide: docs/community-sources.md โ entries are suggestions and are never auto-subscribed.
Entrances by purpose.
| What you want to know | Where to look |
|---|---|
| How it works: architecture, full setup, operations (for engineers) | docs/engineering.md |
| Exact specifications: environment variables, APIs, MCP tools | docs/reference.md |
| Every setting, schedule, and operational detail in one place | docs/operations.md (Japanese) |
| How to contribute | CONTRIBUTING.md |
| How to report a bug or vulnerability | SECURITY.md |
Part of the Caty AI family โ open tools for running a family of AI agents. The full map, including modules still being prepared for release, lives in Family OS.
| Axis | Module | What it does | State |
|---|---|---|---|
| Map | Family OS | The map of the whole family โ every module, its state, and how they fit | published, MIT |
| Rules | Family Dev Handbook | The rules of the road โ issues, PRs, worktrees, handoffs, parallel development | published, MIT |
| Vertical ยท foundation | Caty Agent Harness | Task backbone for AI agents โ retries, checkpoints, and honest completion | published, MIT |
| Vertical | context-kit | Six-piece context hygiene kit for one agent โ bounded output, delegation briefs, safety guards, recall, worktree snapshots | published, MIT |
| Vertical | Persona Engine | Layers relationship and emotion onto an agent's existing persona | published, MIT |
| Vertical | Persona Growth Loop | Grows the persona itself โ minimal, idempotent proposals | published, MIT |
| Vertical | X Collector | Turns X and the web into one daily digest โ for people and agents | published, MIT |
| Vertical | Self Growth Loop | Lets an agent grow its own abilities โ proposals, governance, adoption records | published, MIT |
| Horizontal ยท foundation | Family Memory Architecture | The memory bus โ how the family shares what it knows | published, MIT |
| Horizontal | Sitter | Babysits delegated agent runs โ watches, keeps evidence, restarts only within declared bounds | published, MIT |
| Horizontal | Alpha Nightshift | Nightly autonomous maintenance loop โ isolated night lanes behind a deny-by-default guard; humans cherry-pick in the morning | published, MIT |
X Collector stands on these services: ScrapeCreators (social collection APIs), OpenRouter (AI classification and newspaper composition), Qiita API v2, GitHub REST API, Railway (hosting), and TranscriptAPI (YouTube transcripts).
MIT ยฉ 2026 Sho Jikumaru
We want anyone to use X Collector freely โ run it, modify it, and build it into your own products. As long as the copyright notice stays, commercial use and redistribution are both welcome.
One newspaper a day ๏ฝ 8 source types ๏ฝ For people and AI agents