llms.txt

# extensions/ — LLM context file

This directory contains all data source extensions for Linnet.
Each extension is a self-contained unit: fetch → process → render → FeedSection.

See also: extensions/README.md (human guide), extensions/_template/ (starter)

---

## File map

Each extension is a package directory containing `__init__.py` (the class) and `README.md` (extension-specific docs).

```
base.py              BaseExtension ABC + FeedSection dataclass  ← read this first
__init__.py          REGISTRY list — all active extensions registered here
README.md            Human-readable guide (quickstart, checklist, testing)
llms.txt             This file

arxiv/
  __init__.py        ArxivExtension — fetch + LLM score + summarise + figures
  README.md          Config options, output schema, test commands

hacker_news/
  __init__.py        HackerNewsExtension — fetch + LLM summarise
  README.md          Config options, output schema, test commands

github_trending/
  __init__.py        GitHubTrendingExtension — fetch + LLM summarise
  README.md          Config options, output schema, test commands

us_stocks/
  __init__.py        USStocksExtension — public market data + deterministic scoring + optional LLM synthesis
  collector.py       Quote/news/filing provider fallback chain
  scorer.py          Signal scoring and sector overview aggregation
  summarizer.py      Structured LLM text synthesis
  README.md          Config options, provider notes, output schema, test commands

postdoc_jobs/
  __init__.py        PostdocJobsExtension — RSS + scraping, LLM score + summarise
  README.md          Config options, output schema, enabling steps

supervisor_updates/
  __init__.py        SupervisorExtension — page change detection + LLM diff summary
  README.md          Config options, output schema, enabling steps

_template/
  __init__.py        Fully commented starter for new extensions
  README.md          How to use the template
```

Note: `postdoc_jobs` and `supervisor_updates` are registered extensions, but disabled in the default config. Enable them by setting `enabled: true` in `config/sources.yaml` and adding the needed detail config under `config/extensions/`.

---

## BaseExtension contract

```python
class BaseExtension(ABC):
    key:   str = ""    # unique snake_case — must match config/sources.yaml key
    title: str = ""    # display name used in rendered output

    def __init__(self, config: dict, llm_client: Any = None): ...

    @property
    def enabled(self) -> bool:
        return self.config.get("enabled", True)

    @abstractmethod
    def fetch(self) -> list[dict]: ...      # no LLM; return raw items

    def process(self, items: list[dict]) -> list[dict]:
        return items                         # override for scoring/summarising

    @abstractmethod
    def render(self, items: list[dict]) -> FeedSection: ...  # no network

    def run(self) -> FeedSection:
        # called by orchestrator
        # returns empty FeedSection if not self.enabled
        if not self.enabled:
            return FeedSection(key=self.key, title=self.title)
        items = self.fetch()
        items = self.process(items)
        return self.render(items)
```

---

## FeedSection

```python
@dataclass
class FeedSection:
    key:   str             # snake_case identifier
    title: str             # section heading
    items: list[dict]      # processed items; schema is extension-specific
    meta:  dict            # stats (counts, durations, etc.)
```

---

## self.config — injected keys

The orchestrator builds the config dict by merging sources.yaml + keywords.yaml
and injecting these keys into every extension's config:

| Key | Type | Source |
|---|---|---|
| `enabled` | bool | sources.yaml |
| `language` | str | sources.yaml — BCP-47 code, e.g. "en", "zh" |
| `llm_scoring_model` | str | sources.yaml llm.scoring_model |
| `llm_summarization_model` | str | sources.yaml llm.summarization_model |
| `dry_run` | bool | set when `--dry-run` CLI flag is used |

Extension-specific keys (example for arxiv):
| Key | Source |
|---|---|
| `categories` | keywords.yaml arxiv.categories |
| `must_include` | keywords.yaml arxiv.must_include |
| `boost_keywords` | keywords.yaml arxiv.boost_keywords |
| `llm_score_threshold` | keywords.yaml arxiv.llm_score_threshold |
| `max_papers_per_run` | sources.yaml arxiv.max_papers_per_run |

---

## self.llm — LLM client

OpenAI-compatible client (OpenRouter). Standard usage:

```python
resp = self.llm.chat.completions.create(
    model=self.config["llm_summarization_model"],
    messages=[{"role": "user", "content": prompt}],
    max_tokens=120,
)
result = resp.choices[0].message.content.strip()
```

Always guard with dry_run:

```python
def process(self, items):
    if self.config.get("dry_run"):
        return items          # skip all LLM calls
    ...
```

---

## Item field conventions

Use these field names so templates render correctly without changes:

### Papers (produced by arxiv.py)
```
id, title, authors, affiliations, categories, primary_category,
primary_category_anchor, url, score, abstract, keywords_matched,
figure_url, figure_caption
```

### HN stories (produced by hacker_news.py)
```
id, title, url, score, comments_url, summary
```

### GitHub repos (produced by github_trending.py)
```
full_name, url, description, language, stars_today, total_stars, summary
```

### Jobs (produced by postdoc_jobs extension)
```
title, url, institution, location, deadline, salary, source,
relevance_score, requirements
```

### Supervisor updates (produced by supervisor_updates extension)
```
name, institution, url, change_summary
```

---

## How to add an extension (quick reference)

1. `cp -R extensions/_template extensions/my_source`
2. Set `key` and `title`; implement `fetch()`, optionally `process()`, `render()`
3. Add to `REGISTRY` in `extensions/__init__.py`
4. Add config block in `config/sources.yaml`
5. Write tests in `tests/test_my_source.py`
6. `PYTHONPATH=. pytest tests/ -q` — all must pass

Full guide: extensions/README.md

---

## Registering in __init__.py

```python
from extensions.base import BaseExtension, FeedSection
from extensions.arxiv import ArxivExtension
from extensions.hacker_news import HackerNewsExtension
from extensions.github_trending import GitHubTrendingExtension
# from extensions.my_source import MySourceExtension  ← add here

REGISTRY: list[type[BaseExtension]] = [
    ArxivExtension,
    HackerNewsExtension,
    GitHubTrendingExtension,
    # MySourceExtension,   ← add here
]
```

---

## Testing extensions

```bash
# Run all tests (no network, no LLM)
PYTHONPATH=. pytest tests/ -q

# Run tests for one extension's collector
PYTHONPATH=. pytest tests/test_hn_collector.py -v

# Smoke test with live data, zero LLM cost
python main.py --dry-run
```

Write tests in `tests/` targeting the collector functions (`collectors/*.py`),
not the extension class directly. Use inline fixture data — no live API calls.
See `tests/conftest.py` for shared fixtures (sample_paper, sample_job, etc.).