[agent] collection_stats reports 30 articles but collection has 25,313 chunks — likely undercounting documents

[acertainKnight]

What happened

collection_stats returned the following:

• Total documents (chunks): 25,313
• Document Types → article: 30

The user confirmed they have significantly more than 30 papers in their collection. 25,313 chunks from only 30 articles would mean ~844 chunks per paper on average, which is implausibly high. The article count appears to be severely undercounting the actual number of documents in the collection.

Expected behavior

The article: 30 count under "Document Types" should reflect the true number of distinct papers/documents in the collection, consistent with the chunk count.

Possible causes

• The article count may be reading from a metadata table or index that isn't being updated when new documents are ingested (e.g., PDFs processed by the monitor service may create chunks but not register as articles in the stats source)
• The count might only reflect articles added through a specific pathway (e.g., manual upload, discovery sources) and miss others (e.g., PDF monitor, direct ingestion)
• There may be a distinction between "articles" (as a document_type) and other ingested documents that aren't being surfaced in the stats

Context

• Skill active: onboarding (using collection_stats and search_articles)
• User action: First interaction / collection deep dive
• search_articles with various queries consistently returned results capped at the same ~30 articles, suggesting the article-level index (not the chunk store) is the bottleneck
• The chunk collection (document_chunks via pgvector) appears to have the full corpus at 25,313 chunks

Steps to reproduce

1. Call collection_stats
2. Note the article: 30 count under Document Types
3. Compare against the actual number of distinct papers the user has ingested
4. Observe the discrepancy

[acertainKnight]

Update: collection_stats is now returning much more complete information on a subsequent call:

Knowledge Base:
  - Total papers: 12310
  - Indexed papers: 592
  - Indexed chunks: 25313

Document Categories:
  - research_paper: 12301
  - external: 9

Research Questions:
  - Total: 12
  - Active: 11
  - (5 visible with daily schedules and last run dates)

Research questions now show correctly -- 12 total, 11 active, with names, schedules, and last run timestamps. This part looks complete.

Paper counts are now visible but the "Total papers: 12,310" figure is misleading. This number includes all cited references extracted from the actually-indexed papers, not 12,310 papers the user has uploaded or downloaded. The real collection size is closer to the 592 indexed papers. The stat should either:

• Distinguish between "papers in collection" vs "cited references in graph"
• Or label it more clearly (e.g., "Papers in citation graph: 12,310 | Papers with full text indexed: 592")

Original bug (returning only 30 articles with no research questions) may have been a transient issue or a stale cache. The response format was also completely different between the two calls (different field names, different structure), which suggests the tool may have multiple code paths or the first call hit a fallback/error state silently.

[acertainKnight]

Addressed the feedback in the latest comment. Two further changes made:

"Total papers: 12,310" was misleading -- paper_metadata holds all paper records including extracted citation references, not just papers the user has in their collection. Fixed by breaking the count down by source_of_truth:

• Papers in collection (source_of_truth = 'processed') -- what the user has actually uploaded/downloaded with full text
• Referenced papers (citation graph only) (source_of_truth = 'citation') -- papers that only appear as cited references
• Discovered (not yet downloaded) (source_of_truth = 'discovered') -- papers found via discovery but not yet processed

The "Document Categories" breakdown now also filters to source_of_truth IN ('processed', 'merged') so it only reflects the actual collection, not citation-only entries.

Expected output with this fix:

Knowledge Base:
  - Papers in collection: 592
  - Discovered (not yet downloaded): X
  - Referenced papers (citation graph only): 11,718
  - Indexed for search: 592
  - Indexed chunks: 25,313

The original bugs (research questions at 0, article count capped at 30) were already fixed in the prior commit. This update clarifies the paper counts so the agent has an accurate picture of the collection.