[Feature] Incremental analysis caching, concurrent cronjob, and token-aware repo context

[dwin-gharibi]

Scale-and-cost improvements to the analyzer and the GitLab cronjob so that
organization-wide, repeated documentation runs are cheaper and faster:

• Incremental analysis — skip re-analyzing a repository that hasn't changed.
• Concurrent cronjob — process applicable projects in parallel instead of one at a time.
• Token-aware repo context — build the injected file tree once, respect .gitignore,
  cap it for large monorepos, and detect languages.

No breaking changes; every new behavior is on a sensible default and can be turned off.

Motivation

The cronjob runs across an entire GitLab group on a schedule, and every invocation
previously re-analyzed each repository from scratch, processed projects strictly
sequentially, and injected the entire repository file tree into every one of the
five analyzer agents' prompts (recomputed per agent). On large repos this wastes
tokens/latency, and across an org it wastes a lot of both. This MR addresses the three
"scale & cost" gaps together because they share infrastructure (file enumeration).

Changes

D. Incremental analysis / caching

• Added src/utils/cache.py: an AnalysisManifest (pydantic) plus
  load_manifest / write_manifest / is_up_to_date.
• After a complete analysis run the analyzer writes .ai/docs/.manifest.json
  recording the repo fingerprint, analyzer version, the set of requested analyses,
  and each output file's content hash.
• Before fanning out the agents, the analyzer checks the manifest and, on a valid
  cache hit, skips the entire analysis. Partial runs are intentionally not
  cached, so a previously-failed analysis is retried on the next run.
• Fingerprinting is content-based (not mtime), so it is stable across clones and
  correctly reflects uncommitted working-tree edits.
• The fingerprint and context exclude the tool's own .ai/ output, so writing the
  analysis docs cannot self-invalidate the cache.

E. Parallel cronjob across projects

• src/handlers/cronjob.py: handle() now collects applicable projects sequentially
  (cheap metadata filtering) and then runs the clone → analyze → merge-request pipeline
  concurrently through the existing WorkerPool, bounded by a new max_project_workers
  (default 4, 0 = auto/CPU count).
• Blocking git/GitLab operations (clone, commit/push, MR creation, cleanup) are moved
  off the event loop with asyncio.to_thread so projects genuinely overlap rather than
  serializing behind each other's blocking calls.
• Per-project failures remain isolated: an exception in one project is captured by the
  pool and logged without stopping the others.

F. Token-aware repo context

• Added src/utils/repo_context.py: build_repo_context() returns a RepoContext
  (structure string, languages, file count, truncation flag, file list, fingerprint).
  It is built once per run and reused across all five agents (previously the full
  tree was recomputed for each).
• Respects .gitignore via git ls-files when the repo is a git checkout (fixes
  over-ignoring of legitimately-tracked directories such as docs/, static/,
  public/). Falls back to a filtered directory walk with best-effort .gitignore
  parsing for non-git repositories.
• Caps the injected listing (max_context_files, default 1000) with a clear
  truncation note, plus a hard character backstop, to protect the context window on
  large monorepos.
• Lightweight language auto-detection is surfaced as a header in the injected
  context (e.g. Primary languages: Python, YAML, Markdown).
• Added src/utils/ignore_patterns.py: the 100+ ignore-directory/extension constants
  were moved out of list_files.py into a single dependency-free source of truth
  (re-exported from list_files.py for backward compatibility).

New configuration & CLI flags

All values below are also exposed as CLI flags via the existing config-reflection
(--flag-name) and documented in config_example.yaml.

Scope	Option	Default	Purpose
analyzer	respect_gitignore	true	Honor the repo's .gitignore when building context
analyzer	max_context_files	1000	Max files listed in the injected structure (0 = unlimited)
analyzer	cache_enabled	true	Enable the incremental cache
analyzer	force_reanalysis	false	Force re-analysis on a cache hit (--force-reanalysis)
cronjob.analyze	max_project_workers	4	Projects analyzed concurrently (0 = auto/CPU count)

Testing

• 17 unit tests (stdlib unittest, no new dependency):

  PYTHONPATH=src python -m unittest discover -s tests -t .

  Coverage includes: fingerprint stability & content-sensitivity, the "generated
  .ai/ docs must not invalidate the cache" property, .gitignore handling beyond the
  hardcoded list, truncation, language detection, a repo living under an ignore-named
  parent directory, full manifest validity matrix (fingerprint/version/missing/edited/
  included-set/corrupt), and the analyzer cache-miss → hit → invalidate wiring.
• Lint/format: ruff format and ruff check clean across src/ and tests/.
• Smoke test on this repository (git path): 60 files enumerated, .git/.ai
  excluded, languages detected, a real analyzer prompt rendered, and the cronjob worker
  resolution verified — all without any LLM calls.

Backward compatibility

• Defaults preserve existing behavior for typical repos (small repos are unaffected by
  the context cap; caching is transparent and only skips genuinely-unchanged repos).
• DEFAULT_IGNORED_DIRS / DEFAULT_IGNORED_EXTENSIONS remain importable from
  agents.tools.dir_tool.list_files.
• No dependency or lockfile changes.

Operational notes & caveats

• Concurrency multiplies: effective LLM concurrency ≈
  max_project_workers × analyzer max_workers. Tune both together against provider
  rate limits; the existing 429 retry client backstops bursts. Parallel clones also
  increase peak disk usage (Helm ephemeral-storage).
• The cache's largest wins are local/CI re-runs and idempotency. In the cronjob's steady
  state, any new developer commit changes the fingerprint (correctly → re-analyze); the
  cache mainly avoids redundant work when nothing changed.
• The non-git .gitignore fallback is a pragmatic subset (no negation/anchored
  patterns); git repositories use real git semantics via git ls-files.

[dwin-gharibi]

@Meshkati @miladosos