Some AI suggestions for improvements

[manorit2001]

on v0.21.36 with GPT 5.4

The issue tracker says the same thing the codebase says: Headroom’s main problems are not “lack of ideas,” but too much product surface concentrated into a few brittle integration paths.
Direct tracker read:

• 166 total issues
• 64 open, 102 closed
• labels are dominated by bug (53) and enhancement (33)
• 29/64 open issues are unlabeled
• 33/64 open issues have zero comments
• 24/64 open issues are older than 30 days
  That means two problems coexist:

1. real engineering friction around proxy/provider/integration behavior
2. weak issue routing and triage that turns support load into product-noise
   What trends show up
   The recurring themes are:
3. Proxy and provider compatibility
   • This is the loudest cluster.
   • Repeated reports around OpenAI-compatible flows, Codex, Claude, Azure/LiteLLM, MCP/proxy availability, websocket/response handling, and “traffic routed but not compressed.”
   • Representative examples: Codex /v1/responses traffic is routed but never compressed: msgs=0, transforms=none, tokens_saved=0 #425, Regression: litellm-azure / OpenAI backend traffic can fall to near-zero compression while older Azure/gpt-5.4 sessions achieved 30-40%+ #454, docs: clarify MCP setup when proxy /mcp is unavailable #460, CLI crashes on import when installed without [proxy] extras #441, [Bug] headroom wrap ignores COPILOT_PROVIDER_TYPE=openai override, forcing anthropic #297, Anthropic token-mode compression result discarded after pipeline completes; proxy times out and forwards original request #296.
4. Observability and trust
   • Users often can’t tell whether Headroom is actually working.
   • Dashboard/savings/reporting issues show up repeatedly: misleading savings, blank or unclear dashboards, backend-routed traffic not showing correctly, polling overhead.
   • Representative examples: [BUG] Dashboard is not working #22, fix: cache dashboard stats snapshots and lazy-load heavy polling #225, [BUG] Big drop in cache hitrate ? #327, Dashboard bugs: headline savings can show 0%, Compression Quality can exceed 100%, and recent request summary is misleading for backend-routed traffic #455.
5. Docs, setup, and support friction
   • Many issues are really “how do I make this work?” rather than core defects.
   • MCP setup, proxy startup, deployment model, plugin install, and product-mode confusion keep surfacing.
   • Representative examples: [FEATURE] Document how to deploy for Claude subscription accounts #84, [DOCS] OpenCode integration guide — wrap, MCP, plugin, and manual setup #78, docs: clarify MCP setup when proxy /mcp is unavailable #460, plus the stale GitHub Discussions/doc links from earlier issue [BUG] No Github discussions #388.
6. Performance, cache, and memory regressions
   • Not just correctness bugs; users also hit regressions where compression quality drops, cache behavior degrades, RAM rises, or polling becomes expensive.
   • Representative examples: High Memory Usage (5-9GB RAM) - Unbounded In-Memory Stores & Large ML Models #21, fix: cache dashboard stats snapshots and lazy-load heavy polling #225, [BUG] Big drop in cache hitrate ? #327, Regression: litellm-azure / OpenAI backend traffic can fall to near-zero compression while older Azure/gpt-5.4 sessions achieved 30-40%+ #454.
7. Integration expansion pressure
   • There’s clear demand for more frameworks and agent ecosystems.
   • But the tracker suggests breadth is arriving faster than hardening.
   • Representative examples: [FEATURE] Agno Support #5, [FEATURE] Support for CrewAI Agents #27, [FEATURE] Comprehensive LangChain support #29, [FEATURE] OpenCode MCP server integration — headroom_compress, headroom_retrieve, headroom_stats #75, [FEATURE] OpenCode plugin — headroom-opencode npm package #76, [FEATURE] Dynamic model registry for headroom wrap — auto-discover available models #77.
     Why the codebase produces these issues
     The issue trends map cleanly to a few hotspot files:

• headroom/proxy/server.py (/home/user/Projects/headroom/headroom/proxy/server.py:1) is a 3347-line Python proxy hub.
• headroom/proxy/helpers.py (/home/user/Projects/headroom/headroom/proxy/helpers.py:1) is another 2664-line kitchen-sink helper layer.
• headroom/proxy/handlers/openai.py (/home/user/Projects/headroom/headroom/proxy/handlers/openai.py:1) is 5434 lines.
• headroom/proxy/handlers/anthropic.py (/home/user/Projects/headroom/headroom/proxy/handlers/anthropic.py:1) is 2804 lines.
• headroom/proxy/memory_handler.py (/home/user/Projects/headroom/headroom/proxy/memory_handler.py:1) is 2099 lines.
• crates/headroom-core/src/transforms/live_zone.rs (/home/user/Projects/headroom/crates/headroom-core/src/transforms/live_zone.rs:1) is 2885 lines and central to provider-shape-specific routing/compression logic.
  That architecture explains the tracker:
• one proxy entry surface
• many providers
• many protocol variants
• lots of heuristics
• lots of user-visible “it routed, but didn’t help” failure modes
  There are also maintainability signals behind the issues:
• headroom/perf/analyzer.py (/home/user/Projects/headroom/headroom/perf/analyzer.py:1) explicitly duplicates model-resolution logic from proxy code.
• headroom/integrations/langchain/providers.py (/home/user/Projects/headroom/headroom/integrations/langchain/providers.py:1) uses brittle class-path matching and falls back to OpenAI behavior for some providers.
• headroom/proxy/handlers/gemini.py (/home/user/Projects/headroom/headroom/proxy/handlers/gemini.py:1) still has TODO-backed provider-specific accounting gaps.
• tests/conftest.py (/home/user/Projects/headroom/tests/conftest.py:1) converts some network timeouts into skips, which can hide flaky integration failures.
  Full improvement breakdown

1. Stabilize the proxy surface first
   • This is the main product bottleneck.
   • Treat /v1/chat/completions, /v1/messages, /v1/responses, MCP, websocket/streaming, and provider-routing as contract surfaces.
   • Add explicit contract tests per provider/protocol pair before adding more integrations.
2. Reduce “silent success, no compression” behavior
   • Many issues are really trust failures, not hard crashes.
   • If traffic routes but compression is bypassed, downgraded, or ineligible, surface that loudly in logs, stats, and dashboards.
   • “No savings” must say why.
3. Split the proxy monolith by responsibility
   • The issue tracker is what monolith stress looks like in public.
   • Separate routing, provider adapters, observability, memory, and policy/eligibility decisions.
   • Today, changes in one area are too likely to cause regressions in another.
4. Harden observability as a product feature
   • Users need an answer to:
     • was request intercepted?
     • was it eligible?
     • was it transformed?
     • why not?
     • what provider path was used?
   • This likely removes a large chunk of “is this working?” issue volume.
5. Narrow the public support surface
   • Keep support questions out of Issues where possible.
   • Fix stale links in .github config.
   • Disable blank issues or route them more aggressively.
   • Add issue forms that force category selection: bug, provider compatibility, docs/setup, integration request, observability.
6. Make docs more mode-specific
   • Right now the product has many modes: Python library, proxy, wrap, MCP, TS SDK, plugin, cloud/local.
   • Users seem to get lost at mode boundaries.
   • Docs should be reorganized around “I am using X with Y” flows, not just feature pages.
7. Stop broadening integrations faster than they are tested
   • The repo already shows breadth across SDKs, providers, agent tools, and plugins.
   • The issue tracker suggests hardening is lagging expansion.
   • New integration work should require a compatibility test matrix and a docs path before shipping.
8. Tighten triage discipline
   • Current label usage is too shallow.
   • Add labels like:
     • provider/openai
     • provider/anthropic
     • provider/azure
     • surface/proxy
     • surface/mcp
     • surface/dashboard
     • surface/docs
     • type/regression
     • type/support
   • This would make backlog trends measurable instead of anecdotal.
     Priority order

• P0: proxy/provider compatibility, silent degradation, protocol correctness
• P1: observability/dashboard accuracy, cache/perf regressions
• P2: docs/setup/support routing
• P3: integration expansion and ecosystem breadth
• P4: backlog metadata hygiene and label taxonomy
  Bottom line
  The core trend is: Headroom is ambitious, but its most visible surfaces are still too integration-brittle and too opaque when they fail softly.
  If I had to pick the single highest-leverage move, it would be: treat the proxy as the product kernel, freeze its contracts, and build much stronger compatibility/observability around it before expanding further