feat: /insights command — usage analytics, cost estimation & activity patterns#552
Merged
feat: /insights command — usage analytics, cost estimation & activity patterns#552
Conversation
Inspired by Claude Code's /insights, adapted for Hermes Agent's multi-platform architecture. Analyzes session history from state.db to produce comprehensive usage insights. Features: - Overview stats: sessions, messages, tokens, estimated cost, active time - Model breakdown: per-model sessions, tokens, and cost estimation - Platform breakdown: CLI vs Telegram vs Discord etc. (unique to Hermes) - Tool usage ranking: most-used tools with percentages - Activity patterns: day-of-week chart, peak hours, streaks - Notable sessions: longest, most messages, most tokens, most tool calls - Cost estimation: real pricing data for 25+ models (OpenAI, Anthropic, DeepSeek, Google, Meta) with fuzzy model name matching - Configurable time window: --days flag (default 30) - Source filtering: --source flag to filter by platform Three entry points: - /insights slash command in CLI (supports --days and --source flags) - /insights slash command in gateway (compact markdown format) - hermes insights CLI subcommand (standalone) Includes 56 tests covering pricing helpers, format helpers, empty DB, populated DB with multi-platform data, filtering, formatting, and edge cases.
Comprehensive guide for using Hermes Agent with alternative LLM backends: - Ollama (local models, zero config) - vLLM (high-performance GPU inference) - SGLang (RadixAttention, prefix caching) - llama.cpp / llama-server (CPU & Metal inference) - LiteLLM Proxy (multi-provider gateway) - ClawRouter (cost-optimized routing with complexity scoring) - 10+ other compatible providers table (Together, Groq, DeepSeek, etc.) - Choosing the Right Setup decision table - General custom endpoint setup instructions All of these work via the existing OPENAI_BASE_URL + OPENAI_API_KEY custom endpoint support — no code changes needed.
Custom OAI endpoints, self-hosted models, and local inference should NOT show fabricated cost estimates. Changed default pricing from $3/$12 per million tokens to $0/$0 for unrecognized models. - Added _has_known_pricing() to distinguish commercial vs custom models - Models with known pricing show $ amounts; unknown models show 'N/A' - Overview shows asterisk + note when some models lack pricing data - Gateway format adds '(excludes custom/self-hosted models)' note - Added 7 new tests for custom model cost handling
…, serialization
Issues found and fixed during deep code path review:
1. CRITICAL: Prefix matching returned wrong prices for dated model names
- 'gpt-4o-mini-2024-07-18' matched gpt-4o ($2.50) instead of gpt-4o-mini ($0.15)
- Same for o3-mini→o3 (9x), gpt-4.1-mini→gpt-4.1 (5x), gpt-4.1-nano→gpt-4.1 (20x)
- Fix: use longest-match-wins strategy instead of first-match
- Removed dangerous key.startswith(bare) reverse matching
2. CRITICAL: Top Tools section was empty for CLI sessions
- run_agent.py doesn't set tool_name on tool response messages (pre-existing)
- Insights now also extracts tool names from tool_calls JSON on assistant
messages, which IS populated for all sessions
- Uses max() merge strategy to avoid double-counting between sources
3. SELECT * replaced with explicit column list
- Skips system_prompt and model_config blobs (can be thousands of chars)
- Reduces memory and I/O for large session counts
4. Sets in overview dict converted to sorted lists
- models_with_pricing / models_without_pricing were Python sets
- Sets aren't JSON-serializable — would crash json.dumps()
5. Negative duration guard
- end > start check prevents negative durations from clock drift
6. Model breakdown sort fallback
- When all tokens are 0, now sorts by session count instead of arbitrary order
7. Removed unused timedelta import
Added 6 new tests: dated model pricing (4), tool_calls JSON extraction,
JSON serialization safety. Total: 69 tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an
/insightscommand inspired by Claude Code's/insights, adapted for Hermes Agent's multi-platform architecture. Analyzes session history fromstate.dbto produce comprehensive usage analytics.What it does
Generates a report covering:
Cost Estimation
Includes real pricing data for 25+ models (OpenAI, Anthropic, DeepSeek, Google, Meta) with fuzzy model name matching — strips provider prefixes, handles keyword heuristics (
opus,sonnet,haiku, etc.), falls back to defaults for unknown models.Three Entry Points
/insightsslash command in CLI — full terminal-formatted report with box-drawing and bar charts/insightsslash command in gateway (Telegram/Discord/etc.) — compact markdown formathermes insightsCLI subcommand — standalone usageConfigurable
--days N— time window (default: 30 days)--source <platform>— filter by platform (cli, telegram, discord, etc.)/insights 7sets days=7Example Output
Files Changed
agent/insights.pycli.py/insightsslash command handler in CLIgateway/run.py/insightsslash command handler in gateway (+ help text)hermes_cli/commands.pyhermes_cli/main.pyhermes insightsCLI subcommandtests/test_insights.pyTests
56 new tests covering:
Full test suite: 1982 passed, 5 skipped, 0 failures.
Future Ideas