|
| 1 | +--- |
| 2 | +name: meeseeks-cli-smoketest |
| 3 | +description: End-to-end smoke testing of the Meeseeks CLI via tmux. Use this skill when asked to test the CLI, verify CLI behavior after changes, smoke-test the agent loop, check for regressions, or validate MCP/plugin/session features work correctly through the terminal interface. Also use when debugging CLI crashes, MCP connection issues, or session lifecycle problems that need live reproduction. |
| 4 | +--- |
| 5 | + |
| 6 | +# Meeseeks CLI Smoke Test via Tmux |
| 7 | + |
| 8 | +Automate end-to-end CLI testing by running `meeseeks` inside a tmux pane, sending commands/queries, and analyzing verbose output for errors, warnings, and regressions. |
| 9 | + |
| 10 | +## Why tmux |
| 11 | + |
| 12 | +The CLI is a full-screen Rich/Textual TUI. You cannot run it directly via Bash tool because it requires a PTY and renders interactive widgets. Tmux gives you a real terminal to drive the CLI while capturing output programmatically via `tmux capture-pane`. |
| 13 | + |
| 14 | +## Setup |
| 15 | + |
| 16 | +Find a running tmux session and create a new window: |
| 17 | + |
| 18 | +```bash |
| 19 | +tmux list-sessions |
| 20 | +tmux new-window -t <session>:<next> -n meeseeks-test |
| 21 | +``` |
| 22 | + |
| 23 | +Launch with maximum verbosity and auto-approve (skips permission prompts): |
| 24 | + |
| 25 | +```bash |
| 26 | +tmux send-keys -t <session>:meeseeks-test "uv run meeseeks -vv --auto-approve" Enter |
| 27 | +``` |
| 28 | + |
| 29 | +Wait for startup (MCP connections, plugin loading, skill discovery). Startup typically takes 5-10 seconds depending on MCP server count. Capture and verify the banner appears: |
| 30 | + |
| 31 | +```bash |
| 32 | +sleep 10 && tmux capture-pane -t <session>:meeseeks-test -p -S -100 |
| 33 | +``` |
| 34 | + |
| 35 | +Look for the ready banner showing model, session ID, tool counts, and `meeseeks>` prompt. If the prompt hasn't appeared, wait longer — MCP servers may take time to connect. |
| 36 | + |
| 37 | +## Capture timing |
| 38 | + |
| 39 | +This is the critical non-obvious part. Different operations need different wait times before capturing output: |
| 40 | + |
| 41 | +| Operation | Wait (seconds) | Why | |
| 42 | +|---|---|---| |
| 43 | +| Startup | 8-10 | MCP pool connects, plugins load, skills discover | |
| 44 | +| Slash command (`/help`, `/status`) | 2-3 | Local only, no LLM call | |
| 45 | +| Interactive command (`/mcp`, `/models`) | 2-3 | Opens TUI picker — must send `Escape` to dismiss before next command | |
| 46 | +| Simple query (no tools) | 10-15 | Action plan + LLM call + response | |
| 47 | +| Tool-using query | 15-25 | Plan + LLM + tool execution + synthesis | |
| 48 | +| MCP tool query | 25-40 | Plan + LLM + MCP network call + synthesis | |
| 49 | +| `/compact` | 10-15 | Rebuilds summary via LLM call | |
| 50 | + |
| 51 | +Always use `sleep N && tmux capture-pane` as a single command — do not separate them. Adjust the scroll buffer depth (`-S -N`) based on expected output verbosity. `-S -60` is usually sufficient; use `-S -100` for startup output. |
| 52 | + |
| 53 | +## Test progression |
| 54 | + |
| 55 | +Test in layers, from cheapest to most expensive. If an early layer fails, later layers will too. |
| 56 | + |
| 57 | +### Layer 1: Slash commands (no LLM, no network) |
| 58 | + |
| 59 | +These validate the CLI framework, config loading, and plugin discovery: |
| 60 | + |
| 61 | +``` |
| 62 | +/help — all commands listed, no crashes |
| 63 | +/status — session JSON with valid ID and idle state |
| 64 | +/session — session ID matches banner |
| 65 | +/tokens — budget table renders, context window > 0 |
| 66 | +/skills — skill count matches banner, names listed |
| 67 | +/plugins — installed plugins table renders (note any WARNING lines) |
| 68 | +``` |
| 69 | + |
| 70 | +**What to look for in verbose output**: `WARNING` or `ERROR` log lines during plugin/skill loading. Common issues: |
| 71 | +- `Failed to parse manifest` — stale plugin cache, missing files |
| 72 | +- `No YAML frontmatter` — agent definition files missing required format |
| 73 | +- `Missing or invalid 'name'` — skill SKILL.md files lacking name field |
| 74 | + |
| 75 | +These warnings are non-fatal but indicate plugin integration gaps. |
| 76 | + |
| 77 | +### Layer 2: Interactive commands |
| 78 | + |
| 79 | +Commands that open TUI pickers need special handling: |
| 80 | + |
| 81 | +```bash |
| 82 | +# /mcp opens a selector — verify it renders, then dismiss |
| 83 | +tmux send-keys -t ... "/mcp" Enter |
| 84 | +sleep 3 |
| 85 | +# Capture to verify the picker rendered with server list |
| 86 | +tmux capture-pane -t ... -p -S -40 |
| 87 | +# Dismiss the picker |
| 88 | +tmux send-keys -t ... Escape |
| 89 | +sleep 1 |
| 90 | +``` |
| 91 | + |
| 92 | +### Layer 3: Simple query (tool-use loop, no MCP) |
| 93 | + |
| 94 | +Send a query that exercises the core loop with a local tool: |
| 95 | + |
| 96 | +``` |
| 97 | +List the files in the current directory |
| 98 | +``` |
| 99 | + |
| 100 | +This tests: action plan generation, tool binding, `aider_list_dir_tool` execution, response synthesis. Wait 20 seconds. Verify: |
| 101 | +- Action plan box rendered |
| 102 | +- Tool call shown (look for the tool emoji line) |
| 103 | +- Response box rendered with coherent content |
| 104 | +- No Python tracebacks in verbose output |
| 105 | + |
| 106 | +### Layer 4: MCP tool query |
| 107 | + |
| 108 | +Send a query that forces an MCP tool call: |
| 109 | + |
| 110 | +``` |
| 111 | +Use deepwiki to look up the architecture of bearlike/Assistant |
| 112 | +``` |
| 113 | + |
| 114 | +This tests: MCP tool routing, connection pool, external network call, large response handling. Wait 30-40 seconds. **Critical signals to watch for**: |
| 115 | + |
| 116 | +- `Connected to MCP server` — new on-demand connections (the pool connects lazily for project-level servers) |
| 117 | +- `Disconnected from MCP server` — connection churn during tool-use loop |
| 118 | +- `Failed to reconnect MCP server` — config merge or library compatibility issues |
| 119 | +- `_create_streamable_http_session() got an unexpected keyword argument` — config normalization bug (type/transport collision) |
| 120 | +- `Configuration error: Missing 'transport' key` — plugin MCP config not normalized |
| 121 | + |
| 122 | +These MCP errors often surface only on the SECOND tool call or during `/compact`, because `refresh_if_config_changed` triggers config re-merge. The first call may succeed using the initially-connected pool, while reconnection uses the merged config (which may include CWD `.mcp.json` and plugin configs with different schemas). |
| 123 | + |
| 124 | +### Layer 5: New features |
| 125 | + |
| 126 | +Test recently-added CLI features: |
| 127 | + |
| 128 | +``` |
| 129 | +/fork test-fork — should print "Forked session: <id>" |
| 130 | +/edit What is 2+2? — should re-run with edited prompt and return "4" |
| 131 | +/compact — should produce a summary (may trigger MCP reconnection) |
| 132 | +/budget — should show non-zero token usage after queries |
| 133 | +``` |
| 134 | + |
| 135 | +After `/compact`, check verbose output carefully — compaction re-initializes the tool registry and triggers `refresh_if_config_changed`, which is the most common place for MCP config merge bugs to surface. |
| 136 | + |
| 137 | +## Analyzing results |
| 138 | + |
| 139 | +### Error extraction |
| 140 | + |
| 141 | +After each test, scan the captured output for these patterns: |
| 142 | + |
| 143 | +``` |
| 144 | +WARNING — non-fatal issues (plugin loading, config parsing) |
| 145 | +ERROR — failures that may affect functionality |
| 146 | +Traceback — Python exceptions (CLI crash or near-crash) |
| 147 | +Failed to — connection/reconnection failures |
| 148 | +``` |
| 149 | + |
| 150 | +### MCP config merge issues |
| 151 | + |
| 152 | +When you see MCP reconnection errors, the root cause is usually in the config merge pipeline: |
| 153 | + |
| 154 | +1. **Multiple config formats**: `configs/mcp.json` (Meeseeks native: `servers` + `transport`) vs `.mcp.json` (Claude Code: `mcpServers` + `type`) vs plugin `.mcp.json` (varies) |
| 155 | +2. **Deep merge collision**: `_deep_merge` on individual server configs can produce entries with BOTH `type` and `transport` when CWD overrides global |
| 156 | +3. **Plugin config normalization**: Plugin `.mcp.json` files may use `mcpServers` wrapper or bare server format without `transport` |
| 157 | + |
| 158 | +To trace: read the MCP configs (`configs/mcp.json`, `.mcp.json`, and plugin `.mcp.json` files), then check `_normalize_mcp_config` in `meeseeks_tools/integration/mcp.py` and `load_all_plugin_components` in `meeseeks_core/plugins.py`. |
| 159 | + |
| 160 | +### Session verification |
| 161 | + |
| 162 | +After testing, verify the session was properly tracked: |
| 163 | + |
| 164 | +``` |
| 165 | +/status — should show the current session state |
| 166 | +/budget — token counts should reflect actual usage |
| 167 | +``` |
| 168 | + |
| 169 | +For deeper verification, check Langfuse traces (trace_id == session_id) and MongoDB transcript if available. |
| 170 | + |
| 171 | +## Cleanup |
| 172 | + |
| 173 | +When done testing, exit cleanly: |
| 174 | + |
| 175 | +```bash |
| 176 | +tmux send-keys -t <session>:meeseeks-test "/exit" Enter |
| 177 | +sleep 2 |
| 178 | +tmux kill-window -t <session>:meeseeks-test |
| 179 | +``` |
| 180 | + |
| 181 | +## Non-obvious gotchas |
| 182 | + |
| 183 | +- **Plugin warnings repeat every turn**: Plugin/skill loading runs at each tool-use loop iteration, so the same warnings appear multiple times. This is normal — focus on unique warning messages, not count. |
| 184 | +- **MCP servers connect lazily**: Project-level MCP servers (from `.mcp.json`) may not connect until the first query that needs them. The startup banner only shows globally-configured servers. |
| 185 | +- **`/compact` is the stress test**: It re-initializes the most subsystems (plugins, skills, MCP, tool registry) and is the most likely place to surface integration bugs. |
| 186 | +- **Interactive commands block the prompt**: If you send a query while `/mcp` or `/models` picker is open, it goes to the picker, not the CLI. Always dismiss interactive UI first with `Escape`. |
| 187 | +- **Env var expansion**: CWD `.mcp.json` may contain `${VAR}` references. If the var isn't set in the shell environment where the CLI runs, auth tokens remain as literal strings and MCP calls fail silently. |
0 commit comments