Skip to content

Commit 5058c14

Browse files
konardclaude
konard
and
claude
committed
docs: Add CLI help display research and test scripts
Added comprehensive research findings on stdout vs stderr for help display: - Test scripts comparing popular CLI tools (git, gh, npm) - Research document with findings and recommendations - Evidence that our fix aligns with industry standards 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 4f6e523 commit 5058c14

6 files changed

Lines changed: 397 additions & 0 deletions

File tree

‎experiments/cli-help-research/RESEARCH-FINDINGS.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# CLI Help Display Research: stdout vs stderr
2+
3+
## Question
4+
5+
When a CLI tool is called without required subcommands (like `agent auth`), should the help text be displayed on stdout or stderr?
6+
7+
## Research Methodology
8+
9+
1. Tested popular CLI tools to see how they handle help display
10+
2. Compared behavior between "no subcommand" vs "invalid subcommand"
11+
3. Analyzed yargs framework behavior with `demandCommand`
12+
4. Verified our fix matches industry standards
13+
14+
## Findings
15+
16+
### 1. Popular CLI Tools Behavior (No Subcommand)
17+
18+
| Tool | stdout | stderr | Destination | Status |
19+
| ------ | ---------- | -------- | ----------- | ------ |
20+
| git | 2126 bytes | 0 bytes | STDOUT | ✅ |
21+
| gh | 2442 bytes | 0 bytes | STDOUT | ✅ |
22+
| npm | 1268 bytes | 0 bytes | STDOUT | ✅ |
23+
| docker | 0 bytes | 56 bytes | STDERR | ❌ |
24+
25+
**Conclusion**: 75% of tested tools (git, gh, npm) send help to stdout when called without arguments.
26+
27+
### 2. Invalid Subcommand vs No Subcommand
28+
29+
| Tool | No Subcommand | Invalid Subcommand |
30+
| ---- | ------------------- | ------------------ |
31+
| git | STDOUT (2126 bytes) | STDERR (61 bytes) |
32+
| gh | STDOUT (2442 bytes) | STDERR (378 bytes) |
33+
| npm | STDOUT (1268 bytes) | STDOUT (91 bytes) |
34+
35+
**Key Insight**: Tools distinguish between:
36+
37+
- **Help display** (no args) → STDOUT (informational)
38+
- **Error messages** (invalid args) → STDERR (actual error)
39+
40+
### 3. Our Implementation
41+
42+
#### Before Fix
43+
44+
```bash
45+
agent auth: stdout=0 bytes, stderr=XXX bytes ❌ STDERR
46+
```
47+
48+
#### After Fix
49+
50+
```bash
51+
agent auth: stdout=2242 bytes, stderr=0 bytes ✅ STDOUT
52+
```
53+
54+
### 4. Technical Context
55+
56+
The `agent auth` command uses yargs with `.demandCommand(1, 'Please specify a subcommand')`.
57+
58+
When called without a subcommand, yargs triggers the fail handler with:
59+
60+
- `msg`: "Please specify a subcommand"
61+
- `err`: undefined
62+
63+
This is a **validation failure**, not a true error. The user hasn't done anything wrong - they're likely exploring the CLI or need help.
64+
65+
## Conclusion
66+
67+
**Our approach is CORRECT** based on the following evidence:
68+
69+
1. **Industry Standard**: Leading CLI tools (git, gh, npm) display help on stdout when called without subcommands
70+
2. **User Intent**: When users type `agent auth`, they're seeking information, not encountering an error
71+
3. **Terminal Behavior**: Many terminals display stderr in red, which is inappropriate for help text
72+
4. **Semantic Correctness**: Help display is informational (stdout), not an error condition (stderr)
73+
74+
## The Fix
75+
76+
Changed `src/index.js` line 640-641 from:
77+
78+
```javascript
79+
console.error(msg);
80+
console.error(yargs.help());
81+
```
82+
83+
To:
84+
85+
```javascript
86+
console.log(stripAnsi(msg));
87+
console.log(`\n${stripAnsi(yargs.help())}`);
88+
```
89+
90+
This ensures:
91+
92+
- Help text goes to stdout (like git, gh, npm)
93+
- No red color in terminals
94+
- Clean, colorless output using `stripAnsi()`
95+
96+
## Recommendation
97+
98+
✅ **Keep the current fix** - it aligns with industry standards and best practices.

‎experiments/cli-help-research/test-agent-behavior.sh‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
#!/bin/bash
2+
3+
echo "Testing actual agent binary behavior"
4+
echo "====================================="
5+
echo ""
6+
7+
# Build the agent first
8+
cd /tmp/gh-issue-solver-1766347654233
9+
if [ ! -f "dist/agent" ]; then
10+
echo "Building agent..."
11+
bun run build > /dev/null 2>&1
12+
fi
13+
14+
echo "Test 1: 'agent auth' (no subcommand)"
15+
echo "---"
16+
stdout_file=$(mktemp)
17+
stderr_file=$(mktemp)
18+
19+
./dist/agent auth > "$stdout_file" 2> "$stderr_file" || true
20+
21+
stdout_size=$(wc -c < "$stdout_file")
22+
stderr_size=$(wc -c < "$stderr_file")
23+
24+
echo "stdout size: $stdout_size bytes"
25+
echo "stderr size: $stderr_size bytes"
26+
27+
if [ "$stdout_size" -gt 0 ]; then
28+
echo "stdout preview:"
29+
head -5 "$stdout_file"
30+
fi
31+
32+
if [ "$stderr_size" -gt 0 ]; then
33+
echo "stderr preview:"
34+
head -5 "$stderr_file"
35+
fi
36+
37+
echo ""
38+
echo "Test 2: Compare with 'git' (no subcommand)"
39+
echo "---"
40+
git > "$stdout_file" 2> "$stderr_file" || true
41+
stdout_size=$(wc -c < "$stdout_file")
42+
stderr_size=$(wc -c < "$stderr_file")
43+
echo "git: stdout=$stdout_size bytes, stderr=$stderr_size bytes"
44+
[ "$stdout_size" -gt "$stderr_size" ] && echo " → Help goes to STDOUT ✅" || echo " → Help goes to STDERR ❌"
45+
46+
echo ""
47+
echo "Test 3: Compare with 'gh' (no subcommand)"
48+
echo "---"
49+
gh > "$stdout_file" 2> "$stderr_file" || true
50+
stdout_size=$(wc -c < "$stdout_file")
51+
stderr_size=$(wc -c < "$stderr_file")
52+
echo "gh: stdout=$stdout_size bytes, stderr=$stderr_size bytes"
53+
[ "$stdout_size" -gt "$stderr_size" ] && echo " → Help goes to STDOUT ✅" || echo " → Help goes to STDERR ❌"
54+
55+
echo ""
56+
./dist/agent auth > "$stdout_file" 2> "$stderr_file" || true
57+
stdout_size=$(wc -c < "$stdout_file")
58+
stderr_size=$(wc -c < "$stderr_file")
59+
echo "agent auth: stdout=$stdout_size bytes, stderr=$stderr_size bytes"
60+
[ "$stdout_size" -gt "$stderr_size" ] && echo " → Help goes to STDOUT ✅" || echo " → Help goes to STDERR ❌"
61+
62+
rm -f "$stdout_file" "$stderr_file"
63+
64+
echo ""
65+
echo "====================================="
66+
echo "Summary: With our fix, 'agent auth' now behaves like 'git' and 'gh'"

‎experiments/cli-help-research/test-agent-src.sh‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/bin/bash
2+
3+
echo "Testing agent source behavior"
4+
echo "====================================="
5+
echo ""
6+
7+
cd /tmp/gh-issue-solver-1766347654233
8+
9+
echo "Test 1: 'agent auth' with current fix"
10+
echo "---"
11+
stdout_file=$(mktemp)
12+
stderr_file=$(mktemp)
13+
14+
bun run src/index.js auth > "$stdout_file" 2> "$stderr_file" || true
15+
16+
stdout_size=$(wc -c < "$stdout_file")
17+
stderr_size=$(wc -c < "$stderr_file")
18+
19+
echo "stdout size: $stdout_size bytes"
20+
echo "stderr size: $stderr_size bytes"
21+
22+
if [ "$stdout_size" -gt 50 ]; then
23+
echo ""
24+
echo "First few lines of stdout:"
25+
head -10 "$stdout_file"
26+
fi
27+
28+
echo ""
29+
echo "Comparison with popular CLI tools:"
30+
echo "---"
31+
32+
# Test git
33+
git > "$stdout_file" 2> "$stderr_file" || true
34+
git_stdout=$(wc -c < "$stdout_file")
35+
git_stderr=$(wc -c < "$stderr_file")
36+
37+
# Test gh
38+
gh > "$stdout_file" 2> "$stderr_file" || true
39+
gh_stdout=$(wc -c < "$stdout_file")
40+
gh_stderr=$(wc -c < "$stderr_file")
41+
42+
# Test npm
43+
npm > "$stdout_file" 2> "$stderr_file" || true
44+
npm_stdout=$(wc -c < "$stdout_file")
45+
npm_stderr=$(wc -c < "$stderr_file")
46+
47+
# Test agent auth again
48+
bun run src/index.js auth > "$stdout_file" 2> "$stderr_file" || true
49+
agent_stdout=$(wc -c < "$stdout_file")
50+
agent_stderr=$(wc -c < "$stderr_file")
51+
52+
echo "git: stdout=$git_stdout bytes, stderr=$git_stderr bytes $([ "$git_stdout" -gt "$git_stderr" ] && echo '✅ STDOUT' || echo '❌ STDERR')"
53+
echo "gh: stdout=$gh_stdout bytes, stderr=$gh_stderr bytes $([ "$gh_stdout" -gt "$gh_stderr" ] && echo '✅ STDOUT' || echo '❌ STDERR')"
54+
echo "npm: stdout=$npm_stdout bytes, stderr=$npm_stderr bytes $([ "$npm_stdout" -gt "$npm_stderr" ] && echo '✅ STDOUT' || echo '❌ STDERR')"
55+
echo "agent auth: stdout=$agent_stdout bytes, stderr=$agent_stderr bytes $([ "$agent_stdout" -gt "$agent_stderr" ] && echo '✅ STDOUT' || echo '❌ STDERR')"
56+
57+
rm -f "$stdout_file" "$stderr_file"
58+
59+
echo ""
60+
echo "====================================="

‎experiments/cli-help-research/test-cli-tools.sh‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
#!/bin/bash
2+
3+
# Test how popular CLI tools handle help display
4+
# This script checks whether help goes to stdout or stderr
5+
6+
echo "Testing CLI tools for help display behavior..."
7+
echo "=============================================="
8+
echo ""
9+
10+
# Function to test a command
11+
test_command() {
12+
local cmd="$1"
13+
local description="$2"
14+
15+
echo "Testing: $description"
16+
echo "Command: $cmd"
17+
18+
# Run command and capture stdout/stderr separately
19+
stdout_file=$(mktemp)
20+
stderr_file=$(mktemp)
21+
22+
eval "$cmd" > "$stdout_file" 2> "$stderr_file" || true
23+
24+
stdout_size=$(wc -c < "$stdout_file")
25+
stderr_size=$(wc -c < "$stderr_file")
26+
27+
echo " stdout size: $stdout_size bytes"
28+
echo " stderr size: $stderr_size bytes"
29+
30+
if [ "$stdout_size" -gt "$stderr_size" ]; then
31+
echo " ✅ Help goes to STDOUT"
32+
else
33+
echo " ❌ Help goes to STDERR"
34+
fi
35+
36+
rm -f "$stdout_file" "$stderr_file"
37+
echo ""
38+
}
39+
40+
# Test git (one of the most popular CLI tools)
41+
test_command "git" "git without subcommand"
42+
43+
# Test docker
44+
test_command "docker" "docker without subcommand"
45+
46+
# Test npm
47+
test_command "npm" "npm without subcommand"
48+
49+
# Test gh (GitHub CLI - similar tool to agent)
50+
test_command "gh" "gh without subcommand"
51+
52+
# Test kubectl if available
53+
if command -v kubectl &> /dev/null; then
54+
test_command "kubectl" "kubectl without subcommand"
55+
fi
56+
57+
# Test aws if available
58+
if command -v aws &> /dev/null; then
59+
test_command "aws" "aws without subcommand"
60+
fi
61+
62+
echo "=============================================="
63+
echo "Summary: Most modern CLI tools display help on STDOUT when called without arguments"

‎experiments/cli-help-research/test-invalid-vs-help.sh‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/bin/bash
2+
3+
echo "Testing: INVALID subcommand vs NO subcommand"
4+
echo "=============================================="
5+
echo ""
6+
7+
test_both() {
8+
local tool="$1"
9+
local invalid_cmd="$2"
10+
11+
echo "Tool: $tool"
12+
echo "---"
13+
14+
# Test with no subcommand
15+
stdout_file=$(mktemp)
16+
stderr_file=$(mktemp)
17+
eval "$tool" > "$stdout_file" 2> "$stderr_file" || true
18+
stdout_size=$(wc -c < "$stdout_file")
19+
stderr_size=$(wc -c < "$stderr_file")
20+
21+
echo " No subcommand:"
22+
echo " stdout: $stdout_size bytes, stderr: $stderr_size bytes"
23+
[ "$stdout_size" -gt "$stderr_size" ] && echo " → STDOUT" || echo " → STDERR"
24+
25+
# Test with invalid subcommand
26+
eval "$invalid_cmd" > "$stdout_file" 2> "$stderr_file" || true
27+
stdout_size=$(wc -c < "$stdout_file")
28+
stderr_size=$(wc -c < "$stderr_file")
29+
30+
echo " Invalid subcommand:"
31+
echo " stdout: $stdout_size bytes, stderr: $stderr_size bytes"
32+
[ "$stdout_size" -gt "$stderr_size" ] && echo " → STDOUT" || echo " → STDERR"
33+
34+
rm -f "$stdout_file" "$stderr_file"
35+
echo ""
36+
}
37+
38+
test_both "git" "git invalidcmd123"
39+
test_both "gh" "gh invalidcmd123"
40+
test_both "npm" "npm invalidcmd123"
41+
42+
echo "=============================================="
43+
echo "Key Finding:"
44+
echo "- Help display (no args): Usually goes to STDOUT"
45+
echo "- Error messages (invalid args): Usually goes to STDERR"

0 commit comments

Comments
 (0)