Skip to content

Commit 11dc348

Browse files
UNC1739UNC1739
committed
Add a CLI-surface drift gate on capability-sdk/pkg/clisurface
Adopts capability-sdk/pkg/clisurface so the documented cobra surface cannot silently drift from the live tree.
1 parent 320a3a5 commit 11dc348

10 files changed

Lines changed: 799 additions & 3 deletions

File tree

‎.github/workflows/cli-surface.yml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: CLI Surface Drift
2+
3+
# Fails when the documented CLI surface drifts from the live cobra tree.
4+
# No `paths:` filter: docs-only PRs skip ci.yml.
5+
on:
6+
# No `branches:` filter: `on.pull_request.branches` matches the BASE, so
7+
# `branches: [main]` would skip stacked PRs.
8+
pull_request:
9+
push:
10+
branches: [main]
11+
workflow_dispatch:
12+
13+
concurrency:
14+
group: ${{ github.workflow }}-${{ github.ref }}
15+
cancel-in-progress: true
16+
17+
permissions:
18+
contents: read
19+
20+
jobs:
21+
cli-surface:
22+
name: CLI Surface Drift
23+
runs-on: ubuntu-latest
24+
timeout-minutes: 15
25+
steps:
26+
- name: Checkout code
27+
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
28+
with:
29+
persist-credentials: false
30+
31+
- name: Setup Go
32+
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
33+
with:
34+
go-version-file: go.mod
35+
36+
- name: Snapshot the committed documentation
37+
run: |
38+
set -eo pipefail
39+
find docs README.md -type f -exec sha256sum {} + \
40+
| LC_ALL=C sort > "$RUNNER_TEMP/docs.before"
41+
echo "baselined $(wc -l < "$RUNNER_TEMP/docs.before") file(s)"
42+
43+
- name: Check CLI surface and documentation for drift
44+
run: |
45+
set -eo pipefail
46+
go test ./pkg/runner -run 'TestCLISurface' -count=1 -v \
47+
| tee "$RUNNER_TEMP/gate.log"
48+
for name in TestCLISurface TestCLISurfaceDocLint TestCLISurfaceGateDetectsRename; do
49+
if ! grep -qE "^--- PASS: ${name} " "$RUNNER_TEMP/gate.log"; then
50+
echo "::error::${name} did not run. The drift gate is not executing; check the -run pattern against the test names in pkg/runner/cli_surface_test.go."
51+
exit 1
52+
fi
53+
done
54+
55+
- name: Verify the gate did not rewrite the goldens
56+
run: |
57+
set -eo pipefail
58+
find docs README.md -type f -exec sha256sum {} + \
59+
| LC_ALL=C sort > "$RUNNER_TEMP/docs.after"
60+
if ! diff -u "$RUNNER_TEMP/docs.before" "$RUNNER_TEMP/docs.after"; then
61+
echo "::error::The drift gate wrote to the committed documentation; the diff above names every file added, removed or changed. Check mode must compare without writing, so this is a defect in the gate itself -- pkg/runner/cli_surface_test.go, or a clisurface write reached outside the -update branch. Regenerating with 'make cli-docs' will not fix it."
62+
exit 1
63+
fi

‎.goreleaser.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@ archives:
3030
checksum:
3131
name_template: "checksums.txt"
3232

33+
release:
34+
extra_files:
35+
- glob: docs/cli-surface.json
36+
3337
changelog:
3438
sort: asc
3539
filters:

‎Makefile‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ LDFLAGS := -s -w
2020
# Declare all phony targets
2121
.PHONY: all build build-all build-linux build-darwin build-windows \
2222
release release-checksums \
23-
test test-verbose test-coverage lint clean help
23+
test test-verbose test-coverage lint clean help cli-docs
2424

2525
# =============================================================================
2626
# Build targets (local development)
@@ -79,6 +79,11 @@ test-coverage: ## Run tests with coverage report
7979
lint: ## Run linter
8080
golangci-lint run ./...
8181

82+
cli-docs: ## Regenerate CLI surface docs from the live cobra tree
83+
@GOWORK=off go test ./pkg/runner -list 'TestCLISurface' | grep -qE '^TestCLISurface$$' \
84+
|| { echo "cli-docs: 'go test -list' did not report TestCLISurface in ./pkg/runner. Either the -update writer was renamed, or the package failed to build -- run 'go build ./pkg/runner' to tell which. 'go test -run' exits 0 when its pattern matches nothing, so without this check the target would report success having regenerated nothing at all."; exit 1; }
85+
GOWORK=off go test ./pkg/runner -run 'TestCLISurface' -count=1 -update
86+
8287
clean: ## Clean build artifacts
8388
rm -rf $(BUILD_DIR) $(DIST_DIR)
8489
rm -f coverage.out coverage.html

‎README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,22 @@ julius probe https://target.example.com
8080
+----------------------------+---------+-------------+-------------+--------+-------+
8181
```
8282

83+
### Subcommands
84+
85+
<!-- BEGIN generated: cli-subcommands -->
86+
Julius organizes its functionality into these focused subcommands:
87+
88+
```bash
89+
julius list # List all available probe definitions
90+
julius probe # Probe targets to identify LLM services
91+
julius validate # Validate probe definition files
92+
```
93+
<!-- END generated: cli-subcommands -->
94+
95+
<!-- BEGIN generated: cli-aliases -->
96+
The full reference — every subcommand, alias and flag, including the ones hidden from `--help` — is generated into [docs/CLI.md](docs/CLI.md).
97+
<!-- END generated: cli-aliases -->
98+
8399
## Supported LLM Services
84100

85101
Julius identifies 64 LLM platforms across self-hosted, gateway, MCP, RAG/orchestration, and cloud-managed categories:

‎docs/CLI.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
<!-- Generated from the live cobra command tree by 'make cli-docs'. Do not edit by hand. -->
2+
3+
# julius CLI reference
4+
5+
Every command, alias and flag below is derived from the cobra command tree, not from prose.
6+
Schema version 1, surface hash `sha256:0b90ec6c9ac2f198300eb2bb9b760049de4257ac3240da746f689915930e3299`.
7+
8+
Regenerate with `make cli-docs` after adding, removing or renaming a command or a flag.
9+
10+
## Command index
11+
12+
| Command | Aliases | Description |
13+
| --- | --- | --- |
14+
| [`julius`](#julius) | *(none)* | Julius - LLM Service Fingerprinting Tool |
15+
| [`julius list`](#julius-list) | *(none)* | List all available probe definitions |
16+
| [`julius probe`](#julius-probe) | *(none)* | Probe targets to identify LLM services |
17+
| [`julius validate`](#julius-validate) | *(none)* | Validate probe definition files |
18+
19+
## `julius`
20+
21+
Julius - LLM Service Fingerprinting Tool
22+
23+
- Usage: `julius`
24+
- Aliases: *(none)*
25+
- Requires a subcommand
26+
27+
### Flags
28+
29+
| Flag | Short | Type | Default | Description |
30+
| --- | --- | --- | --- | --- |
31+
| `--banner` | | bool | `true` | Show ASCII banner |
32+
| `--ca-cert` | | string | | Path to custom CA certificate file |
33+
| `--concurrency` | `-c` | int | `10` | Maximum concurrent probe requests per target |
34+
| `--insecure` | | bool | `false` | Skip TLS certificate verification |
35+
| `--max-response-size` | | int64 | `10485760` | Maximum response body size in bytes (default 10MB) |
36+
| `--no-color` | | bool | `false` | Disable color output |
37+
| `--output` | `-o` | string | `table` | Output format (table, json, jsonl) |
38+
| `--probes-dir` | `-p` | string | | Override probe definitions directory |
39+
| `--quiet` | `-q` | bool | `false` | Suppress non-match output |
40+
| `--timeout` | `-t` | int | `5` | HTTP timeout in seconds |
41+
| `--verbose` | `-v` | bool | `false` | Verbose output |
42+
43+
## `julius list`
44+
45+
List all available probe definitions
46+
47+
- Usage: `julius list`
48+
- Aliases: *(none)*
49+
50+
### Inherited flags
51+
52+
| Flag | Short | Type | Default | Description |
53+
| --- | --- | --- | --- | --- |
54+
| `--banner` | | bool | `true` | Show ASCII banner |
55+
| `--ca-cert` | | string | | Path to custom CA certificate file |
56+
| `--concurrency` | `-c` | int | `10` | Maximum concurrent probe requests per target |
57+
| `--insecure` | | bool | `false` | Skip TLS certificate verification |
58+
| `--max-response-size` | | int64 | `10485760` | Maximum response body size in bytes (default 10MB) |
59+
| `--no-color` | | bool | `false` | Disable color output |
60+
| `--output` | `-o` | string | `table` | Output format (table, json, jsonl) |
61+
| `--probes-dir` | `-p` | string | | Override probe definitions directory |
62+
| `--quiet` | `-q` | bool | `false` | Suppress non-match output |
63+
| `--timeout` | `-t` | int | `5` | HTTP timeout in seconds |
64+
| `--verbose` | `-v` | bool | `false` | Verbose output |
65+
66+
## `julius probe`
67+
68+
Probe targets to identify LLM services
69+
70+
- Usage: `julius probe [targets...]`
71+
- Aliases: *(none)*
72+
73+
### Flags
74+
75+
| Flag | Short | Type | Default | Description |
76+
| --- | --- | --- | --- | --- |
77+
| `--augustus` | | bool | `false` | Include Augustus generator configs in output |
78+
| `--base-paths` | | string | | Comma-separated path prefixes to prepend to probe paths (e.g., /api,/proxy) |
79+
| `--file` | `-f` | string | | Read targets from file |
80+
| `--header` | `-H` | stringArray | `[]` | Custom HTTP header (e.g., "Authorization: Bearer token"). Can be specified multiple times |
81+
82+
### Inherited flags
83+
84+
| Flag | Short | Type | Default | Description |
85+
| --- | --- | --- | --- | --- |
86+
| `--banner` | | bool | `true` | Show ASCII banner |
87+
| `--ca-cert` | | string | | Path to custom CA certificate file |
88+
| `--concurrency` | `-c` | int | `10` | Maximum concurrent probe requests per target |
89+
| `--insecure` | | bool | `false` | Skip TLS certificate verification |
90+
| `--max-response-size` | | int64 | `10485760` | Maximum response body size in bytes (default 10MB) |
91+
| `--no-color` | | bool | `false` | Disable color output |
92+
| `--output` | `-o` | string | `table` | Output format (table, json, jsonl) |
93+
| `--probes-dir` | `-p` | string | | Override probe definitions directory |
94+
| `--quiet` | `-q` | bool | `false` | Suppress non-match output |
95+
| `--timeout` | `-t` | int | `5` | HTTP timeout in seconds |
96+
| `--verbose` | `-v` | bool | `false` | Verbose output |
97+
98+
## `julius validate`
99+
100+
Validate probe definition files
101+
102+
- Usage: `julius validate [directory]`
103+
- Aliases: *(none)*
104+
105+
### Inherited flags
106+
107+
| Flag | Short | Type | Default | Description |
108+
| --- | --- | --- | --- | --- |
109+
| `--banner` | | bool | `true` | Show ASCII banner |
110+
| `--ca-cert` | | string | | Path to custom CA certificate file |
111+
| `--concurrency` | `-c` | int | `10` | Maximum concurrent probe requests per target |
112+
| `--insecure` | | bool | `false` | Skip TLS certificate verification |
113+
| `--max-response-size` | | int64 | `10485760` | Maximum response body size in bytes (default 10MB) |
114+
| `--no-color` | | bool | `false` | Disable color output |
115+
| `--output` | `-o` | string | `table` | Output format (table, json, jsonl) |
116+
| `--probes-dir` | `-p` | string | | Override probe definitions directory |
117+
| `--quiet` | `-q` | bool | `false` | Suppress non-match output |
118+
| `--timeout` | `-t` | int | `5` | HTTP timeout in seconds |
119+
| `--verbose` | `-v` | bool | `false` | Verbose output |

‎docs/cli-surface-allow.txt‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Flag names documentation may mention even though the CLI does not accept them.
2+
#
3+
# The CLI-surface doc lint (see pkg/runner/cli_surface_test.go) checks every julius invocation in
4+
# a fenced code block, every backticked flag name in prose, and every flag name
5+
# in a Go comment under the configured Go directories against the flags cobra
6+
# actually registers. This file is the escape hatch for deliberate mentions.
7+
# Every entry MUST carry a '#' reason: the gate rejects an entry without one,
8+
# because an unexplained exception is how a stale flag reference survives forever.
9+
#
10+
# Keep this list as short as possible. If a mention is not deliberate, fix the
11+
# document instead of allowing it.
12+
13+
--api-key # vLLM's own flag, named in SECURITY.md as the recommended configuration for that service

0 commit comments

Comments
 (0)