Skip to content

Commit 7c3434e

Browse files
rsteube-botrsteube-bot
committed
added AGENTS.md
Assisted-by: Crush:glm-5.2
1 parent fb4560c commit 7c3434e

1 file changed

Lines changed: 136 additions & 0 deletions

File tree

‎AGENTS.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# AGENTS.md
2+
3+
Guide for agents working in the `carapace-shlex` repository — a multi-shell command-line lexer (fork of go-shlex) that splits and re-joins command lines while tracking quotation state for shell completion.
4+
5+
## Commands
6+
7+
```bash
8+
# Build everything (library + CLI)
9+
go build -v ./...
10+
11+
# Run tests with coverage (matches CI)
12+
go test -v -coverprofile=profile.cov ./...
13+
14+
# Formatting check enforced by CI — note the -s (simplify) flag
15+
gofmt -d -s .
16+
17+
# Static analysis enforced by CI
18+
go install honnef.co/go/tools/cmd/staticcheck@latest && staticcheck ./...
19+
20+
# Run the CLI directly (no build step needed)
21+
go run ./cmd/carapace-shlex --format bash --completion-context "echo foo | grep hel"
22+
go run ./cmd/carapace-shlex --format elvish --current-pipeline --words "bat | {|"
23+
24+
# Test a single format
25+
go test -run TestElvish -v ./...
26+
```
27+
28+
Go 1.24.0. The CI image is `ghcr.io/carapace-sh/go:1.25.4`. Tags trigger GoReleaser builds (see `.goreleaser.yml`).
29+
30+
## Repository Layout
31+
32+
- **Root package `shlex`** — the library: tokenizer state machine (`shlex.go`), `Format` interface (`format.go`), per-shell formats (`format_<shell>.go`), token slice operations (`tokenslice.go`), wordbreak types (`wordbreak.go`), quoting helpers (`quote.go`), completion context (`completion.go`).
33+
- **`cmd/carapace-shlex/`** — a **separate Go module** (`cmd/go.mod`) that imports the library and wraps it as a cobra CLI. It depends on `carapace` and `carapace-bridge` for its own completion.
34+
- **`go.work`** — workspace including both the root module and `./cmd`. Contains a `replace` directive: `github.com/carapace-sh/carapace v1.11.0 => ../carapace`. This means local development expects a sibling `../carapace` checkout. The `cmd/go.mod` has its own `replace ... => ../` for the shlex library itself.
35+
- **`skills/shlex/`** — in-depth reference docs (architecture, cross-shell comparison, per-format references). Load these via the `shlex` skill when doing substantial format work.
36+
- **`plan.md`** — design doc for elvish lambda-pipe handling; useful context for the `PostProcessor` interface and `WORDBREAK_LAMBDA_PIPE`.
37+
38+
## Architecture
39+
40+
### Data flow
41+
42+
```
43+
command line string
44+
→ SplitWith(s, format) or SplitForCompletion(s, format)
45+
→ format.Classifier() (rune → rune class)
46+
→ tokenizer.scanStream() (flat state machine, shared across all formats)
47+
→ TokenSlice (typed tokens with Span + quotation State)
48+
→ [optional] format.PostProcess(tokens) (post-pass reclassification)
49+
→ CompletionContext (current word, prefix, quoting state, pipeline)
50+
```
51+
52+
### The tokenizer is a flat state machine
53+
54+
The core state machine in `shlex.go` (`scanStream`) has **no nesting awareness**. It classifies runes one at a time into `TokenType`s (WORD/SPACE/COMMENT/WORDBREAK) and tracks quotation state via `LexerState`. Every shell format plugs into the **same** machine via the `Format` interface — there is no per-shell parser.
55+
56+
**Consequence**: format-specific behavior that requires context the flat machine can't track (e.g. elvish `|` inside `{|params|}` being a parameter delimiter, not a pipeline pipe) is handled via the optional `PostProcessor` interface, which runs a post-pass over the `TokenSlice` after tokenization. Do **not** add nesting/brace tracking into `scanStream` — add a `PostProcess` method on the format instead.
57+
58+
### The `Format` interface and its optional companions
59+
60+
`format.go` defines:
61+
- **`Format`** (required): `Classifier`, `ClassifyOperator`, `KeywordOperators`, quote-behavior flags (`NonEscapingQuoteEscapes`, `NonEscapingQuoteBackslashEscapes`, `EscapeNotBareword`, `EscapeNotInEscapingQuote`, `EscapingQuoteEscapeChars`), `TripleQuoteSupport`, `RawPrefixSupport`, `QuoteWord`.
62+
- **Optional interfaces** (asserted via type assertion in `tokenizer.Next` / `SplitWith`):
63+
- `PostProcessor` — post-pass token reclassification (elvish, nushell)
64+
- `BlockCommenter` — multi-line block comments (PowerShell `<# #>`)
65+
- `StopParsingToken` — raw lexing mode after a token (PowerShell `--%`)
66+
- `LineContinuationEscaper` — escape+newline as line continuation (PowerShell backtick)
67+
- `EscapingQuoteUnescaper` — custom unescape inside double quotes beyond simple backslash-dropping
68+
69+
When adding a new format, implement `Format` plus whichever optional interfaces apply. No-op returns (e.g. `KeywordOperators() nil`) are the norm for formats that don't need a feature.
70+
71+
### Token model
72+
73+
```go
74+
type Token struct {
75+
Type TokenType // WORD_TOKEN, WORDBREAK_TOKEN, etc.
76+
Value string // dequoted value
77+
RawValue string // raw source text including quotes/escapes
78+
Span Span // rune offsets {Start, End} — NOT byte offsets
79+
State LexerState // quotation state after this token
80+
WordbreakType WordbreakType // operator type for WORDBREAK_TOKENs
81+
WordbreakIndex int // index of last opening quote in Value (prefix boundary)
82+
}
83+
```
84+
85+
`Span` offsets are **rune offsets**, not byte offsets — relevant when inspecting multi-byte input. `TokenSlice.Words()` merges tokens by `Span` adjacency (End==Start), so quote-openers, wordbreaks, and word fragments that touch get merged into one word.
86+
87+
### WordbreakType drives TokenSlice operations
88+
89+
`WordbreakType.IsPipelineDelimiter()` and `IsRedirect()` determine how `Pipelines()`, `CurrentPipeline()`, `FilterRedirects()`, and `WordbreakPrefix()` behave. When adding a new operator type, decide deliberately whether it should split pipelines or be filtered as a redirect — `WORDBREAK_LAMBDA_PIPE` intentionally returns false for both so elvish lambda parameter lists don't break pipeline splitting.
90+
91+
`FilterRedirects()` has a special case: a numeric token (e.g. `2`) immediately adjoining a redirect operator (e.g. `>`) is filtered out as the fd prefix. Don't break this when touching redirect logic.
92+
93+
### Public API surface
94+
95+
- `Split(s)` / `SplitWith(s, format)` → `TokenSlice, error`
96+
- `SplitForCompletion(s, format)` → `*CompletionContext` (never errors; returns empty context with `START_STATE` on failure)
97+
- `Join(s)` / `JoinWith(s, format)` → quoted string
98+
- `CompletionContext` — the completion-oriented API: `Words`, `CurrentWord`, `RawCurrentWord`, `Prefix`, `QuotingState`, `IsRedirect`, `InLambdaParams`, and `Pipeline` (raw token escape hatch)
99+
100+
`SplitForCompletion` is the primary entry point for completion callers (carapace). It internally calls `SplitWith` then derives the context fields. `InLambdaParams` is detected via an odd count of `WORDBREAK_LAMBDA_PIPE` in the current pipeline (toggle heuristic — see limitations in `plan.md`).
101+
102+
## Adding a New Shell Format
103+
104+
1. Create `format_<shell>.go` implementing `Format` (+ optional interfaces as needed).
105+
2. Add a `<shell>QuoteWord` function in `quote.go` and reference it from the format's `QuoteWord` method. Quoting helpers are kept in `quote.go`, not in the format file.
106+
3. If the shell has operators that differ from the bash grammar, add a `<shell>WordbreakType` function in `wordbreak.go` (see `bashWordbreakType`, `tcshWordbreakType` as templates).
107+
4. Add the format to `formatFromFlag` in `cmd/carapace-shlex/cmd/root.go` and to the `--format` flag's completion values.
108+
5. Add the format to the table in `README.md`.
109+
6. Create `format_<shell>_test.go` (see existing test files for the pattern).
110+
7. If the shell needs behavior the flat state machine can't express, implement `PostProcessor` rather than modifying `scanStream`.
111+
112+
## Testing Patterns
113+
114+
Tests live alongside their format: `format_<shell>_test.go`. The established pattern:
115+
116+
```go
117+
tokens, err := SplitWith(input, SomeFormat())
118+
if err != nil { t.Fatal(err) }
119+
words := tokens.Words().Strings()
120+
// assert on words, and on token State / WordbreakType for quoting/operator cases
121+
```
122+
123+
Tests assert on **dequoted `Value`** via `Words().Strings()`, and on `State` (e.g. `IN_WORD_STATE`, `QUOTING_STATE`) and `WordbreakType` for quotation/operator behavior. The `Equal` method on `Token` compares all fields — useful for golden-style tests.
124+
125+
`completion_test.go` tests `SplitForCompletion` and the `CompletionContext` fields. `join_test.go` tests `JoinWith` roundtrips. `shlex_test.go` tests the core tokenizer/state machine.
126+
127+
## Gotchas
128+
129+
- **`go.work` replace expects `../carapace`** as a sibling checkout. The root `go.mod` also has the replace, so even non-workspace `go build` pulls the local carapace. CI uses the image `ghcr.io/carapace-sh/go` which has the dependency available.
130+
- **`cmd/carapace-shlex` is a separate module** with its own `go.mod`. Changes to the library's public API must be reflected in `cmd/go.mod`'s `require` (often via a replace to `../`).
131+
- **gofmt `-s` (simplify) is enforced**, not just plain gofmt. Run `gofmt -d -s .` before committing.
132+
- **`staticcheck` is enforced** in CI. Install and run it locally — it's not in the standard toolchain.
133+
- **`bufio.Reader` only supports one `UnreadRune`**. The triple-quote peek helpers (`checkTripleQuote`, `checkTripleClose`) handle this constraint by returning a `consumedRune` when the second peek fails to match — the first peeked rune can't be unread, so callers must add it to `RawValue`. Preserve this pattern when extending peek-based logic.
134+
- **`BashFormat().Classifier()` reads `COMP_WORDBREAKS`** from the environment at call time, not at init. Tests that assert on bash wordbreaks should set/unset `COMP_WORDBREAKS` explicitly or they inherit the ambient value.
135+
- **`plan.md` is a design doc, not implemented spec**. The elvish lambda-pipe `PostProcess` is implemented, but `InLambdaParams` uses a toggle heuristic that doesn't handle nested lambdas — documented as a known limitation in `plan.md`.
136+
- **Don't add comments to code** unless explaining *why* (and only when non-obvious). The codebase uses minimal comments; match that style.

0 commit comments

Comments
 (0)