Add csv-star importer: Starlark-configured CSV import pipelines - #147
Merged
Conversation
yugui
force-pushed
the
claude/csv-importer-design-k7pbft
branch
4 times, most recently
from
July 20, 2026 07:53
2f01385 to
c5b245d
Compare
Experience with the csv-sexp experiment surfaced two usability problems: S-expression programs embedded in TOML strings get no editor support, so matching parentheses is hard, and pipeline nodes can only be named through let* bindings, making the primary activity — wiring nodes — feel second-class. csv-star keeps the same csvbase substrate and the same evaluation model (the program runs exactly once at construction and builds a static, applicative step graph; rows are evaluated by the frozen pipeline), but the program is Starlark. Assignments bind pipeline steps as first-class values, plain literals auto-lift to constants, and regexes read as raw strings. Per-row logic escapes to real Starlark through exactly two carriers — apply(f, ...) for scalar computation and emit(implementation = f, ...) as the terminal — so Key-level construction forms and selective conditional combinators are dropped entirely; branching lives in ordinary if/elif inside callbacks, and directive assembly uses value-level constructors on resolved values. Callbacks receive and return plain values, never keys, which keeps the dependency graph static and analyzable at load time. Each step records its .star call site, so load-time errors and row diagnostics point at the program source instead of at anonymous nodes. The surface is callback-centric: implementation functions receive a Bazel-rule-style ctx (rowhash, raw row, hints, warnings) as their first argument; referentially transparent basics (parse_date, parse_amount, date_offset, amount) are phase-polymorphic — applied to keys they register steps, applied to plain values they compute immediately, so the same vocabulary works in both worlds; and a first-class decimal type backs amount arithmetic, with floats accepted only at a shortest round-trip conversion boundary so money never passes through binary float. Errors raised inside callbacks drop only the offending row with a diagnostic, mirroring how bad data behaves elsewhere in the pipeline; a loading-phase builtin invoked from a callback fails immediately with a didactic error rather than corrupting the frozen graph. Loading-phase builtins are deliberately limited to operations that contribute diagnostic or nil-identity semantics a callback cannot express; pure computation belongs to callbacks. Programs may be inlined in TOML or referenced via program_file, with load() shared libraries confined to the program's directory. Alternatives considered: Bazel-style name-attr node declaration (rejected — its name indirection serves external addressability needs this DSL does not have, at the cost of reintroducing mandatory binding ceremony) and per-row Starlark execution of whole programs (rejected — it would bypass csvbase's diagnostic and idempotency machinery). The kind registers as "csv-star" and coexists with csv-sexp; the two remain parallel experiments until one wins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6piKCgFNqYpcecRkEU2xz
Side-by-side use on real bank statements settled the experiment: csv-star covers every csv-sexp use case with better editor support, first-class node binding, and ordinary control flow in callbacks, so the S-expression surface no longer pays for its maintenance. Along with the package, drop the csvbase combinators that existed only to serve it: the selective conditional layer (If and the predicate steps) and NilDirective. Conditional per-row logic now lives in csv-star's callbacks, where it needs no combinator vocabulary at all. The remaining csvbase steps stay untouched as library primitives; csvimp continues to build on them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6piKCgFNqYpcecRkEU2xz
yugui
force-pushed
the
claude/csv-importer-design-k7pbft
branch
from
July 20, 2026 09:40
c5b245d to
61daa17
Compare
govulncheck fails CI with GO-2026-5856 (Encrypted Client Hello privacy leak in crypto/tls), which is a standard-library vulnerability fixed in Go 1.25.12. CI derives its toolchain from go.mod (setup-go go-version-file), so raising the go directive is the complete fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6piKCgFNqYpcecRkEU2xz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experience with the csv-sexp experiment surfaced two usability problems:
S-expression programs embedded in TOML strings get no editor support, so
matching parentheses is hard, and pipeline nodes can only be named
through let* bindings, making the primary activity — wiring nodes — feel
second-class.
csv-star keeps the same csvbase substrate and the same evaluation model
(the program runs exactly once at construction and builds a static,
applicative step graph; rows are evaluated by the frozen pipeline), but
the program is Starlark. Assignments bind pipeline steps as first-class
values, plain literals auto-lift to constants, and regexes read as raw
strings. Per-row logic escapes to real Starlark through exactly two
carriers — apply(f, ...) for scalar computation and
emit(implementation = f, ...) as the terminal — so Key-level
construction forms and selective conditional combinators are dropped
entirely; branching lives in ordinary if/elif inside callbacks, and
directive assembly uses value-level constructors on resolved values.
Callbacks receive and return plain values, never keys, which keeps the
dependency graph static and analyzable at load time. Each step records
its .star call site, so load-time errors and row diagnostics point at
the program source instead of at anonymous nodes.
Programs may be inlined in TOML or referenced via program_file, with
load() shared libraries confined to the program's directory. Alternatives
considered: Bazel-style name-attr node declaration (rejected — its name
indirection serves external addressability needs this DSL does not have,
at the cost of reintroducing mandatory binding ceremony) and per-row
Starlark execution of whole programs (rejected — it would bypass
csvbase's diagnostic and idempotency machinery).
The kind registers as "csv-star" and coexists with csv-sexp; the two
remain parallel experiments until one wins.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01F6piKCgFNqYpcecRkEU2xz