feat: static typing / type annotations (port of starlark-rust's spec)#11
Open
jcmfernandes wants to merge 28 commits into
Open
feat: static typing / type annotations (port of starlark-rust's spec)#11jcmfernandes wants to merge 28 commits into
jcmfernandes wants to merge 28 commits into
Conversation
Port the annotation syntax of starlark-rust: parameter annotations (x: T, x: T = v, *args: T, **kwargs: T) via a new TypedParam node, return types (def f()! -> T:, with this fork's ! marker first), and annotated assignments (x: T = e, single targets only). New ARROW and ELLIPSIS tokens. Type expressions are validated against the restricted grammar of starlark-rust's TypeExprUnpackP (paths, T[...], | unions, tuples, ..., legacy [T1, T2] unions; literals and calls rejected). Annotations are gated by FileOptions.Types (TypesDisabled default, TypesParseOnly, TypesEnabled), mirroring DialectTypes. Lambdas cannot be annotated: a colon always ends their parameter list. Also adds the missing Walk cases for this fork's error-handling nodes (TryExpr, CatchExpr, DeferStmt, ErrDeferStmt, RecoverStmt), which previously panicked.
…nments Like parameter defaults, annotations are resolved in the enclosing scope: they are evaluated when the def statement executes. In TypesParseOnly mode annotations are skipped entirely, so undefined names within them are not errors (starlark-rust parity). resolve.Function gains ReturnType and HasTypes; the parameter classification loop is refactored around shared bind helpers so TypedParam dispatches to the existing ident/default/star logic.
Annotation expressions compile like defaults: evaluated at def time in the enclosing scope and carried in the MAKEFUNC operand tuple, now (defaults..., types..., freevars...). Funcode gains TypeParams (local slots of annotated parameters) and HasReturnType to split the tuple at runtime. Annotated assignments compile to a new TYPECHECK<name> opcode that checks the value against the evaluated type. The ... token becomes a new EllipsisConst constant kind. Bumps the serialization Version to 15. Also fixes a pre-existing bug: Funcode.CanReturnError was not serialized, so a program round-tripped through Encode/DecodeProgram silently lost the !-ness of its functions.
Add the runtime half of the type annotation system, following starlark-rust's semantics: - New *starlark.Type value (the result of evaluating an annotation) with a Matches method; TypeOf converts annotation values to types (the analogue of rust's TypeCompiled::new). Matching is deep for containers; str displays as 'str' but matches runtime 'string'. - Types are first-class: indexing the canonical constructors yields parameterized types (list[int], dict[str, int], tuple[int, ...]), and | between type-like operands builds unions (int | None). isinstance and eval_type join the universe; the new lib/typing module provides typing.Any/Never/Callable/Iterable. - Function values carry compiled parameter/return matchers, split out of the MAKEFUNC tuple and exposed via ParamType/ReturnType. Arguments are checked after setArgs (*args per element, **kwargs per value), results at RETURN -- skipped when a ! function returns through the error channel -- and annotated assignments via the TYPECHECK opcode. Mismatches are uncatchable failures, like fail(), with rust's exact message format. In TypesParseOnly mode nothing is emitted or checked; unannotated code takes no new paths beyond nil guards.
Port starlark-rust's record and enum library extensions as opt-in
packages (starlarkrecord, starlarkenum), each exposing a Predeclared
StringDict for the embedder, mirroring LibraryExtension::RecordType
and ::EnumType.
record(host=str, port=field(int, 80)) builds a callable record type;
construction is named-only with per-field type checks, and field()
defaults are checked eagerly. enum("red", "green") interns its
elements, which expose .value/.index and hash like their strings; the
type supports len, indexing, iteration, .values(), and element access
by name. Both act as type annotations matching instances of that
exact type, and both implement the new starlark.Exportable interface:
the interpreter's SETGLOBAL now tells a value the name of the global
it is first assigned to, so types display as record[Rec](...) or
Color("red") instead of anonymous forms.
Port the static analysis pass of starlark-rust's typing crate as the
optional typecheck package: Check(file, env, loads) runs on a resolved
file, entirely outside execution, and returns errors, a per-binding
TypeMap, the module's Interface (for checking dependents' loads), and
the Approximations it made.
The Ty model is a canonical union of basic types (prims, containers,
callables with ParamSpecs, iterables, type values). Inference collects
binding expressions over the resolved AST -- assignments, tuple
destructuring, for loops, augmented assignment, and the statement-level
x.append(e)/x.extend(e) mutation rule -- and solves them by naive
fixpoint, seeding annotated bindings with their declared types. Calls
are validated against signatures from annotated defs and a hand-written
table covering the universe and string/list/dict/set methods
(typecheck/universe.go, kept in sync with starlark/library.go).
Compatibility is intersection, not subtyping, and the checker is
deliberately lenient: anything it cannot model becomes typing.Any plus
a recorded Approximation, so it aims never to reject a program that
would run. Error messages mirror rust's golden output ("Expected type
X but got Y", etc.).
Fork-specific constructs are understood: try e types as e, e catch v
unions both sides, block-form catch unions in its recover values, and
catch error variables type as error. Tests include a no-crash sweep
over the full starlark/syntax testdata corpus and an agreement test
pinning the static annotation interpretation to the runtime's.
-types=off|parse|on selects the annotation dialect mode for files and the REPL, and the typing module joins the universe. -typecheck (which implies -types=on) runs the static typechecker after parsing and aborts before execution if it reports errors.
TYPES.md describes the feature end to end (dialect modes, syntax, runtime semantics, types as values, record/enum, the static typechecker, and the Go API); TYPES_TODO.md lists known follow-ups. The contributor guide gains a summary section pointing at both.
Match starlark-rust's numeric coercion: a float annotation (and list[float] etc., via deep matching) accepts int values at runtime. The reverse does not hold: an int annotation still rejects floats. The static typechecker's intersection rule follows suit so the two layers agree.
Support Python-style positional-only parameter markers, which starlark-rust's extended dialect also provides, gated behind a new FileOptions.PositionalOnly flag since standard Starlark lacks them (CLI: -positionalonly; tests: # option:positionalonly). The marker parses like bare * (a UnaryExpr with Op SLASH, valid in defs and lambdas), and the resolver enforces its placement: not first, at most once, and before * or **. Parameters preceding it cannot be bound by name; per Python semantics their names remain available to **kwargs. Funcode gains NumPositionalOnly (serialized under the still-unreleased Version 15), setArgs skips positional-only names during keyword binding with a dedicated error message, and the static typechecker marks such parameters PosOnly so calls like f(x=1) are reported at lint time. Positional-only markers compose with type annotations: def f(x: int, /, y: str) -> int.
…e builtins
Three precision improvements to the static checker, all rust-parity
or rust-shaped:
- == and != between certainly-disjoint types are now reported, like
rust's expr_bin_op for Equal/NotEqual, which validates the rhs
against the lhs ('Expected type `int` but got `str`'). Numeric
coercion and union narrowing (x == None on int | None) still pass.
- Callable types intersect by signature instead of always: a port of
rust's params_intersect. The precise check applies when at least
one spec is in 'simple' form (all-required positional-only/named-
only, the shape of typing.Callable[[...], R]); a simple spec is
validated as a call against the general one; two general specs
always intersect (leniency). Catches passing def f(s: str) where
typing.Callable[[int], str] is annotated.
- Builtin results are now argument-aware via a specialFn hook on
callable types: sorted/reversed/min/max/list/set/tuple/zip/
enumerate/dict propagate their arguments' element types (zip is
arity-aware), abs preserves int/float, and dict.get/pop/setdefault
incorporate the default argument's type. *args/**kwargs call forms
fall back to the declared (sound) result.
A harness in the style of starlark-rust's typing/tests/golden: each
typecheck/testdata/golden/NAME.star input is checked and the reported
errors must match NAME.golden. Regenerate with:
go test ./typecheck -run TestGolden -update
Seeded with inputs covering call shape/argument errors, attribute and
index errors, returns and annotated assignments, operators (including
the pointless-comparison lint), higher-order callable annotations,
argument-aware builtin results, and a clean module.
-typecheck now follows load() statements: each dependency is parsed, resolved, and checked once (with cycle detection), and its inferred Interface feeds its dependents, so load()ed symbols have precise types instead of Any. Errors from all modules in the graph are reported, and any error aborts before execution. Module names are interpreted as file names, exactly like the executor's load (repl.MakeLoadOptions). Also fixes a bug in the -typecheck path: it passed starlark.Universe.Has as the isPredeclared predicate to SourceProgramOptions, so universal names (print, len, ...) resolved as predeclared and were uninitialized when the program was later run with Init(thread, nil). The exec path declares nothing predeclared, and now the typecheck path agrees.
x: T = e re-evaluates T on every execution of the statement. When the
type expression is constant — every name in it resolves to a universal
or predeclared binding, neither of which can be assigned during
execution — the compiler now emits a caching sequence instead:
TYPEFETCH<slot>; DUP; CJMP check
POP; <evaluate T>; TYPESTORE<slot>
check: TYPECHECK<name>
The value is evaluated once per Function value and stored in an atomic
per-slot cache (a frozen Function may be called concurrently;
concurrent first executions store equal values). Non-constant
expressions — e.g. a local alias, which can legitimately be rebound
between executions — keep the re-evaluating code, preserving the
documented semantics exactly. With a constant list[int] | None
annotation in a 1e6-iteration loop, annotation overhead drops about
6x versus full re-evaluation.
The restricted type-expression grammar guarantees the absence of side
effects; predeclared values used in annotations are assumed constant,
as in starlark-rust, which compiles every annotation exactly once.
Funcode gains NumTypeCaches (serialized; the format doc is updated).
Also fixes an off-by-one in Opcode.String that excluded the last
opcode from name lookup.
TYPES.md documents the deep-matching cost (deliberately unlimited, matching rust), the constant-annotation matcher cache, per-load typechecking in cmd/starlark, and the new static-checker precision (pointless comparisons, callable signature intersection, argument- aware builtin results). TYPES_TODO.md drops the completed tier-1/2/3 items; what remains is the partial-evaluation/CustomTy track and the deferred items.
…tags awareness Implement the three 'big rocks' of TYPES_TODO.md: teach the static checker that values computed at module load time carry type information. Module-level partial evaluation (peval.go), the analogue of starlark-rust's fill_types_for_lint, replaces collectAliases' single name-keyed scan: a binding-keyed pre-pass maps each binding to the static value its assignments compute, with agree-or-unknown merging (if/else branches that agree keep the value; disagreement or an unmodeled rebinding degrades to unknown, never a guess). This distinguishes the type of a binding (IntList: type) from the type value it denotes (list[int]); the denoted channel is exposed as Interface.Denoted, so aliases flow across load() with the per-load CLI flow. Aliases under conditionals, inside functions, loaded from other modules, and minted by calls all work now. CustomTy (custom.go), the analogue of rust's TyUser, is a new Basic alternative for externally defined types: a display name plus an attribute table, with optional callable/indexable/iterable capabilities. Identity is nominal (per-mint serial), mirroring the runtime's pointer-identity matchers: structurally identical record types do not intersect. Custom types are minted by TypeFactory hooks attached to env entries (WithFactory); binary operators on customs are lenient (a runtime type value supports e.g. Rec | None). Clients: - error_tags (built into UniverseEnv): the tag set's attribute table is exactly the tags, so errors.NotFonud is a static error. This required two fixes the old Any typing hid: returns in ! functions accept error/error_tag besides the declared success type (the runtime skips the check for error returns), and the error_tag prim is callable, constructing an error. - starlarkrecord/typed and starlarkenum/typed: adapter packages contributing record/field/enum env entries (typecheck still imports only resolve and syntax). They catch unknown fields and elements (r.hosst, Color.bleu), constructor argument errors, and give x: Rec annotations exact instance matching, across load() too. TestRecordEnumAgreement pins that the runtime TypeMatcher of a record or enum type and its static CustomTy accept the same values and display the same way, alongside TestAnnotationAgreement.
catch_error was confusing given that the language features a catch keyword.
Expose the assertion vocabulary required by the spec-suite harness contract (assert, trap, matches, freeze) as a predeclared environment. trap and matches were previously unexported and absent from assert.star's globals.
Per the Starlark spec, indexing a bytes yields the byte's value as an int, and two bytes values concatenate with +. The implementation returned a 1-byte bytes for b[i] and rejected bytes + bytes. The static typechecker already modeled both per the spec, so this also restores runtime/static agreement.
…ifier parseCatchSuffix tried to backtrack after probing for the block form by saving and restoring p.in, but p.in is a *scanner, so the restore was a self-assignment: the probed identifier was lost and the value-form fallback parse began one token late. 'catch DEFAULT + "x"' dropped DEFAULT, 'catch fb()' was a parse error, and a bare 'catch DEFAULT' swallowed the following statement as the fallback, silently changing control flow. Parse the value-form fallback by continuing the expression from the already-consumed identifier instead, via two refactored helpers: parseSuffixes (dot/index/call suffix loop, extracted from parsePrimaryWithSuffix) and parseBinopExprFrom (precedence climbing from a given left operand, extracted from parseBinopExpr).
AGENTS.md and doc/bckground_extensions.md claimed catch blocks do not create a new lexical scope and that their assignments are visible outside. The implementation (and its catch_scope.star tests) do the opposite: the error variable and names bound in the block are local to it, shadowing enclosing bindings without overwriting them, while enclosing values remain readable and mutable.
A ruby/spec-style suite of Starlark programs under spec/, written against the harness contract in spec/harness.md: - core/: 28 files over the core language (doc/spec-original.md): scalar and container types, strings/bytes, functions, scope, freezing, expressions, statements, builtins, and chunked *_errors.star files for parse/resolve/runtime errors. - optional/: one unit per dialect option or extension, each with a normative spec.md and an index.md mapping headings to files: set, while, recursion, toplevelcontrol, globalreassign, positionalonly, defer, error_handling, types. - interactions/: behavior defined only when several units combine (error_handling+types, globalreassign+toplevelcontrol). doc/spec-original.md is the snapshot of the upstream Starlark spec that core/ files cite via their '# spec:' headers.
Implements the runner obligations of spec/harness.md: discovers
*.star files, gates them on required units (location plus
'# requires:' headers) with skips reported, executes each program in
a fresh module with the assertion vocabulary predeclared, matches
chunked '###' error expectations against parse, resolve, and runtime
errors, and enforces suite integrity ('# spec:' header present,
chunk notation only in *_errors.star).
Also supports the contract's known-failures overlay
(known_failures.txt): a listed file that fails reports as a skip
with its failures logged; one that passes is an error, so entries
cannot go stale. The suite runs as TestSuite via go test ./spectest.
marckysharky
reviewed
Jul 6, 2026
marckysharky
reviewed
Jul 6, 2026
Comment on lines
+183
to
+190
| ```python | ||
| T = int if legacy else int # branches agree: T denotes int | ||
| load("lib.star", "IntList") # aliases flow across load() | ||
|
|
||
| def f(): | ||
| Row = list[int] # aliases inside functions | ||
| def g(r: Row): ... | ||
| ``` |
There was a problem hiding this comment.
This isn't clear to me - can I use IntList as an annotation?
def g(r: IntList): ...
Member
Author
There was a problem hiding this comment.
You can, yes, because IntList = list[int] so just like Row = list[int] allows you to write def g(r: Row): ....
marckysharky
reviewed
Jul 6, 2026
marckysharky
reviewed
Jul 6, 2026
Comment on lines
+245
to
+247
| # interaction with error-returning (!) functions: | ||
| # an error return bypasses the return type check | ||
| # option:types |
There was a problem hiding this comment.
I don't think the following is covered, here: Explicit error values which would require 'typing.Never`
Member
Author
There was a problem hiding this comment.
Not sure I understand mate. Mind clarifying?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the type annotation system of starlark-rust (used by Buck2) to this implementation, following that spec as closely as possible and integrating it with this fork's Zig-style error handling. Full description in
TYPES.md; known follow-ups inTYPES_TODO.md.An executable spec suite for the core language and this fork's extensions was developed on this branch and now lives in #12 (88 spec files, stacked on this one); two conformance fixes it uncovered remain here, see below.
Syntax & dialect
syntax.FileOptions.Typesmirrors rust'sDialectTypes:TypesDisabled(default, parse error),TypesParseOnly(parsed, ignored at runtime),TypesEnabled.TypeExprUnpackPgrammar: paths,T[...],|unions, tuples,..., legacy[T1, T2]unions; string literals and calls are parse errors.Runtime checking
Annotations are evaluated at
deftime in the enclosing scope (like defaults, sharing the MAKEFUNC tuple) and compiled to matchers. Arguments are checked after binding (*argsper element,**kwargsper value), results at every return — skipped when a!function returns through the error channel — and annotated assignments via a new TYPECHECK opcode. Mismatches are uncatchable failures (likefail()), with rust's exact message format:Types are first-class values:
list[int],int | Noneevaluate totypevalues;isinstance/eval_typejoin the universe;lib/typingprovidesAny/Never/Callable/Iterable. Embedders extend matching viastarlark.TypeMatcher/starlark.TypeName.record() and enum()
Opt-in ports of rust's library extensions (
starlarkrecord,starlarkenum): typed record constructors withfield()defaults, interned enum elements, both usable as annotations matching instances of that exact type. A newstarlark.Exportablehook on SETGLOBAL gives them their assigned global name (rust'sexport_as).Static typechecker
New optional
typecheckpackage, the analogue ofAstModule.typecheck: fixpoint inference over the resolved AST (assignment sites, destructuring, thex.append(e)mutation rule), call validation against annotated defs plus a hand-written universe/method signature table, intersection-based compatibility, and deliberate leniency (unknowns →Any+ recordedApproximation). Understandstry/catch/recover, composes across files viaInterface, and is pinned to agree with the runtime by an agreement test. Exposed asstarlark -typecheck, which catches errors in never-called code.It also performs a module-level partial-evaluation pre-pass (
peval.go) tracking the type values bindings denote — aliases, conditionals, acrossload()— and an extension API (CustomTy,TypeFactory) under whichrecord/enum(adapters instarlarkrecord/typed,starlarkenum/typed) anderror_tagsget nominal static types that agree with their runtime matchers.Conformance fixes (found by the spec suite, #12)
b[i]now yields the byte's value as anintandbytes + bytesconcatenates, per the spec (the static typechecker already modeled both — this restores runtime/static agreement).*scannerpointer, restoring nothing; fallbacks beginning with an identifier were parsed one token late —f() catch DEFAULT + "!"droppedDEFAULT,f() catch fb()was a parse error, and a baref() catch DEFAULTsilently swallowed the next statement as the fallback. Fixed by continuing the expression from the consumed identifier (parseSuffixes+parseBinopExprFrom).doc/bckground_extensions.mdis split intodoc/defer.mdanddoc/error_handling.md, anddoc/spec_original.md(the upstream spec snapshot) joins as the reference the suite cites.Drive-by fixes
Funcode.CanReturnErrorwas not serialized: round-tripping a compiled program silently dropped!-ness (fixed under the Version 14→15 bump).syntax.Walkpanicked on all of this fork's error-handling nodes (cases added).starlarktest: thecatch_errorhelper is renamedtrap(it traps failures, the channelcatchcannot intercept);SpecPredeclared()exposes the assertion vocabulary the spec-suite runner (test: executable Starlark spec suite (ruby/spec-style) with reference runner #12) predeclares.Testing
master