test: executable Starlark spec suite (ruby/spec-style) with reference runner#12
Open
jcmfernandes wants to merge 13 commits into
Open
test: executable Starlark spec suite (ruby/spec-style) with reference runner#12jcmfernandes wants to merge 13 commits into
jcmfernandes wants to merge 13 commits into
Conversation
Nine files across seven new interaction directories, each pinning behavior observable only when units combine: - defer+error_handling: a deferred !-call's error is discarded but the call runs; deferred mutation of the returned value is visible. - defer+types: deferred-call arguments are captured eagerly but the callee's annotations are checked when the deferred call runs, at function exit, against the captured value. - defer+toplevelcontrol: top-level if/for bodies still reject defer. - error_handling+recursion: try through deep recursion; per-frame errdefer unwinding innermost-first; an intermediate catch stopping the unwind. - error_handling+while: retry loops, re-raising by returning the caught error value, and a parenthesized catch in a while condition. - error_handling+set: tag identity in sets; classifying caught errors against a tag set. - globalreassign+types: assignment annotations belong to the statement, not the variable; reassigned type aliases do not affect defs that already captured them. - positionalonly+types: '/' composes with annotations on every parameter kind. - set+types: set[T] as a first-class deep-matching type value.
A selective port of the implementation's testdata into the suite -- normative behavior only, rewritten to the harness conventions, with implementation-shaped content (small-int boundaries, shift/repeat caps, embedder modules, load variants) left behind. 14 new files, 17 extended; the suite grows from 74 to 88 files. - float: special values (infinity/NaN arithmetic, total order, negative zero, NaN dict keys, stable sort), conversions (round-half-to-even at 2^53, exact comparison vs rounding arithmetic, oversized-int errors), %e/%f/%g/%d formatting, sign tables, and the operand/result type matrix. - string: unicode (chr/ord, U+FFFD replacement, case conversion with titlecase digraphs, is* predicates), the not-iterable sweep, split/rsplit asymmetry, whitespace splitting, splitlines, %(name)s and %c, format() errors, Java-hashCode pins. - assignment/names: evaluation order (RHS first; LHS once in augmented assignment), destructuring arity errors, use-before- assignment in all flavors (including shadowed builtins), comprehension single-block scoping. - function: closure independence and state, freeze of defaults, recursion check on syntactic bodies (Y-combinator, bounded-but- rejected helpers), lambda parsing corners, calls through arbitrary expressions. - dict/list/int: mutation-during-iteration variants, the destructure-then-mutate non-violation, **kwargs non-aliasing, key-position stability across rehash, frozen-dict edge cases, int() base-handling matrix, big-magnitude exactness, negative shift counts. - types: float-accepts-int coercion, implicit-None return checks, default checked at binding, tuple[T, ...], legacy [T1, T2] unions. - error_handling: no-arg tag calls, str(error), tag/error falsiness, cause chains, destructuring through try/catch (unpacking.star).
go test caches package results even when only spec/*.star files change, as they are not tracked as test inputs, so a local run could report a stale pass for the spec suite. Exclude spectest from the bulk go test invocation and run it separately with -count=1. (CI already runs everything with -count=1.) Also fix a latent 'CI: unbound variable' error under set -u when CI is not set at all, and switch the shebang to bash.
The workflow already runs the tests itself; the only thing internal/test.sh added was the tidiness check. Replace it with a workflow step using go mod tidy -diff, which prints the necessary changes and exits non-zero if the files are not tidy, without ever modifying them. Nothing referenced the script.
Documents spec/ and spectest for contributors: the harness.md contract, the core/optional/interactions layout, spec file conventions (headers, chunked *_errors.star files, the predeclared vocabulary, default dialect options in core), the runner and its known-failures overlay, and the normative-only discipline. Notably, Go's test cache is .star-aware (it content-hashes opened files and walked directory listings), so cached spectest results remain valid across spec edits; no -count=1 is needed.
failures.star now asserts the in-language half explicitly: passing a caught error value to fail() turns it into a failure whose message uses the tag's name, not the error's message -- the handling a module-level try is specified to perform. Also fix API drift in doc/error_handling.md's Go example: the FailError field is StarlarkError and tag access is the Tag() method (failErr.StarlarkError.Tag(), not failErr.Value.Tag).
New 'The Go boundary' section in doc/error_handling.md: failures reach the embedder as *EvalError with the cause on its Unwrap chain; *FailError marks deliberate, fail-style aborts (set StarlarkError to carry an error value, as fail(e) and module-level try do); any other builtin error is an incidental fault; an uncaught recoverable error surfaces as *ReturnedError. The two never overlap. A Go builtin is Starlark code that happens to be written in Go: the section maps its two return values onto every outcome of the error model, including returning *FailError to mimic a call to fail() -- also stated on the FailError doc comment in value.go. The FailError example moves out of Module-level try, which now cross-references.
TestFailError verifies, end to end, what doc/error_handling.md's 'The Go boundary' section promises: fail() surfaces to Go as an *EvalError wrapping a *FailError, with StarlarkError carrying the error value when fail was given one (and nil when not); module-level try produces the same shape; a Go builtin returning *FailError raises the equivalent failure; and an incidental builtin error does not match *FailError. Complements TestReturnedError, which covers the recoverable half.
The fail() builtin now formats only its arguments; (*FailError).Error owns the presentation: it adds the "fail: " prefix and, when Msg is empty but StarlarkError is set, falls back to the error value's representation (its tag name). Observable messages are unchanged. This relieves Go builtins that return a *FailError to mimic fail() from hand-writing the prefix, and gives a payload-only FailError a sensible message for free. TestFailError pins the composition rules; the doc example carries the bare message accordingly.
fail() now has two modes. Message mode (no argument is an error value or error tag) keeps the core behavior: arguments rendered like str, joined by sep. Payload mode takes a single error value or error tag -- a bare tag is wrapped in an error value, as in a ! function's return -- and the failure carries it to the embedder; the message is the tag's name. Mixing an error or error tag with other arguments, or passing more than one, previously picked a payload silently (last wins) while stringifying all of them; it is now itself a failure describing the misuse. FailError reshaped to match: Msg is the message content, and Error() renders every FailError with the "fail: " prefix, however it was constructed -- the prefix marks the failure as fail-style, like the type itself. Payload-mode construction sets no Msg, so message and payload cannot disagree. NewFailError implements both modes and is total: a mixed call returns a misuse-describing FailError rather than panicking, so builtins forwarding script-controlled values cannot crash the host. The fail builtin is a thin wrapper over it; observable messages are unchanged. Pinned in TestFailError (parity with fail() in both modes, misuse shapes, prefix composition), spec/optional/error_handling (failures.star pins payload, wrapping, and all four misuse shapes; spec.md states the refined contract), and doc/error_handling.md (new 'fail' subsection; Go-boundary bullet updated).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an executable, implementation-independent spec suite for Starlark — modeled on ruby/spec — covering the core language and this fork's extensions, plus the reference runner for this implementation. Stacked on #11 (the suite specs the typing and error-handling extensions, and its
*_errors.starexpectations pin messages introduced there).The suite (
spec/)Written in Starlark only; an implementation participates by providing the small harness described in
spec/harness.md, which defines the contract: layout, file format (# requires:/# spec:headers; chunked*_errors.starfiles with### "regex"expectations), the predeclared assertion vocabulary (assert,trap,matches,freeze), the three observation channels (values, recoverable errors in-language, aborts), the normative-occurrence vs advisory-message error tiers, and runner obligations.spec/core/— 47 files over the original Starlark spec (doc/spec_original.md): scalar and container types, strings/bytes, functions, scope, freezing, expressions, statements, builtins, and static/dynamic error expectations. Includes behavior that is easy to get wrong: the float total order and round-half-to-even conversions,+=list aliasing, assignment evaluation order, mutation-during-iteration, the Java-hashCodepin forhash, Unicode case conversion, comprehension scoping, and theint()base-handling matrix — much of it ported from the implementation's testdata, normative behavior only.spec/optional/<unit>/— one independently adoptable unit per dialect option or extension:set,while,recursion,toplevelcontrol,globalreassign,positionalonly,defer,error_handling,types. Each carries a normativespec.mdand anindex.mdmapping spec headings to the files that exercise them.spec/interactions/— 11 files of behavior defined only when units combine:error_handling+types(the! ->form; return annotations check the success value only),defer+types(deferred-call annotations check the eagerly-captured argument at function exit),defer+error_handling(a deferred!-call's error is discarded),error_handling+recursion(per-frameerrdeferunwinding),error_handling+while(retry loops),error_handling+set(tag classification),globalreassign+types(annotations belong to the statement, not the variable),set+types,positionalonly+types,defer+toplevelcontrol,globalreassign+toplevelcontrol.The runner (
spectest/)go test ./spectestruns the suite asTestSuite, one subtest per file. It implements the contract's runner obligations:*.starunderspec/; gate each file on its required units (location plus header) with skips reported, never silent; isolate every program (or chunk) in a fresh module.### "regex"lines are verified against parse, resolve, and runtime errors at the declared line; plain files must run clean.# spec:header or chunk notation outside*_errors.staris an error even for overlay-listed files.spectest/known_failures.txt): a listed file that fails reports as a skip with its failures logged; one that passes is an error, so entries cannot go stale. Currently empty — the two conformance bugs the suite caught (bytes indexing/concatenation, the catch value-form parser bug) are fixed in feat: static typing / type annotations (port of starlark-rust's spec) #11.Testing
go test ./spectest -vfor the per-file report). Go's test cache tracks the.starfiles the runner opens (contents and directory listings), so cached results stay valid across spec edits.fail/Go-boundary thread:failures.starpinsfail()applied to an error value;doc/error_handling.mdgains a "The Go boundary" section (failures surface as*EvalErrorwrapping their cause —*FailErrorfor deliberate, fail-style aborts, which Go builtins may also return to mimicfail(); uncaught recoverable errors surface as*ReturnedError);TestFailErrorinstarlark/eval_test.gopins the wrapping end to end.go mod tidy -diffin the workflow, replacinginternal/test.sh(deleted; the workflow already ran the tests itself).master