Compile KL to a bytecode VM (~15× faster tak, ~60× faster tail loops) - #50
Merged
tiancaiamao merged 3 commits intoJun 8, 2026
Merged
Conversation
Phase 0 — Correctness fix: - Fix <, <=, >, >= to use mustNumber() instead of mustInteger(), so float comparisons like (< 1.5 1.8) return true correctly. Regression test added. Phase 2 — Bytecode VM: - kl/compiler.go: KL → bytecode compiler. Handles defun, lambda, freeze, let, if, and/or, cond, do, type, trap-error, function calls (tail + non-tail). Closures capture upvalues by value at closure-creation time. - kl/vm.go: stack-based VM with per-call flat locals[] frame. Opcodes: LOAD_CONST, LOAD_LOCAL, STORE_LOCAL, LOAD_GLOBAL, LOAD_UPVAL, CALL, TAIL_CALL, RETURN, JUMP, JUMP_FALSE, MAKE_CLOSURE, POP. - eval.go: defun special form now compiles to bytecode (CompileFunc). apply() has a fast path for scmBytecodeFunc. Try() no longer requires scmNative, accepting any callable (enables trap-error in compiled code). scmHeadBytecodeFunc is self-evaluating. - primitives.go: primDefun compiles scmProcedure → bytecode on binding. PrimStr handles scmBytecodeFunc. - library.go: equal() handles scmBytecodeFunc (pointer equality). - types.go: ObjString handles scmBytecodeFunc. Benchmark (tak 18 12 6): Baseline (tree-walker): 0.088s Phase 2 (VM): 0.013s → 6.6× speedup Tail-call loop (sum 0 5000000): Baseline: ~3.0s Phase 2: ~0.24s → 12.5× speedup All kl unit tests pass (29/29). https://claude.ai/code/session_01RFe4YsuWLhZMdG5XDUewiq
Phase 3 — Arithmetic intrinsics: - Compiler detects calls to +, -, *, <, <=, >, >=, =, not and emits specialized opcodes (OP_ADD, OP_SUB, OP_MUL, OP_LT, OP_LE, OP_GT, OP_GE, OP_EQ, OP_NOT) that bypass the trampoline entirely. - Fixnum fast paths in numAdd/numSub/numMul/numCmp/numCmpLE avoid float64 boxing for the common integer case. - Symbols for each intrinsic are interned at startup (symAdd, symSub, …). Phase 5 — Self-tail-call loop: - OP_SELF_TAIL_CALL: when the compiler detects a recursive call to the same function in tail position (with the same arity), it emits args onto the stack then OP_SELF_TAIL_CALL N. The VM copies the new args into locals[0..N-1] and resets pc=0 — no trampoline round-trip, no new locals allocation. Benchmark update: tak(18,12,6): 0.006s (was 0.088s baseline → 14.7×) sum(0, 5M) loop: 0.05s (was 2.99s baseline → 60×) fib(30): 0.29s All 29 unit tests pass. https://claude.ai/code/session_01RFe4YsuWLhZMdG5XDUewiq
P1: compileDefun now threads the outer compiler so nested defuns can
close over lexical variables from the enclosing scope.
P1: OP_JUMP_FALSE panics on non-boolean values instead of treating
anything non-False as truthy, matching KL strict boolean semantics.
P2: In call position, a symbol with a global function binding now takes
precedence over a same-named local variable, matching the interpreter's
evalFunction lookup order.
Cleanup: remove unused isBytecodeFunc, drop unused upvals param from
vmPartialApply, replace numCmp(x,y,-1) with a dedicated numCmpLT,
fix OP_EQ comment ("numeric only" → "structural equality").
Tests: add 6 new TestBytecodeVM cases covering each regression and
multi-level closure chains, over-application, and float comparisons
through compiled defuns.
https://claude.ai/code/session_01RFe4YsuWLhZMdG5XDUewiq
Owner
|
Once upon a time, there was a bytecode VM implementation The latest implementation use ast-interpreter + compile to Go, just because the ast interpreter performance does not really matter, it is only used for bootstrap. Any way, in the AI era, code is cheap now. I think it's acceptable to get it back, as long as it does not break anything |
tiancaiamao
approved these changes
Jun 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds a stack-based bytecode VM for KL, replacing the tree-walking
interpreter as the execution path for every
defun(both REPL/KLdefunand the Shen-level
define, which lowers to KLdefun). The tree-walkerremains in place and is still used for top-level forms and as a fallback.
Each function is compiled once, at define time, to a flat
[]Instrsequencewith integer-indexed local slots:
envGet(O(depth) alist scan) → array index O(1).2×aritycons cells per call to extend theenvironment.
switchoveruint8opcodes.lambda/freeze) capture upvalues by value at creation time.place without growing the Go stack.
+ - * < <= > >= = notcompile to dedicatedopcodes with a fixnum fast path, avoiding boxing and the trampoline.
A correctness fix is also included (Phase 0):
<,<=,>,>=usedmustInteger(which truncated floats) instead ofmustNumber.Performance (S39.2 kernel)
tak(18,12,6)(sum 0 5000000)tail loopfib(30)See
bench/RESULTS.mdandthoughts/shen-go-compiler-design.mdfor the fulldesign write-up and methodology.
Semantics preserved
The VM matches the interpreter's
evalFunction/evalsemantics, includingthe Lisp-2 detail that a global function binding takes precedence over a
same-named local in call position. Other preserved behaviors: strict
boolean
if(non-boolean condition errors), nested-defunlexical capture,partial application, over-application, and
trap-error(lowered totry-catchover afreezed body so panic/recover still has a real frame).Testing
go test ./...passes, including newTestBytecodeVMcases coveringclosures (incl. multi-level upvalue chains),
letshadowing,trap-error,tail recursion, currying, over-application, nested-defun capture, strict
boolean
if, global-vs-local precedence, and float comparisons.go vet ./kl/clean.defungoes throughthe compiler), and a suite of real Shen programs — pattern-matched
defines withwhereguards, higher-order functions, Ackermann, deep(200k-iteration) tail recursion, string/float ops — all produce correct
results.
Notes for reviewers
+/-/etc., which means redefining thoseprimitives globally would not be picked up by already-compiled code. This
matches typical Lisp compiler behavior and the kernel never redefines them.
thoughts/shen-go-compiler-design.mdis included as design rationale —happy to drop it if you'd prefer it not live in the repo.
🤖 Generated with Claude Code