Compile KL to a bytecode VM (~15× faster tak, ~60× faster tail loops) (#50)

* Phase 0+2: bytecode VM compiler, float comparison fix

Phase 0 — Correctness fix:
- Fix <, <=, >, >= to use mustNumber() instead of mustInteger(), so float
  comparisons like (< 1.5 1.8) return true correctly.  Regression test added.

Phase 2 — Bytecode VM:
- kl/compiler.go: KL → bytecode compiler.  Handles defun, lambda, freeze,
  let, if, and/or, cond, do, type, trap-error, function calls (tail + non-tail).
  Closures capture upvalues by value at closure-creation time.
- kl/vm.go: stack-based VM with per-call flat locals[] frame.  Opcodes:
  LOAD_CONST, LOAD_LOCAL, STORE_LOCAL, LOAD_GLOBAL, LOAD_UPVAL, CALL,
  TAIL_CALL, RETURN, JUMP, JUMP_FALSE, MAKE_CLOSURE, POP.
- eval.go: defun special form now compiles to bytecode (CompileFunc).
  apply() has a fast path for scmBytecodeFunc.  Try() no longer requires
  scmNative, accepting any callable (enables trap-error in compiled code).
  scmHeadBytecodeFunc is self-evaluating.
- primitives.go: primDefun compiles scmProcedure → bytecode on binding.
  PrimStr handles scmBytecodeFunc.
- library.go: equal() handles scmBytecodeFunc (pointer equality).
- types.go: ObjString handles scmBytecodeFunc.

Benchmark (tak 18 12 6):
  Baseline (tree-walker): 0.088s
  Phase 2 (VM):           0.013s  → 6.6× speedup

Tail-call loop (sum 0 5000000):
  Baseline: ~3.0s
  Phase 2:  ~0.24s  → 12.5× speedup

All kl unit tests pass (29/29).

https://claude.ai/code/session_01RFe4YsuWLhZMdG5XDUewiq

* Phase 3+5: arithmetic fast paths and self-tail-call loop

Phase 3 — Arithmetic intrinsics:
- Compiler detects calls to +, -, *, <, <=, >, >=, =, not and emits
  specialized opcodes (OP_ADD, OP_SUB, OP_MUL, OP_LT, OP_LE, OP_GT,
  OP_GE, OP_EQ, OP_NOT) that bypass the trampoline entirely.
- Fixnum fast paths in numAdd/numSub/numMul/numCmp/numCmpLE avoid
  float64 boxing for the common integer case.
- Symbols for each intrinsic are interned at startup (symAdd, symSub, …).

Phase 5 — Self-tail-call loop:
- OP_SELF_TAIL_CALL: when the compiler detects a recursive call to the
  same function in tail position (with the same arity), it emits args
  onto the stack then OP_SELF_TAIL_CALL N.  The VM copies the new args
  into locals[0..N-1] and resets pc=0 — no trampoline round-trip, no
  new locals allocation.

Benchmark update:
  tak(18,12,6):          0.006s  (was 0.088s baseline → 14.7×)
  sum(0, 5M) loop:       0.05s   (was 2.99s baseline  → 60×)
  fib(30):               0.29s

All 29 unit tests pass.

https://claude.ai/code/session_01RFe4YsuWLhZMdG5XDUewiq

* Fix three semantic regressions from code review + cleanup

P1: compileDefun now threads the outer compiler so nested defuns can
close over lexical variables from the enclosing scope.

P1: OP_JUMP_FALSE panics on non-boolean values instead of treating
anything non-False as truthy, matching KL strict boolean semantics.

P2: In call position, a symbol with a global function binding now takes
precedence over a same-named local variable, matching the interpreter's
evalFunction lookup order.

Cleanup: remove unused isBytecodeFunc, drop unused upvals param from
vmPartialApply, replace numCmp(x,y,-1) with a dedicated numCmpLT,
fix OP_EQ comment ("numeric only" → "structural equality").

Tests: add 6 new TestBytecodeVM cases covering each regression and
multi-level closure chains, over-application, and float comparisons
through compiled defuns.

https://claude.ai/code/session_01RFe4YsuWLhZMdG5XDUewiq

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: pyrex41

bench/RESULTS.md

@@ -0,0 +1,66 @@
+# shen-go Benchmark Results
+
+Machine: Linux amd64  
+Kernel: S39.2  
+Date: 2026-05-05  
+
+## Measurements
+
+### `tak(18,12,6)` single call (wall time, seconds)
+
+| Milestone | Time (s) | vs baseline |
+|---|---|---|
+| Baseline (tree-walker, interpreter only) | 0.088 | 1× |
+| Phase 0 (float-fix only, no perf change) | 0.088 | 1× |
+| Phase 2 (bytecode VM, indexed slots) | 0.013 | **6.6×** |
+| Phase 3+5 (arithmetic fast paths + self-tail loop) | 0.006 | **14.7×** |
+
+### `(sum 0 5000000)` tail-recursive integer loop (wall time in Go tests)
+
+| Milestone | Time (s) |
+|---|---|
+| Baseline | 2.99 |
+| Phase 2 VM | 0.24 |
+| Phase 3+5 (self-tail + fast integer =,-) | 0.05 |
+
+Speedup on tight tail-call loop: **60×**
+
+### `fib(30)` (non-tail double recursion)
+
+| Milestone | Time (s) |
+|---|---|
+| Phase 3+5 | 0.29 |
+
+---
+
+## Method
+
+```
+# Define tak via KL defun, then time with get-time run:
+printf '(defun tak (X Y Z) ...) (get-time run) (tak 18 12 6) (get-time run)\n' \
+  | ./shen-go/shen
+# tak time = t2 - t1
+```
+
+---
+
+## Notes
+
+- Phase 0: fixed float comparison bug (`mustInteger` → `mustNumber` for `<`, `<=`, `>`, `>=`).
+  No performance change.
+- Phase 2: `defun` now compiles to a bytecode VM with flat indexed slots (no alist env).
+  Both REPL-defined KL `defun` and Shen-level `define` are compiled.
+  `lambda`/`freeze`/`trap-error`/`let`/`cond` all compile to bytecode.
+  Closures capture upvalues by value at creation time.
+
+---
+
+## Remaining gap to shen-cl
+
+shen-cl (SBCL) typically runs `tak(18,12,6)` in under 0.002s.  
+Current gap: ~6.5× (0.013s vs ~0.002s).  
+Target: within 3–5× of shen-cl.
+
+Next steps to close the gap:
+- Phase 4: decision-tree pattern matching compilation
+- Inline allocation pooling to reduce GC pressure

bench/bench.shen

@@ -0,0 +1,57 @@
+\ Benchmark suite for shen-go compiler work.
+\ Measures tak (arithmetic/recursion), fib (recursion), nrev (list), map-double (HOF).
+\ Run: printf '(load "bench/bench.shen")\n' | ./shen
+
+(define bench-tak
+  X Y Z N ->
+    (if (= N 0)
+        done
+        (let _ (tak X Y Z)
+          (bench-tak X Y Z (- N 1)))))
+
+(define tak
+  X Y Z ->
+    (if (not (< Y X))
+        Z
+        (tak (tak (- X 1) Y Z)
+             (tak (- Y 1) Z X)
+             (tak (- Z 1) X Y))))
+
+(define fib
+  0 -> 0
+  1 -> 1
+  N -> (+ (fib (- N 1)) (fib (- N 2))))
+
+(define iota-help
+  N Acc -> Acc where (= N 0)
+  N Acc -> (iota-help (- N 1) (cons N Acc)))
+
+(define iota
+  N -> (iota-help N []))
+
+(define nrev
+  [] -> []
+  [H | T] -> (append (nrev T) [H]))
+
+(define map-double
+  [] -> []
+  [H | T] -> (cons (* H 2) (map-double T)))
+
+(define run-bench
+  Name Thunk ->
+    (let T0 (get-time run)
+         _  (Thunk)
+         T1 (get-time run)
+      (do (output "~A: ~A s~%" Name (- T1 T0))
+          (- T1 T0))))
+
+(output "~%--- shen-go benchmarks ---~%")
+
+(run-bench "tak(18,12,6)x200" (freeze (bench-tak 18 12 6 200)))
+(run-bench "fib(25)" (freeze (fib 25)))
+(let L (iota 1000)
+  (run-bench "nrev(1000)" (freeze (nrev L))))
+(let L (iota 10000)
+  (run-bench "map-double(10000)" (freeze (map-double L))))
+
+(output "~%--- done ---~%")