Epic: Better XTalk compatibility — canonical source and linter

[hoebat]

Objective

Make XTalk source code canonical, readable, and portable again.

The first phase is deliberately narrow:

1. define and enforce the canonical XTalk source language;
2. build a linter that explains which layer a form belongs to;
3. clean the xt.lang.* source files first;
4. use JavaScript as the reference implementation;
5. only then address Python, Dart, and other target-language differences.

The goal is to stop target-language constraints leaking into canonical XTalk source.

Why this is needed

XTalk currently has several overlapping layers:

XTalk source
  -> grammar/staging and macro expansion
  -> typed lowering/inference
  -> target-specific rewrites
  -> target emitter

The implementation already contains enough evidence that the language works across targets, but source code has accumulated compatibility forms such as:

• large amounts of explicit x:get-key;
• explicit x:not-nil? added for target truthiness constraints;
• block forms appearing in value positions;
• target-specific lambda and conditional workarounds;
• inconsistent handling of object fields, array destructuring, and snake_case.

The immediate problem is not that XTalk lacks features. It is that the canonical source layer is no longer clearly separated from the lower-level semantic IR and target adapters.

Current findings

1. Dot access already has the intended canonical shape

The typed lowerer already maps:

(. obj ["key"])
=> (x:get-key obj "key")

(. obj ["a"] ["b"])
=> (x:get-path obj ["a" "b"] nil)

See src/hara/typed/xtalk_lower.clj.

The grammar also defines . as the index/access operator, while x:get-key, x:get-path, and x:set-key are macro-level access operations in src/hara/common/grammar_xtalk.clj.

The canonical authoring rule should therefore be:

(. obj ["key"])
(. obj [key])
(. obj ["a"] ["b"])

x:get-key and x:get-path should remain valid semantic/internal forms, but should not be required for ordinary source code.

A redundant simple form such as:

(x:get-key obj "key")

should be lintable as:

(. obj ["key"])

provided it has no non-nil default and no special dynamic semantics.

2. Python currently exposes the access-layer leak

Live emission currently differs:

(. a ["key"])
=> a["key"]

(x:get-key a "key")
=> a.get("key")

The second form preserves XTalk nil-safe access semantics; the first does not necessarily do so.

This is exactly the kind of difference that must be resolved in the target rewrite layer rather than in canonical source. Dot access and x:get-key should first normalize to one semantic access representation, then Python should decide how to emit that representation.

3. Destructuring has two distinct canonical meanings

The typed environment already distinguishes:

(var #{name age} m)

as object/field destructuring, from:

(var [name age] arr)

as positional/array destructuring.

The canonical checker must preserve this distinction:

• set target -> object fields;
• vector target -> array positions;
• map target -> explicitly supported map destructuring;
• anything else -> diagnostic.

Object destructuring should normalize field names through the same field-key function used by record typing and map literals.

For example, these should canonicalize to the same external field:

:arr-name
:arr_name
arr-name
arr_name

The checker must also detect collisions after normalization, such as:

#{arr-name arr_name}

Local binding names and external field keys must remain separate concepts. A local binding may be emitted as arr_name, while its canonical source symbol remains arr-name.

4. Field-key normalization is partly implemented but not consistently enforced

hara.typed.xtalk-common/field-key already normalizes field keys to snake_case, and typed map inference already uses it.

However, normalization is not consistently applied to:

• dot access;
• explicit x:get-key;
• object destructuring;
• emitted map keys;
• collision detection;
• target-specific access rewrites.

The canonical linter must make field-key normalization visible and deterministic.

There is also an important boundary to define: declared XTalk record fields should be normalized, but arbitrary external protocol keys such as clientInfo, protocolVersion, or other wire-format identifiers must not be silently rewritten. The implementation needs a clear distinction between canonical field keys and opaque external object keys.

5. Block/value context is not currently checked strongly enough

The grammar classifies if, cond, when, let, do, while, and for as :block.

:? is the value-oriented conditional.

The intended rule is:

(def a (if b 1 2))       ; illegal
(var a (if b 1 2))      ; illegal
(return (if b 1 2))      ; illegal

Use:

(def a (:? b 1 2))
(var a (:? b 1 2))
(return (:? b 1 2))

The current inference path can infer the branch union for if, but it does not carry enough value-position context to report this as a canonical error. Direct emission can therefore produce invalid target code.

The linter must add an explicit value/block context pass before target emission.

6. Dart truthiness belongs in the Dart adapter

The Dart rewrite already detects boolean-compatible expressions and wraps general XTalk truthiness using:

(and (x:not-nil? value)
     (not= false value))

The canonical source should remain:

(:? value then else)

Dart-specific bool adaptation should happen in hara.model.spec-dart.rewrite.

Explicit x:not-nil? remains valid when the programmer means “present even if false”; it should not be required merely because Dart has stricter conditional typing.

7. Python lambda lifting already belongs in the target rewrite

Python already has a lambda compatibility predicate and a hoisting rewriter. Complex canonical lambdas can be lifted into named helper functions during Python rewriting.

That machinery should be retained and improved later. It should not force Python-shaped forms into canonical XTalk.

8. Python conditional emission needs parity coverage

Live Python emission currently turns:

(:? b 1 2)

into an and/or expression. That is unsafe when the true branch is falsey.

This should become a target conformance test and be fixed in the Python adapter after the canonical JavaScript contract is locked.

Canonical source contract for phase one

The first implementation should establish these forms as canonical:

;; object access
(. obj ["name"])
(. obj [key])
(. obj ["meta"] ["role"])

;; value conditionals
(:? test then else)

;; object destructuring
(var #{name age} m)

;; positional destructuring
(var [name age] arr)

;; ordinary lambdas
(fn [x] ...)
(fn:> [x] expression)

The following remain valid but are lower-level or exceptional:

(x:get-key obj key default)
(x:get-path obj path default)
(x:get-idx arr index default)
(x:not-nil? value)
(x:nil? value)

Linter design

The linter should operate on canonical XTalk source before target-specific rewriting.

Initial diagnostics:

• XT001 block-in-value
• XT002 redundant-get-key
• XT003 access-kind-mismatch
• XT004 field-key-collision
• XT005 invalid-destructuring-target
• XT006 field-key-normalization
• XT007 unsupported-canonical-form
• XT008 target rewrite required (informational only)

The linter should report:

• source location;
• original form;
• canonical interpretation;
• suggested rewrite where safe;
• layer responsible for the issue:
  • source/canonical;
  • typed semantic;
  • target rewrite;
  • emitter/runtime.

It should expose a programmatic API suitable for a Kimi guard, rather than putting the semantic logic entirely in clj-kondo. clj-kondo can provide syntax and lightweight style integration, while the Foundation XTalk analyzer should own semantic checks.

Staged implementation

Stage 0 — Canonical fixtures

Create a small golden corpus covering:

• dot access;
• nested paths;
• dynamic keys;
• defaults;
• object and array destructuring;
• snake_case normalization;
• field collisions;
• block-in-value errors;
• :? truthiness;
• nil checks;
• lambdas.

Each fixture should record:

• source form;
• canonical normalized form;
• inferred type;
• JavaScript output.

Stage 1 — Build the linter

Add a source-level XTalk linter with:

• context-aware value/block checking;
• access normalization;
• destructuring classification;
• field-key normalization;
• redundant x:get-key detection;
• structured diagnostics;
• JSON output for Kimi/editor integration.

Initially run it in warning mode over src-lang/xt.

Stage 2 — Clean xt.lang.*

Use the linter to clean src-lang/xt/lang/* first.

The target is not broad mechanical replacement. Every rewrite must preserve:

• nil/default behavior;
• dynamic versus static keys;
• object versus array access;
• truthy versus non-nil semantics.

JavaScript is the reference target for this stage.

Stage 3 — Lock JavaScript as the canonical reference

For every canonical fixture:

• source must pass the linter;
• typed analysis must succeed;
• JavaScript emission must be stable;
• runtime tests must pass.

This stage establishes what XTalk means before target languages are allowed to influence the source contract.

Stage 4 — Clean the remaining src-lang/xt source

Apply the same canonical rules to the rest of the XTalk source tree.

Generated test-lang/xtbench files should not be edited manually; regenerate them from canonical seeds where applicable.

Stage 5 — Cross-language conformance

Only after the canonical JavaScript layer is concrete. Lua is included alongside Python and Dart in this conformance stage:

• compare Python, Dart, Lua, and other targets against the canonical fixtures;
• classify failures as target rewrite, emitter, runtime, or actual language limitation;
• add target-specific rewrites without changing canonical source.

Stage 6 — Kimi guard and CI enforcement

Promote diagnostics gradually:

• canonical errors -> blocking;
• redundant syntax -> warning/autofix;
• target adaptation notices -> informational;
• unsupported target capability -> target-specific failure.

Definition of done

This epic is complete when:

• canonical XTalk source forms are documented and mechanically checked;
• xt.lang.* no longer requires target-specific compatibility syntax;
• dot access and x:get-key have one semantic meaning;
• object and array destructuring are unambiguous;
• field-key normalization and collisions are deterministic;
• block forms cannot silently appear in value positions;
• JavaScript is a stable reference implementation;
• Python/Dart differences are isolated in target adapters;
• the linter can explain whether a failure belongs to source, typing, rewriting, or emission;
• Kimi can run the linter as a guard before edits are accepted.

Initial implementation issue candidates

1. Define canonical XTalk fixture format and JavaScript golden tests.
2. Add context-aware canonical source walker.
3. Add access normalization and redundant x:get-key diagnostics.
4. Add destructuring and field-key checks.
5. Add block-in-value diagnostics.
6. Add JSON linter API for Kimi.
7. Clean src-lang/xt/lang/*.
8. Fix JavaScript canonical emission/runtime parity.
9. Add Python/Dart/Lua conformance matrix after the canonical phase.

Update — 2026-07-22

Implemented and pushed in commit fd927be1.

• Added typed canonical lowering context for free XTalk forms.
• Local bindings are inferred before structural access lowering.
• Python now receives canonical x:get-key and emits nil-safe .get(...) access.
• Added lazy dependent-namespace type registration.
• Removed the obsolete +intrinsic-targets+ remapping.
• Added regressions for method-call preservation, runtime helper resolution, and canonical-stage behavior.
• Verified xt.event.base-route-test with 54 passing checks.

The remaining epic work is the staged cleanup of canonical xt.lang.* sources and later target conformance work.

[hoebat]

Progress update — 2026-07-22

The first implementation slice is now working and is being kept deliberately incremental.

Linter and hook integration

• Added the programmatic canonical XTalk linter under hara.typed.xtalk-lint.
• The initial rules are active:
  • XT001 block forms in value position;
  • XT002 redundant simple x:get-key;
  • XT004 normalized field-key collisions;
  • XT005 invalid vector/set destructuring targets.
• Renamed the clj-kondo hook namespace/file to hooks.hara-xtalk / hara_xtalk.clj.
• Renamed diagnostics from :xtalk/* to :hara.xtalk/*.
• Renamed PostgreSQL diagnostics from :gwpg/* to :hara.postgres/*.
• Updated both local and exported clj-kondo configuration/hook namespaces. A repository search confirms the old hook and diagnostic names are gone from those configurations.

Current canonical-source result

The linter now runs over all 23 src-lang/xt/lang/*.clj files:

• 85 diagnostics;
• 0 canonical errors;
• 85 XT002 warnings.

The count was 86 before the first source rewrite.

Test-tree linting has also been exercised. The canonical test findings currently include two XT002 warnings in test-lang/xt/lang/common_protocol_test.clj. The wider test tree still has expected clj-kondo noise from the custom code.test macros and generated bench namespaces; those are separate from the canonical XTalk diagnostics and should be handled as a test-lint configuration layer.

First dependency-ordered rewrite

Started with src-lang/xt/lang/common_sort_by.clj, the lowest-dependency file with a canonical finding:

(return (xt/x:get-key e key))

is now:

(return (. e [key]))

The REPL emission probe confirms canonical dot access emits indexed access consistently for JavaScript, Python, and Lua. After reload:

• canonical xt.lang.common-sort-by-test: 3 passed;
• generated Lua bench: 1 passed;
• generated Python bench: 1 passed;
• generated Dart bench: 1 passed;
• post-write canonical linter result for the file: 0 findings.

Seedgen coverage

seedgen benchadd was used (not langadd) for Lua, Python, and Dart. It generated:

• test-lang/xtbench/lua/lang/common_sort_by_test.clj;
• test-lang/xtbench/python/lang/common_sort_by_test.clj;
• test-lang/xtbench/dart/lang/common_sort_by_test.clj.

The compatibility matrix already registers Lua, Python, and Dart; no language registration change was needed.

Next slices will continue from least to most dependent, using JavaScript as the canonical reference and regenerating the target benches after each source-owned change.

[hoebat]

Follow-up finding: one rewrite was correctly rejected

A second low-dependency candidate in xt.lang.common-lib/wrap-callback was tested through the full cycle:

(xt/x:get-key callbacks key)

When changed to (. callbacks [key]), the canonical JS test still passed and Lua passed, but Python failed with KeyError: 'missing'. The source was reverted, and the canonical, Lua, Python, and Dart suites are green again.

This confirms the linter’s XT002 result must remain a warning/suggestion until access normalization preserves nil-safe lookup semantics across targets. The first common_sort_by rewrite remains safe because its key lookup is used on known record fields and all four target suites pass. The next implementation task is therefore to unify dot access and x:get-key in the semantic lowering/rewrite layer before promoting more source rewrites.

[hoebat]

Progress checkpoint: commit e04daa8e — Normalize canonical XTalk access lowering — has been pushed to main.

This snapshot covers the current canonical XTalk cleanup: grammar/lowering changes, shared language updates, clj-kondo configuration, regression coverage, and selected xt.event seed-test access cleanup. git diff --check is clean. The next stage remains the incremental xt.lang.* cleanup with canonical source as the authority and target-language differences handled in lowering/spec layers.