Code Completion Redesign

[tkobayas]

Code Completion Redesign Plan

Problem Statement

Current Architecture

The code completion pipeline for semantic completions (after .) is:

Broken MVEL text
  → ANTLR4 Parser (produces ErrorNodes for incomplete code)
  → TolerantMvel3ToJavaParserVisitor (patches ErrorNodes into JavaParser AST)
  → JavaSymbolSolver (resolves type of scope expression)
  → Completion items (fields, methods, properties)

TolerantMvel3ToJavaParserVisitor (312 lines) extends Mvel3ToJavaParserVisitor and overrides
three methods (visitBlock, visitBlockStatement, visitMemberReferenceExpression) to handle
incomplete expressions by detecting ANTLR ErrorNodes, reconstructing expressions from raw tokens,
inserting __COMPLETION_FIELD__ sentinel markers, and merging fragmented block statements.

Why It Is Fragile

1. Two overlapping recovery paths that fire unpredictably:

   • visitMemberReferenceExpression handles the case where ANTLR successfully creates a
     MemberReferenceExpressionContext with a trailing dot (detected by hasTrailingDot()).
   • visitBlockStatement + buildIncompleteExpression handles the case where ANTLR failed
     to produce any rule context and created raw ErrorNode children instead.
   • Which path fires depends on ANTLR's error recovery heuristics, which are grammar-version-dependent
     and vary with surrounding code. The same input can trigger different paths.

2. buildIncompleteExpression only understands identifier.identifier.identifier:

   • Creates NameExpr and FieldAccessExpr only — no parentheses, no arguments, no method calls.
   • Cannot handle method chains: list.stream().filter(x -> x > 5).
   • Cannot handle MVEL null-safe access: person!.address!. (EXCL_DOT is unrecognized).

3. Every new syntax construct before a trailing dot needs new tolerant handling:

   • MVEL inline cast (l#ArrayList#.), null-safe (person!.), method reference (Foo::), etc.
   • Each requires case-by-case code in the tolerant visitor.

4. Pre-existing flaw in tokenIdJPNodeMap lookup (in Mvel3CompletionHelper, lines 191-213):

   int scopeTokenIndex = previousTokenIndex - 1;  // token before the DOT
   Expression scopeNode = (Expression) tokenIdJPNodeMap.get(scopeTokenIndex);

   This assumes the token immediately before . is mapped. But for list.stream()., the token
   before . is ), and tokenIdJPNodeMap typically maps only identifiers — so scopeNode
   is null for method chains. This must be fixed regardless of approach.

Considered Approaches

A. Text Pre-Processing / Placeholder Injection (Simple)

Inject __placeholder__ after trailing dot to produce syntactically valid text, then use the
normal Mvel3ToJavaParserVisitor.

• Works for: person.address., l#ArrayList#., list.stream().map(...).
• Fails when: other broken code elsewhere in the file causes cascading ANTLR error recovery;
  cursor is mid-code without semicolon injection; multiple incomplete expressions exist.

B. Scope Expression Extraction from Token Stream

Walk backward from cursor in the token stream, extract the scope expression text, parse it in
a controlled wrapper. Used by Eclipse JDT and kotlin-language-server.

• Immune to other broken code (isolated sub-parse).
• Requires a backward-walk parser for balanced parens, generics, MVEL tokens.
• Needs variable/type context injected into the wrapper.

C. Truncation + Placeholder (Kotlin-Style) — Recommended Primary Path

Truncate the file at the cursor position, inject placeholder + closing syntax, parse with
the normal visitor.

• Eliminates all broken-code-after-cursor problems (the most common scenario during editing).
• Combined with placeholder injection, covers the majority of real-world cases.

D. AST Caching with Localized Recovery

Cache the last successfully-parsed JavaParser AST. Use it as fallback when error recovery fails.

• Orthogonal to other approaches; improves robustness.

Chosen Architecture: Layered Hybrid

Layer 1 — Truncation + Placeholder (Primary Path)

Handle the common case: user is typing at or near the end of an expression.

Steps in Mvel3CompletionHelper.createSemanticCompletions():

1. Truncate the input text at the cursor position (discard everything after the cursor line).
2. Scan the trailing characters of the truncated text to detect the completion trigger pattern
   (e.g., trailing ., !., #Type#.).
3. Inject a synthetic identifier (__completion__) after the trigger:
   • Trailing . → append __completion__
   • Trailing !. → append __completion__
   • Trailing #. (end of inline cast) → append __completion__
4. Inject a semicolon if in statement context (not inside parentheses or brackets).
5. Close open braces by counting unmatched { in the truncated text and appending matching }.
6. Parse the patched text with the normal Mvel3ToJavaParserVisitor (not tolerant).
7. Walk the resulting JavaParser AST to find the __completion__ node (a NameExpr or
   FieldAccessExpr with that name). Navigate to its parent to get the scope expression.
8. Resolve the scope type via JavaSymbolSolver and produce completion items.

Why this works for the main cases:

Input	After truncation + injection	Parses as
person.address.	person.address.__completion__; + closing braces	Valid statement
l#ArrayList#.	l#ArrayList#.__completion__; + closing braces	Valid inline cast + member access
list.stream().	list.stream().__completion__; + closing braces	Valid method chain + member access
person!.	person!.__completion__; + closing braces	Valid null-safe + member access
foo(bar.) (mid-expression)	foo(bar.__completion__ + close parens/braces	Valid nested expression

Layer 2 — AST Walk Instead of tokenIdJPNodeMap

Replace the fragile token-index lookup:

// BEFORE (fragile):
int scopeTokenIndex = previousTokenIndex - 1;
Expression scopeNode = (Expression) tokenIdJPNodeMap.get(scopeTokenIndex);

// AFTER (robust):
// Walk the JavaParser AST to find the __completion__ sentinel node.
// The scope is the parent expression of that node.
Expression scopeNode = findCompletionScope(compilationUnit, "__completion__");

Implement findCompletionScope() as a JavaParser VoidVisitorAdapter that:

• Visits all FieldAccessExpr nodes looking for name __completion__
• Visits all NameExpr nodes looking for name __completion__
• Returns the scope (parent expression of the found node)

This fixes method-chain completion because the AST naturally represents list.stream() as a
MethodCallExpr, which is the correct scope node for type resolution.

Layer 3 — Fallback to ANTLR4-C3 Syntactic Completion

If the parse fails (broken code before the cursor that cascading error recovery cannot handle),
fall back to keyword/type suggestions from CodeCompletionCore, which already works today
(the getCompletionItemsFromTokens path).

This graceful degradation means the user always gets some completion — syntactic keywords at
minimum, semantic members when the parse succeeds.

Layer 4 (Future) — AST Caching

Cache the last successfully-parsed CompilationUnit (JavaParser AST). On each completion request:

1. Attempt the truncation + placeholder parse (Layer 1).
2. If it produces a valid AST, use it and update the cache.
3. If it fails, use the cached AST for type resolution where possible.

This is an optimization for later — not required for the initial redesign.

What Gets Deleted

• TolerantMvel3ToJavaParserVisitor.java (312 lines) — entirely removed from mvel repo.
• tokenIdJPNodeMap lookup in createSemanticCompletions() — replaced by AST walk.
• The __COMPLETION_FIELD__ sentinel concept moves from the visitor layer to the text
  pre-processing layer in Mvel3CompletionHelper.

What Gets Added / Modified

Mvel3CompletionHelper.java (modified)

1. New private method: prepareTextForCompletion(String text, Position caretPosition)

   • Truncates text at cursor line
   • Detects completion trigger (., !., #.)
   • Injects __completion__ identifier
   • Injects semicolon if in statement context
   • Closes open braces
   • Returns the patched text

2. New private method: findCompletionScope(CompilationUnit cu, String sentinel)

   • Walks JavaParser AST to find the sentinel node
   • Returns the scope expression (parent of sentinel)

3. Modified createSemanticCompletions()

   • Uses prepareTextForCompletion() instead of TolerantMvel3ToJavaParserVisitor
   • Uses findCompletionScope() instead of tokenIdJPNodeMap lookup
   • Uses the normal Mvel3ToJavaParserVisitor for AST conversion

Estimated Complexity

Component	Lines (approx)
prepareTextForCompletion()	~40-60
findCompletionScope()	~20-30
Modified createSemanticCompletions()	~30 (simplified from current)
Net change	~100 lines added, ~312 lines deleted

Test Plan

Existing Tests (must continue to pass)

• Mvel3CompletionHelperTest — complete/valid code completions
• Mvel3CompletionHelperIncompleteCodeTest — the three existing incomplete-code tests:
  • emptyInput() — empty document
  • testInlineCast() — l#ArrayList#. completion
  • incompleteClass_PropertyAccessor() — p.address. completion

New Tests to Add

Test	Input	Expected
Method chain completion	list.stream().	filter, map, forEach, ...
Nested method call	foo(bar.) where bar is typed	members of bar's type
Null-safe completion	person!.	members of Person
Broken code before cursor	int x = ;\nperson.address.	members of Address (or graceful fallback)
Cursor mid-file	person. followed by int y = 42;	members of Person
Multiple dots in chain	a.b.c.	members of c's type
After method with args	list.subList(0, 5).	members of List

Implementation Order

1. Implement prepareTextForCompletion() with unit tests for the text transformation itself
2. Implement findCompletionScope() AST walker
3. Rewire createSemanticCompletions() to use the new methods with normal Mvel3ToJavaParserVisitor
4. Verify all existing tests pass
5. Add new tests (method chains, null-safe, mid-file cursor, etc.)
6. Delete TolerantMvel3ToJavaParserVisitor.java from the mvel repo
7. Remove tokenIdJPNodeMap usage from completion code (the field remains in the base visitor
   for other use cases in the mvel compiler)

Risks

• Brace counting for closing open braces may be tricky with string literals and comments
  that contain { characters. Use a simple state machine (track whether inside string/comment)
  rather than naive character counting.
• ANTLR error recovery for code before the cursor remains unpredictable. The fallback to
  ANTLR4-C3 syntactic completion mitigates this.
• JavaSymbolSolver resolution failures for complex generic types or MVEL-specific AST nodes
  may require targeted fixes. Handle with try-catch and graceful degradation.

[baldimir]

First set of changes done as part of PRs:
mvel/mvel#410
#7