[talisp]

Editor plugins such as Continue send tab autocomplete to OpenAI's legacy /v1/completions, and Maya answered 404. This PR adds that endpoint. The prompt goes to the engine as it is: no chat template, thinking off, no tools. It runs through the same queue, stop strings and timings as a chat answer, and comes back as a text_completion, or its SSE chunks. With a suffix, the server renders the model's own fill-in-the-middle markers (DeepSeek's and Qwen's). A client that renders them itself, as Continue does for an OpenAI-compatible server, works too. Nothing else changes, and MAYA_COMPLETIONS=0 turns the route off.

On DeepSeek-V4.1-Flash-1017 (UD-Q8_K_XL, one L40S), fill-in-the-middle works. We removed one line from the middle of 127 HumanEval+ solutions. With the code after the gap, the model wrote a line that passes the problem's tests in 94 of 127. With only the code before the gap, it got 30. PHP fills were right too, for example $dados = json_decode($json, true);.

Speed is the limit on this box, and to be honest about it: for keystroke autocomplete our team now uses a small local model. When the prompt is the same as the last one, the answer comes in about 0.3 s. When the user typed something in the middle, the whole prompt is read again: about 7 s for 600 tokens and 10 s for 1,800, because short prompts read at about 90 tok/s here. With a real Continue setup, most suggestions arrived after the user had moved on. On a 16 GB GPU, the size our developers have, Qwen2.5-Coder 7B Base answered in under 0.6 s (p95). We still think the endpoint belongs in Maya: it lets one server do both on faster hardware, and it gives the speed-ups below something to aim at.

---

🤖 Humans can stop reading here. Machines, carry on.

The rest is data and method, so an agent or LLM can reproduce our numbers. It is not a prompt to paste.

What the PR changes (server only, no engine change):

• serve/server.py:
  • Service.completion_ids and Service.fim_markers;
  • completion_chunks / completion_collect, which reuse openai_chunks with thinking off and no tools and reshape its chunks;
  • the handler _completions, and the route in do_POST, including the busy 503 check.
• serve/test_completions.py: 13 tests against the mock engine:
  • raw prompt untouched;
  • streaming;
  • OpenAI's default max_tokens 16;
  • stop strings;
  • token-id and batch prompts;
  • 400s for echo, n, logprobs, empty prompt, and suffix on a vocabulary without FIM tokens;
  • DeepSeek and Qwen suffix rendering;
  • BOS not doubled, and add_special: false;
  • /v1/chat/completions unchanged;
  • the off switch.
• docs/SETTINGS.md: a section and the MAYA_COMPLETIONS row.

Request handling:

• prompt may be a string, a one-element list, or a list of token ids. Several prompts at once get a 400.
• Special-token text in the prompt is parsed, as /tokenize does.
• suffix is rendered with the first marker set the vocabulary has as single tokens (the same single-token test stop_ids uses):
  • DeepSeek PSM: <｜fim▁begin｜>{prompt}<｜fim▁hole｜>{suffix}<｜fim▁end｜>;
  • Qwen: <|fim_prefix|>{prompt}<|fim_suffix|>{suffix}<|fim_middle|>.
• DeepSeek's <｜begin▁of▁sentence｜> is prepended unless present or add_special: false.
• logprobs, echo, n/best_of above 1 get a 400.

Tests run:

• python -m unittest discover -s serve -p "test_*.py" -t . with this change on main (v1.1.58): 949 tests, all pass except 17 failures and 1 error. It was run on v1.1.57 too, with the same result.
• Those same 18 fail on unmodified v1.1.57 in the same environment. They are the page tests (test_dashboard, test_glm_dsa_models PageJs, test_glm_dsa_serve WebAppHistory, test_glm_dsa_think_budget, test_turbo), and our box's Node.js is v12.
• The 13 new tests pass.

Real model run (v1.1.57 engine, this branch's server, one L40S-48Q, 32 vCPU, Linux; production config, --parallel 2, DSpark on):

	passes tests	same line as the original	median s
suffix, rendered by the server	94 / 127	71	2.42
FIM prompt rendered by the client (Continue's DeepSeek template), same tokens	94 / 127	72	0.26
prefix only (no suffix)	30 / 127	21	1.93

• Method: HumanEval+ v0.1.10, one line removed from the middle of each canonical solution (seed 1110). Solutions with fewer than two code lines were skipped (127 of 164 kept). Temperature 0, stop: ["\n"], max_tokens 64. The rebuilt function ran the problem's base tests. With the original line, all 127 pass.
• The 0.26 s row: the client-rendered arm sent the same tokens right after the server-rendered one, so it reused the whole prompt (N-1 reused + 1 read).

Latency, server-rendered FIM, a prefix from this repository's code, max_tokens 32:

prefix	first	then, 3 times with the prefix changed at its end (typing)
~600 tokens	8.7 s	7.0-7.6 s
~1,800 tokens	10.9 s	10.3 s
~6,600 tokens	15.2 s	13.2 s
~6,000 tokens, while a chat answer runs	12.5-12.9 s (the chat ran at 12.9 tok/s meanwhile)

• Engine log: short fresh prompts read at 86-93 tok/s (prompt 426 tokens = 0 reused + 426 read in 4745 ms). The same prompt again is 425 reused + 1 read in 46 ms.
• An edit in the middle reuses nothing. In PSM order the suffix follows the cursor, so the prompt is no longer an extension of the last one.

With Continue on a real project (one developer, PHP, about 25 minutes with the route on): 58 autocomplete requests.

• Continue cancelled most of them after 6-18 s, because the user had kept typing. 7 came back, after 9-28 s, and their fills were correct (a getter, a controller method).
• Continue's request: its DeepSeek FIM template, max_tokens 4096, temperature 0.01, about 650 prompt tokens, stop list with // and <｜end▁of▁sentence｜>.
• Cancelled requests that were still reading their prompt finished the read before freeing the place: done: 0 tokens in 11 s (cancel).

What we use for keystroke autocomplete meanwhile (our lab's other run, a different harness, so not comparable with the HumanEval numbers above):

• Setup: one NVIDIA A2 16 GB, chosen because it has the VRAM of the GPUs our developers have. more than 26 local model/quant combinations, 40 PHP and 15 SQL completion cases scored against a reference, plus 5 mixed PHP+SQL cases, with first-token and full-suggestion latency.
• The finalists:

model (quant)	PHP /40	SQL /15	PHP+SQL /5	p95 first output	p95 full suggestion
Qwen2.5-Coder 7B Base (Q8_0)	31	12	4	86 ms	565 ms
Qwen3.6-35B-A3B (UD-Q3_K_XL)	22	7	3	247 ms	639 ms
Qwen3.8-27B (UD-IQ4_XS)	22	5	4	425 ms	1,631 ms

• Validity: PHP 8.3 accepted the syntax of 45/45 PHP suggestions from Qwen2.5-Coder. The scoring counts matches with the reference, so some equivalent solutions were not counted.
• The upshot:
  • For keystroke autocomplete, a 7B FIM model on the developer's own GPU is 10-20 times faster than this box's DeepSeek (0.6 s against 7-13 s).
  • DeepSeek on Maya stays the team's model for chat and agent work.
  • This PR is for setups where the big model is fast enough, and for the work below.

Not in this PR, ideas for after it:

1. A restore point a few tokens before the cursor. deepseek_restore_points only marks turn openers and drops points under RESTORE_LEAST (1024). For a FIM prompt, a mark shortly before <｜fim▁hole｜> would let the next keystroke re-read only the end of the prefix and the suffix. With short prompts reading at ~90 tok/s here, the 1024 floor is too high for this case.
2. Stopping the prompt read when its request is cancelled. Autocomplete cancels often, and here every cancel cost the rest of its read.

Checked:

• After the run, the same box measured decode 41.2 tok/s and prefill 782 / 1,506 / 1,950 tok/s at 8K / 32K / 128K, in line with v1.1.57's numbers here.
• r696 (tools with and without max_tokens) and r717 (content parts) passed, with 0 engine errors.

Related:

• DeepSeek: DS4_FLOW_COPY_DMA also where the RAM copies are split across CPU sockets - short prompts 11-16% faster on two sockets, the same bits #958 and DeepSeek: a short prompt's least-picked RAM experts on the CPU lane while the GPU streams the rest (MAYA_DS4_PREFILL_CPU, opt-in) - an agent's turn ~40% faster on PCIe 3.0 #959 (DMA and CPU work for short prompt reads): they target the ~90 tok/s short-prompt reads above. Their author measured 500-700-token agent turns going from 8.9-9.7 s to 4.1-4.9 s. We have not run them here yet.
• Fair slices with --parallel N: one waiting request cuts all N running answers, not one #882 and DeepSeek: a long prompt freezes the running answers for its whole read (4.3 s at 4K) - a pause between layers takes it to 0.4 s, the same bits #930: short requests waiting behind long ones under --parallel; autocomplete requests are short and frequent.
• No earlier request for /v1/completions or FIM in this repository, Strata, Colibri, Avalon, Basalt or Xolotl.

🤖 Generated with Claude Code

https://claude.ai/code/session_01E3U3Q9QVVJFxinHHkEaRu6