WIP: prepl backend (architectural exercise, not for merge)#3899
Draft
WIP: prepl backend (architectural exercise, not for merge)#3899
Conversation
Sketch implementation on the prepl-support feature branch. Not yet
wired into cider-connect / cider-jack-in proper; the goal of this
commit is to show that the architecture works against a synthetic
input stream, with no JVM dependency in CI.
New files:
- lisp/cider-conn.el cl-defgenerics for the connection-protocol
boundary: cider-send-eval, -eval-sync, -op,
cider-supports-op-p, cider-conn-interrupt,
cider-conn-close.
- lisp/cider-conn-nrepl.el Thin wrappers implementing the generics on
top of the existing nREPL request layer.
Existing call sites unchanged for now.
- lisp/cider-prepl.el Process filter that incrementally reads
EDN responses from a Clojure prepl
(`clojure.core.server/io-prepl'), demuxes
by tag (:out/:err/:tap/:ret/:exception),
and feeds them into nrepl-make-eval-handler
via a synthesized response dict. FIFO of
pending eval handlers handles ordering-
based correlation. Sketch
cider-connect-prepl as a manual entry
point.
- test/cider-prepl-tests.el Five Buttercup specs driving the filter
with synthetic input: tag routing,
:exception path, partial-chunk buffering,
two-evals-in-flight demuxing, stray-
response handling.
Also update dev/design/prepl-support.md to use the agreed-on
`cider-send-*' naming throughout (we no longer prefix with `-conn-'
for wire methods; only the lifecycle methods keep the prefix to avoid
colliding with existing user commands like `cider-interrupt').
Real find from the test: prepl connection buffers need the same
nrepl-pending/completed-requests hashes the eval handler bookkeeping
expects. Fixed in both `cider-connect-prepl' and the test helper.
…der-connection.el
Original draft used names that collided with existing CIDER conventions:
- `lisp/cider-conn.el' was visually too close to the existing
`lisp/cider-connection.el'. Reader had to remember which was the
abstraction layer and which was the manager.
- `lisp/cider-conn-nrepl.el' read as two namespaces mashed together
for what was really just thin nREPL glue.
Both replaced. The new layout:
- `lisp/cider-backend.el' The `cl-defgeneric' boundary. Defines
`cider-send-eval' / `cider-send-eval-sync' /
`cider-send-op' (wire methods, dropping the
redundant `-conn-' tag), plus
`cider-supports-op-p',
`cider-backend-interrupt',
`cider-backend-close', the
`cider-backend-type' buffer-local, and the
`cider-backend-op-unsupported' error
symbol.
- `lisp/cider-connection.el' Existing surface, plus the nREPL
`cl-defmethod' block at the tail. These
are short wrappers around the existing
nrepl-client.el functions and belong next
to the rest of CIDER's nREPL-side glue.
- `lisp/cider-prepl.el' prepl protocol decoding + prepl
`cl-defmethod' block. prepl is a
self-contained protocol implementation, so
it justifies its own file.
Wire methods keep `cider-send-*' (the disambiguator from the dense
`cider-eval-*' user-command namespace). Lifecycle methods get
`cider-backend-*' to disambiguate from `cider-interrupt' /
`cider-close-buffer' user commands.
Tests still pass: 552/554 specs (same as before the rename, the 2
skipped are platform-gated).
…uffers
Two things land together because they're entangled:
1. Mark nREPL connection buffers with `cider-backend-type = nrepl' in
`cider-repl-create'. Without this, the nREPL `cl-defmethod's added
in the previous commit don't dispatch (they assert on the type).
The new generics still aren't called by anything in tree, so this
is invisible until the migration starts -- but it's a prerequisite.
2. Sketch `cider-prepl--info-via-eval' as the first eval-form fallback
for the `info' op. Wires up an `op-fallbacks' alist so adding more
ops (Step 3 of dev/design/prepl-support.md) is a one-line entry +
helper function.
Pattern for fallbacks:
- Build a Clojure form from PARAMS that gathers the data.
- Wrap the user HANDLER with an intermediate eval-handler that
parses the eval `:ret', reshapes it into the response dict the
op's nREPL response would have used, and feeds it to HANDLER.
- Call `cider-send-eval' with the form + intermediate handler.
`cider-send-op' on a prepl conn now dispatches through the alist;
unknown ops still signal `cider-backend-op-unsupported'.
`cider-supports-op-p' checks the alist.
Two real bugs found by writing the test:
- `plist-get' uses `eq', so it can't look up string-keyed op params.
Added a small `cider-prepl--params-get' helper using `equal'.
- The prepl filter normally dispatches handlers from inside
`with-current-buffer (process-buffer proc)' so the eval-handler's
bookkeeping reaches the connection's buffer-local hashes. The
test's spy on `cider-send-eval' didn't replicate that and tripped
over `nrepl--mark-id-completed' looking up nil. Spy fake now does
the same buffer setup.
Tests: 7/7 prepl specs pass, 554/556 in the full suite (+2 new).
Three pieces, in one commit because they're small and they make the
prepl prototype usable end-to-end:
1. Two more eval-form fallbacks following the same pattern as `info'.
- `apropos': `(mapv str (clojure.repl/apropos #"query"))'
-> response with `apropos-matches' slot.
- `ns-vars': `(mapv (comp str key) (ns-publics 'foo))'
-> response with `ns-vars' slot.
Both are simpler than `info' (no metadata gathering), so the
shared boilerplate factored out into a `cider-prepl--simple-via-eval'
helper. Each fallback now amounts to ~5 LOC: a Clojure form
string, a reshape function, an alist entry.
2. `cider-prepl-eval-string': minimal user-facing eval command.
Reads CODE from the minibuffer, sends to the current prepl
connection via `cider-send-eval-sync', echoes the value (or
the err on failure) to the minibuffer. No REPL UI yet -- this
is the smallest thing that lets a user actually try the
prototype against a running prepl.
3. Sesman registration in `cider-connect-prepl'. The connection
buffer now sets `sesman-system' to `CIDER' and registers itself
via `sesman-add-object', so prepl connections show up alongside
nREPL ones in `sesman-current-sessions'. Full sesman integration
(e.g. project-aware connection resolution) is later work; this
is just enough for `cider-prepl-current-conn' to find the
connection and for `cider-quit'-style commands to discover it.
Tests: 11/11 prepl specs pass, 558/560 in the full suite (+4).
Three thin wrappers on top of `cider-prepl-eval-string', mirroring the shape of `cider-eval-region' / `cider-eval-last-sexp' / `cider-eval-defun-at-point' on the nREPL side. Each just resolves a text region in the current buffer and forwards. Factored the value/err display logic into `cider-prepl--display-result' and the connection check into `cider-prepl--ensure-conn' so all four commands share one path. While writing tests for these I tripped over a real design issue with the `cl-defmethod' dispatch -- both nREPL and prepl methods specialize on `(conn buffer)', so cl picks whichever was defined last (load order), not whichever matches the runtime `cider-backend-type'. Today prepl wins by load order, but that's luck, not real dispatch. The tests work around it by spying at the wrapper boundary (`cider-prepl-eval-string') instead of the cl-defgeneric. Documented as Open Question #0 in dev/design/prepl-support.md with two proposed fixes; needs to land before any in-tree migration uses these generics.
Three more user-facing commands rounding out the basic prepl session: - `cider-prepl-load-file': sends `(load-file "...")' to the prepl. Caveat noted in the docstring that the path must be resolvable by the prepl process. - `cider-prepl-quit': closes the connection by delegating to the generic `cider-backend-close', so the same key (when bound) works regardless of backend. - `cider-prepl-doc': uses the existing `info' op fallback to look up a symbol's docstring and prints it to the echo area. Reads from the symbol at point as the default. `cider-prepl-doc' is the first command that actually exercises the op-fallback layer end-to-end (build a Clojure form, dispatch through the prepl filter, parse the result, hand back to the user-facing command). It does so synchronously by polling the response with `accept-process-output' and a 5s deadline -- a more polished implementation would pop a *cider-doc*-style buffer and accept the response asynchronously, but the sketch is enough to confirm the shape works. Tests: 17/17 prepl specs pass; 564/566 in the full suite.
…spatch
Two changes coming together because they fed each other:
1. New test/utils/cider-prepl-mock-server.el: a minimal in-Emacs
stand-in for `clojure.core.server/io-prepl'. Listens on an
ephemeral TCP port; each line of input maps to a small canned
table of `:tag/:val' EDN responses. Added three integration
tests that connect via `cider-connect-prepl' to the mock and run
eval forms end-to-end (value, stdout capture, exception path).
No JVM needed -- works in CI.
2. The integration tests immediately surfaced the dispatch bug
flagged as Open Question #0 in the design doc. Both backends'
`cl-defmethod' specialized on `(conn buffer)', so cl picked
whichever was defined last (the nREPL one happened to win),
not the runtime `cider-backend-type'.
Rather than the originally-sketched `(eql 'foo)' specializer
approach, replace the cl-generic layer entirely with a plain
dispatch table:
- `cider-backend.el' now defines `cider-backend-impl' struct (one
slot per protocol method) and `cider-backend-register'.
- `cider-send-eval' / `-eval-sync' / `-op' / `cider-supports-op-p' /
`cider-backend-interrupt' / `-close' are plain defuns that look
up `cider-backend-type', fetch the impl, and forward.
- Each backend (cider-connection.el for nREPL, cider-prepl.el for
prepl) defines plain `*--send-eval' helpers and registers via
`cider-backend-register'.
Simpler than the cl-generic approach, easier to extend (one
register call per backend), and crucially: dispatch actually
honors `cider-backend-type' now.
Tests: 20/20 prepl specs pass, 567/569 in the full suite. The three
new mock-server integration specs each take ~15ms because they
exercise the real network round-trip.
Updated dev/design/prepl-support.md to mark Open Question #0
resolved with the actual fix that landed.
User-facing documentation page at doc/.../repl/prepl.adoc plus a navigation entry. Covers what works, what doesn't, how to start an io-prepl from a deps.edn alias, the op-fallback table for the three ops we currently translate, and a short architecture pointer to the files involved. Front-loaded the experimental warning so nobody mistakes this for production-quality. Honest list of limitations: no interrupt (prepl-level), no clj+cljs bundling (nREPL+Piggieback shape), no REPL prompt buffer yet, no jack-in.
Three more entries in the eval-form fallback table, all following the
established `cider-prepl--simple-via-eval' shape:
- `ns-list': `(mapv (comp str ns-name) (all-ns))' -> {"ns-list": [...]}.
- `source': `(clojure.repl/source-fn (symbol ...))' -> {"source": "..."}.
Wraps in `(or ... "")' so a missing source returns an empty string
rather than an unparseable nil-as-edn-string.
- `macroexpand': `(pr-str (macroexpand-1 'code))' by default; honors
the `expander' param for `macroexpand' / `macroexpand-all' (the
-all variant requires `clojure.walk', which the form requires
inline).
Six fallbacks now: info, apropos, ns-vars, ns-list, source, macroexpand.
That covers the most-used non-trivial nREPL ops. More can join the
table without touching the dispatcher.
Updated the doc page's fallback table.
Tests: 24/24 prepl specs, 571/573 in the full suite.
Three small additions, all in the same area of the connection lifecycle: 1. Process sentinel. If the prepl socket closes (server crash, network drop, or the user explicitly closes it), `cider-prepl-- sentinel' fires. Any handlers still in `cider-prepl--pending-evals' that were waiting for a `:ret'/`:exception' get a synthesized `err' + `eval-error'/`done' so callers don't hang forever. 2. `cider-prepl--connect-params' buffer-local that records the original host/port at connect time, so we can reconnect with the same parameters later. 3. `cider-prepl-restart': close the current connection and reconnect using the recorded params. Uses the public `cider-backend-close' generic for the close, so the restart logic doesn't leak prepl-internal cleanup details. New spec drops the server process under a pending eval and verifies the handler gets drained with the right shape (err + eval-error + done).
Adds a basic interactive REPL surface so prepl connections aren't purely echo-area driven. Built on comint-mode for prompt protection, multi-line input, and history; the input-sender hands typed forms to `cider-send-eval' rather than `process-send-string' so responses still flow through our protocol decoder. Two specs: mode-on-connect + inline render of an eval response.
Three small wins for the prepl REPL buffer: - Keymap: C-c C-q (quit), C-c M-r (restart), C-c C-d C-d (doc), C-c C-o (clear-output). Mirrors the corresponding cider-repl-mode bindings on a much smaller surface. - `cider-prepl-clear-output' wipes the buffer up to the active prompt, matching the standard CIDER REPL behavior. - Value output now goes through `cider-font-lock-as-clojure', so result strings get the same syntax highlighting as in cider-repl.
`:tap'-tagged responses from io-prepl can arrive at any time -- they correspond to whatever the prepl process happens to call `tap>' on, not the eval in flight. Routing them onto the head pending handler (the previous behavior) was wrong: tap values from background work showed up as fake stdout for an unrelated eval. Now every `:tap' is appended to `*cider-prepl-tap <conn>*', a lazily created buffer linked back to its connection via a buffer-local pointer. C-c M-t / cider-prepl-show-tap-buffer pops it up.
Three more nREPL ops now have eval-form fallbacks on prepl:
- `eldoc' resolves the var, packages :arglists/:doc/:type. Enough
for the standard eldoc display.
- `complete' prefix-filters `ns-map' (publics + refers) and returns
`[{:candidate ... :type ...}]'. No compliment-style context, but
good enough for trivial completion in a REPL.
- `classpath' just splits `java.class.path' on the path separator.
Also adds `cider-prepl--hash->dict-form', a small helper that wraps
the existing key/value flattener with the leading `dict' symbol, for
ops (like complete) that need a list of fully-formed dicts inside a
parent dict.
Spawns a JVM with `clojure.core.server/io-prepl' on a free port (or `cider-jack-in-prepl-port' if set), waits for the port to be reachable, and runs `cider-connect-prepl' to attach. Deliberately not extending the existing `cider-jack-in-tools' registry: that shape is nREPL-flavored (params, middleware injection, etc.). prepl gets its own narrow command for now. A future tools-registry generalization could subsume both -- noted in the file's header.
Prepl was setting up `nrepl-pending-requests' / `nrepl-completed-requests' purely so the eval-handler returned by `nrepl-make-eval-handler' wouldn't crash when its done-branch called `nrepl--mark-id-completed'. Move the tolerance into nrepl--mark-id-completed itself: when the pending table is nil (because the buffer's backend isn't nREPL), the function is a no-op. Prepl no longer touches those tables. The eval-handler shape is already a documented reuse point; this makes the abstraction honest -- backends only opt in to the parts they actually need.
Three swaps: - `cider-prepl--params-get' was reimplementing equal-keyed plist-get; drop it for `nrepl-plist-get'. - `cider-prepl--hash->dict' returned a flat kv-list; `--hash->dict-form' wrapped it in `(dict ...)'. Replace the pair with one helper that builds an `nrepl-dict' directly via `nrepl-dict-put' (still strips the leading `:' from parseedn keyword keys -- the only reason we can't call `nrepl-dict-from-hash' verbatim). - `cider-prepl--simple-via-eval' reshape-fn now returns a full dict rather than a kv-list-to-be-spliced. The single-purpose info-via- eval shrinks from 40 lines of bespoke handler plumbing down to a call to simple-via-eval with `cider-prepl--hash->dict' as the reshape-fn -- structurally the same as eldoc. All inlined `(dict ...)' constructions go through `nrepl-dict' so the shape stays in one place.
The FIFO previously used `(append list (list new))', which is O(n) in the queue depth on every send. In normal interactive use the depth is 0-1 so it's invisible, but bulk operations (load-file splitting forms, scripted eval-region runs) make it quadratic. Use the `queue' package directly (CIDER already depends on it via nrepl-client.el). Same flat plist entries, just behind the queue struct -- enqueue/dequeue are constant-time. Required `queue' explicitly in cider-prepl.el rather than relying on the nrepl-client transitive load.
- `cider-prepl-doc' was polling for up to 5s with 50ms ticks just so it could `message' the result. The op handler is invoked asynchronously per response anyway -- track shown/done state in closures and fire the message inline. No blocking, no deadline. - `cider-prepl--close' now kills the per-connection tap buffer alongside the connection. The tap buffer was lazily linked via a buffer-local pointer but never torn down, so it leaked across reconnects. - `cider-prepl--send-eval-sync' was busy-polling at 50ms; pass the process to `accept-process-output' with a 1s timeout so the call blocks until that specific process produces output. - Drop redundant `(buffer-live-p)' / `(process-live-p)' guards in the comint-side helpers and `--close' -- the surrounding flow guarantees the values are good, and `kill-buffer' tears down a process all by itself.
Comment fluff that referenced the feature branch, design-doc Step numbers, or "polishing comes later" framing didn't belong in the shipped code -- it was meta about the change, not about why the code looks the way it does. Also moved the `cider-prepl--connect-params' defvar to the buffer-local state section so it sits next to the other locals and reads top-down rather than referenced-then-declared.
Silences byte-compile warnings about defcustoms not specifying a containing group.
io-prepl tags every `:ret'/`:exception' response with the namespace the eval ran in. Stash that on the connection buffer, plumb it through the eval handler's `:on-ns' callback, and use it as the prefix for subsequent prompts. So `(in-ns 'foo)' now flips the prompt from `user=> ' to `foo=> ', matching how the standard cider-repl-mode behaves. Mock server gains a `:tag/ns' suffix syntax in canned responses so end-to-end tests can exercise namespace transitions without spinning up a JVM.
Two small wins: - `cider-prepl-set-ns' (C-c M-n) -- mirrors `cider-repl-set-ns'. Reads a namespace from the minibuffer (default: source-buffer ns) and sends `(in-ns 'foo)' through the regular eval path. Combined with the prompt-tracking change, this gives the standard ns-switch ergonomics on prepl. - `cider-prepl-current-conn' now goes through the sesman-linked sessions for the current buffer first, falling back to a buffer- list scan only when no session is linked. The same machinery cider-current-repl uses, just narrower (filters by cider-backend-type).
- CHANGELOG.md: experimental-feature bullet listing the new commands and pointing at the prepl page. - repl/prepl.adoc: add `cider-prepl-set-ns', clear-output, and show-tap-buffer to the commands list; add a keybinding table. - dev/design/prepl-support.md: mark open questions 1, 2, 5 as resolved (sync semantics, tap routing, EDN reading); update 3 to reflect the partial jack-in implementation; leave 4 (registry generalization) explicitly deferred.
`cider-prepl-mode' bound RET to comint's default `comint-send-input', which submits whatever's after the prompt as a single form. Typing `(let [x' RET sent the fragment to the prepl, which would block on the unfinished read. `cider-prepl-return' (now bound to RET) parses the input region with the same `cider-repl--input-complete-p' the standard CIDER REPL uses: balanced -> submit; unbalanced -> newline-and-indent so the user can keep typing. Outside the input region, defer to comint's default (copy-old-input).
When a process chunk arrives and we have nothing pending in the partial-line accumulator, the existing `(concat acc string)' was copying STRING for no reason. Branch on the empty case so the common path (clean line boundaries) avoids the copy.
If the underlying socket has dropped (server crash, network blip, explicit `cider-prepl-quit'), `process-send-string' on the dead process previously raised a generic "process not running" Emacs error. Detect the case up front and `user-error' with the actual remediation.
Pure docstring change to make the registry forward-compatible with non-nREPL backends. Default `:backend' is `nrepl'; `:server-args-fn' is the prepl-backend hook for building server-startup argv. Mark `:params-var' / `:inject-fn' / `:jack-in-type' as nREPL-only -- they don't apply to prepl entries (prepl has no middleware injection and no clj/cljs distinction in the same session).
- `cider-jack-in-prepl' now reads its tool spec from the registry: binary via `:command-var' / `:default-command-fn', argv builder via `:server-args-fn'. Defaults to a new `clojure-cli-prepl' tool entry, but accepts any registered prepl-backend tool. - `cider-jack-in-universal' dispatches on `:backend': nREPL tools go through the existing `cider-jack-in-clj' / `-cljs' paths, prepl tools go through `cider-jack-in-prepl'. - Register `clojure-cli-prepl' (deps.edn, prefix arg 6). No `:project-files' on the entry -- a deps.edn project may be jacked in either way, and project-file detection picks one, so prepl is opt-in via the prefix arg or the explicit command. Backward compatible: existing third-party tool registrations omit `:backend' and pick up the `nrepl' default.
`cider-current-backend' (in cider-session.el) inspects the linked sesman session and returns `nrepl' / `prepl' / nil, with nREPL priority when both are linked. Plumbed into: - cider-eval-last-sexp - cider-eval-region - cider-eval-defun-at-point (skipped when DEBUG-IT is on -- prepl has no debugger fallback) - cider-load-file - cider-doc A Clojure source buffer connected only to a prepl session now responds to the regular keybindings (C-x C-e, C-M-x, C-c C-k, C-c C-d C-d) without the user having to reach for the cider-prepl-* siblings. No nREPL path changes for users with an nREPL session.
`start-file-process' instead of `start-process' so a remote `default-directory' transparently spawns the JVM on the remote host. For local jack-in the call is identical to before. Free-port discovery is still local-only (`make-network-process' binds locally); when `default-directory' is remote and `cider-jack-in-prepl-port' is 0, error up front so the user knows to set a fixed port instead of seeing an opaque "address in use" later. Two new customs to handle the network shape: - `cider-jack-in-prepl-bind-address' threads through to io-prepl's `:address' arg. Defaults to nil (= io-prepl default of 127.0.0.1). Setting "0.0.0.0" makes the remote JVM reachable directly from the local Emacs. - `cider-jack-in-prepl-host' overrides the connect-side host. Auto-detect uses the TRAMP host when remote, 127.0.0.1 otherwise -- override to "127.0.0.1" when an SSH tunnel forwards the port locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch prototypes a Clojure prepl (
clojure.core.server/io-prepl) backend for CIDER. It is not intended for merge; it's a thinking-out-loud branch left as a reference artifact.1. Useful as a separation exercise. Adding a second client forced the question "what's actually nREPL-specific in CIDER vs. what's general?". The eval-handler keyword API (#3892, already merged) was step one; building a second consumer surfaced a few more leaky-abstraction warts.
nrepl--mark-id-completedquietly assumed the request tables existed on every connection buffer.cider-jack-in-toolswas entirely nREPL-shaped (params, middleware injection). The eval-* user commands called nREPL request functions directly rather than going through an abstraction. Even with prepl shelved, those are real findings.2. Prototyping confirmed prepl isn't a good fit. Two reasons:
cider-current-backend. The jack-in registry needed a:backendslot. None of this is hard individually; all of it is overhead future features would have to keep paying.cljfor (debugger, structured stacktrace, inspector, profiler, test runner UI) are the same ones prepl can't host. A prepl-on-CIDER user would get a degraded experience compared to nREPL-on-CIDER while paying the same dispatch tax in the codebase.3. Things worth bringing forward independently:
nrepl--mark-id-completedtolerating nil request tables. A small defensive fix, correct even without a second backend.dev/design/prepl-support.mddocuments "we considered prepl, here's what it'd cost". Worth keeping as a reference for the next time the question comes up.make-network-process :server t :host "127.0.0.1" :service t) is generic enough to live incider-util.elif a second consumer ever shows up.Leaving the branch archived. If demand ever materializes (a serious Basilisp prepl push, say), the work is on file.