Collaboration with CIDER

[bbatsov]

As we briefly discussed on Slack I'm opening here a ticket to discuss future potential collaboration between the two projects. I'm pretty happy to see that someone is working on viable alternative to CIDER, as I haven't been able to dedicate much time to project recently, but I also know how fragile the balance is on most Elisp projects which are basically a one-man show most of the time. I'm now looking for opportunities to bring people and ideas to make sure that in the long run the Clojure community is stronger because of this.

I've long wanted to support the socket REPL (and unrepl) by association in CIDER, but lack of time and contributors have made this a pipe dream at this point.

While there are different ways to do this (the "right" one requiring a lot of time to abstract away certain implementation details in CIDER), I'm actually willing to consider any approach that might get you interested in this - even just moving the unrepl.el project under cider, continuing to develop it as you do, but with the ability to just reuse directly the relevant CIDER code and just gradually abstracting away connection differences. I'm guessing that if this path is taken in the end of the day the only major differences with respect to using unrepl instead of nREPL would be the improved REPL interactions (as far as end users are concerned) and some functionality not being supported with unrepl at least initially.

While I do see the value of REPL-centric approach to development, I think that ultimately it'd be nice if people had more flexibility how to interaction with code and the results it produces and having access to ready made UIs for pretty much everything would save a lot of time in the long run.

Just a crazy idea, I'm always open for other ideas as well. :-) Simply wanted to say I think you're doing some awesome work and I'd be happy for us to be on the same team!

[volrath]

Thanks for creating this issue. I had in my TODO to create a similar issue in CIDER to discuss possible ways to run the implemenation.

I think we're both on the same mindset: I've always been a CIDER user, and I would love it if CIDER supports socket REPLs (and Unrepl) because it would make the Emacs-Clojure community way stronger.

Regarding UI: I think it's important to clarify that my idea is not to promote REPL-centric development, but to make a UI that uses the REPL buffer as its primary tool for user interaction with anything that can't be handled directly on a Clojure buffer. With this I mean to favor using the REPL buffer instead of temporary buffers as the primary UI. So far two big examples come to my mind regarding switching from temp buffers to the REPL buffer: exceptions (whenever possible) and data inspection.
I guess I need to clarify this in the README.

I also took the path of making the REPL a bit more flashy: adding a history number to the prompt, adding an "eval result indicator" (the blue >), adding features like the grouped stdout thing, etc. While I understand most of these things are easily customizable in CIDER, I decided to go with these defaults as to make the UI more appealing for newcomers, and as to convey a certain sense of the UNREPL protocol rich capabilities.

To be honest, the opportunity to think and design this alternative UI is the decisive factor that drove me forward from just implementing UNREPL.el as a proof of concept, to trying to release a working IDE. Now, of course I'm completely open to criticism and would love to hear your thoughts on it.

I fear that by taking UNREPL.el under CIDER, this driving factor of having an alternative UI would disappear, since I'm guessing we would have to at least present a unified UI for nrepl and unrepl users.

I've also thought about the possibility of helping to get socket repl/unrepl support into CIDER while continue working on UNREPL.el, but I worry it might be too much for me and I end up doing a half ass job on any (or both) of them.

Anyway, I wanted to express all my concerns and ideas so far.

Just a crazy idea, I'm always open for other ideas as well. :-) Simply wanted to say I think you're doing some awesome work and I'd be happy for us to be on the same team!

This means a lot to me coming from you. I love CIDER and I would love to help bringing Socket REPL support to it, let's just keep discussing to make it happen =)

[bbatsov]

Regarding UI: I think it's important to clarify that my idea is not to promote REPL-centric development, but to make a UI that uses the REPL buffer as its primary tool for user interaction with anything that can't be handled directly on a Clojure buffer. With this I mean to favor using the REPL buffer instead of temporary buffers as the primary UI. So far two big examples come to my mind regarding switching from temp buffers to the REPL buffer: exceptions (whenever possible) and data inspection.

Yeah, that's a good approach in general. Frankly, I didn't think much about this in the past as I was so used to SLIME that I just tried to replicate as much as I could of it in CIDER. Also this seemed like a good transition for SLIME users, as many of them were all pretty used to it.

I do agree that for many things we can also render the data in the REPL buffer. There's a fork (of an old version) of CIDER that does something similar https://github.com/sanel/monroe

I fear that by taking UNREPL.el under CIDER, this driving factor of having an alternative UI would disappear, since I'm guessing we would have to at least present a unified UI for nrepl and unrepl users.

Not necessarily. While I think that many of your ideas can work with nREPL as well, I think in general it's not a big deal if one underlying REPL doesn't support the entire possible feature set or something like this.

Of course, it's always more work when you have to reconcile the capabilities of the different underlying protocols. :-)

This means a lot to me coming from you. I love CIDER and I would love to help bringing Socket REPL support to it, let's just keep discussing to make it happen =)

I think that if you can get the socket support REPL support in CIDER to a bootable state (there's some REPL which boots and evals stuff) finding people to port the rest of the core functionality won't be that hard. That would mostly amount to splitting cider-nrepl into two libraries and adding connection based command dispatch mechanism to CIDER. Imagine this ugly pseudocode as something more generic that's in REPL client functions (for instance).

if conn-type == nREPL
  invoke-middleware-op(x)
else if conn-type == socket-repl
  eval-function-x
else if ...
end

Anyways, my point is that if all you do is lay the foundation for socket REPL support, I'm relatively certain there would be some volunteers to work on the grunt work of gradually making more and more commands REPL-independent.

[arichiardi]

The last message is exactly the same I had in mind for inf-clojure + spiral merging. Dispatching is the easiest way to make it easy to support this or that protocol, even text only.

An RPC scheme for character-based communications.
While writing this I started to think: isn't it what unrepl already is? Well yes ..is it worth supporting text only repls, all in the same project? Mmm...maybe not. I cannot think of any reason at the moment.

[volrath]

@bbatsov I agree with you. At this moment, I feel I don't know enough about CIDER to be sure where to start. I also think that after setting that foundation, the rest might be a smoother ride. Maybe it will help if you could provide more detailed insight on how would you go to build this bootable version, or how do you see it working in terms of UI.

@arichiardi tbh I'm not entirely sure if I follow you. In general, unrepl differs from an RPC scheme by being synchronous and stream based. I guess I have to learn a bit about inf-clojure and inferior emacs repl processes anyway :)

I already wrote this in slack but I'll leave it here as well only for persistence.

Besides the pseudo-code above, there are some other things to think about:

1. both unrepl and nrepl respond differently, we'd need an abstraction layer between parseclj and nrepl-client.
2. cider-nrepl would need to be split in two, separating nrepl-related stuff away.

[bbatsov]

@bbatsov I agree with you. At this moment, I feel I don't know enough about CIDER to be sure where to start. I also think that after setting that foundation, the rest might be a smoother ride. Maybe it will help if you could provide more detailed insight on how would you go to build this bootable version, or how do you see it working in terms of UI.

Basically what's needed to get started is an alternative to the nrepl-client.el package where's the low-level connection bootstrapping and connection management. Together with cider-repl.el that's the only code that will require some changes initially (as the REPL buffer is basically the connection process, I assume it's something similar in SPIRAL a well).

What I see as most simple would basically be:

• some code to connect to a socket REPL endpoint and be able to send data and process results
• tweaking a bit cider-repl.el to be able to bootstrap a REPL buffer with the alternative client code and eval stuff there
• this would entail adding a type property to connections - right now they have a notion of type, but it's Clojure and ClojureScript and we need a different type (something like server-type or whatever). This can be just another buffer-local variable for connection buffers. This would be able to branch eval operations (and others), based on the type of the connection
• I'm already working (slowly) on https://github.com/clojure-emacs/orchard, so providing the core functionality over the socket repl should be relatively easy afterwards, just in terms of eval
• Probably cider-jack-in should eventually prompt about the type of server to start and connect to, but in the beginning you can just add some different command that boots and connects to the socket server.

I hope all of this makes sense. There's a lot of other code in CIDER, but it's not important for this first step at all. Once some evaluation is feasible over a socket REPL the rest is (mostly) mechanical work.

both unrepl and nrepl respond differently, we'd need an abstraction layer between parseclj and nrepl-client.

Yeah, that should obviously be handled in the low-level communication library. Perhaps it's possible to extract something shared between SPIRAL and CIDER? It'd be nice if something submitted parseclj to MELPA, so it can used more easily as a dependency.

cider-nrepl would need to be split in two, separating nrepl-related stuff away.

Yeah, that's true. As mentioned here and in the Clojurians, the work is finally beginning https://github.com/clojure-emacs/orchard I hope in a couple of weeks it'd be in some usable state.