Completion not working in hy-mode

[noobymcnoob]

Hy version: 0.18.0

The following code does not seem to complete in hy-mode. I do have completion for builtins and as you can see, it does work with jedhy directly. I'd love to debug so if you can give some tips on where to start, that would be great.

(import jedhy.api)
(setv api (jedhy.api.API))

(defclass Foo []
  (defn --init-- [self a b]
    (print self a b))
  (defn test [self a b]
    (print (+ a b))))

(setv foo (Foo 1 2))

(api.set-namespace :locals- (locals))
(api.complete "foo.t") ;;; -> ('foo.test',)

(foo.) ;; no completion

[allison-casey]

From what I can discern, I think when hy mode constructs the jedhy instance its making the call to locals in the internal hy repl and not the user repl where the locals are defined:
(hy-shell--redirect-send-internal hy-jedhy--reset-namespace-code)

I don't know how to fix it yet, but maybe this will help

[noobymcnoob]

That sounds plausible, will give it a run

[allison-casey]

so did some more digging and I don't know if we can get local var completions without running jedhy in the user's shell. Jedi gets around it be doing some crazy static code analysis, but jedhy is just doing a (dir <var>) which needs to be done in the user shell. I'll have a stab at running it in the user shell and see what happens, but idk what side effects that would have 🤷‍♀️

[ekaschalk]

It doesn't use the regular shell because of eg. long running computations and polluting completions with variables you set in the shell.

We get some of the way there by providing a command to send the imports in the current file (identified by a regex elisp side). This resolves completions for big imports like numpy and pandas.

If you look at the regex it also sends sys.path extensions.

This can be used to accomplish local completions in effect. (Eg. Use 'import * from myfile').

I used this when I was still working actively with hy. It is not perfect or even a good solution but it was effective for me compared to the alternative at the time (hy-mode/tooling was basically nothing).

[noobymcnoob]

so did some more digging and I don't know if we can get local var completions without running jedhy in the user's shell. Jedi gets around it be doing some crazy static code analysis, but jedhy is just doing a (dir <var>) which needs to be done in the user shell. I'll have a stab at running it in the user shell and see what happens, but idk what side effects that would have woman_shrugging

I'd be OK if jedhy ran in my user shell. Let me know if you have some ideas I can try on my end (simply changing the buffer doesn't do it.)

[ekaschalk]

In theory I don't see anything stopping Hy from substituting the python AST it constructs directly into jedi or a python LSP implementation.

Then we could access Python's tooling - save for Hy's unique constructs like macros. It certainly is in-line with what I see attractive about Hy (that it interops with python's ecosystem).

This would be quite involved though.

I'd be OK if jedhy ran in my user shell.

The internal shell is also technically a user shell. In particular you could type (setv foo 1) in that shell and run the command Allison correctly identified to complete foo in any Hy buffer.

If updating hy-shell--name-internal and hy-shell--buffer-name-internal does not work, you have some options like:

1. Adding a variation (or just always) of the send-to-shell commands that does hy-shell--redirect-send-internal with the text too.

2. Adding a comint hook in the shell buffer that calls redirect-send with the text you are sending

3. Maintain a "working file" that has a list of sys.path extensions and "import * from my-current-working-project.file-1, ...".

Note that 1 and 3 can be utilized together - and is what I've done in the past.

[allison-casey]

Feeding the generated ast directly into Jedi world be extremely cool, and pretty in line with the whole python interop philosophy. I don't know if it would be possible though as an incomplete attribute access (something.) is a syntax error in Hy and would fail to generate the ast at all. I guess the question is, can we generate an invalid AST? Would suppressing that syntax error break the rest of the compilation? Sounds like it's experiment time 🥳

[allison-casey]

This is honestly such a dumb solution but it works for some reason and I find it hilarious that it does.

(setv source "
(setv something [1 2 3 4])
(print (something.))
"
      cursor-line 3
      cursor-column 18)
(lfor completion (complete source cursor-line cursor-column)
  completion.name)
;; => ['append', 'clear', 'copy', 'count', 'extend', ... ]

(setv source "
(import json)
(json.lo)
"
      cursor-line 3
      cursor-column 8)

(lfor completion (complete source cursor-line cursor-column)
  completion.name)
;; => ['load', 'loads']

Basically, i'm using hy2py to compile to python keeping track of the object and attribute we want completions for through the hy => py compilation. If theres no attribute, I changed the compiler to replace the empty attribute with ___COMPLETION_PLACEHOLDER___ (like i said very dumb). Once its compiled and we've found the line number and column of the obj and attribute (or the placeholder which i strip out), Feed those into jedi and boom! we have completions.

You can find the gist here: https://gist.github.com/allison-casey/33da1d7777bc7727a3140b7d328d7ebb
Again this is super dumb, badly written, and I wrote it in a half hour so keep that in mind

[allison-casey]

It works! It's clunky and lags a bit but for a proof of concept it works well enough. If I can get this to perform better and be more robust I'll open a pull request here and in the hy compiler repo.

[noobymcnoob]

That's pretty freaking clever @allison-casey!!

[ekaschalk]

+1 here! Feel free to reach out over anything.