"RET" binding in a vterm buffer not working

[johannes-mueller]

Consider the following transient prefix

(defun hello-world () (interactive) (message "Hello world!"))
(defun saluton-mondo () (interactive) (message "Saluton mondo!"))

(transient-define-prefix hello-world-menu ()
  [("s" "Diru saluton" saluton-mondo)
   ("RET" "Say hello world" hello-world)])

(global-set-key [C-f12] 'hello-world-menu)

Works fine: hit C-f12 → menu pops up → hit <return> → message "Hello World!" is issued.

Except if the current buffer is a vterm. Then it says

Unbound suffix: ‘<return>’ (Use ‘C-g’ to abort, ‘?’ for help) [vterm-send-return]

Setting transient--debug to t gives the following:

<hit C-<f12>>
-- setup                  (cmd: hello-world-menu, event: "C-<f12>", exit: nil)
-- stack-zap              (cmd: hello-world-menu, event: "C-<f12>", exit: nil)
-- setup-transient        (cmd: hello-world-menu, event: "C-<f12>", exit: nil)
     push transient--transient-map
     push transient--redisplay-map
-- post-command           (cmd: hello-world-menu, event: "C-<f12>", exit: nil)
<hit <return>>
-- pre-command            (cmd: vterm-send-return, event: "<return>", exit: nil)
Unbound suffix: ‘<return>’ (Use ‘C-g’ to abort, ‘?’ for help) [vterm-send-return]
-- post-command           (cmd: transient-undefined, event: "<return>", exit: nil)
<hit C-g>
-- pre-command            (cmd: transient-quit-one, event: "C-g", exit: nil)
-- pre-exit               (cmd: transient-quit-one, event: "C-g", exit: t)
     pop  transient--transient-map
     pop  transient--redisplay-map
-- post-command           (cmd: transient-quit-one, event: "C-g", exit: t)
-- post-exit              (cmd: transient-quit-one, event: "C-g", exit: t)

My first thought was, that vterm remaps the <return> key to vterm-send-return but it also remaps almost all other keys to vterm--self-insert. And the above transient prefix works when s is hit. Then the debug looks like this:

<hit C-<f12>>
-- setup                  (cmd: hello-world-menu, event: "C-<f12>", exit: nil)
-- stack-zap              (cmd: hello-world-menu, event: "C-<f12>", exit: nil)
-- setup-transient        (cmd: hello-world-menu, event: "C-<f12>", exit: nil)
     push transient--transient-map
     push transient--redisplay-map
-- post-command           (cmd: hello-world-menu, event: "C-<f12>", exit: nil)
< hit s>
-- pre-command            (cmd: saluton-mondo, event: "s", exit: nil)
-- stack-zap              (cmd: saluton-mondo, event: "s", exit: nil)
-- pre-exit               (cmd: saluton-mondo, event: "s", exit: t)
     pop  transient--transient-map
     pop  transient--redisplay-map
Saluton mondo!
-- post-command           (cmd: saluton-mondo, event: "s", exit: t)
-- post-exit              (cmd: saluton-mondo, event: "s", exit: t)

I tried to track this down a bit further, but it gets quickly into Emacs internals I am not familiar with.

The question is now how – or better where – to fix this. It does not seem to be related to any other quirk in my Emacs setup. I tried it with --no-init just setting the load-path to find vterm. Same phenomenon.

This all happened with the builtin transient version 0.7.2.2 from Emacs 30.1

Any ideas?

Thanks for any kind of support.

[tarsius]

My first thought was, that vterm remaps the <return> key to vterm-send-return but it also remaps almost all other keys to vterm--self-insert.

I seems to remap that key differently from how it remaps other keys. Emacs then sees <return>, with the expected result. I am sorry but I do not have time to investigate.

[johannes-mueller]

Thanks for your answer. It has given a very helpful hint. I found out that vterm defines the return key actually twice:

https://github.com/akermu/emacs-libvterm/blob/master/vterm.el#L681-L682

    (define-key map [return]                    #'vterm-send-return)
    (define-key map (kbd "RET")                 #'vterm-send-return)

Removing the definition for [return] fixes the issue. So the snippet to workaround this is:

(define-key vterm-mode-map [return] nil t)

Issue reported in akermu/emacs-libvterm#765