Shell command completion

[ElectreAAS]

Follow-up to #14636, which bumped the version of cmdliner to 2.1.1
This version bump allows us to use the command completion features of cmdliner for supported shells: bash, zsh, and powershell.

What is included

For supported shells

• (sub)command completion: dune b<tab> -> dune build
• (sub)command suggestion dune init <tab> -> executable library ...
• flag completion & suggestion dune build --pro <tab> -> --profile --promote-install-files

What is NOT included

• Target completion dune build <tab> -> @test @install .... The code infrastructure for this isn't present in dune yet, from what I understand it'd be a significant undertaking.
• Completion for fish 🐟 😢

Experimental testing

I was only able to test that this works in bash on my machine, reviewer help for zsh and more importantly powershell would be welcome.
To test this PR, simply run dune build @install, and if your local share directory is in _opam, run eval $(opam env) (or equivalent). You should be able to see completions!

Reproduction testing

This PR doesn't yet contain tests for this behaviour, I'll look into what's possible to make

• tested on bash
• tested on zsh
• tested on pwsh

Thanks

All my thanks to The cmdliner programmers whose work made this possible. @dbuenzli ❤️

[Alizter]

What is cmdliner doing that it takes 7 seconds to generate all these files? Let's investigate and try to speed this up a little.

{
  "cat": "process",
  "name": "finish",
  "ts": 1780090345.0997474,
  "args": {
    "process_args": [
      "install",
      "tool-support",
      "../../bin/main.exe:dune",
      "cmdliner-support"
    ],
    "pid": 3697916,
    "categories": [],
    "prog": "../../vendor/cmdliner/src/tool/cmdliner_main.exe",
    "dir": "_build/.sandbox/ea3e637369c0ec1fb679b2ec0a531450/default/editor-integration/completions",
    "exit": 0,
    "target_dirs": [
      "_build/default/editor-integration/completions/cmdliner-support"
    ],
    "rusage": {
      "user_cpu_time": 3920148000,
      "system_cpu_time": 3049940000,
      "maxrss": 219684,
      "minflt": 1266988,
      "majflt": 0,
      "inblock": 0,
      "oublock": 11712,
      "nvcsw": 2205,
      "nivcsw": 839
    }
  },
  "dur": 7.1887091779999999
}

[Alizter]

Suggested change

	(rule
	(target
	(dir cmdliner-support))
	(deps
	(:cmdliner %{project_root}/vendor/cmdliner/src/tool/cmdliner_main.exe))
	(action
	(ignore-stdout
	;; This installs both manpages and the scaffolding for command completion
	(run %{cmdliner} install tool-support %{bin:dune}:dune cmdliner-support))))
	(rule
	(targets
	(dir cmdliner-support))
	(deps
	(sandbox always)
	(:cmdliner %{project_root}/vendor/cmdliner/src/tool/cmdliner_main.exe))
	(action
	(progn
	;; This installs both manpages and the scaffolding for command completion.
	;; stderr is captured and diffed so that any cmdliner diagnostics remain
	;; visible and error.
	(no-infer
	(with-stderr-to
	cmdliner-install.stderr
	(ignore-stdout
	(run
	%{cmdliner}
	install
	tool-support
	%{bin:dune}:dune
	cmdliner-support))))
	(diff? cmdliner-install.stderr.expected cmdliner-install.stderr))))

• (sandbox always) ensures that we run this in a sandbox
• (no-infer ) ensures that targets are not inferred
• (diff?) lets you diff against a non-existant file

The net result is that we don't produce any silly targets and we have errors if there is ever stderr from cmdliner, as we had with the unescaped $ in the manpages.

[Alizter]

needs a changes entry too

[Alizter]

Having a look at the cmdliner executable there are a few options that might be worth going through and considering. I see options for standalone completion scripts and also avoiding generating man pages.

The man page generation appears to be the slow part in my complain above. We should avoid doing that here and just focus on completion. In a future PR we can consider a way to do it well.

[dbuenzli]

All my thanks to The cmdliner programmers whose work made this possible. @dbuenzli ❤️

Especially @WardBrian! who is instrumental in helping with the completion scripts (which are beyond my sanity).

• Completion for fish 🐟 😢

It's up for grabs dbuenzli/cmdliner#213

What is cmdliner doing that it takes 7 seconds to generate all these files?

That sounds quite long indeed, but it's likely the manpages. Basically if you have n (sub)commands it will spawn your tool n+1 times via Sys.command with --help=groff. I don't know how the main of dune is structured but is there perhaps setup stuff dune odes unconditionally regardless of cli parsing (and thus also on --help) that would make these spawns egregriously slow ?

[WardBrian]

Especially @WardBrian! who is instrumental in helping with the completion scripts (which are beyond my sanity).

🥳. I must admit I'm glad we figured out pretty robust answers for cmdliner, if for no other reason than I never need to try to hand write these sorts of things again...

but is there perhaps setup stuff dune odes unconditionally regardless of cli parsing (and thus also on --help) that would make these spawns egregriously slow ?

Interestingly, I just ran into this in my own cmdliner-using project (stan-dev/stanc3#1623). A few strategic lazy statements sped up the do-nothing case by 2.7x. This kind of thing could definitely be impacting the build time, but it is also nice to speed up because the executable gets invoked by the completion script and unnecessary startup work could make <TAB> sluggish as well