Enable Zsh completions

[j-g00da]

No description provided.

[johnslavik]

Thanks!

[pawamoy]

Thanks a lot @j-g00da! It's really cool to get completions for Zsh too 😄

One thing I'd like to see for Zsh completions is actual description of the completion words: Zsh's completion system allows to attach a description to each term/word, that is then displayed when hitting TAB in the shell. That means we'd have to add a bit of code to our logic for generating word candidates here, so that descriptions are returned too. I see two options:

• the logic always returns tuples (word, description), and higher-up in the stack we filter out descriptions if the shell doesn't support those
• the logic accepts a shell argument that lets it know whether it should return descriptions as well as words

We would have to check how this integrates with the actual completion script (completions.zsh) and compctl command.

[pawamoy]

I understand you may have taken inspiration from the Bash example above, but this doesn't seem right to me. The Bash example uses standard locations that Bash immediately understands. This location would require users to fiddle with the Zsh configuration to load the completion from ~/.duty I believe. Furthermore, not using standard locations is generally frowned-upon by users (me included 😄) as that leads to a cluttered HOME directory 🙂

In short, could you try to see if there's a standard location we could write the file in, so that Zsh's completion system natively finds it?

[j-g00da]

In short, could you try to see if there's a standard location we could write the file in, so that Zsh's completion system natively finds it?

From what I understand, zsh doesn't have any "standard" location for completions other than /usr/local/share/zsh/site-functions, which is in fpath by default, but I don't think we want to install it system-wide. I based my approach on oh my zsh, which puts completions under ~/.oh-my-zsh/completions and adds it to fpath. Still, I think cluttered HOME is a good point. Let me know what you think.

One thing I'd like to see for Zsh completions is actual description of the completion words

Good point, I will work on this later today.

[pawamoy]

Thanks! We could then document that and let the user decide. I found https://github.com/zsh-users/zsh-completions/blob/master/zsh-completions-howto.org#telling-zsh-which-function-to-use-for-completing-a-command to be very readable, maybe we could link to it.

[j-g00da]

I've modified the docs yestarday and changed the completions.zsh to use compdef (rich completions) instead of old compctl. Still no descriptions, but now it will be possible to add them. Will work on adding descriptions later today or tomorrow. You can take a look at the docs and let me know if it's enough information (I will polish the text later).
Sources: this answer on stackoverflow, and ofc the zsh-completions-howto.org.

[j-g00da]

Also - completions.zsh is WIP, it's just a minimal example for now, but it works, and I know opts.complete parsing as of now breaks the bash complete implementation.
Also 2 - We can make it compatible with old zsh versions, but I don't know if this is that important since user can always use bash completions in zsh if native approach doesn't work (explained in the docs).

[pawamoy]

Thank you so much! Don't bother about supporting old Zsh versions, latest is fine 🙂

[johnslavik]

I'm yearning for some automatic tests of those scripts. WDYT @pawamoy

[pawamoy]

@bswck yes, testing is always good! I'm definitely not sure how to test auto-completion though! Any idea? Wouldn't it be enough to just test the output of our Python code that generates words?

[johnslavik]

Wouldn't it be enough to just test the output of our Python code that generates words?

Definitely no, I strongly feel we need white-box regression tests of completions working on the user end.

I'm definitely not sure how to test auto-completion though! Any idea?

We can get some inspiration from typer tests which seem to do exactly that.

[j-g00da]

I think of doing a bit of extra work on it, since the docs on how to enable completions grew in the Zsh section.

Duty is simple, and installing completions for it should also be simple. It’s fine to inform the user about options, but for the most part… it's too much and I really would like something like duty --install-completions that would just install it in some default place, for auto-detected shell so I don't have to think about it.

For Bash, it would be really easy - just run what’s in the docs. For Zsh I think the best option would be to put it in /usr/local/share/zsh/site-functions, because:

1. This is what most tools use - homebrew, uv, cargo to name a few.
2. Doesn't require adding to fpath, as it is there by default; which normally would be tricky, as compinit must be run after adding a dir to fpath, so we would have to parse .zshrc and do some unholy stuff or ask user to manually do it, which misses the point.

In fact, I see only one disadvantage of such an approach - /usr/local/share is by default owned by root on linux. This is not a problem on macos, where /usr/local/share is owned by the current user. So, in short - this command would require sudo on Linux. I still think it's 100% worth it, let me know what you think @pawamoy, @bswck.

[j-g00da]

Also - I just checked Typer source code, since @bswck linked it, and this is exactly what they do:

• --install-completion: Install completion for the current shell.
• --show-completion: Show completion for the current shell, to copy it or customize the installation.

But when it comes to installation dir, it's ~/.zfunc. So as I said, there really is no "default" directory for that with Zsh...
source:

def install_zsh(*, prog_name: str, complete_var: str, shell: str) -> Path:
    # Setup Zsh and load ~/.zfunc
    zshrc_path = Path.home() / ".zshrc"
    zshrc_path.parent.mkdir(parents=True, exist_ok=True)
    zshrc_content = ""
    if zshrc_path.is_file():
        zshrc_content = zshrc_path.read_text()
    completion_line = "fpath+=~/.zfunc; autoload -Uz compinit; compinit"
    if completion_line not in zshrc_content:
        zshrc_content += f"\n{completion_line}\n"
    style_line = "zstyle ':completion:*' menu select"
    # TODO: consider setting the style only for the current program
    # style_line = f"zstyle ':completion:*:*:{prog_name}:*' menu select"
    # Install zstyle completion config only if the user doesn't have a customization
    if "zstyle" not in zshrc_content:
        zshrc_content += f"\n{style_line}\n"
    zshrc_content = f"{zshrc_content.strip()}\n"
    zshrc_path.write_text(zshrc_content)
    # Install completion under ~/.zfunc/
    path_obj = Path.home() / f".zfunc/_{prog_name}"
    path_obj.parent.mkdir(parents=True, exist_ok=True)
    script_content = get_completion_script(
        prog_name=prog_name, complete_var=complete_var, shell=shell
    )
    path_obj.write_text(script_content)
    return path_obj

On a side note they might have a potential bug there...

[pawamoy]

Completely agree with your comment @j-g00da. I was exactly going to say that the only downside is that users will probably have to use sudo. IMO that's an acceptable tradeoff. If they don't want to use sudo, they can deal with their own configuration by writing the completion script where they prefer.

OK so it looks like Typer modifies .zshrc (in addition to cluttering home with a non-standard .zfunc directory it seems) 🤢 Please let us not do that. EDIT: PDM also suggests writing in ~/.zfunc, but I can't find any official docs about such a directory, so it looks like a convention?

Lets go with the approach you suggested:

• an --install-completions flag detects the shell (and accepts the shell as argument for user control)
• for Bash it runs the example we currently have (using equivalent instructions in Python)
• for Zsh it tries to write in /usr/local/share/zsh/site-functions, and the docs say that one must probably run the command with sudo

[pawamoy]

@bswck @j-g00da do you know if Python has any standard or discussion about distributing shell completions inside packages (source dists or wheels)? Looks like venvs have the opportunity to store a, I don't know, completions folder somewhere. Then tool installers like pipx and uv could expose command to easily locate / load such completions. Each package could distribute completions for all (supported) shells, so that users actually never have to run commands to either install completions or redirect completions. CLIs could then free themselves of --completion-related flags.

I mean pipx managed to handle manpages so surely they could handle completion scripts 🤔

Basher offers such a mechanism for example. Your project defines completion files for Bash, Zsh, whatever, and Basher puts them in a dedicated folder. Users can then easily point their shell at these folders.

[pawamoy]

I opened pypa/pipx#1604 and astral-sh/uv#11354, let see what people think 😊

[johnslavik]

I opened pypa/pipx#1604 and astral-sh/uv#11354, let see what people think 😊

Related pawamoy/git-changelog#66 (comment)

[j-g00da]

EDIT: PDM also suggests writing in ~/.zfunc, but I can't find any official docs about such a directory, so it looks like a convention?

Poetry suggests .zfunc too, but also lists omz directory as a good option.

Still - it's not backed by any standard AFAIK (it's often described as example of custom directory for completions) and because of that, the directory is not by default in fpath and requires adding it manually in .zshrc. Maybe I'm too deep into this, but I still have strong feelings that site-functions would be the best place for that. On the other hand, since it's popular, we can use .zfunc as an example of custom dir instead of a random .duty. Let me know what you think.

I still want to take a look into uv source code later, on my mac it installed completions in site-functions, I want to know if it does the same for linux.

[j-g00da]

• the logic always returns tuples (word, description), and higher-up in the stack we filter out descriptions if the shell doesn't support those

I went with this approach in 592f938

[pawamoy]

Awesome 😍

[pawamoy]

• functions docstrings: good, it's reasonable to keep only the first line, as docstrings' first line should be a complete sentence anyway.
• parameter docstrings: we don't parse them yet anyway (from looking at code below). When we do (using Griffe?), I don't think there's a more robust way than keeping the first line anyway. We don't want to enter natural language processing territory (splitting sentences is hard). Maybe we could recommend to use single-line descriptions for parameters, or at least to have a complete sentence as first line.

[johnslavik]

Let's group behaviors by shells instead of topics.

Instead of fine-grained repetitive interfaces

topic1

• bash
• zsh
• other shells...

topic2

• bash
• zsh
• other shells...

we want modular strategies, like

bash

• topic1
• topic2

zsh

• topic1
• topic2

other shells follow the same pattern...

This approach is known under many names; the I from SOLID, the strategy design pattern, the loose coupling, the high cohesion.

Notice that the current solution doesn't really leverage the use of classes for grouping methods by common state; an interface for a "shell" can be easily described with an abstract class/protocol, distinct implementations gathered within a class-level registry modified via __init_subclass__ of the abstract class/protocol. That would cause future implementations of other shells to be cohesive, discoverable, comprehensive and complete, with a potential of sharing useful state across "topic" methods; something along the lines of:

class Shell(metaclass=abc.ABCMeta):
    name: ClassVar[str]
    implementations: Final[ClassVar[dict[str, Shell]]] = {}

    @abc.abstractmethod
    def parse_completions(self, candidates: Sequence[CompletionCandidateType]) -> str:
        ...

    @abc.abstractmethod
    def install_completions(self, candidates: Sequence[CompletionCandidateType]) -> str:
        ...

    def __init_subclass__(cls) -> None:
        cls.implementations[cls.name] = cls


class Bash(Shell):
    name = "bash"

    def parse_completions(self, candidates: Sequence[CompletionCandidateType]) -> str:
        # implementation for bash

    def install_completions(self) -> str:
        # implementation for bash


class Zsh(Shell):
    name = "zsh"

    def parse_completions(self, candidates: Sequence[CompletionCandidateType]) -> str:
        # implementation for zsh

    def install_completions(self) -> str:
        # implementation for zsh


# other shells follow the same pattern

Implementing a new shell/changing an existing shell implementation will only require changes in one area in the code, instead of a number of them, all over the place, separated by other, unrelated fragments.
Notice how this approach makes the new implementations also portable and easier to plug into the pre-existing runtime—simply implement a strategy for your shell and ship it. The library can find it in the Shell.implementations registry and you can override the existing ones as well, and then maybe suggest a patch to upstream.

I'm up for pair programming on this one, hit me up on Discord if you're interested :)

[j-g00da]

Thanks for feedback @bswck. Generally there are only tests left to do now @pawamoy. I'll do some research and experiment with it today. There are still some things worth pointing out though:

1. The $SHELL check I'm doing when no shell is provided is very error prone. I think we can just use a parameter in docs example to encourage users to provide it, the help text already points out that $SHELL is used when no param provided (or maybe we should just require it?)
2. I already talked with @bswck about it - I used prints to be consistent with the rest of the codebase, but maybe we should create an issue to use a proper logger?
3. Do we want to test for backward compatibility of shell completion scripts? Eg. I had to change completions.bash and make sure the old script will still work when someone upgrades from 1.5.0 and has bash completions installed already (without manually deleting the old script from the system and reinstalling completions.) Regular unit tests wouldn't catch that.

[pawamoy]

Thank you so much for all your hard work on this @j-g00da 🙏

1. We can keep the frail logic, but indeed recommend using the --shell option, and always use it in the docs. I'm curious though, what makes you say it's error-prone?
2. Let's stay with print for now. I've not made up my mind yet on a way to use and configure logging for both logging and terminal display. Happy to hear about your own experience on that matter though!
3. Good question. We definitely do not want completions to break with ugly errors when users update duty. I think that as long as we respect the completion interface of the supported shells, and that we don't make breaking changes in the CLI (which is called by the completion scripts), we should be fine. For now the only backward compatibility precaution you added is us defaulting to bash for --complete, right? Do you see other things that would need such backward compatibility precautions right now or in the near future? In any case, it's easy enough for users to just re-install completions when upgrading duty (at least if/when they get an error). We just have to document this prominently. Also, if/when pipx/uv start both supporting man pages and completion scripts, this point becomes moot, as completions will always be up-to-date. So in the end I don't think we should go out of our way to test this.

[j-g00da]

I'm curious though, what makes you say it's error-prone?

$SHELL is user shell, not current shell - if you for example launch bash from zsh, it will still say /bin/zsh.

[j-g00da]

Do you see other things that would need such backward compatibility precautions right now or in the near future?

There are some, but since we are using symlinks (in --install-completion) - as long as paths to completion scripts are the same, they should update automatically. So the main issue can happen if we move/rename completions.* files.

On the other hand should we really care if completion scripts were installed manually? Sure, we can make test specifically for backward compatibility with 1.5.0, as currently this is the only way to install completions, but with the next update we can promote usage of --install-completion and add warning in docs, that if you choose to install completions manually, you are on your own. I think that's fair and then we can just make a test that checks if completions.* exist for each shell in proper location...

WDYT? @pawamoy @bswck

[pawamoy]

So the main issue can happen if we move/rename completions.* files.

Hmm.

• Symlinks: auto-updates, work as long as path stay the same (both the completion file path within our project, and the installation path on the user machine, which depends on the Python version! reinstall same duty version on different Python version and the symlink will be broken)
• Hard links: depending on how pip updates work, inode could change, separating hard link from source file, meaning no auto-update. Uninstall and reinstalling would isolate the hard link for sure.

🤔

Package updates are likely more frequent that reinstalls on a different Python versions, so lets continue with symlinks 👍

In the end though it looks to me like we should always recommend users to update the completion script upon updating duty. Also, if the file already exists, don't error out, force rewrite it. I'm waiting for the PR to get out of draft for a full review 🙂

[pawamoy]

if you choose to install completions manually, you are on your own

Yes.

[j-g00da]

Just pushed some work in progress (only bash for now) tests to show what I'm trying to do here. Since some of these modify the system, I decided to mark them with custom pytest mark and run them in isolated containers.

Just a proof of concept for now, not optimized, I'm still not entirely sure if it's the way to go.

[j-g00da]

Also a side note - completions always use duties.py in main directory to generate candidates because -d param is not passed to completion script. I can try to fix it when I'm done with tests, but maybe let's make it another PR.

[pawamoy]

IIUC the current solution would create lots of jobs. Also, lots of boilerplate 😕

My suggestion: remove all the boilerplate. Skip relevant tests if env var is unset. Set the env var in CI.

[pawamoy]

Hmmm, yeah, there's parallelism to handle too. I'm thinking about it.

[pawamoy]

We could use pytest-xdist's --dist=loadfile option, to make sure all tests within a given module are executed by the same worker. Maybe there's a way to set this option automatically when our custom env var is set. Each test would clean up after itself. I believe this would prevent any race condition.

[j-g00da]

IIUC the current solution would create lots of jobs.

Yes, that's why I'm not really sure about it. On the other hand this is the only way I see to make these tests deterministic. If we run multiple tests on the same machine, we can create some cleanup mechanism but it won't be failproof and will depend on implementation - and if a change in implementation needs a change in corresponding test, then it's not a good test.

Also there is a problem of parallelism as you said. Even if we run them in one job (per python version and os), we can't parallelize it, so we still need a second matrix for them.

Boilerplate can be reduced for sure, I just wanted to share what I'm trying to do, because this needs to be discussed.

[j-g00da]

If we were to ditch this idea and run them like the other tests, easiest solution would be to just call {Shell}().install_path.unlink(). It would work for Bash and Zsh, not sure about for example PowerShell.

[pawamoy]

and if a change in implementation needs a change in corresponding test, then it's not a good test.

That's a tradeoff I'm willing to accept.

easiest solution would be to just call {Shell}().install_path.unlink()

Sounds very reasonable to me 🙂

[pawamoy]

I mean, yeah, integration tests are nice to have (detecting early when completion install commands do not work as expected anymore), but at the same I don't expect shells to change frequently (and that's an euphemism). So I'm on the side of "lets not waste energy writing and maintaining integration tests that aren't relevant to the project specifically". If it works now, chances are very high that it'll keep working for a long time. So, yes please, no boilerplate, no custom handling of these tests, just clean up / teardown and use --dist=loadfile when the env var is set 🙂 (can be done in the test duty)

[j-g00da]

Sure, I will get back to this, when I have some free time. I think we can also force pytest to run the installation test first (before edge cases like overriding previous installation.)

[pawamoy]

I would not recommend forcing the order of tests as this can hide dependencies between them. Random order forces us to make sure each test is atomic and complete and correctly cleans up after itself.

And of course, no rush, thank you so much for all your work already 😄

[j-g00da]

Good point!