feat(python-lsp)!: replace the old pylsp with zuban and ruff

[Cloud0310]

Note

See previous PR #1489 and #1172 for previous details.
Due to pyrefly is still not reached beta yet, we choose to wait its first beta release, as it doesn't have any comlicated refactor capibility for now.
The PR now is converted into a draft now, the information about pyrefly could retrived here and facebook/pyrefly#344. And this would likely to be a BREAKING CHANGE once merged.

Summary

As we are searching for a better Python LSP, and there has been updates on pyrefly as a better choice for this, which is now mature and solid, I think its time to move on for pyrefly as the default python lsp.

Details

1. deleted: previous dependencies for pylsp, remove extra config
2. added: pyrefly (main python lsp) and ruff (format and lint) and keep the default behavior backward compatible.

[charliie-dev]

does pyrefly's and ruff's default config gd enough?
otherwise we might need to add some language server config in lua/modules/configs/completion/servers.(check other language server's config as reference.

[Cloud0310]

does pyrefly's and ruff's default config gd enough? otherwise we might need to add some language server config in lua/modules/configs/completion/servers.(check other language server's config as reference.

Thanks for your kindly remind, I check up the previous config for pylsp, and pyrefly config docs, and ruff config doc.
I commited agian to fully remove the pylsp.lua config, and align the behavior of ruff with previous config.
If there's something we can config further for user experience, please let me know.

[Cloud0310]

Another thing is that, I suggest we should ask other maintainers for their opinion on this PR about whether its time to switch and what can we improve about lsp/linter/formatter config.

[ayamir]

Maybe I can't review it now.

1. Recently I'm busy on working for my job and have no enough spare time to review it. ByteDance you know, byte and heart, only one can dance.
2. It’s been ages since I touched python, my use case is not confident.

[Cloud0310]

Now there's another option for Python LSP, Zuban claims to support more lsp features, and look promising. We should take one that into consideration as well.

[charliie-dev]

Announcing Pyrefly Beta

[Cloud0310]

Announcing Pyrefly Beta

I saw it as well, but it's kinda late (UTC+8 1a.m.) now, tomorrow I will try to test it.

[Cloud0310]

I guess for my use case for scripting in python with types, the mentioned refactor feature is working properly and all other things are working as intended. So I guess its ready for review and test.

Another thing is the new zuban lsp, should we consider it instead of pyrefly?
There's a feature comparison table here by zuban (kinda old for current pyrefly version 0.4x), and it claims to be faster then pyrefly and having better compatibility.

ref: @misumisumi @charliie-dev

[charliie-dev]

As zuban's readme stated:

Zuban is 20–200× faster than Mypy, while using roughly half the memory and CPU compared to Ty and Pyrefly.

Mason support:

yeah, sure. why not lol

[Cloud0310]

Found this amazing test set from python/typing, I will try to re-test, and then I guess we can choose a base on this test.

[Cloud0310]

results.html
I guess as the test demonstrates, zuban is better?

[charliie-dev]

results.html I guess as the test demonstrates, zuban is better?

can we add ty to the list?

[Cloud0310]

results.html I guess as the test demonstrates, zuban is better?

can we add ty to the list?

Guess nothing is working on that one. Even basic array type infer.

[charliie-dev]

then +1 for zuban, lets gooo

[misumisumi]

I also think it would be better to choose zuban now.

P.S.
There are some concerns, such as zuban being a personal project of the creator of jedi, so there are concerns about sustainability, and pyrefly getting more attention (from repo star).
But if pyrefly become better tool, we just switch to it.

[Cloud0310]

I also think it would be better to choose zuban now.

P.S. There are some concerns, such as zuban being a personal project of the creator of jedi, so there are concerns about sustainability, and pyrefly getting more attention (from repo star). But if pyrefly become better tool, we just switch to it.

As for the personal project problem, the current solution python-lsp is based on jedi as well, so I guess the community solution is absolutely fine.

Personally, I would like the solution of using zuban for now. In the long term, if there's better solution with python lsp, I guess it wouldn't be a problem to switch to it.

P.S. I happen to know one of the contributors of zuban and pyrefly, @asukaminato0721. The situation about all mentioned lsp inplementations is as this blog (Simplified Chinese) suggests.

[Cloud0310]

So maybe we could merge this? @charliie-dev @misumisumi

[charliie-dev]

just tested on my private python repo, seems right lol
LGTM

[asukaminato0721]

imo, both are excellent choices.

Zuban did very well in the standard.

Pyrefly just released a beta version and will be the CI tool for PyTorch.

d = {"2": 2}
if d.get("1") is None:
    raise
x = d.get("1")

Now only Pyrefly can infer that x is int. facebook/pyrefly@70b058e

[misumisumi]

LGTM

[ayamir]

LGTM 💯