Break in compatibility with jupyterlab-lsp + python-lsp-ruff since 0.0.285

[felix-cw]

I'm not 100% sure this is the right project to raise this, since the problem comes from interaction with python-lsp-ruff and juptyerlab-lsp. I only raised it here since the issue arose from a change in ruff.

Since 0.0.285, the jupyterlab-lsp plugin now gives the following diagnostic message:

SyntaxError: Expected a Jupyter Notebook (.ipynb), which must be internally stored as JSON, but found a Python source file: expected value at line 1 column 1

I believe this is because jupyterlab-lsp extracts the python code from the notebook cells, then python-lsp-ruff passes it to ruff via stdin, and specifies --stdin-filename. Since #6628, this makes ruff expect ipynb JSON, rather than python code, which causes the error.

[dhruvmanila]

Hey, thanks for opening the issue!

First of all, your observation is spot on in that the --stdin-filename is used by python-lsp-ruff¹ and that JupyterLab is probably sending the source code as Python (instead of JSON) to the server² (They have a concept of Virtual Document). Here, the server is python-lsp-ruff which uses the Ruff executable.

You can see the URI for the document which has an extension of .ipynb:

A few solutions which come to my mind:

• python-lsp-ruff would always update the document URI to use .py extension but that would mean any per-file-ignores using the .ipynb extension won't work
• A new CLI flag to specify the source type explicitly (similar to the --parser flag in Prettier):

   --parser <flow|babel|babel-flow|babel-ts|typescript|acorn|espree|meriyah|css|less|scss|json|json5|json-stringify|graphql|markdown|mdx|vue|yaml|glimmer|html|angular>
                            Which parser to use.
  

Feel free to share any other suggestions :)

Footnotes

1. https://github.com/python-lsp/python-lsp-ruff/blob/f6707dfab9c39cbab23b8bc89646341164435afc/pylsp_ruff/plugin.py#L503-L505 ↩

2. https://jupyterlab-lsp.readthedocs.io/en/latest/Architecture.html#positioning-system ↩

[felix-cw]

Hi all. Sorry for posting on an old issue. For me, I would prefer option 2: a new CLI flag, so that options such as per-file-ignores can work both in a jupyter lsp context as well as when running ruff on notebooks in command lines, pre-commit etc.

An example of a per-file-ignore that can definitely be scoped to ipynb files is B018.

[zanieb]

Could our notebook parser (or a step upstream of it) just be updated to wrap the content in a single cell if it's not already? I don't see a parser option as a user-facing setting.

[felix-cw]

I think that could work. My concern was that since ruff output includes cell numbers, it would interfere with the part of jupyterlab-lsp which maps line numbers in the linter output back to the original cells.
However, it looks like when --format=json it just gives line numbers so should be fine.

[charliermarsh]

I think the exclusion of cell numbers is a bug in the JSON format is a bug (see, e.g., #7664). Broadly, I'm worried that treating Python source as Jupyter could cause other problems, some unanticipated (e.g., if you accepted the code via stdin, we'd then output notebook JSON when formatting or autofixing).

Without doing deeper research, my current preferences would be something like:

1. Figure out how to get jupyterlab-lsp to send us the full notebook. This would the most reliable and it would avoid unnecessarily converting from Jupyter to Python back to Jupyter.
2. Add a dedicated flag to determine the source type (like --parser). I think this is reasonable, even if only intended for LSPs and other programmatic clients.
3. If we parse a notebook, and it's not JSON, fall back to parsing as Python? This is sort of similar to Zanie's suggestion but internally we'd treat the whole thing as if it were Python rather than trying to convert it to Jupyter. This feels like a structural error though, I don't love it.

[zanieb]

Agree that 1. would be preferable. 2. is fine with me if we hide it from the CLI help menu until there are more user-focused use-cases.

[felix-cw]

Thanks everyone! I thought that it might be too complicated to get python-lsp-ruff/jupyterlab-lsp to use linters that work on notebooks directly, so I made a PR implementing the --parser option. Thanks for the opportunity to contribute to such a useful and amazing project!

[MichaReiser]

My concern with 2 is that, compared to Prettier, we don't support different parsers. I prefer a CLI flag that is more high level and instead allows overriding the file type (mime type remapping), which can be useful in other use cases other than the one mentioned here.

[charliermarsh]

Can you spell out in a bit more detail what this CLI flag would look like and/or how it would differ from the above?

[dhruvmanila]

overriding the file type (mime type remapping)

Wouldn't that be in conflict with the inferred filetype from the filename's extension provided via --stdin-filename? Or, would that mean that the inference will be avoided?

FWIW, in ruff-dev, I added an option --jupyter which would just mean that the content of the Python file is from a Jupyter Notebook. This helped in development where I didn't need to run a Jupyter server for testing purposes. I think this is a similar case where we want to tell Ruff that the content of this file is from a different filetype.

[MichaReiser]

Wouldn't that be in conflict with the inferred filetype from the filename's extension provided via --stdin-filename? Or, would that mean that the inference will be avoided?

What I have in mind is that you can map extensions to other files. I don't have a good python example but JSON supports different standards and a user may decide that all their .json files are JSON5. They could then configure *.json = JSON5. I guess it's similar to parser except that we define the language rather than the parser (because we don't support different Python parser).

We could expose the same option on the CLI and it could then be used as --extension ipnby=python or similar.

[charliermarsh]

Interesting, so --extension ipynb=python could be repeated?