Unknown language: "Tree-sitter Query" when searching by language

[mkatychev]

Describe the bug

Clicking on Tree-sitter Query in Languages section produces an error stating:

Unknown language: "Tree-sitter Query"
repo:nvim-treesitter/nvim-treesitter language:"Tree-sitter Query"

• This is true for the tsq language alias as well

Expected behaviour

All Tree-sitter Query files should be shown (just like other langauges) when omnisearch is filtered by language.

Related discussion

#7033

Additional notes

PR introducing tsq: #7243

[mkatychev]

@lildude thanks for moving this to discussions, I was hoping to also get some guidance on improving detection of tree-sitter query.

Even though I provided hard and fast regex qualifies/disquialifies there's still a confidence interval and sometimes detection would spill over into a completely unrelated language

Example - an unrelated filestem (.sls) got misclassified as a result of this commit:
I had to revert some r6rs stuff because it spilled over into another language:
Commit addition: 38411c5
GHA that triggered a false classification as a result: https://github.com/github-linguist/linguist/actions/runs/13423748354/job/37502275404#step:5:675

Say I wanted 100% accuracy for the queries found in this repo:
https://github.com/nvim-treesitter/nvim-treesitter/tree/master/queries

Is the best way to iterate involve running github-linguist ./nvim-treesitter/queries?

And is there a way to improve the "weight" of the heuristic regexes here I feel like these prositive/negative matches are hard qualifiers:
https://github.com/mkatychev/linguist/blob/8c36a6ce4b49ebb0a4a6a1abbe358a65f4cf0679/lib/linguist/heuristics.yml#L721-L733

[lildude]

Clicking on Tree-sitter Query in Languages section produces an error stating:

Unknown language: "Tree-sitter Query"
repo:nvim-treesitter/nvim-treesitter language:"Tree-sitter Query"

This is a known limitation with the way GitHub Search works and is precisely why the very last sentence in the troubleshooting doc and similarily on every release PR, like the v9.1.0 release which is the first to include this language, is:

Note: New languages will not appear in GitHub's search results for some time after the pull request has been merged and the new Linguist release deployed to GitHub.com. This is because GitHub's search uses a separate internal library for language detection which tends to lag behind Linguist by a few weeks to months.

I linked to this document in the approval message of your PR and you even acknowledged it with 👍 😁

I was hoping to also get some guidance on improving detection of tree-sitter query.

One thing to keep in mind is your changes will only apply to repos that have been pushed to after your changes have been deployed. We do not go back and re-analyse all repos as this would be insanely resource intensive with little benefit.

Say I wanted 100% accuracy for the queries found in this repo: https://github.com/nvim-treesitter/nvim-treesitter/tree/master/queries

Is the best way to iterate involve running github-linguist ./nvim-treesitter/queries?

Sort of. That would only use the installed version of Linguist. In dev, bundle exec bin/github-linguist ... will allow you to analyze repos as you change the code.

And is there a way to improve the "weight" of the heuristic regexes here

The only way to sort of add some weight is to re-order them in the heuristics (I advise against that given how much more complex your regexes are), but that still wouldn't help if your files don't match any of the regexes which appears to be the case for some of your files right now. Things are falling through to the classifier which will make a guess based on the samples, hence we ask for representative real-world examples.

[mkatychev]

Thanks for the writeup, apologies about filing another discussion about the Unknown language section 🙇 , the other parts (misclassifications) of the writeup are still super relevant so your responses are still quite valuable.

The only way to sort of add some weight is to re-order them in the heuristics (I advise against that given how much more complex your regexes are), but that still wouldn't help if your files don't match any of the regexes which appears to be the case for some of your files right now. Things are falling through to the classifier which will make a guess based on the samples, hence we ask for representative real-world examples.

So is your suggestion here to simply add more samples to scheme and tsq? I'm worried about triggering domino effects such as the .sls false positive, is the best way to deal with that involve also adding SaltStack files?

[lildude]

Almost. The most reliable solution is to improve the regexes but if this isn't possible adding more samples will help the classifier. Keep in mind, we're not trying to be 100% accurate all the time (it would be almost impossible), that's why we have overrides.

The easiest solution for you is to accept the limitations of your implementation as good enough and use an override, but feel free to keep fighting if you want 😁