Update to Pug Customized Grammar

.github/workflows/test.yml

@@ -9,11 +9,14 @@ on:
 jobs:
   test:
     runs-on: ubuntu-latest
-    name: go-tree-sitter
+    strategy:
+      matrix:
+        go: ["1.23", "1.22", "1.21", "1.20"]
+    name: Test on ${{ matrix.go }}
     steps:
-      - name: Check out source code
-        uses: actions/[email protected]
-      - uses: actions/setup-go@v3.3.1
+      - uses: actions/checkout@v4
+      - name: Setup go
+        uses: actions/setup-go@v5
         with:
           go-version: 1.19
       - name: Download dependencies

.gitignore

@@ -1 +1,57 @@
-vendor
+.idea
+
+# Created by https://www.toptal.com/developers/gitignore/api/visualstudiocode,macos
+# Edit at https://www.toptal.com/developers/gitignore?templates=visualstudiocode,macos
+
+### macOS ###
+# General
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Icon must end with two \r
+Icon
+
+# Thumbnails
+._*
+
+# Files that might appear in the root of a volume
+.DocumentRevisions-V100
+.fseventsd
+.Spotlight-V100
+.TemporaryItems
+.Trashes
+.VolumeIcon.icns
+.com.apple.timemachine.donotpresent
+
+# Directories potentially created on remote AFP share
+.AppleDB
+.AppleDesktop
+Network Trash Folder
+Temporary Items
+.apdisk
+
+### macOS Patch ###
+# iCloud generated files
+*.icloud
+
+### VisualStudioCode ###
+.vscode/*
+!.vscode/settings.json
+!.vscode/tasks.json
+!.vscode/launch.json
+!.vscode/extensions.json
+!.vscode/*.code-snippets
+
+# Local History for Visual Studio Code
+.history/
+
+# Built Visual Studio Code Extensions
+*.vsix
+
+### VisualStudioCode Patch ###
+# Ignore all local history of files
+.history
+.ionide
+
+# End of https://www.toptal.com/developers/gitignore/api/visualstudiocode,macos

README.md

@@ -5,14 +5,98 @@
 
 Golang bindings for [tree-sitter](https://github.com/tree-sitter/tree-sitter)
 
+## How to Update Pug Grammar
+
+### 1. Update Automation Grammars Configuration File
+Update the grammar to the `codepen/go-tree-sitter/_automation/grammars.json` file. The format is:
+
+```json
+{
+"language": "pug",
+"url": "https://github.com/codepen/tree-sitter-pug",
+"files": ["parser.c", "scanner.cc"],
+"reference": "master",
+"revision": "757e95a5fbf26058e38f9beb1fd2f05c140410a7"
+},
+```
+
+### 2. Run Script to Update Grammar
+
+This will pull the latest generated C parser files from the GitHub repo.
+
+```bash
+go run _automation/main.go update pug -force
+```
+
+### 3. Add Go Binding
+
+Manually add a file called `binding.go` to the `codepen/go-tree-sitter/pug` directory. This file should contain the following code, specific to your grammar:
+
+```go
+package pug
+
+//#include "parser.h"
+//TSLanguage *tree_sitter_pug();
+import "C"
+
+import (
+	"unsafe"
+
+	sitter "github.com/codepen/go-tree-sitter"
+)
+
+func GetLanguage() *sitter.Language {
+	ptr := unsafe.Pointer(C.tree_sitter_pug())
+	return sitter.NewLanguage(ptr)
+}
+```
+
+This logic adds a `GetLanguage` method that allows you to call the underlying C code from Go.
+
+Be sure to set the correct language name in two places in this file:
+Line 4: `//TSLanguage *tree_sitter_<language>();`
+Line 14: `ptr := unsafe.Pointer(C.tree_sitter_<language>())`
+
+### 4. Add Binding Test
+Test the syntax tree your newly added parser generates by creating a test file in the `codepen/go-tree-sitter/<language>` directory. This file should contain the following code, specific to your grammar:
+
+```go
+package pug_test
+
+import (
+	"context"
+	"testing"
+
+	sitter "github.com/codepen/go-tree-sitter"
+	"github.com/codepen/go-tree-sitter/pug"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestGrammar(t *testing.T) {
+	assert := assert.New(t)
+
+	n, err := sitter.ParseCtx(context.Background(), []byte(`extends layout.pug`), pug.GetLanguage())
+	assert.NoError(err)
+	assert.Equal("(source_file (extends_statement (path)))", n.String())
+}
+```
+
+Run the test from the root of the repo with the following command:
+```bash
+go test -v pug/binding_test.go
+```
+
 ## Usage
 
 Create a parser with a grammar:
 
 ```go
 import (
-	sitter "github.com/codepen/go-tree-sitter"
-	"github.com/codepen/go-tree-sitter/javascript"
+	"context"
+	"fmt"
+
+	sitter "github.com/smacker/go-tree-sitter"
+	"github.com/smacker/go-tree-sitter/javascript"
 )
 
 parser := sitter.NewParser()
@@ -23,7 +107,7 @@ Parse some code:
 
 ```go
 sourceCode := []byte("let a = 1")
-tree := parser.Parse(nil, sourceCode)
+tree, _ := parser.ParseCtx(context.Background(), nil, sourceCode)
 ```
 
 Inspect the syntax tree:
@@ -51,6 +135,7 @@ Known external grammars:
 
 - [Salesforce grammars](https://github.com/aheber/tree-sitter-sfapex) - including Apex, SOQL, and SOSL languages.
 - [Ruby](https://github.com/shagabutdinov/go-tree-sitter-ruby) - Deprecated, grammar is provided by main repo instead
+- [Go Template](https://github.com/mrjosh/helm-ls/tree/master/internal/tree-sitter/gotemplate) - Used for helm
 
 ### Editing
 
@@ -149,7 +234,7 @@ Update vendor files:
 
 - open `_automation/grammars.json`
 - modify `reference` (for tagged grammars) or `revision` (for grammars from a branch)
-- run `c <grammar-name>`
+- run `go run _automation/main.go update <grammar-name>`
 
 It is also possible to update all grammars in one go using
 

_automation/README.md

@@ -0,0 +1,62 @@
+# Grammar Version Checker
+
+This Go program automates the process of checking and updating grammar versions for various programming languages. It targets specific grammars defined in a JSON configuration file and supports operations like checking for updates and performing updates.
+
+## Features
+
+- **Check for Updates**: Scan through defined grammars and check if there are new versions available.
+- **Update Grammar**: Update a specific grammar to its latest version.
+- **Update All**: Batch update all grammars to their latest versions.
+
+## Prerequisites
+
+- Go installed on your machine.
+- Git installed and accessible from your command line.
+
+## Configuration
+
+The program relies on a JSON file (`_automation/grammars.json`) for the list of grammars to manage. Each grammar should specify its `language`, `url`, `files`, `reference` (tag or branch), and `revision` (commit hash).
+
+Example structure of `grammars.json`:
+
+```json
+[
+  {
+    "language": "golang",
+    "url": "https://github.com/tree-sitter/tree-sitter-go",
+    "files": [
+      "parser.c"
+    ],
+    "reference": "master",
+    "revision": "64457ea6b73ef5422ed1687178d4545c3e91334a",
+    "updatedBasedOn": "commit", //OPTIONAL: Use "tag" (instead of "commit") to restrict updates to the latest tag and not latest commit
+  }
+]
+```
+
+`updatedBasedOn` is an OPTIONAL property. If not provided, updating from the latest commit will be presumed.
+
+## Usage 
+
+### To Check for Updates
+
+```bash
+go run ./_automation/main.go check-updates
+```
+
+### To update a specific grammar
+```bash
+go run ./_automation/main.go update <language> [--force]
+```
+
+### To update all grammars
+```bash
+go run ./_automation/main.go update-all
+```
+
+## Test After Updating
+Once you've updated a grammar, it is important to test that the binding tests still pass (`binding_test.go`). Then run:
+
+```bash
+go test -v ./...
+```