[sqlite] fails to parse SELECT with a function

[walterwanderley]

The query

select sql from sqlite_schema where lower(name) = lower('MyTable')

got error

unexpected error: 1:48: mismatched input '=' expecting :

[kaby76]

I can't reproduce the error. Here is how I tested the grammar/input.

01/12-07:03:21 ~/issues/g4-current/sql/sqlite
$ trgen -t CSharp
C:\msys64\home\Kenne\issues\g4-current\sql\sqlite
CSharp  C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/SQLiteParser.g4 success 0.0524824
CSharp  C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/SQLiteLexer.g4 success 0.0102405
Dependency graph of grammar:
2 vertices, 1 edges
SQLiteParser: SQLiteParser->SQLiteLexer
SQLiteLexer:

Top-level grammars SQLiteLexer SQLiteParser
Start rule parse
Copying template file from CSharp/EarleyAtnParser.cs to ./Generated-CSharp/EarleyAtnParser.cs
Copying template file from CSharp/EpsilonFreeNfa.cs to ./Generated-CSharp/EpsilonFreeNfa.cs
Copying template file from CSharp/MyAbstractPredicateTransition.cs to ./Generated-CSharp/MyAbstractPredicateTransition.cs
Copying template file from CSharp/MyActionTransition.cs to ./Generated-CSharp/MyActionTransition.cs
Copying template file from CSharp/MyATN.cs to ./Generated-CSharp/MyATN.cs
Copying template file from CSharp/MyATNState.cs to ./Generated-CSharp/MyATNState.cs
Copying template file from CSharp/MyAtomTransition.cs to ./Generated-CSharp/MyAtomTransition.cs
Copying template file from CSharp/MyEpsilonTransition.cs to ./Generated-CSharp/MyEpsilonTransition.cs
Copying template file from CSharp/MyNotSetTransition.cs to ./Generated-CSharp/MyNotSetTransition.cs
Copying template file from CSharp/MyParserInterpreter.cs to ./Generated-CSharp/MyParserInterpreter.cs
Copying template file from CSharp/MyPrecedencePredicateTransition.cs to ./Generated-CSharp/MyPrecedencePredicateTransition.cs
Copying template file from CSharp/MyPredicateTransition.cs to ./Generated-CSharp/MyPredicateTransition.cs
Copying template file from CSharp/MyRangeTransition.cs to ./Generated-CSharp/MyRangeTransition.cs
Copying template file from CSharp/MyRuleTransition.cs to ./Generated-CSharp/MyRuleTransition.cs
Copying template file from CSharp/MySetTransition.cs to ./Generated-CSharp/MySetTransition.cs
Copying template file from CSharp/MyStateType.cs to ./Generated-CSharp/MyStateType.cs
Copying template file from CSharp/MyTransition.cs to ./Generated-CSharp/MyTransition.cs
Copying template file from CSharp/MyWildcardTransition.cs to ./Generated-CSharp/MyWildcardTransition.cs
Copying template file from CSharp/Other.csproj to ./Generated-CSharp/Other.csproj
Rendering template file from CSharp/st.BinaryCharStream.cs to ./Generated-CSharp/BinaryCharStream.cs
Rendering template file from CSharp/st.build.ps1 to ./Generated-CSharp/build.ps1
Rendering template file from CSharp/st.build.sh to ./Generated-CSharp/build.sh
Rendering template file from CSharp/st.clean.ps1 to ./Generated-CSharp/clean.ps1
Rendering template file from CSharp/st.clean.sh to ./Generated-CSharp/clean.sh
Rendering template file from CSharp/st.Encodings.cs to ./Generated-CSharp/Encodings.cs
Rendering template file from CSharp/st.ErrorListener.cs to ./Generated-CSharp/ErrorListener.cs
Rendering template file from CSharp/st.makefile to ./Generated-CSharp/makefile
Rendering template file from CSharp/st.MyParser.cs to ./Generated-CSharp/MyParser.cs
Rendering template file from CSharp/st.perf.sh to ./Generated-CSharp/perf.sh
Rendering template file from CSharp/st.ProfilingCommonTokenStream.cs to ./Generated-CSharp/ProfilingCommonTokenStream.cs
Rendering template file from CSharp/st.run.ps1 to ./Generated-CSharp/run.ps1
Rendering template file from CSharp/st.run.sh to ./Generated-CSharp/run.sh
Rendering template file from CSharp/st.test-ambiguity.sh to ./Generated-CSharp/test-ambiguity.sh
Rendering template file from CSharp/st.test-cover.sh to ./Generated-CSharp/test-cover.sh
Rendering template file from CSharp/st.Test.cs to ./Generated-CSharp/Test.cs
Rendering template file from CSharp/st.test.ps1 to ./Generated-CSharp/test.ps1
Rendering template file from CSharp/st.test.sh to ./Generated-CSharp/test.sh
Rendering template file from CSharp/Test.csproj.st to ./Generated-CSharp/Test.csproj
Copying template file from C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/SQLiteParser.g4 to ./Generated-CSharp/SQLiteParser.g4
Copying template file from C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/SQLiteLexer.g4 to ./Generated-CSharp/SQLiteLexer.g4
Copying template file from C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/pom.xml to ./Generated-CSharp/pom.xml
Copying template file from C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/desc.xml to ./Generated-CSharp/desc.xml
01/12-07:03:26 ~/issues/g4-current/sql/sqlite
$ cd Generated-CSharp/
01/12-07:03:28 ~/issues/g4-current/sql/sqlite/Generated-CSharp
$ make
bash build.sh
Restore complete (1.0s)

Build succeeded in 1.6s
Restore complete (0.5s)
  Test net10.0 succeeded (5.5s) → bin\Debug\net10.0\Test.dll

Build succeeded in 6.6s
01/12-07:03:42 ~/issues/g4-current/sql/sqlite/Generated-CSharp
$ e 4725.sql
01/12-07:04:10 ~/issues/g4-current/sql/sqlite/Generated-CSharp
$ ./bin/Debug/net10.0/Test.exe 4725.sql
CSharp 0 4725.sql success 0.0583
Total Time: 0.1410668
01/12-07:04:17 ~/issues/g4-current/sql/sqlite/Generated-CSharp
$ cat 4725.sql
select sql from sqlite_schema where lower(name) = lower('MyTable')
01/12-07:07:11 ~/issues/g4-current/sql/sqlite/Generated-CSharp
$ trparse 4725.sql | trtree
CSharp 0 4725.sql success 0.0610814
parse
├── sql_stmt_list
│   └── sql_stmt
│       └── select_stmt
│           └── select_core
│               ├── SELECT_
│               │   └── "select"
│               ├── result_column
│               │   └── expr
│               │       └── column_name
│               │           └── any_name
│               │               ├── Attribute SPACES Value ' ' chnl:HIDDEN
│               │               └── IDENTIFIER
│               │                   └── "sql"
│               ├── Attribute SPACES Value ' ' chnl:HIDDEN
│               ├── FROM_
│               │   └── "from"
│               ├── table_or_subquery
│               │   └── table_name
│               │       └── any_name
│               │           ├── Attribute SPACES Value ' ' chnl:HIDDEN
│               │           └── IDENTIFIER
│               │               └── "sqlite_schema"
│               ├── Attribute SPACES Value ' ' chnl:HIDDEN
│               ├── WHERE_
│               │   └── "where"
│               └── expr
│                   ├── expr
│                   │   ├── function_name
│                   │   │   └── any_name
│                   │   │       ├── Attribute SPACES Value ' ' chnl:HIDDEN
│                   │   │       └── IDENTIFIER
│                   │   │           └── "lower"
│                   │   ├── OPEN_PAR
│                   │   │   └── "("
│                   │   ├── expr
│                   │   │   └── column_name
│                   │   │       └── any_name
│                   │   │           └── IDENTIFIER
│                   │   │               └── "name"
│                   │   └── CLOSE_PAR
│                   │       └── ")"
│                   ├── Attribute SPACES Value ' ' chnl:HIDDEN
│                   ├── ASSIGN
│                   │   └── "="
│                   └── expr
│                       ├── function_name
│                       │   └── any_name
│                       │       ├── Attribute SPACES Value ' ' chnl:HIDDEN
│                       │       └── IDENTIFIER
│                       │           └── "lower"
│                       ├── OPEN_PAR
│                       │   └── "("
│                       ├── expr
│                       │   └── literal_value
│                       │       └── STRING_LITERAL
│                       │           └── "'MyTable'"
│                       └── CLOSE_PAR
│                           └── ")"
├── Attribute SPACES Value '\r' chnl:HIDDEN
├── Attribute SPACES Value '\n' chnl:HIDDEN
└── EOF
    └── ""


01/12-07:09:26 ~/issues/g4-current/sql/sqlite/Generated-CSharp

[walterwanderley]

Ok, @kaby76 Thanks. I'm using https://github.com/antlr4-go/antlr. I will do some investigation now.

[kaby76]

The Go target works for me.

01/12-15:39:02 ~/issues/g4-current/sql/sqlite
$ trgen -t Go
C:\msys64\home\Kenne\issues\g4-current\sql\sqlite
CSharp  C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/SQLiteParser.g4 success 0.0512338
CSharp  C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/SQLiteLexer.g4 success 0.0105353
Dependency graph of grammar:
2 vertices, 1 edges
SQLiteParser: SQLiteParser->SQLiteLexer
SQLiteLexer:

Top-level grammars SQLiteLexer SQLiteParser
Start rule parse
Rendering template file from Go/st.BinaryCharStream.go to ./Generated-Go/BinaryCharStream.go
Rendering template file from Go/st.build.ps1 to ./Generated-Go/build.ps1
Rendering template file from Go/st.build.sh to ./Generated-Go/build.sh
Rendering template file from Go/st.clean.ps1 to ./Generated-Go/clean.ps1
Rendering template file from Go/st.clean.sh to ./Generated-Go/clean.sh
Rendering template file from Go/st.EncodingFileStream.go to ./Generated-Go/EncodingFileStream.go
Rendering template file from Go/st.go.mod to ./Generated-Go/go.mod
Rendering template file from Go/st.makefile to ./Generated-Go/makefile
Rendering template file from Go/st.perf.sh to ./Generated-Go/perf.sh
Rendering template file from Go/st.run.ps1 to ./Generated-Go/run.ps1
Rendering template file from Go/st.run.sh to ./Generated-Go/run.sh
Rendering template file from Go/st.test.ps1 to ./Generated-Go/test.ps1
Rendering template file from Go/st.test.sh to ./Generated-Go/test.sh
Rendering template file from Go/Test.go.st to ./Generated-Go/Test.go
Copying template file from C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/SQLiteParser.g4 to ./Generated-Go/parser/SQLiteParser.g4
Copying template file from C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/SQLiteLexer.g4 to ./Generated-Go/parser/SQLiteLexer.g4
Copying template file from C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/pom.xml to ./Generated-Go/pom.xml
Copying template file from C:/msys64/home/Kenne/issues/g4-current/sql/sqlite/desc.xml to ./Generated-Go/desc.xml
01/12-15:39:10 ~/issues/g4-current/sql/sqlite
$ cd Generated-Go/
01/12-15:39:13 ~/issues/g4-current/sql/sqlite/Generated-Go
$ make
bash build.sh
go: github.com/antlr4-go/antlr/v4@v4.13.1 requires go >= 1.22; switching to go1.24.11
go: upgraded go 1.20 => 1.22
go: added toolchain go1.24.11
go: added github.com/antlr4-go/antlr/v4 v4.13.1
go: added golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842
go: upgraded go 1.22 => 1.23.0
go: added golang.org/x/text v0.26.0
01/12-15:39:21 ~/issues/g4-current/sql/sqlite/Generated-Go
$ cat run.sh
./Test.exe "$@"
01/12-15:39:28 ~/issues/g4-current/sql/sqlite/Generated-Go
$ bash run.sh ../Generated-CSharp/4725.sql -tree
(parse (sql_stmt_list (sql_stmt (select_stmt (select_core select (result_column (expr (column_name (any_name sql)))) from (table_or_subquery (table_name (any_name sqlite_schema))) where (expr (expr (function_name (any_name lower)) ( (expr (column_name (any_name name))) )) = (expr (function_name (any_name lower)) ( (expr (literal_value 'MyTable')) ))))))) <EOF>)Go 0 ../Generated-CSharp/4725.sql success 0.013
Total Time: 0.016
01/12-15:39:42 ~/issues/g4-current/sql/sqlite/Generated-Go
$ bash run.sh ../Generated-CSharp/4725.sql
Go 0 ../Generated-CSharp/4725.sql success 0.012
Total Time: 0.014
01/12-15:39:49 ~/issues/g4-current/sql/sqlite/Generated-Go
$

The driver app for testing is generated from templates, which are here: https://github.com/antlr/grammars-v4/tree/8e44761887bb5fbf92f8d77682436dbf97bd72ec/_scripts/templates/Go

[walterwanderley]

Got this using the templates (from trgen tool and current main branch):

$ cat 4725.sql 
select sql from sqlite_schema where lower(name) = lower('MyTable')
$ bash run.sh 4725.sql -tree
line 1:48 mismatched input '=' expecting <EOF>
(parse (sql_stmt_list (sql_stmt (select_stmt (select_core select (result_column (expr (expr_recursive (expr_or (expr_and (expr_not (expr_binary (expr_comparison (expr_bitwise (expr_addition (expr_multiplication (expr_string (expr_collate (expr_unary (expr_base (column_name_excluding_string (any_name_excluding_string sql))))))))))))))))) from (join_clause (table_or_subquery (table_name (any_name sqlite_schema)))) where (expr (expr_recursive (function_name (any_name_excluding_raise lower)) ( (expr (expr_recursive (expr_or (expr_and (expr_not (expr_binary (expr_comparison (expr_bitwise (expr_addition (expr_multiplication (expr_string (expr_collate (expr_unary (expr_base (column_name_excluding_string (any_name_excluding_string name)))))))))))))))) ))))))) = lower ( 'MyTable' ))Go 0 4725.sql fail 0.006
Total Time: 0.009
$ bash run.sh 4725.sql
line 1:48 mismatched input '=' expecting <EOF>
Go 0 4725.sql fail 0.004
Total Time: 0.005

Using a parser before this PR #4700 the results are OK!

[walterwanderley]

@kaby76 Thank you for the help here.

Analyzing your parse tree, I see only expr nodes and not expr_recursive nodes, which were introduced in PR 4700. I believe the bug is in the expr parser. I’ll investigate further.

[kaby76]

Yes, it looks like I was testing the grammar before the last change. I did a git reset --hard but didn't do git fetch; git pull.