[Bug]: The lexer breaks if the placeholder is within a comment or a quoted literal

[nogitting]

What happened?

Hello & kudos.

The error occurs with this (or a similar) configuration:

• templater = placeholder
• param_style = dollar (but not only)
• -- This is ${my_param} or WHERE full_name = '${some_name}' (in the given SQL file)

Here is what it outputs:

thread 'main' panicked at crates/lib-core/src/parser/lexer.rs:787:25:
not implemented
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

SQLFluff parses it correctly though.

One hacky workaround is to add a space before the parameter name. So, -- This is ${ my_param} and WHERE full_name = '${ some_name}' work, regardless of the dialect.

Version

Latest (0.25.10)

Checked in Playground

• I have checked this bug in the playground.

[benfdking]

Hey, sorry you are seeing that. Could you paste an exact config and SQL for me to double check?

[nogitting]

Configuration:

[sqruff]
dialect = databricks
templater = placeholder

[sqruff:templater:placeholder]
param_style = dollar

SQL code:

-- debugging, col2_value = ${col2_value}

SELECT col1 from my_table 
WHERE col2 = "${col2_value}";

[nogitting]

I see you are taking care of it in PR#1395. 👍
I just want to clarify that:

• It is not specific to the dollar style, i.e. other styles fail too.
• It is not specific to quoted literals, placeholders in comments make it break similarly.

[benfdking]

Should be deployed, let me know if it works for you.

[nogitting]

Getting better, but now if you have both a comment and a quoted literal that include a placeholder (not necessarily the same param), it will still panic:

thread 'main' panicked at crates/lib-core/src/parser/lexer.rs:808:13:
Unable to process slice: TemplatedFileSlice { slice_type: "literal", source_slice: 40..83, templated_slice: 37..80 }

(Same configuration and SQL code.)