C++ lexer: ");" in function params breaks the lexer, rest of file is lexed as string

[KasperBasalte]

Name of the lexer
C++ (Rouge::Lexers::Cpp).

Code sample

Foo f(");");
int after = 1;  // This line and everything below is lexed as a string.

I can reproduce it as follows:

docker run --rm -i ruby:3.3 sh -c '
    gem install rouge -q >/dev/null
    ruby -e "
      require \"rouge\"
      Rouge::Lexers::Cpp.new.lex(STDIN.read).each { |t, v| puts \"#{t.qualname}\t#{v.inspect}\" }
    "' << 'EOF'
Foo f(");");
int after = 1;
EOF

Output:

Name	"Foo"
Text	" "
Name.Function	"f"
Punctuation	"("
Literal.String	"\")"
Punctuation	";"
Literal.String	"\");"
Error	"\n"
Literal.String	"int after = 1;"
Error	"\n"

Additional context
Seems to be caused by this rule in state :root of lib/rouge/lexers/c.rb:

rule %r(
  ([\w*\s]+?[\s*]) # return arguments
  (#{id})          # function name
  (\s*\([^;]*?\))  # signature
  (#{ws}?)({|;)    # open brace or semicolon
)mx

It matches up to the first ) followed by ;, even when that ); is inside a string.

My suggestion is to make it string aware:

rule %r(
  ([\w*\s]+?[\s*]) # return arguments
  (#{id})          # function name
  (\s*\((?:"(?:[^"\\\n]|\\.)*"|[^;"'])*?\))  # signature
  (#{ws}?)({|;)    # open brace or semicolon
)mx

Note that this doesn't handle r-strings. I also noticed that multicharacters are not supported by the lexer. If they were ');' would also trip the lexer.

[jneen]

Confirmed, live example: https://rouge.jneen.ca/v5.1.0/cpp/aW50IGYoIik7Iik7CmludCBhZnRlciA9IDE7Cg

[jneen]

These giant pygments-style regexes for parsing function declarations and the like are misguided IMO (even if I wrote them myself...). If it's possible, we should use states to accomplish the same thing, and lookahead regexes if we need to disambiguate complex cases.