Skip to content

Add option to automatically try all parse modes#228

Open
seanlinsley wants to merge 1 commit into15-latestfrom
parse-all
Open

Add option to automatically try all parse modes#228
seanlinsley wants to merge 1 commit into15-latestfrom
parse-all

Conversation

@seanlinsley
Copy link
Member

@seanlinsley seanlinsley commented Dec 21, 2023

This adds a PG_QUERY_PARSE_ALL parse mode to automatically try every parse mode, which is helpful if you're parsing a query and don't know ahead of time which parse mode is valid for the query.

Additionally, this adds pg_query_normalize_opts which was missed by #216.

Note that the parse tree and fingerprints are different for some queries because multiple parse modes are able to parse the query. The NEW.name and NEW.author.first_name parse trees are actually incorrect so we may need to change the order in which the parse modes are tried.

# parse_opts
INVALID result for "v3.c1 := 4" with 6 mode
expected: {"version":150001,"stmts":[{"stmt":{"PLAssignStmt":{"name":"v3","indirection":[{"String":{"sval":"c1"}}],"nnames":2,"val":{"targetList":[{"ResTarget":{"val":{"A_Const":{"ival":{"ival":4},"location":9}},"location":9}}],"limitOption":"LIMIT_OPTION_DEFAULT","op":"SETOP_NONE"}}}}]}
  actual: {"version":150001,"stmts":[{"stmt":{"PLAssignStmt":{"name":"v3","indirection":[{"String":{"sval":"c1"}}],"nnames":1,"val":{"targetList":[{"ResTarget":{"val":{"A_Const":{"ival":{"ival":4},"location":9}},"location":9}}],"limitOption":"LIMIT_OPTION_DEFAULT","op":"SETOP_NONE"}}}}]}
INVALID result for "NEW.name = upper(cleanString(NEW.name))" with 6 mode
expected: {"version":150001,"stmts":[{"stmt":{"PLAssignStmt":{"name":"new","indirection":[{"String":{"sval":"name"}}],"nnames":2,"val":{"targetList":[{"ResTarget":{"val":{"FuncCall":{"funcname":[{"String":{"sval":"upper"}}],"args":[{"FuncCall":{"funcname":[{"String":{"sval":"cleanstring"}}],"args":[{"ColumnRef":{"fields":[{"String":{"sval":"new"}},{"String":{"sval":"name"}}],"location":29}}],"funcformat":"COERCE_EXPLICIT_CALL","location":17}}],"funcformat":"COERCE_EXPLICIT_CALL","location":11}},"location":11}}],"limitOption":"LIMIT_OPTION_DEFAULT","op":"SETOP_NONE"}}}}]}
  actual: {"version":150001,"stmts":[{"stmt":{"SelectStmt":{"targetList":[{"ResTarget":{"val":{"A_Expr":{"kind":"AEXPR_OP","name":[{"String":{"sval":"="}}],"lexpr":{"ColumnRef":{"fields":[{"String":{"sval":"new"}},{"String":{"sval":"name"}}]}},"rexpr":{"FuncCall":{"funcname":[{"String":{"sval":"upper"}}],"args":[{"FuncCall":{"funcname":[{"String":{"sval":"cleanstring"}}],"args":[{"ColumnRef":{"fields":[{"String":{"sval":"new"}},{"String":{"sval":"name"}}],"location":29}}],"funcformat":"COERCE_EXPLICIT_CALL","location":17}}],"funcformat":"COERCE_EXPLICIT_CALL","location":11}},"location":9}}}}],"limitOption":"LIMIT_OPTION_DEFAULT","op":"SETOP_NONE"}}}]}
INVALID result for "NEW.author.first_name = upper(cleanString(NEW.author.first_name))" with 6 mode
expected: {"version":150001,"stmts":[{"stmt":{"PLAssignStmt":{"name":"new","indirection":[{"String":{"sval":"author"}},{"String":{"sval":"first_name"}}],"nnames":3,"val":{"targetList":[{"ResTarget":{"val":{"FuncCall":{"funcname":[{"String":{"sval":"upper"}}],"args":[{"FuncCall":{"funcname":[{"String":{"sval":"cleanstring"}}],"args":[{"ColumnRef":{"fields":[{"String":{"sval":"new"}},{"String":{"sval":"author"}},{"String":{"sval":"first_name"}}],"location":42}}],"funcformat":"COERCE_EXPLICIT_CALL","location":30}}],"funcformat":"COERCE_EXPLICIT_CALL","location":24}},"location":24}}],"limitOption":"LIMIT_OPTION_DEFAULT","op":"SETOP_NONE"}}}}]}
  actual: {"version":150001,"stmts":[{"stmt":{"SelectStmt":{"targetList":[{"ResTarget":{"val":{"A_Expr":{"kind":"AEXPR_OP","name":[{"String":{"sval":"="}}],"lexpr":{"ColumnRef":{"fields":[{"String":{"sval":"new"}},{"String":{"sval":"author"}},{"String":{"sval":"first_name"}}]}},"rexpr":{"FuncCall":{"funcname":[{"String":{"sval":"upper"}}],"args":[{"FuncCall":{"funcname":[{"String":{"sval":"cleanstring"}}],"args":[{"ColumnRef":{"fields":[{"String":{"sval":"new"}},{"String":{"sval":"author"}},{"String":{"sval":"first_name"}}],"location":42}}],"funcformat":"COERCE_EXPLICIT_CALL","location":30}}],"funcformat":"COERCE_EXPLICIT_CALL","location":24}},"location":22}}}}],"limitOption":"LIMIT_OPTION_DEFAULT","op":"SETOP_NONE"}}}]}

# fingerprint_opts
INVALID result for "v3.c1 := 4" with 6 mode
expected: "a8c86658ce26a653"
actual: "7f65202632663ba0"
actual tokens: INVALID result for "NEW.name = upper(cleanString(NEW.name))" with 6 mode
expected: "bb52450e4d46f7a1"
actual: "184cff1a84037010"
actual tokens: INVALID result for "NEW.author.first_name = upper(cleanString(NEW.author.first_name))" with 6 mode
expected: "a148e3f78b53c252"
actual: "b7de0b0d4f636a16"

Todo:

  • if possible, fix the incorrect parse trees
  • update normalize tests to include literal values in every example to ensure they become bind params

Per 1/9 meeting:

  • verify if pg_stat_statements includes these parse mode variants
  • verify if auto_explain gives us enough information to know exactly which parse mode variant is needed

@seanlinsley seanlinsley requested a review from a team December 21, 2023 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments