Detect wether query just filters rows or is more complex with sqlglot

[anuunchin]

Description

This is a research PR with an implementation of a utility function that analyzes a select query and detects whether it's complex or not.

Related Issues

• Resolves Detect wether query just filters rows or is more complex with sqlglot #2557

Additional Notes:

• A union all statement is parsed as a Union expression rather than a select one.
• Queries with CTEs are flagged as complex no matter the underlying structure.

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	48640ad
🔍 Latest deploy log	https://app.netlify.com/projects/dlt-hub-docs/deploys/683ec3dc864c9900080bba3e
😎 Deploy Preview	https://deploy-preview-2619--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[anuunchin]

This will probably need to be TTableSchemaColumns 👀

[sh-rp]

I'm not quite sure, maybe this will be an SQLGlotSchema, but let's keep a list of known columns for now and we can later change it when we use this code in the transformations work.

[anuunchin]

Another thing I tried here is using sqlglot's diff function which outputs what needs to be done to create a given query from the base query. In our case, the base query could be SELECT * FROM my_table, or SELECT <all_columns explicitly> FROM my_table and we can check if the required actions involve removing or adding a column and some other allowed actions - but this I realized might not be as straightforward as simply going through the parsed query

[zilto]

I remembered reading in the SQGlot AST primer that find(), find_all() and walk() are not always reliable

Here's a common pitfall of the walk methods:

ast.find_all(exp.Table)

At first glance, this seems like a great way to find all tables in a query. However, Table instances are not always tables in your database. Here's an example where this fails:

ast = parse_one("""
WITH x AS (
  SELECT a FROM y
)
SELECT a FROM x
""")

# This is NOT a good way to find all tables in the query!
for table in ast.find_all(exp.Table):
    print(table)

# x  -- this is a common table expression, NOT an actual table
# y

The post follows with how to use the Scope object to traverse the AST. At each "node" of the scope (think of traversing a graph), you can directly query the .stars, .tables, etc. properties to retrieve what's in the scope.

[anuunchin]

@zilto Do we take into account only physical tables? I just thought if the query already has multiple table expressions, no matter whether each of them corresponds to an actual table or not, we still consider the query complex - because it's complex in terms of syntax if not in terms of semantics 👀

[anuunchin]

Added the sqlglot example just in case: 🫡

    {
        "query": "WITH temp_table AS (SELECT * FROM my_table) SELECT * FROM temp_table;",
        "complex": True,
        "description": "cte + two table refs",
    },
    {
        "query": "WITH x AS (SELECT a FROM y) SELECT a FROM x",
        "complex": True,
        "description": "cte + two table refs + sqlglot example",
    },

[zilto]

I meant to say that the resource I linked proposes an approach that could simplify our code. Manipulating instances of the Scope class might be more convenient than manipulating raw Expression objects. Scope has many useful properties

Probably something like this:

from sqlglot.optimizer.scope import build_scope

ast = parse_one("""
WITH x AS (
  SELECT a FROM y
)
SELECT a FROM x
""")

root = build_scope(ast)
# `.traverse()` recursively walks the graph
for scope in root.traverse():
   # 1- Scope<SELECT a FROM y>
   # 2- Scope<WITH x AS (SELECT a FROM y) SELECT a FROM x>

    if scope.is_cte:
       return True
       
    if scope.is_derived_table:
       return True
       
    if scope.is_subquery:
       return True
       
    if scope.is_union:
       return True
       
    if scope.pivots:
       return True
       
    if scope.joint_hints:
       return True

   # and more

[sh-rp]

very nice, thanks :)

[sh-rp]

This is very good. I think you can keep the scope approach and fix the few things I have highlighted.

[sh-rp]

I'm not quite sure, maybe this will be an SQLGlotSchema, but let's keep a list of known columns for now and we can later change it when we use this code in the transformations work.