perf(go): optimize GetObjects by reading SHOW results directly#105
Open
perf(go): optimize GetObjects by reading SHOW results directly#105
Conversation
Reduce Snowflake round-trips for GetObjects at ObjectDepthTables and ObjectDepthDBSchemas when catalog is specific (non-wildcard). Instead of executing SHOW TERSE commands only to collect queryIDs then running a complex SQL template that re-reads those results via RESULT_SCAN joins, the new direct path reads SHOW rows via driver.Rows and builds the GetObjectsInfo tree in Go. Query count reductions: - Specific catalog + schema (Tables depth): 4 -> 1 - Specific catalog + wildcard schema (Tables depth): 4 -> 2 - Specific catalog (DBSchemas depth): 3 -> 1 - Wildcard catalog: unchanged (falls through to existing path)
lidavidm
reviewed
Apr 2, 2026
| @@ -0,0 +1,382 @@ | |||
| // Copyright (c) 2025 ADBC Drivers Contributors | |||
Contributor
There was a problem hiding this comment.
Suggested change
| // Copyright (c) 2025 ADBC Drivers Contributors | |
| // Copyright (c) 2026 ADBC Drivers Contributors |
Comment on lines
+338
to
+347
| cols := rows.Columns() | ||
| nameIdx, dbIdx := -1, -1 | ||
| for i, col := range cols { | ||
| switch col { | ||
| case "name": | ||
| nameIdx = i | ||
| case "database_name": | ||
| dbIdx = i | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Why is the column order not fixed?
| return readSchemaEntries(rows) | ||
| } | ||
|
|
||
| func readSchemaEntries(rows driver.Rows) ([]schemaEntry, error) { |
Contributor
There was a problem hiding this comment.
This is only ever used in one place; just consolidate the two functions?
| rows, err := c.cn.QueryContext(ctx, query, nil) | ||
| if err != nil { | ||
| var sfErr *gosnowflake.SnowflakeError | ||
| if errors.As(err, &sfErr) && sfErr.Number == 2043 { |
| } | ||
|
|
||
| // execShowSchemas executes a SHOW TERSE SCHEMAS command and reads the results directly. | ||
| func (c *connectionImpl) execShowSchemas(ctx context.Context, pattern *string, suffix string) (entries []schemaEntry, err error) { |
Contributor
There was a problem hiding this comment.
(Why are the results named when we never use that?)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GetObjectsatObjectDepthTablesandObjectDepthDBSchemaswhen catalog is specific (non-wildcard)SHOW TERSErows viadriver.Rowsand builds theGetObjectsInfotree in Go, instead of collecting queryIDs and running complex SQL templates that re-read results throughRESULT_SCANjoinsMotivation
Benchmark comparison (report) between the ADBC ODBC bridge and the official Snowflake ODBC driver showed
SQLTableswas 2.06x slower. The root cause:GetObjects(ObjectDepthTables)executed 4 Snowflake queries (3 parallelSHOW TERSE+ 1 complex SQL joining them viaRESULT_SCAN) where the official driver uses essentially 1 query.Query count reductions
ObjectDepthTables)ObjectDepthTables)ObjectDepthDBSchemas)Changes
go/get_objects_direct.go(new): OptimizedGetObjectsimplementations that executeSHOW TERSEcommands and read rows directly, building the result tree in Go viadriverbase.BuildGetObjectsRecordReader.go/connection.go: Added fast-path call inGetObjects()before the existing SQL template path. Falls through when the direct path is not applicable.