Skip to content

improve allocations in expression generator#444

Open
pquerna wants to merge 1 commit into
doug-martin:masterfrom
pquerna:pq/sql_expression_allocs
Open

improve allocations in expression generator#444
pquerna wants to merge 1 commit into
doug-martin:masterfrom
pquerna:pq/sql_expression_allocs

Conversation

@pquerna

@pquerna pquerna commented Aug 16, 2025

Copy link
Copy Markdown
  • We noticed heavy allocation generation in a production workflow, which then generates lots of garbage collection pressure, coming from the dataset -> toSQL generation.

  • Adds benchmarks around the Expression SQL generator

  • Adds pooling of instances

  • Adds GrowBuffer/GrowArgs to upsize buffers when we know large writes are coming

  • Adds more native / non-reflection based writers

  • other minor optimizations

benchmark                                                                       old allocs     new allocs     delta
BenchmarkExpressionSQLGenerator_Insert/Unprepared/SingleRowManyCols-8           1626           1596           -1.85%
BenchmarkExpressionSQLGenerator_Insert/Unprepared/MultiRowBulk-8                35730          34695          -2.90%
BenchmarkExpressionSQLGenerator_Insert/Prepared/SingleRowManyCols-8             868            864            -0.46%
BenchmarkExpressionSQLGenerator_Insert/Prepared/MultiRowBulk-8                  11263          11239          -0.21%
BenchmarkExpressionSQLGenerator_Insert/PreparedWithNums/SingleRowManyCols-8     1025           864            -15.71%
BenchmarkExpressionSQLGenerator_Insert/PreparedWithNums/MultiRowBulk-8          19357          11239          -41.94%
BenchmarkExpressionSQLGenerator_Update/Unprepared-8                             1754           976            -44.36%
BenchmarkExpressionSQLGenerator_Update/Prepared-8                               1595           1340           -15.99%
BenchmarkExpressionSQLGenerator_Update/PreparedWithNums-8                       2240           1340           -40.18%
BenchmarkExpressionSQLGenerator_Select/Unprepared-8                             3961           2520           -36.38%
BenchmarkExpressionSQLGenerator_Select/Prepared-8                               3585           3330           -7.11%
BenchmarkExpressionSQLGenerator_Select/PreparedWithNums-8                       4912           3331           -32.19%

@pquerna pquerna force-pushed the pq/sql_expression_allocs branch from 1ecc04b to f2bce03 Compare August 18, 2025 19:08
@pquerna pquerna force-pushed the pq/sql_expression_allocs branch from f2bce03 to 2488067 Compare August 18, 2025 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant