Skip to content

feat: extend udf_overhead benchmark w/ sub-str example#358

Open
crepererum wants to merge 1 commit intomainfrom
crepererum/sub-str-overhead-bench
Open

feat: extend udf_overhead benchmark w/ sub-str example#358
crepererum wants to merge 1 commit intomainfrom
crepererum/sub-str-overhead-bench

Conversation

@crepererum
Copy link
Collaborator

@crepererum crepererum commented Feb 5, 2026

For #28.

Results

=== cost model ===                                                                                                                                                                                                 
y = cost_row * n_rows + cost_call * n_batches + cost_cached                                                                                                                                                        
                                                                                                                                                                                                                   
   cost_row: Cost per input row/cell.                                                                                                                                                                              
  cost_call: Cost per UDF call / record batch.                                                                                                                                                                     
cost_cached: Cost amortized by calling the UDF multiple times.                                                                                                                                                     
      score: How well the linear model fits (0=not, 1=perfect).                                                                                                                                                    
┌─────────┬────────┬──────────┬───────────┬─────────────┬───────┐                                                                                                                                                  
│ payload ┆ mode   ┆ cost_row ┆ cost_call ┆ cost_cached ┆ score │                                                                                                                                                  
╞═════════╪════════╪══════════╪═══════════╪═════════════╪═══════╡                                                                                                                                                  
│ addone  ┆ native ┆      110 ┆         0 ┆      15_142 ┆  1.00 │                                                                                                                                                  
│ addone  ┆ wasm   ┆      477 ┆   936_577 ┆     980_115 ┆  0.99 │                                                                                                                                                  
│ addone  ┆ python ┆    2_733 ┆   968_911 ┆     853_064 ┆  1.00 │                                                                                                                                                  
│ substr  ┆ native ┆    3_369 ┆         0 ┆           0 ┆  1.00 │                                                                                                                                                  
│ substr  ┆ wasm   ┆   12_390 ┆ 2_611_558 ┆     489_419 ┆  1.00 │                                                                                                                                                  
│ substr  ┆ python ┆   19_973 ┆ 2_964_281 ┆     657_907 ┆  1.00 │                                                                                                                                                  
└─────────┴────────┴──────────┴───────────┴─────────────┴───────┘    

=== delta rel2 ===
delta = y / max(x, 1)
┌─────────┬────────┬──────────┬──────────┬──────────────┬─────────────┐
│ payload ┆ mode   ┆ baseline ┆ cost_row ┆    cost_call ┆ cost_cached │
╞═════════╪════════╪══════════╪══════════╪══════════════╪═════════════╡
│ addone  ┆ wasm   ┆ native   ┆     4.34 ┆   936_577.00 ┆       64.73 │
│ addone  ┆ python ┆ native   ┆    24.85 ┆   968_911.00 ┆       56.34 │
│ addone  ┆ python ┆ wasm     ┆     5.73 ┆         1.03 ┆        0.87 │
│ substr  ┆ wasm   ┆ native   ┆     3.68 ┆ 2_611_558.00 ┆  489_419.00 │
│ substr  ┆ python ┆ native   ┆     5.93 ┆ 2_964_281.00 ┆  657_907.00 │
│ substr  ┆ python ┆ wasm     ┆     1.61 ┆         1.14 ┆        1.34 │
└─────────┴────────┴──────────┴──────────┴──────────────┴─────────────┘

The per-row overhead looks better if we pass larger data around, likely because the cost to manage python objects gets relatively lower. However I'm somewhat surprised that the per-call overhead for both Python and the plain WASM wrapper is soo large. Gonna need to check the reason for this in a follow-up.

@crepererum crepererum force-pushed the crepererum/sub-str-overhead-bench branch from 12e88dd to 4074d79 Compare February 11, 2026 16:58
@crepererum crepererum marked this pull request as ready for review February 12, 2026 08:46
@crepererum crepererum requested review from Sl1mb0 and mhilton February 12, 2026 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant