Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve appender speed #351

Merged
merged 2 commits into from
Jan 24, 2025

Conversation

JAicewizard
Copy link
Contributor

@JAicewizard JAicewizard commented Jan 17, 2025

Rough performance before:

BenchmarkAppenderSingle/int8-16         	    8288	    142109 ns/op

Rough performance after caching chunk size:

BenchmarkAppenderSingle/int8-16         	   35270	     32326 ns/op

Full (existing API without reusing the appender for fairness) vs the new TS (TableSource) implementation

BenchmarkAppenderSingleFull/int8-16         	   11076	    109986 ns/op
BenchmarkAppenderSingleFull/int8-16         	   10000	    103503 ns/op
BenchmarkAppenderSingleFull/int8-16         	   10000	    103835 ns/op
BenchmarkAppenderSingleFull/int8-16         	   11526	    107647 ns/op
BenchmarkAppenderSingleFull/int8-16         	   10000	    107233 ns/op
BenchmarkAppenderSingleTS/int8-16           	   12663	     99329 ns/op
BenchmarkAppenderSingleTS/int8-16           	   12016	     97747 ns/op
BenchmarkAppenderSingleTS/int8-16           	   12054	     98896 ns/op
BenchmarkAppenderSingleTS/int8-16           	   12721	     96048 ns/op
BenchmarkAppenderSingleTS/int8-16           	   12434	    102425 ns/op

This benchmark only tests the single threaded row based tablesource, if this gets looked at and there are no major problems I can implement the multi-threaded and chunk APIs as well. These will be even faster!

@taniabogatsch taniabogatsch marked this pull request as ready for review January 24, 2025 09:28
@taniabogatsch
Copy link
Collaborator

Thanks!

@taniabogatsch taniabogatsch merged commit a0a16d2 into marcboeker:main Jan 24, 2025
7 checks passed
taniabogatsch added a commit to taniabogatsch/go-duckdb that referenced this pull request Mar 11, 2025
taniabogatsch added a commit that referenced this pull request Mar 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants