[General] Benchmarking ParadeDB vs Elastic

[inevolin]

What would you like to discuss?

Hi team 👋

I've been benchmarking ParadeDB vs Elasticsearch for full-text search performance, and I wanted to share the findings and start a discussion. In my tests I consistently see Elastic outperforming ParadeDB, so I'm curious whether this is expected or not (since there are not many independent benchmarks available online). Maybe I missed something in my testing suite. Looking forward to hearing some thoughts and feedback: https://github.com/inevolin/ParadeDB-vs-ElasticSearch

Best regards

Full Name:

Ilja N

Affiliation:

n.a.

[inevolin]

One finding I am observing is that some settings/configs are not being applied. I am providing custom config values, but when querying the database (pg_settings) it has different values from what I provided.

EDIT: my pg_settings query was wrong, it was using setting instead of current_setting, hence the false-positive warnings.

[philippemnoel]

Hi @inevolin. Thank you for reporting this. Your benchmark is nicely done. We're planning to release a similar benchmarking tool for ParadeDB as well, and may draw inspiration from your work. A few things to jump out to me:

• You put latest as the ParadeDB version. You should be explicit about which version you're testing against for ParadeDB as well.
• You mention that the settings are not being respected. I did not dive into your codebase, but in general these needs to be applied in the postgresql.conf file and Postgres needs to be restarted. More on this here: https://docs.paradedb.com/documentation/performance-tuning/overview
• The dataset used to test is small. Most of our performance optimization work of the past year has been centered around TBs+ workloads. That said, we are actively planning to focus on smaller workloads in Q1 and Q2 of this year, and you can expect the overhead you're noticing in ParadeDB to decrease significantly at this data volume.

Thank you for making this project and sharing it with us :)

[inevolin]

Thanks @philippemnoel and happy new year!

• I've fixed the version to the used tag (0.20.5 instead of latest)
• The config/settings were working just fine, the issue was how I was cross-validating them (see my edit above)
• The dataset is indeed small, but I do think with the upgrade to PG18 it might outperform ES. But also looking forward to expanding to larger tests with more queries. Let's collab on this.
  Cheers

[philippemnoel]

We'll also be working to improve performance for small datasets. We should be releasing PG18 this week, see #3850 for the PG18 release.

[inevolin]

I've setup a PG18 (with pg_search + GINs) vs ElasticSearch benchmark suite that we can use to compare as well when the PG18 release comes out for ParadeDB - https://github.com/inevolin/Postgres-FTS-vs-ElasticSearch

[philippemnoel]

If you're interested in continuing to work on benchmarking ParadeDB, we'd love to collaborate. I'd recommend you join our community Slack, so you can interact more easily with the engineers on our team who will be working on this initiative

[inevolin]

@philippemnoel does paradeDB utilize a different storage mechanism for large JSON documents, or do they also get TOASTed by default? The reason why I'm asking is this use case: https://www.reddit.com/r/PostgreSQL/comments/1q5ts8u/postgres_with_large_jsonbs_vs_elasticsearch/

[philippemnoel]

Tantivy just breaks down the json into sub fields and indexes them separately, it’s not indexed as a blob

[inevolin]

@philippemnoel thanks for clarifying! I'll explore that some more.