Performance regressions in duckdbfs

[myrrc]

What happened?

When #5767 was merged, we saw TPC-h (S3) severe regressions in benchmarks. Then we decided to turn it off in #6564. I've been investigating this regression recently, but there are a couple of issues with this approach:

• Our CI can't run benchmarks for a duckdb hash Support building duckdb from a hash, including CI #6658 DuckDB: test curl impl, don't auto-load extensions #6677
• Moreover, Duckdb 1.5 introduced API changes so we depend on that (tracking changes in Trying out duckdb-1.5 with duckdbfs #6627)
• My guess was that regression may be negligible in using duckdbfs + httpfs's curl backend (default from 1.5) as opposed to httplib (1.4-). However, our benchmarks in CI fail with curl backend due to 1.4 httpfs curl S3 globbing (list) bug Fix incorrect AWS list request in 1.4 duckdb/duckdb-httpfs#265 and we also have a bug in our code for httplib backend use curl backend and duckdbfs for benchmarks #6685 Fix S3 globbing for duckdbfs with httplib #6709, which further complicates testing.