GH-37630: [C++][Python][Dataset] Allow disabling fragment metadata caching

[pitrou]

Rationale for this change

Parquet file fragments currently cache their (Parquet) metadata for later accesses when scanning has finished.
This can produce surprisingly high memory consumption in cases where:

1. the dataset is only scanned once, rather than repeatedly (this is very common)
2. there is a high metadata-to-data ratio; this can happen when the schemas on disk are very wide, with few rows per file and/or a low number of columns selected for reading

What changes are included in this PR?

Add an option to disable metadata caching on Parquet file fragments.

Are these changes tested?

Yes, by new unit tests. Also, reading a wide dataset locally has been confirmed to consume much less memory when the new option is toggled.

Are there any user-facing changes?

No.

• GitHub Issue: [C++] Potential memory leak in Parquet reading with Dataset #37630

[github-actions]

⚠️ GitHub issue #37630 has been automatically assigned in GitHub to PR creator.

[pitrou]

In #45287 (comment) it was mentioned that clearing physical_schema_ further improved the memory profile, but it's not obvious that it would be correct.

[icexelloss]

I wonder if we can have a mode to release the fragment once the data for that fragment has been read in a "scan once" usage pattern. But also I don't know how hard it is to change that.

Per physical_schema_ - I am not familiar with the details but it seems fine to me if this is "scan once" and the fragment is not used again?

[timothydijamco]

I posted some thoughts about clearing physical_schema_ in #45287 (comment)

[pitrou]

I added a change that clears the cached physical schema, but keeps the original schema when it was passed via the constructor.

[pitrou]

@zanmato1984 I don't know if you would like to take a quick look at this PR.

(sorry if you received a ping on the issue, I initially chose the wrong tab :))

[zanmato1984]

Hi @pitrou , I'm on vacation for now and I will be able to review it next week. Will that be too late for you? Thanks.

[pitrou]

@zanmato1984 No, it's ok of course! Thank you.

[zanmato1984]

the dataset is only scanned once, rather than repeatedly (this is very common)

A quick question: by "very common" you mean scan once or repeatedly?

[pitrou]

A quick question: by "very common" you mean scan once or repeatedly?

I mean scan once.

[pitrou]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 918c50e

Submitted crossbow builds: ursacomputing/crossbow @ actions-36b0a9da17

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 47e56fa

Submitted crossbow builds: ursacomputing/crossbow @ actions-a9b6c9cf52

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

I plan to merge this soon if there are no further comments.

[pitrou]

Thanks for the reviews!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit f8a0902.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.