test: add memory tests (via DHAT)

[zachschuermann]

What changes are proposed in this pull request?

exploring DHAT tests for memory profiling. this adds two ignored tests in a separate (new) mem-test internal crate.

run with:

cargo test -p mem-test -- --ignored --nocapture

How was this change tested?

test-only

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.83%. Comparing base (39c440d) to head (e77caac).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1009   +/-   ##
=======================================
  Coverage   84.83%   84.83%           
=======================================
  Files         113      113           
  Lines       28735    28735           
  Branches    28735    28735           
=======================================
  Hits        24376    24376           
  Misses       3203     3203           
  Partials     1156     1156           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[scovich]

I don't think I know how DHAT works... can you given an example of expected output in the PR description?

[OussamaSaoudi]

Seems useful. We can iterate as needed.

[OussamaSaoudi]

oof

Failed to get latest snapshot: ObjectStore(Generic { store: "LocalFileSystem", source: UnableToOpenFile { source: Os { code: 24, kind: Uncategorized, message: "Too many open files" }, path: "/var/folders/cr/cl0ybfwx6kjd8bybfc2mr4rw0000gp/T/.tmpapqcBg/_delta_log/00000000000000000749.json" } })

[zachschuermann]

ah macos? i think might need to bump up ulimit

[OussamaSaoudi]

@scovich Here's the output that didn't crash:

   Running tests/dhat_large_table_data.rs (target/debug/deps/dhat_large_table_data-55715a37bc576271)

running 1 test
File size (compressed file size):    1073334219 bytes
Total size (uncompressed file size): 1073183367 bytes
Heap stats after writing parquet:
HeapStats { total_blocks: 3939016, total_bytes: 7698447898, curr_blocks: 3, curr_bytes: 88, max_blocks: 6245, max_bytes: 2237041899 }
Heap stats after creating snapshot:
HeapStats { total_blocks: 3940234, total_bytes: 7700146377, curr_blocks: 378, curr_bytes: 70344, max_blocks: 6245, max_bytes: 2237041899 }
Heap stats after building scan:
HeapStats { total_blocks: 3940250, total_bytes: 7700147415, curr_blocks: 390, curr_bytes: 71080, max_blocks: 6245, max_bytes: 2237041899 }
Heap stats after scan execution:
HeapStats { total_blocks: 3979394, total_bytes: 10933070181, curr_blocks: 764, curr_bytes: 98934, max_blocks: 6245, max_bytes: 2237041899 }
Total rows read: 1000000
test test_dhat_large_table_data ... ok

[nicklan]

approved for now as this is purely additive and we can iterate on exactly what we want to do with them