Memory errors

[datbth]

Problem

Lately, our app has been seeing sporadic memory errors with recent duckdb-go versions:

• failed to execute the prepared arrow: Invalid Error: basic_string::_M_create
• failed to execute the prepared arrow: Out of Memory Error: Allocation failure
• failed to execute the prepared arrow: Out of Memory Error: Allocation failure LINE 5: READ_PARQUET('/opt/canal_lake/production/7738/2199027312554... ^
• failed to execute the prepared arrow: Out of Memory Error: Allocation failure LINE 15: READ_PARQUET('/opt/canal_lake/production/2199023266005/2199... ^
• failed to execute the prepared arrow: Out of Memory Error: Allocation failure LINE 5: READ_PARQUET('/opt/canal_lake/production/7738/2199027310769... ^

We haven't been able to reproduce or narrow down to specific problematic data. But all errors are returned from Arrow.QueryContext when querying various local Parquet files.

So we need some pointers/guidance on how to troubleshoot this further.

More context

Our app uses duckdb-go for 2 purposes:

1. Querying local parquet files
2. As MotherDuck client

These were our recent changes:

1. On Nov 26th, we upgraded go-duckdb from v2.2.1 (duckdb 1.2.2) to v2.5.2 (duckdb 1.4.2) to match MotherDuck's client version requirement
2. Since then, we observed a memory accumulation issue, so we restarted the service once on Dec 5th.
3. After some testing and bisecting, we found that the memory accumulation issue is connected to the new External File Cache feature introduced in duckdb 1.3.0
4. After setting enable_external_file_cache = false (line 52) on Dec 8th, we are seeing less of an accumulating memory issue.

However, we also then started seeing the memory issues mentioned at the beginning

Another note: We are using a pool of a duckdb instances (i.e. re-using instances for multiple queries).

[datbth]

Update: We haven't seen new errors lately.
So it's possible that this is due to

• Some incompatibility between
  • Parquet files written by old arrow-go (v18.2.0)
  • And new arrow-go (v18.4.1) or new duckdb-go
• Or somehow duckdb had managed to write some cache while enable_external_file_cache was enabled, then that cache caused some issues when enable_external_file_cache became disabled.