Is HashAggregation spilling enabled by default?

[LiaCastaneda]

👋 Hi, I had the doubt if HashAggregation has spilling enabled by default (didn't report this as an issue since I saw #7400 ). We are running some heavy queries and are encountering OOM errors:
Resources exhausted: Failed to allocate additional 10935575608 bytes for GroupedHashAggregateStream[3] with 0 bytes already allocated for this reservation - 9130676356 bytes remain available for the total pool
I looked at AggregateExec metrics and see no spilling is being made.

[2010YOUY01]

Spilling is always enabled, the difference is:

• If no memory limit is specified (default), memory accounting inside operators never fails.
• If a memory limit is specified, memory accounting might fail, and a fallback spilling path will be triggered.

It's possible to trigger an error (or bug) before the first spilling file is written, perhaps you can share the reproducer by opening an issue.

[LiaCastaneda]

Thanks for the answer, yes this makes sense & that's exactly what is happening to me, when I set a concrete memory pool it fails otherwise using a UnboundedMemoryPool it works just fine

[LiaCastaneda]

So, I understand is that this is not an OOM error but a memory accounting error? in any case I think there is no fallback spilling being made since I have:
spill_count = 0
spilled_bytes=0
spilled_rows=0
But yeah I will try share a reproducer for this

[2010YOUY01]

Yes, this is an error thrown by DataFusion (instead of OS) during internal memory tracking updates.
The stats will only be updated after the first spill file is written with success, if anything goes wrong during writing the first spill, it's possible to get an error with such 0 stats.

[2010YOUY01]

Also, just a reminder: external execution is still experimental, so bugs are likely. We plan to improve it over the coming months.

[LiaCastaneda]

I see, I also forgot to mention Im using a GreedyMemoryPool so that might not be the best one to handle queries that need to spillI guess 😄 guiding myself on https://datafusion.apache.org/comet/user-guide/tuning.html (i'm not using cometh though)