[TPC-H] Configure temporary directory in duckdb for out-of-core processing

[hendrikmakait]

Closes #1509

[ntabris]

This would be a temp directory on the VM? If so, I'd recommend doing something under /scratch so it uses NVMe if present.

[hendrikmakait]

This would be a temp directory on the VM? If so, I'd recommend doing something under /scratch so it uses NVMe if present.

Good point, I'll adjust the code to check if we're running on the cloud. I suppose we should create the temporary directory within the workspace of the Dask worker that runs the coiled.function?

[jrbourbeau]

I'd recommend doing something under /scratch so it uses NVMe if present

TIL, thanks @ntabris. Is there a way I can figure out when NVMe is present? Is it only specific instance types?

[ntabris]

I suppose we should create the temporary directory within the workspace of the Dask worker that runs the coiled.function?

Yeah, that would work—dask worker workspace is already set to be in /scratch.

Is there a way I can figure out when NVMe is present? Is it only specific instance types?

It's specific instance types on AWS, in particular, instance types with d in the family, such as m6id or g4dn (plus a few others, such as g5 which does have NVMe).

[hendrikmakait]

Closing as stale