How to configure temporary directory to spill data

[djouallah]

I am trying to run daft with ray runner in a fabric notebook, I would like to configure the location of the temporary directory where data is spilled

[samster25]

Hi @djouallah,

Great question! You should be able to do via

import json
import ray
import daft

ray.init(
    _system_config={
        "object_spilling_config": json.dumps(
            {"type": "filesystem", "params": {"directory_path": "/tmp/spill"}},
        )
    },
)

daft.context.set_runner_ray()

check out these docs if you want to try it out in a distributed fashion.

https://docs.ray.io/en/latest/ray-core/objects/object-spilling.html#single-node