What is the correct and optimal way to do a full shuffle?

[conceptofmind]

I guess this is mostly the title. I need to shuffle around 8TBs of data.

[conceptofmind]

Maybe:

@daft.func.batch(return_dtype=DataType.float64())
def get_random(series: Series) -> Series:
    rng = np.random.default_rng()
    return Series.from_numpy(rng.random(len(series)))

ddf = ddf.with_column("rand_key", get_random(col("text")))
ddf = ddf.sort(col("rand_key"))

[colin-ho]

This ain't a bad solution actually

[conceptofmind]

Ok. So if this adequate do you think there is any way that is more optimal or I should just run with this udf?

A node only has around 512GB of RAM.

[conceptofmind]

Also, does write_parquet shuffle at all?

[conceptofmind]

Or is doing repartition the best?

import daft

ddf = daft.read_parquet("/common-pile_sampled/raw/**")
ddf = ddf.repartition(15565)
ddf.write_parquet("/common-pile_sampled/shuffled")

[colin-ho]

Repartition is a decent approximation of a full shuffle, but it wont be as truly random as the solution you had with the udf and sort though.

[colin-ho]

Though note that repartition only works when running on ray, not native.

[conceptofmind]

Ok. So I had run this with the native runner which I assume will not have done anything haha.

[universalmind303]

@colin-ho @srilman any advice here?