max_concurrent_tasks not applicable in non-partitioned z-order

[marsupialtail]

Environment

Delta-rs version: 0.10.2

Binding: Python

Environment:
Ubuntu 22:

---

Bug

see title.

[marsupialtail]

@wjones127

[wjones127]

In order to fix this, we need to pass down a memory limit to DataFusion. Also we might want to consider passing down a limit on the number of threads used, though perhaps a memory limit is sufficient.

[edgBR]

Hi,

I have noticed this recently. Changing this parameter doesn't have any impact in performance. In fact even using a larger compute to run z order seems to not decrease the time that z order consumes.

[justkr]

Hello,
Please work on this as I would like to make my performance better with this param.
Thanks!

[pmb-b2i]

I’m facing the same issue—max_concurrent_tasks doesn’t improve z_order performance for me either

[obiii]

Hi,

I've had similar findings—the parameter seems to have no impact on performance. I’d be grateful if this could be looked into soon; the issue is almost like an old friend at this point!

[carlos6moya]

Hi,

I am conducting various tests in AWS to improve the performance of the Z-order operation. I performed a Z-order operation using Fargate with 16 vCPUs and 120 GB of memory, but the runtime was the same as when using SageMaker on AWS with 96 vCPUs and 768 GB of memory.

Additionally, I adjusted the maximum concurrency settings, but the execution time remained unchanged.

[ion-elgreco]

I guess you are all on same team, would be useful if any of you provides stats about your table.

The streams are created per BIN of files, so if there aren't many bins there is no point in increasing the max concurrent tasks.

Just slapping more compute to something is almost never a promise of faster runtime unless it was the bottleneck

[edgBR]

Hi @ion-elgreco,

@justkr and @obiii and I work on the same team. Currently we have 2 projects in production running:

deltalake==0.19.2

and

deltalake==0.19.2

Respectively, both of them run AzureML workloads using as base docker image:

mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04

However I detected this problem in the last project. Which dependencies are as follows:

channels:
  - defaults
dependencies:
  - python=3.10.14
  - pip:
      - numpy==1.26.4
      - pandera==0.20.4
      - pandas==2.2.2
      - polars==1.10.0
      - pyarrow==17.0.0
      - deltalake==0.19.2
      - mltable==1.6.1
      - tsfeatures==0.4.5
      - holidays==0.58
      - scikit-learn==1.5.1
      - azure-ai-ml==1.21.0
      - azure-core==1.31.0
      - azure-identity==1.9.0
      - azure.keyvault==4.2.0
  - conda:
      - conda=23.3.0

We are using azureML environments but as mentioned above the base image is: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04

Currently we have 2 modes of loading data in our delta tables:

• Full load: that means we fetch all of the data from our transactional system.
• Incremental load: we fetch just the new records.
• Ad-hoc: this is an special workload that is not yet ready for production as we are optimizing it, more details below.

For the full load and incremental load workloads the statistics are as follows:

bronze: 11523107 rows
silver: 2008737 rows
gold monthly table: 1479060 rows
gold daily table: 8559852

Our underlying parquet files are around 50MBs (not the optimum size but we are using zstd compression, we have not decided yet in how to handle this).

Currently our processing jobs take around 2m 40s in a Standard_F32s_v2 (32 cores, 64 GB RAM, 256 GB disk) machine.

For the ad-hoc workloads the details of the table are as follows:

bronze table 1: 1460M rows (approx) / 6 columns
bronze table 2: 365 M rows (approx) / 23 columns.
silver table: around 2.6B rows / 29 columns
gold table: 42 M rows (approx)/ 40 columns

Runtime: approx 15mins in Standard_FX48mds (48 cores, 1008 GB RAM, 2016 GB disk) .

Currently we would like to understand which parameters affect the Z order performance and clarify the usage of max_concurrent_tasks as well as max_spill_size.

BR
E

[ion-elgreco]

@edgBR the bins are based on how many partitions you have afaik, so are these tables partitioned?

Max_spill_size is used for greedy memory pool size IIRC, can't verify this ATM but should be easy to check if the parameter is passed in the memory pool then that's the case

[edgBR]

Hi @ion-elgreco

Based on your last comment I believe I have missunderstood completely the Z order then.

Right now my tables are not partitioned. With this I mean that when writting my tables:

Im not passing partitions to the Writer properties. The reason because I did not implement this is because in my case there is not that "great partition to choose" what do I mean by that:

• My data consist of transactional data that is unevenly timewise distributed. 2 consecutive days can have a completely different volumes of transactions. It is true however that most of the transactions happen between day 20 and day 5 (of next month).
• I deal with financial portfolios which have a unique id, however it could be that for some portfolios I dont have any transactions at all for some days.

I then just Z ordered the table as I tought that it will redistribute the files accordinly to the statistics of the data columns I have in my files and as I tought that Z order and partitioning should not be combined.

Regarding the underlying files:

My bronze table tough has 31 underlying files.
My silver table has 15 files.
My gold table has < 10 files.

This are of course after file compatction an z ordering.