[Bug] FDedup Fails on Windows

[touma-I]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/universal/fdedup

What happened + What you expected to happen

When running FDedup on windows, I get an error:
13:50:52 ERROR - Exception during execution 'processing_time': None
13:50:52 INFO - Completed execution in 0.0 min, execution result 1
13:50:52 ERROR - ClusterAnalysis failed with status 1, aborting ...

A complete trace can be found here:

https://github.com/IBM/data-prep-kit/blob/v1.0.0.post1/transforms/transforms-1.0-lang-Windows.ipynb

Reproduction script

Run the following notebook on Windows:

https://github.com/IBM/data-prep-kit/blob/v1.0.0.post1/transforms/transforms-1.0-lang-Windows.ipynb

Anything else

No response

OS

Windows WSL

Python

Other

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[cmadam]

@touma-I commit 86bbded is the first try to fix windows problem. Can you please test it using the local file system and see if it works? I still think it will not work for s3 storage; is there a way, inside a transform, to determine whether it uses the local file system or S3

[touma-I]

@cmadam with the lastest PR, I get the following error below. It seems the filename issue was solved but now another error pops up on windows:

17:40:40 INFO - Starting SignatureCalculation step
17:40:40 INFO - Got parameters for SignatureCalculation
17:40:40 INFO - minhash parameters are : {'document_id_column': 'int_id_column', 'contents_column': 'contents', 'seed': 42, 'num_permutations': 112, 'jaccard_similarity_threshold': 0.75, 'word_shingle_size': 5, 'num_bands': 14, 'num_minhashes_per_band': 8, 'num_segments': 1, 'shingle_option': 'word'}
17:40:40 INFO - data factory scdata_ is using local configuration without input/output path
17:40:40 INFO - data factory scdata_ max_files -1, n_sample -1
17:40:40 INFO - data factory scdata_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']
17:40:40 INFO - pipeline id pipeline_id
17:40:40 INFO - code location None
17:40:40 INFO - data factory data_ is using local data access: input_folder - files-doc-id output_folder - files-fdedup
17:40:40 INFO - data factory data_ max_files -1, n_sample -1
17:40:40 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']
17:40:40 INFO - orchestrator minhash started at 2025-01-31 17:40:40
17:40:40 INFO - Number of files is 1, source profile {'max_file_size': 0.030454635620117188, 'min_file_size': 0.030454635620117188, 'total_file_size': 0.030454635620117188}
17:40:40 INFO - Completed 1 files (100.0%) in 0.0 min
17:40:40 INFO - Done processing 1 files, waiting for flush() completion.
17:40:40 INFO - Starting flush()
17:40:40 INFO - Wrote 14 tables with a total size of 215,040 bytes
17:40:40 INFO - done flushing in 0.066 sec
17:40:40 INFO - Completed execution in 0.002 min, execution result 0
17:40:40 INFO - SignatureCalculation completed successfully
17:40:40 INFO - Starting ClusterAnalysis step
17:40:40 INFO - Got parameters for ClusterAnalysis
17:40:40 INFO - cluster parameters are : {'jaccard_similarity_threshold': 0.75, 'num_bands': 14, 'num_segments': 1, 'sort_output': False}
17:40:40 INFO - pipeline id pipeline_id
17:40:40 INFO - code location None
17:40:40 INFO - data factory data_ is using local data access: input_folder - files-fdeduands output_folder - files-fdedup\docs_to_remove
17:40:40 INFO - data factory data_ max_files -1, n_sample -1
17:40:40 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']
17:40:40 INFO - orchestrator cluster started at 2025-01-31 17:40:40
17:40:40 INFO - Number of folders is 14
17:40:40 WARNING - Exception processing file band=0\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 1 files (7.14%) in 0.001 min
17:40:40 WARNING - Exception processing file band=1\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 2 files (14.29%) in 0.001 min
17:40:40 WARNING - Exception processing file band=2\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 3 files (21.43%) in 0.001 min
17:40:40 WARNING - Exception processing file band=3\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 4 files (28.57%) in 0.001 min
17:40:40 WARNING - Exception processing file band=4\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 5 files (35.71%) in 0.001 min
17:40:40 WARNING - Exception processing file band=5\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 6 files (42.86%) in 0.001 min
17:40:40 WARNING - Exception processing file band=6\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 7 files (50.0%) in 0.001 min
17:40:40 WARNING - Exception processing file band=7\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 8 files (57.14%) in 0.001 min
17:40:40 WARNING - Exception processing file band=8\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 9 files (64.29%) in 0.001 min
17:40:40 WARNING - Exception processing file band=9\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 10 files (71.43%) in 0.001 min
17:40:40 WARNING - Exception processing file band=10\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 11 files (78.57%) in 0.001 min
17:40:40 WARNING - Exception processing file band=11\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 12 files (85.71%) in 0.001 min
17:40:40 WARNING - Exception processing file band=12\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 13 files (92.86%) in 0.001 min
17:40:40 WARNING - Exception processing file band=13\segment=0: Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\transform_file_processor.py", line 85, in process_file
    out_files, stats = self.transform.transform(folder_name=f_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 148, in transform
    cluster_dataframe, cluster_stats = self._get_clusters(band_segment_dataframe)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\dpk_fdedup\cluster_analysis\transform.py", line 180, in _get_clusters
    groupby_dataframe = band_segment_dataframe.group_by("band_hash").agg("document_data")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\dataframe\group_by.py", line 232, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\marou\AppData\Roaming\Python\Python312\site-packages\polars\lazyframe\frame.py", line 2050, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: band_hash

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'group_by' <---
DF []; PROJECT */0 COLUMNS; SELECTION: None

17:40:40 INFO - Completed 14 files (100.0%) in 0.001 min
17:40:40 INFO - Done processing 14 files, waiting for flush() completion.
17:40:40 INFO - done flushing in 0.0 sec
Traceback (most recent call last):
  File "C:\Users\marou\anaconda3-a\envs\DPK2\Lib\site-packages\data_processing\runtime\pure_python\transform_orchestrator.py", line 131, in orchestrate
    stats["processing_time"] = round(stats["processing_time"], 3)
                                     ~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'processing_time'
17:40:40 ERROR - Exception during execution 'processing_time': None
17:40:40 INFO - Completed execution in 0.001 min, execution result 1
17:40:40 ERROR - ClusterAnalysis failed with status 1, aborting ...