Skip to content

Commit ce2c0e9

Browse files
authored
DPT-2956: Pin awswrangler==3.9.1 and pyarrow==17.0.0 in Glue jobs (#1425)
Latest awswrangler (3.16.1) dropped Python 3.9 support and latest pyarrow (18+) no longer ships pre-built wheels for Python 3.9, causing pip install failures in our Glue Python Shell 3.9 jobs. Pin awswrangler to 3.9.1 (last version well-tested on Glue Python Shell 3.9) and pyarrow to 17.0.0 (last version with cp39 manylinux wheels).
1 parent 4ae1ecd commit ce2c0e9

1 file changed

Lines changed: 4 additions & 4 deletions

File tree

iac/main/resources/state-machine.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -429,7 +429,7 @@ DataQualityPythonGlueJob:
429429
ScriptLocation: !Sub s3://${ELTMetadataBucket}/txma/data_quality_metrics_script/data_quality_metrics_generation.py
430430
DefaultArguments:
431431
'--JOB_NAME': !Sub ${Environment}-dap-data-quality-metrics-generation
432-
'--additional-python-modules': 'awswrangler'
432+
'--additional-python-modules': 'pyarrow==17.0.0,awswrangler==3.9.1'
433433
'--env': !Sub ${Environment}
434434
'--raw_db': !Sub ${Environment}-txma-raw
435435
'--stage_db': !Sub ${Environment}-txma-stage
@@ -766,7 +766,7 @@ RawStageTransformProcessPythonGlueJob:
766766
DefaultArguments:
767767
'--JOB_NAME': !Sub ${Environment}-dap-raw-stage-transform-process
768768
'--LOG_LEVEL': 'INFO'
769-
'--additional-python-modules': 'awswrangler,aws_lambda_powertools'
769+
'--additional-python-modules': 'pyarrow==17.0.0,awswrangler==3.9.1,aws_lambda_powertools'
770770
'--config_bucket': !Sub ${ELTMetadataBucket}
771771
'--config_key_path': 'txma/raw_stage_optimisation_solution/configuration_rules/raw_to_stage_config_rules.json'
772772
'--raw_database': !Sub ${Environment}-txma-raw
@@ -867,7 +867,7 @@ DataQualityStageLayerOptimisedPythonGlueJob:
867867
ScriptLocation: !Sub s3://${ELTMetadataBucket}/txma/data_quality_metrics_script/data_quality_new_stage_metrics_generation.py
868868
DefaultArguments:
869869
'--JOB_NAME': !Sub ${Environment}-dap-data-quality-new-stage-metrics-generation
870-
'--additional-python-modules': 'awswrangler'
870+
'--additional-python-modules': 'pyarrow==17.0.0,awswrangler==3.9.1'
871871
'--env': !Sub ${Environment}
872872
'--stage_db': !Sub ${Environment}-txma-stage
873873
'--stage_parent_tbl': 'txma_stage_layer'
@@ -976,7 +976,7 @@ SplunkMigratedRawStageTransformProcessPythonGlueJob:
976976
ScriptLocation: !Sub s3://${ELTMetadataBucket}/txma/raw_stage_optimisation_solution/scripts/raw_to_stage_process_glue_job.py
977977
DefaultArguments:
978978
'--JOB_NAME': !Sub ${Environment}-dap-splunk-migration-raw-stage-transform-process
979-
'--additional-python-modules': 'awswrangler'
979+
'--additional-python-modules': 'pyarrow==17.0.0,awswrangler==3.9.1'
980980
'--config_bucket': !Sub ${ELTMetadataBucket}
981981
'--config_key_path': 'txma/raw_stage_optimisation_solution/configuration_rules/splunk_to_stage_config_rules.json'
982982
'--raw_database': !Sub ${Environment}-txma-raw

0 commit comments

Comments
 (0)