In case this helps anyone: my dataflow job on Google Cloud was restricted to 1 worker, so I added the following argument when launching create_data.py:
--num_workers 8
Now the job was restricted to 4 workers instead due to the quota on external IP addresses. So I turned them off with:
--no_use_public_ips
and enabling Private Google Access for the used region's subnet (us-central1 for me) in the VPC networks section.
In case this helps anyone: my dataflow job on Google Cloud was restricted to 1 worker, so I added the following argument when launching create_data.py:
--num_workers 8Now the job was restricted to 4 workers instead due to the quota on external IP addresses. So I turned them off with:
--no_use_public_ipsand enabling Private Google Access for the used region's subnet (us-central1 for me) in the VPC networks section.