To iterate quickly, I've been deploying this recipe manually from my laptop with the following config:
# config-dataflow-gcs.py
c.Bake.bakery_class = "pangeo_forge_runner.bakery.dataflow.DataflowBakery"
bucket = "gs://pangeo-forge-runner-ci-testing"
c.DataflowBakery.temp_gcs_location = bucket + "/temp"
c.TargetStorage.fsspec_class = "gcsfs.GCSFileSystem"
c.TargetStorage.root_path = bucket + "/target/{job_name}"and, e.g., this command:
$ pangeo-forge-runner bake --repo="." -f=config-dataflow-gcs.py --Bake.recipe_id="goes" --Bake.job_name="goes-2019-2023-attempt2"
The job succeeds in about 35 mins, building a roughly 300mb zarr store with 37,000+ timesteps.
Just working to get an optimally chunked version built now, will check back here once I have that.
To iterate quickly, I've been deploying this recipe manually from my laptop with the following config:
and, e.g., this command:
$ pangeo-forge-runner bake --repo="." -f=config-dataflow-gcs.py --Bake.recipe_id="goes" --Bake.job_name="goes-2019-2023-attempt2"The job succeeds in about 35 mins, building a roughly 300mb zarr store with 37,000+ timesteps.
Just working to get an optimally chunked version built now, will check back here once I have that.