S3 buckets for our transformed data

[esheehan-gsl]

@ian-noaa I was thinking we might want to set up a few buckets for our Zarr files. Besides the one for the production Zarr, could we set up a separate bucket for development? This would allow us to run the dev environment locally without having to download the production Zarr to our laptops, and would give us a space to experiment with changes to the Zarr structure without disrupting production.

What do you think?

[ian-noaa]

I think that's a good idea. I'll ask VLab to create a new "production" bucket for us to use since the current one already has some dev artifacts in it.

[esheehan-gsl]

Oh, great. Which bucket is ours? I’ll upload the dev Zarr I made this morning for our use.

[ian-noaa]

We currently have use of s3://osti-modeling-dev-rtma-vis. The initial diag files we received are at s3://osti-modeling-dev-rtma-vis/diag-demo-files/ already and are currently being used by the "production" app. I expect that'll change when we update the backend to use the new Zarr array from the data pipeline in the new s3 bucket. Feel free to put the Zarr anywhere else that makes sense to you!

...if that sounds too messy, I could have VLab create something like s3://osti-modeling-dev-rtma-vis-dev and s3://osti-modeling-dev-rtma-vis-prod and retire the current one so it's explicit which bucket is which.

[esheehan-gsl]

For now, I think we’re fine. I can give a separate key to the Zarr files that we use for testing and production, if we need to separate them.

Do you think it would be better to have completely separate buckets (perhaps for managing permissions?) for dev and prod, or do we just use keys to separate the two? We could just create environment prefixes for our keys like prod/, dev/, and test/ in one bucket, if you like. I don’t think I have a problem with that.