Open
Description
The Feature
Currently, the include block works fine when using the LFS to access the config and can look up the included files. However, when using an S3/GCS bucket to load the config, the include is not considered.
What should happen:
- It should make consequent requests to fetch the included keys/files from the bucket
Proposal:
- Use the bucket name and path specified in the config to download all the files in the include, and create a final config object. When using
include, the path should be relative to the bucket i.e. bucket key.
Motivation, pitch
We use an S3 bucket to load our config as it allows us to not rebuild Docker images again and again. Given we have 20+ models in our config, we wanted to leverage include, which unfortunately doesn't work as expected and we end up not loading any models.
Are you a ML Ops Team?
No