Open
Description
What I need help with / What I was wondering
I want to read multiple s3 bucket within a single process.
Here is psudo code:
```
import tensorflow_datasets as tfds
import tensorflow_io as tfio
os.environ["AWS_ACCESS_KEY_ID"] = "xxxxxxxxx"
os.environ["AWS_SECRET_ACCESS_KEY"] = "xxxxxxxxxx"
# os.environ["AWS_REGION"] = "us-east-1"
os.environ["S3_ENDPOINT"] = "http://ip1:80"
os.environ["S3_USE_HTTPS"] = "0"
os.environ["S3_VERIFY_SSL"] = "0"
builder = tfds.builder_from_directory(builder_dir='s3://open_x_embodiment_origin/cmu_stretch/0.1.0/')
os.environ["AWS_ACCESS_KEY_ID"] = "yyyyy"
os.environ["AWS_SECRET_ACCESS_KEY"] = "yyyyyyyyyy"
# os.environ["AWS_REGION"] = "us-east-1"
os.environ["S3_ENDPOINT"] = "http://ip2:80"
builder = tfds.builder_from_directory(builder_dir='s3://droid/1.0.0/')
```
I can only get the first bucket. When I builder from the second bucket, I fails with the message: FileNotFoundError: Could not load dataset info from s3://droid/1.0.0/dataset_info.json.
It seems like tfds can not read two buckets with different access keys. Is it right? or I should do something to connect two different buckets.
Best,
Looking forward to your reply.
Activity