Drain service attempts to create bucket that already exists

[kralicky]

It appears that the drain service is attempting to create a bucket that already exists. Here are the relevant logs:

2023-05-26 17:23:49,669 - INFO - Connected to S3 client
2023-05-26 17:23:49,683 - INFO - opni-drain-model bucket does not exist so creating it now
Traceback (most recent call last):
  File "/app/./drain_modules.py", line 11, in <module>
    drain_training_inferencing.main()
  File "/app/drain_training_inferencing.py", line 377, in main
    persistence = FilePersistence("workload_drain_model.bin")
  File "/app/drain3/file_persistence.py", line 26, in __init__
    self.connect_to_s3()
  File "/app/drain3/file_persistence.py", line 56, in connect_to_s3
    self.s3_client.create_bucket(Bucket=S3_BUCKET)
  File "/opt/venv/lib64/python3.9/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/opt/venv/lib64/python3.9/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/opt/venv/lib64/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/venv/lib64/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket name can not be an existing collection, and the bucket namespace is shared by all users of the system. Please select a different name and try again.

Restarting the pods does not fix the issue.

[AmartC]

This appears to be an issue with Seaweed when nodes are rebooted. The plan is once the Python nats wrapper is updated to include object storage, then that will be used in place of Seaweed.