PBM-1776 GCS parallel upload (with storage close)#1344
Conversation
jcechace
changed the title
| // HMAC credentials continue to use the MinIO client and are not affected by this setting. | ||
| type ParallelUpload struct { | ||
| // Enabled enables Writer parallel uploads when clientType is grpc. | ||
| Enabled bool `bson:"enabled,omitempty" json:"enabled,omitempty" yaml:"enabled,omitempty"` |
There was a problem hiding this comment.
This is redundant parameter, we can disable/enable it with MaxConcurrency
|
|
||
| // PartSize is the size of each temporary part in bytes. | ||
| // If unset, PBM uses the computed ChunkSize for the uploaded object. | ||
| PartSize int `bson:"partSize,omitempty" json:"partSize,omitempty" yaml:"partSize,omitempty"` |
There was a problem hiding this comment.
ChunkSize can be reused here, or maybe even better use PartSize for both (json and grpc) but in that case it should be out of ParallelUpload block.
| if pu.PartSize > 0 { | ||
| partSize = fmt.Sprintf("%v (%v)", pu.PartSize, storage.PrettySize(int64(pu.PartSize))) |
There was a problem hiding this comment.
storage.ComputePartSize should be used for grps also
|
|
||
| stopHB chan struct{} | ||
| stopCleanupHB chan struct{} | ||
| hbWG sync.WaitGroup |
There was a problem hiding this comment.
Rather create local storage instance just for the HB and close it when HB is stopped. So, that we do not need to do this type of synchronization.
| storage storage.Storage | ||
| } | ||
|
|
||
| type storageReadCloser struct { |
There was a problem hiding this comment.
It seems that it necessary to refactor logical restore to use single storage instance. This is workaround that's hard to follow, and I am not sure that it closes all flows.
There was a problem hiding this comment.
That would be basically reverting https://github.com/percona/percona-backup-mongodb/pull/703/changes
I think the reason why each file get's it's own storage is shared state in Downloaders. We would be sharing this field arenas []*storage.Arena
f390fbf to
fb06a23
Compare
2 participants
Status will remain experimental due to ongoing work in GCS.
At least these need to be released: