Conversation
21649c0 to
4564f6a
Compare
- Introduced `S3RosbagBackupAgent` for managing S3 backups of
rosbag recordings
- Use `multiprocessing.Process` to run a parallel
process `_backup_routine()` to periodically check
a queue for ROSbag directories to back up
- The module is designed to be used as a non-blocking operation,
thus the need of a queue in case multiple directories are added
at once or if another upload is in progress.
- Defined `enqueue()` method to add new directories to the queue
and trigger the backup process routine if necessary.
- Support standalone execution via `main()` function which parses
command line arguments for configuration and run the agent.
- Accept a list of ROSbag recordings with `--recordings_list` arg
- Accept a `--debug` flag to enable debug logging
- Accept a `--dryrun` flag to simulate the backup process
without uploading files
- Use `boto3` `TransferConfig` to set multipart upload threshold
to allow ROS bag splits larger than 10GB to be uploaded to S3.
- Define a default threshold of 3 GB and use up to 4 parallel workers
- Change `boto3.resource('s3')` to `boto3.client('s3')`
to allow for multipart uploads with `TransferConfig`.
- Set TODO comments with issue number for future reference.
4564f6a to
108b763
Compare
Member
GreatAlexander
left a comment
There was a problem hiding this comment.
Looks good! The tiniest of comments plz! 🙏 🚀
GreatAlexander
approved these changes
Jul 15, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
S3RosbagBackupAgentfor managing S3 backups ofrosbag recordings
multiprocessing.Processto run a parallelprocess
_backup_routine()to periodically checka queue for ROSbag directories to back up
thus the need for a queue in case multiple directories are added
at once, or if another upload is in progress.
enqueue()method to add new directories to the queueand trigger the backup process routine if necessary.
main()function which parsescommand line arguments for configuration and running the agent.
--recordings_listarg--debugflag to enable debug logging--dryrunflag to simulate the backup processwithout uploading files
s3_backup_agenttopyproject.tomlto enable pip installations3_backup_agentcommandAllow files 5GB+ to be uploaded to S3
boto3TransferConfigto set multipart upload thresholdto allow ROS bag splits larger than 5GB to be uploaded to S3.
boto3.resource('s3')toboto3.client('s3')to allow for multipart uploads with
TransferConfig.Add the S3 backup agent to the DataManager pipeline