Skip to content

Add ROS bag backup support#66

Merged
hect95 merged 5 commits intomainfrom
add-s3-backup-agent
Jul 15, 2025
Merged

Add ROS bag backup support#66
hect95 merged 5 commits intomainfrom
add-s3-backup-agent

Conversation

@hect95
Copy link
Collaborator

@hect95 hect95 commented Jul 1, 2025

  • Introduced S3RosbagBackupAgent for managing S3 backups of
    rosbag recordings
    • Use multiprocessing.Process to run a parallel
      process _backup_routine() to periodically check
      a queue for ROSbag directories to back up
      • The module is designed to be used as a non-blocking operation,
        thus the need for a queue in case multiple directories are added
        at once, or if another upload is in progress.
    • Defined enqueue() method to add new directories to the queue
      and trigger the backup process routine if necessary.
    • Support standalone execution via main() function which parses
      command line arguments for configuration and running the agent.
      • Accept a list of ROSbag recordings with --recordings_list arg
      • Accept a --debug flag to enable debug logging
      • Accept a --dryrun flag to simulate the backup process
        without uploading files
    • Add s3_backup_agent to pyproject.toml to enable pip installation
      • Define a s3_backup_agent command

Allow files 5GB+ to be uploaded to S3

  • Use boto3 TransferConfig to set multipart upload threshold
    to allow ROS bag splits larger than 5GB to be uploaded to S3.
    • Define a default threshold of 3 GB and use up to 4 parallel workers
  • Change boto3.resource('s3') to boto3.client('s3')
    to allow for multipart uploads with TransferConfig.
  • Set TODO comments with issue number for future reference.

Add the S3 backup agent to the DataManager pipeline

@hect95 hect95 self-assigned this Jul 1, 2025
@hect95 hect95 changed the title WIP: ROS bag backup agent Add ROS bag backup support Jul 1, 2025
@mergify mergify bot requested a review from GreatAlexander July 1, 2025 20:27
@hect95 hect95 force-pushed the add-s3-backup-agent branch 2 times, most recently from 21649c0 to 4564f6a Compare July 4, 2025 18:24
hect95 added 4 commits July 7, 2025 13:25
- Introduced `S3RosbagBackupAgent` for managing S3 backups of
  rosbag recordings
  - Use `multiprocessing.Process` to run a parallel
    process `_backup_routine()` to periodically check
    a queue for ROSbag directories to back up
    - The module is designed to be used as a non-blocking operation,
      thus the need of a queue in case multiple directories are added
      at once or if another upload is in progress.
  - Defined `enqueue()` method to add new directories to the queue
    and trigger the backup process routine if necessary.
  - Support standalone execution via `main()` function which parses
    command line arguments for configuration and run the agent.
    - Accept a list of ROSbag recordings with `--recordings_list` arg
    - Accept a `--debug` flag to enable debug logging
    - Accept a `--dryrun` flag to simulate the backup process
      without uploading files
- Use `boto3` `TransferConfig` to set multipart upload threshold
  to allow ROS bag splits larger than 10GB to be uploaded to S3.
  - Define a default threshold of 3 GB and use up to 4 parallel workers
- Change `boto3.resource('s3')` to `boto3.client('s3')`
  to allow for multipart uploads with `TransferConfig`.
- Set TODO comments with issue number for future reference.
@hect95 hect95 force-pushed the add-s3-backup-agent branch from 4564f6a to 108b763 Compare July 7, 2025 12:25
@hect95 hect95 marked this pull request as ready for review July 7, 2025 12:56
Copy link
Member

@GreatAlexander GreatAlexander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! The tiniest of comments plz! 🙏 🚀

@hect95 hect95 requested a review from GreatAlexander July 14, 2025 16:14
@hect95 hect95 merged commit b653b8a into main Jul 15, 2025
3 checks passed
@hect95 hect95 deleted the add-s3-backup-agent branch July 15, 2025 18:44
@hect95 hect95 linked an issue Aug 13, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement ROS bag backup agent

2 participants