Bagman

Bagman is a ROS 2 bag (.mcap) management tool.

Table of Contents

• Features
• Getting started
  • Prerequisites
  • Run Dashboard
  • Run CLI
  • Database Integration
• Contributing

Features

• CLI Tool: includes administrative commands for managing recordings
• NoSQL Database Support: compatible with MongoDB, Elasticsearch, and TinyDB
• Streamlit Dashboard: interactively search for recordings and explore their content
• Prefect Pipeline (in development): define and execute workflows by selecting tasks and applying them to recordings

Getting started

Prerequisites

• Docker (apt install docker.io)
• Docker Compose (apt install docker-compose)
• yq (snap install yq)

Clone the repository:

git clone https://github.com/yannikmotzet/bagman.git && cd bagman

Run Dashboard

1. Build the Docker image:

   docker build -t bagman .

2. Set environment variables for docker-compose:

   echo "RECORDINGS_STORAGE=$(yq '.recordings_storage' config.yaml)" > .env
   echo "DASHBOARD_PORT=$(yq '.dashboard_port' config.yaml)" >> .env
   echo "CONFIG_PATH=config.yaml" >> .env

3. Start the application:

   docker-compose up -d

4. Open Dashboard in browser: localhost:8502

Run CLI

1. Install the package:

    pip install .

   Note: For development use pip install -e . which creates a symbolic link to the source code.

2. Execute the CLI:

   bagman

   bagman CLI
   
   positional arguments:
   {upload,download,add,update,delete,remove,exist,connection,metadata,map,video}
       upload              upload local recording to storage (optional: add to database)
       download            download a recording from storage to local machine
       add                 add a recording to database or update existing one
       update              update an existing recording in database
       delete              delete a recording from storage (optional: remove from database)
       remove              remove a recording from database
       exist               check if recording exists in storage and database
       connection          check connection to the storage and database
       metadata            (re)generate metadata file for a local recording
       map                 generate a map plot from GNSS data in the recording
       video               generate a video file from the camera data in the recording
   
   options:
     -h, --help            show this help message and exit
     -c CONFIG, --config CONFIG
                           path to config file, default: config.yaml in current directory
   

User Authentication

To enable user authentication for the dashboard, create a YAML configuration file as described in the Streamlit-Authenticator documentation. Save this file and reference it in your config.yaml under the dash_auth_file field. In the dash_auth_pages field you can specify the pages that require authentication.

Database Integration

Currently, the following databases are supported:

• MongoDB
• Elasticsearch
• TinyDB (based on .json file)

Database integration is managed through the config.yaml file and environment variables. The database_uri field specifies the connection details for the selected database, while authentication credentials can be provided via a .env file.

TinyDB

For TinyDB, set the database_uri field in config.yaml to the path of the .json file:

database_uri: path/to/database.json

Elasticsearch

For Elasticsearch, set the database_uri field in config.yaml to the URL of the database:

database_uri: http://localhost:9200

Additionally, specify the database_name field in config.yaml to define the Elasticsearch index name:

database_name: your_index_name

If authentication is required, add the credentials to a .env file:

For username/password authentication:

DATABASE_USER=your_username
DATABASE_PASSWORD=your_password

For API key authentication, use a Base64-encoded key:

DATABASE_TOKEN=your_base64_encoded_api_key

Note: Use either username/password or API key, but not both.

MongoDB

For MongoDB, set the database_uri field in config.yaml to the connection URL:

database_uri: mongodb://localhost:27017

Additionally, specify the database_name field in config.yaml to define both the MongoDB database name and the collection name:

database_name: your_collection_name

If authentication is required, add the credentials to a .env file:

DATABASE_USER=your_username
DATABASE_PASSWORD=your_password

Notes

• Ensure the .env file is located in the same directory as your application.
• The application will automatically load the environment variables from the .env file during runtime.
• Using username and password inside the database URL is not recommended for security reasons.
• For more details, refer to the documentation of the respective database.

Prefect Integration

To integrate Prefect 3 with Bagman, follow these steps:

1. Start the Prefect server:

   prefect server start

2. Create a deployment:

   prefect deploy

   This command will automatically detect the pipeline/flow_bagman.py:flow_default flow. Assign it a name, such as default. When prompted to create a work pool, select process as the type and provide a name, e.g., work_pool.

3. Start the work pool:

   prefect worker start --pool "work_pool"

4. Add the PREFECT_API_URL environment variable (is printed in terminal when starting Prefect server) under the prefect_api_url field in config.yaml to enable the pipeline dashboard page to interact with the Prefect server.

For additional details, refer to the Prefect documentation.

Contributing

Use pre-commit:

1. Install pre-commit:

   pip install pre-commit

2. Run pre-commit:

   pre-commit