Adding background services to BRAND

[yahiaali]

Overview

This proposal introduces the concept of "services" (or background graphs) to BRAND. Services are specialized graphs that run perpetually in the background until explicitly stopped. Unlike standard graphs that are meant to run for a limited amount of time, services are designed for continuous operation alongside other graphs or services.

Motivation

Current BRAND architecture limits users to running a single graph at a time. However, there are several use cases where continuous background processes would be valuable:

• Personal use interfaces
• Automated model training
• Updating normalization parameters
• System monitoring
• Communication with external systems

Implementation

Service Definition

Services would be defined using the existing YAML graph format. This maintains compatibility with the current configuration system while clearly identifying graphs intended to run as services.

nodes:
  - name: monitor_node
    nickname: monitor
    module: ../brand-modules/monitoring
    parameters:
      interval: 5000
      log: INFO
  # Additional nodes...

Supervisor Commands

Two new commands would be added to the supervisor:

1. startService [file <path_to_file>] [service <service_json>]: Start a service from a YAML file path or JSON string
2. stopService [nickname <service_nickname>]: Stop a running service by its nickname

Architectural Changes

The implementation requires several changes to the supervisor:

1. Support for multiple concurrent graph/service execution
2. Service registry to track and manage running services
3. Modification of the termination logic to allow selective shutdown of nodes

Example Usage

# Start a monitoring service
XADD supervisor_ipstream * commands startService file services/system_monitor.yaml

# Start another service for external API communication
XADD supervisor_ipstream * commands startService file services/api_interface.yaml

# Run a regular experiment graph alongside services
XADD supervisor_ipstream * commands startGraph file graphs/experiment/task1.yaml

# Stop a specific service
XADD supervisor_ipstream * commands stopService nickname system_monitor

Technical Considerations

1. Error Handling: Service crashes should not affect other running services or graphs
2. Memory management: Nodes running within services should cap stream length to avoid accumulating memory
3. Data saving: Some data export derivatives assume that all data in Redis belongs to the current block. If services produce data that persists across blocks, then data export derivatives will need to account for that.
4. Monitoring: Supervisor should track and report the health of services

Backward Compatibility

The introduction of services will not affect existing graphs. Current commands like startGraph and stopGraph will continue to work as before, with stopGraph stopping only the main experimental graph, not any running services.