Skip to content

MPROCS_QUICK_REFERENCE.md

Latest commit

 

History

History
140 lines (105 loc) · 5.34 KB

MPROCS_QUICK_REFERENCE.md

File metadata and controls

140 lines (105 loc) · 5.34 KB

Quick Reference: mprocs Support in Breeze

Basic Command

Start airflow uses mprocs by default:

breeze start-airflow

You can switch to using tmux instead by specifying the option:

breeze start-airflow --terminal-multiplexer tmux

Once you select the terminal multiplexer, Breeze will remember your choice for future runs.

You can also switch terminal multiplexers by running breeze setup config:

breeze setup config --terminal-multiplexer mprocs
breeze setup config --terminal-multiplexer tmux

When Breeze will start Airflow components using mprocs to manage multiple processes in a single terminal window. It will generate dynamically the required configuration based on the selected executor and options and use it, the generated configuration file is stored in a files folder inside the container, so that you can also use it outside of Breeze if needed.

Common Usage Patterns

Command Description
breeze start-airflow --terminal-multiplexer mprocs Start Airflow with mprocs (and remember the choice)
breeze start-airflow --terminal-multiplexer tmux Start Airflow with tmux (and remember the choice)
breeze start-airflow --debug scheduler Debug scheduler with last selected multiplexer
breeze start-airflow --dev-mode --terminal-multiplexer mprocs Use mprocs in dev mode (and remember the choice)

mprocs Keyboard Shortcuts

Key Action
↑↓ Navigate between processes
r Restart selected process
x Stop selected process
s Start selected process
a Add new process
q Quit mprocs
? Show help

Components Managed

  • scheduler - Airflow scheduler
  • api_server (3.x+) / webserver (2.x) - Web interface
  • triggerer - Handles deferred tasks
  • dag_processor - Standalone DAG processor (when enabled)
  • celery_worker - Celery worker (with CeleryExecutor)
  • flower - Celery monitoring (when enabled)
  • edge_worker - Edge worker (with EdgeExecutor)

Environment Variables

Variable Purpose
TERMNAL_MULTIPLEXER Use mprocs when set to "mprocs"
INTEGRATION_CELERY Enable Celery components
CELERY_FLOWER Enable Flower UI
STANDALONE_DAG_PROCESSOR Enable standalone DAG processor
BREEZE_DEBUG_* Enable component debugging
DEV_MODE Enable development mode

Debug Ports (when debugging enabled)

Component Port
Scheduler 50231
Dag Processor 50232
Triggerer 50233
API Server 50234
Celery Worker 50235
Edge Worker 50236
Web Server 50237

Installation

mprocs is pre-installed in the Breeze CI image as of Airflow 3.x development. No additional setup is required.

For older images or custom setups, you can install mprocs manually:

# Download and install mprocs
MPROCS_VERSION="0.7.3"
SYSTEM=$(uname -s | tr '[:upper:]' '[:lower:]')
PLATFORM=$([ "$(uname -m)" = "aarch64" ] && echo "arm64" || echo "amd64")
curl -L "https://github.com/pvolok/mprocs/releases/download/v${MPROCS_VERSION}/mprocs-${MPROCS_VERSION}-${SYSTEM}-${PLATFORM}.tar.gz" \
  | tar -xz -C /usr/local/bin/ mprocs
chmod +x /usr/local/bin/mprocs

Mac OS X and iTerm2

Mouse clicks are not captured correctly by default in iTerm2 (which is often used by developers on MacOS). You need to configure "Enable Mouse reporting" to take advantage of the copying feature and mouse handling:

Enable mouse reporting

Standalone execution

You can run mprocs outside of Breeze for custom setups. Create a mprocs.yaml configuration file defining your processes, then start mprocs:

mprocs -f mprocs.yaml

An example mprocs.yaml file for Airflow components can be found in this directory - it has default configurations for all major Airflow components, but you can customize it as needed by copying it elsewhere and modifying the process definitions, uncommenting the commented or adding new processes.