Upsun Metrics Collection System

A complete metrics collection and visualization system for Upsun projects using InfluxDB (managed service) and Grafana (self-hosted).

Architecture

┌─────────────────┐
│  Cron Job       │
│  (every 5 min)  │
└────────┬────────┘
         │
         │ runs
         ▼
┌─────────────────────┐
│ collect_metrics.py  │
│                     │
│ - Executes SQL      │
│ - Parses results    │
│ - Writes to InfluxDB│
└─────────┬───────────┘
          │
          │ writes
          ▼
    ┌──────────────┐
    │  InfluxDB    │
    │  (managed)   │
    └──────▲───────┘
           │
           │ queries
           │
    ┌──────────────┐
    │   Grafana    │
    │ (self-hosted)│
    └──────────────┘

Features

✅ Automated metric collection via cron
✅ Uses Upsun CLI (or OEM versions: Pimcore, Ibexa, Shopware)
✅ API token authentication (no interactive login needed)
✅ InfluxDB time-series storage (managed Upsun service)
✅ Self-hosted Grafana using composable image
✅ Pre-configured Grafana dashboards with Flux queries
✅ Extensible metric definitions
✅ Automatic collection duration tracking for performance monitoring
✅ 30-day data retention

Setup

1. Configure API Token

Set your Upsun CLI token as a project-level environment variable (not committed to code):

# Using Upsun CLI
upsun variable:create \
  --level project \
  --name env:UPSUN_CLI_TOKEN \
  --value "your-token-here" \
  --sensitive=true \
  --visible-build=false \
  --visible-runtime=true

# Or via Upsun Console:
# Project Settings → Variables → Add Variable
# Name: env:UPSUN_CLI_TOKEN
# Value: your-api-token-here
# Sensitive: ✓
# Available during build: ✗
# Available at runtime: ✓

To generate an API token:

upsun auth:api-token-login
# Or for OEM CLIs:
pimcore auth:api-token-login

2. Configure Projects and Metrics

Edit metrics-config.yaml to define your projects and metrics:

projects:
  - name: "pimcore-production"
    cli: "pimcore-cloud"
    project_id: "your-project-id"
    environment: "main"
    enabled: true

    metrics:
      # SQL query metric
      - name: "queue_size"
        type: "sql"
        app_name: "pimcore"  # Required: which container to run this on
        description: "Queue size"
        query: "SELECT COUNT(*) FROM queue_table;"
        labels:
          table: "queue_table"

      # SSH command metric
      - name: "log_files_count"
        type: "ssh"
        app_name: "pimcore"  # Required: which container to run this on
        description: "Number of log files"
        command: "ls -1 /var/log/*.log 2>/dev/null | wc -l"
        labels:
          directory: "var_log"

      # Metric with custom unit formatting
      - name: "disk_usage_bytes"
        type: "ssh"
        app_name: "pimcore"
        description: "Disk usage"
        unit: "bytes"  # Optional: formats as B, KB, MB, GB, TB
        command: "du -sb /app | awk '{print $1}'"

Metric Types:

sql: Execute SQL queries via <cli> sql command
ssh: Execute any shell command via <cli> ssh command
php_fpm_status: Query the PHP-FPM status page over FastCGI (built-in; see PHP-FPM Worker Monitoring)

Required Fields per Metric:

name - Metric identifier (used in InfluxDB)
type - sql, ssh, or php_fpm_status
app_name - Container name to run the command on (required)
description - Human-readable description
query or command - Depending on type (not needed for php_fpm_status)

Optional Fields:

worker - Worker name (for worker containers, adds --worker <name> flag)
labels - Custom tags for InfluxDB
unit - Grafana unit for display (e.g., bytes, percent, ms, s). Defaults to short (raw number). See Grafana units for all options.
outputs - Emit several measurements from one command run. Each entry has a measurement (series name), a match (substring selecting a line of the output), and optional unit/description. The value is the last number on the first matching line. Ideal when one expensive command returns many figures (e.g. aws s3 ls --summarize, or a PHP-FPM status page).
long_running - If true, the metric is collected only by the slow cron (--group slow), keeping multi-minute commands off the frequent 5‑minute cron.
interval_seconds - Minimum seconds between successful collections. The slow cron can fire often (e.g. hourly) while this gates real work to, say, every 6h; a failed run is retried sooner. Backed by a small state file on the /data mount, with a per-metric lock so a still-running collection is never started twice.
timeout_seconds - Local timeout backstop in seconds (default 60). For long commands, keep it above any remote timeout wrapper in the command itself.
dashboard_only - No collection; only render a panel. Handy for the auto-generated <metric>_collection_duration series.

Collection cadence (fast vs. slow): The cron runs the collector with --group fast (every 5 min, skips long_running metrics) and --group slow (infrequent, only long_running metrics). Without --group, all metrics run. See .upsun/config.yaml for the two cron entries.

Multiple Containers per Project: To monitor different containers within the same project:

projects:
  - name: "MyProject"
    cli: "pimcore-cloud"
    project_id: "abc123"
    environment: "main"
    enabled: true

    metrics:
      # Main container metric
      - name: "main_queue_size"
        type: "sql"
        app_name: "myapp"
        query: "SELECT COUNT(*) FROM queue;"
        labels:
          container: "main"

      # Worker container metric
      # For workers: use --app <app> --worker <worker_name>
      - name: "worker_errors"
        type: "ssh"
        app_name: "myapp"
        worker: "data_hub"  # Translates to: --app myapp --worker data_hub
        command: "grep -c ERROR /var/log/app.log || echo 0"
        labels:
          container: "worker"

All metrics for a project appear in one dashboard, with all environments visible as separate series on the same graphs.

Benefits:

Support multiple projects and environments
Compare metrics across environments (dev, main, stage) on the same graph
Each project can have different metrics
Mix SQL and SSH metrics
Enable/disable projects individually
One dashboard per project showing all environments

3. Generate Dashboards

After configuring metrics, generate Grafana dashboards:

# Locally (recommended)
python3 -m pip install --user pyyaml  # one-time
python3 generate_dashboards.py

# Commit the generated dashboards
git add grafana/dashboards/
git commit -m "Add/update dashboards"
git push

This creates one dashboard per project:

grafana/dashboards/
├── My Pimcore Project.json   # Environment: main
└── Another Project.json      # All environments: dev, main

Dashboard Organization:

One dashboard per project
All environments for a project appear on the same graphs as separate series
This allows easy comparison between dev, main, and stage environments
Dashboard queries don't filter by environment, showing all data together

Dashboard Updates:

New dashboards appear automatically within 60 seconds of deployment
UI customizations (panel sizes, time ranges) persist until you regenerate
Regenerating dashboards with generate_dashboards.py will overwrite UI changes
For heavy customization, save a copy via UI: Dashboard → Save as copy

Adding New Projects or Environments:

Add the project/environment configuration to metrics-config.yaml
Run python3 generate_dashboards.py to update the dashboards
The script automatically:
- Groups all environments by project name
- Creates/updates one dashboard per unique project name
- Includes all environments as tags for filtering
- Metrics from all environments appear on the same graphs

4. Deploy to Upsun

git add .
git commit -m "Add metrics collection system"
git push upsun main

5. Access Grafana

After deployment:

Grafana: https://grafana.{your-project-url}/
- Default credentials: admin/admin
- InfluxDB datasource is pre-configured
- Each project has one dashboard showing all environments

Security: Grafana is publicly accessible. Change the default credentials immediately after first login, or set them upfront via Upsun environment variables (these take priority over the defaults in .grafana.env):
upsun variable:create \
  --level project \
  --name env:GF_SECURITY_ADMIN_USER \
  --value "your-admin-username" \
  --visible-build=false \
  --visible-runtime=true

upsun variable:create \
  --level project \
  --name env:GF_SECURITY_ADMIN_PASSWORD \
  --value "your-strong-password" \
  --sensitive=true \
  --visible-build=false \
  --visible-runtime=true

Services Used

InfluxDB (Managed Service)

Type: influxdb:2.7
Purpose: Time-series database for storing metrics
Retention: 30 days
Organization: upsun
Bucket: metrics

Grafana (Self-Hosted)

Type: composable:2.4
Purpose: Visualization and dashboards
Port: 8888
Database: SQLite (persistent mount)

Metric Types & Examples

SQL Metrics

Execute database queries to collect metrics:

- name: "pending_orders"
  type: "sql"
  description: "Number of pending orders"
  query: "SELECT COUNT(*) FROM orders WHERE status = 'pending';"

- name: "average_order_value"
  type: "sql"
  description: "Average order value today"
  query: "SELECT AVG(total) FROM orders WHERE created_at > CURRENT_DATE;"

SSH Metrics

Execute any shell command to collect metrics:

# Count files in a directory
- name: "cache_files_count"
  type: "ssh"
  description: "Number of cache files"
  command: "find /app/var/cache -type f | wc -l"

# Count running processes
- name: "php_processes"
  type: "ssh"
  description: "Number of PHP-FPM processes"
  command: "ps aux | grep php-fpm | grep -v grep | wc -l"

# Memory usage (in MB)
- name: "memory_usage_mb"
  type: "ssh"
  description: "Memory usage in MB"
  command: "free -m | awk 'NR==2 {print $3}'"

# Count log entries by severity and date
- name: "errors_today"
  type: "ssh"
  description: "ERROR entries in app.log today"
  command: |
    grep -c "^\[$(date +%Y-%m-%d).*\.ERROR:" /var/log/app.log || echo 0

- name: "warnings_today"
  type: "ssh"
  description: "WARNING entries in app.log today"
  command: |
    grep -c "^\[$(date +%Y-%m-%d).*\.WARNING:" /var/log/app.log || echo 0

Multiple Outputs from One Command

Some commands return several figures at once. Instead of running the command once per figure, run it once and split the output into several measurements with outputs — each picks a line by match and takes the last number on it:

# One `aws s3 ls --summarize` listing -> object count AND total size.
# Marked long_running so it runs on the slow cron, gated to ~every 6h.
- name: "s3_stats"
  type: "ssh"
  app_name: "app"
  long_running: true
  interval_seconds: 21000
  timeout_seconds: 3660
  description: "S3 bucket stats (objects + size)"
  command: |
    AWS_ACCESS_KEY_ID='...' AWS_SECRET_ACCESS_KEY='...' AWS_DEFAULT_REGION='...' \
    timeout -k 30 3600 aws s3 ls s3://default/ --endpoint-url http://s3.internal:8080 \
    --recursive --summarize | grep -E "Total Objects|Total Size"
  labels:
    source: "s3"
  outputs:
    - measurement: "s3_total_objects"
      match: "Total Objects"
      description: "Total number of objects in S3 storage"
    - measurement: "s3_total_size_bytes"
      match: "Total Size"
      unit: "bytes"
      description: "Total size of objects in S3 storage"

A single <metric>_collection_duration series is recorded per run (not one per output). Failed/timed-out collections are skipped (never recorded as 0).

PHP-FPM Worker Monitoring

The php_fpm_status type reads the PHP-FPM status page (active/idle/total workers, listen queue, slow requests, etc.) by speaking FastCGI directly to the FPM unix socket on the target container. The FastCGI client is built into the collector, so the config stays short — just declare the outputs you want.

Requirements on the target: python3 available, and PHP-FPM configured with pm.status_path (default /-/status). Optional overrides: socket (default /run/app.sock) and status_path.

- name: "php_fpm_status"
  type: "php_fpm_status"
  app_name: "app"
  description: "PHP-FPM worker pool status"
  # socket: "/run/app.sock"       # optional
  # status_path: "/-/status"      # optional
  labels:
    source: "php-fpm"
  outputs:
    - measurement: "php_fpm_active_processes"     # workers in use
      match: "active processes:"
    - measurement: "php_fpm_idle_processes"
      match: "idle processes:"
    - measurement: "php_fpm_total_processes"
      match: "total processes:"
    - measurement: "php_fpm_listen_queue"         # requests waiting (saturation)
      match: "listen queue:"
    - measurement: "php_fpm_slow_requests"
      match: "slow requests:"
    # ...also available: max active processes, max listen queue,
    #    listen queue len, max children reached, accepted conn, start since

See metrics-config.yaml for the full 11-field example.

Filesystem Monitoring

Monitor the size of directories on your application container using du. The -s flag summarizes the total for the given path (instead of listing each subdirectory), and -b outputs the size as a raw byte count — required because InfluxDB expects a plain integer, not a human-readable string like 1.2G. Grafana then auto-scales the value to KB/MB/GB using the bytes unit.

# Size of a specific directory in bytes
- name: "var_tmp_size_bytes"
  type: "ssh"
  app_name: "myapp"
  description: "Size of /app/var/tmp directory"
  unit: "bytes"
  command: "du -sb /app/var/tmp | awk '{print $1}'"
  labels:
    path: "/app/var/tmp"

# Size of the full application directory
- name: "app_size_bytes"
  type: "ssh"
  app_name: "myapp"
  description: "Total size of /app"
  unit: "bytes"
  command: "du -sb /app | awk '{print $1}'"
  labels:
    path: "/app"

# Available disk space on the filesystem
- name: "disk_available_bytes"
  type: "ssh"
  app_name: "myapp"
  description: "Available disk space"
  unit: "bytes"
  command: "df /app | tail -1 | awk '{print $4 * 1024}'"

RabbitMQ Queue Monitoring

Monitor RabbitMQ queues using the Management HTTP API. Note: The Management API port (15672) is not exposed in the relationship, so it must be hardcoded.

# Total messages across all active queues (excluding failed)
- name: "rabbitmq_active_queues_total"
  type: "ssh"
  description: "Total messages in active queues (excluding failed)"
  command: |
    RABBITMQ_URL=$(echo $PLATFORM_RELATIONSHIPS | base64 -d | jq -r '.queue[0] | "http://\(.username):\(.password)@\(.host):15672"')
    curl -s "$RABBITMQ_URL/api/queues" | jq '[.[] | select(.name != "failed") | .messages] | add // 0'
  labels:
    source: "rabbitmq"

# Ready messages across all active queues
- name: "rabbitmq_active_queues_ready"
  type: "ssh"
  description: "Ready messages in active queues (excluding failed)"
  command: |
    RABBITMQ_URL=$(echo $PLATFORM_RELATIONSHIPS | base64 -d | jq -r '.queue[0] | "http://\(.username):\(.password)@\(.host):15672"')
    curl -s "$RABBITMQ_URL/api/queues" | jq '[.[] | select(.name != "failed") | .messages_ready] | add // 0'
  labels:
    source: "rabbitmq"

# Number of active connections (reliable alternative to consumer count —
# some PHP AMQP clients use basic.get instead of basic.consume, which means
# RabbitMQ never registers them as consumers even though messages are being processed)
- name: "rabbitmq_connections"
  type: "ssh"
  description: "Number of active RabbitMQ connections"
  command: |
    RABBITMQ_URL=$(echo $PLATFORM_RELATIONSHIPS | base64 -d | jq -r '.queue[0] | "http://\(.username):\(.password)@\(.host):15672"')
    curl -s "$RABBITMQ_URL/api/connections" | jq 'length'
  labels:
    source: "rabbitmq"

# Message delivery rate across all queues (reflects actual worker throughput)
- name: "rabbitmq_deliver_rate"
  type: "ssh"
  description: "Message delivery rate across all queues (msgs/s)"
  command: |
    RABBITMQ_URL=$(echo $PLATFORM_RELATIONSHIPS | base64 -d | jq -r '.queue[0] | "http://\(.username):\(.password)@\(.host):15672"')
    curl -s "$RABBITMQ_URL/api/queues" | jq '[.[] | .message_stats.deliver_get_details.rate // 0] | add'
  labels:
    source: "rabbitmq"

# Number of queues
- name: "rabbitmq_queue_count"
  type: "ssh"
  description: "Number of RabbitMQ queues"
  command: |
    RABBITMQ_URL=$(echo $PLATFORM_RELATIONSHIPS | base64 -d | jq -r '.queue[0] | "http://\(.username):\(.password)@\(.host):15672"')
    curl -s "$RABBITMQ_URL/api/queues" | jq '. | length'
  labels:
    source: "rabbitmq"

Filtering queues:

# Exclude specific queue by name
select(.name != "failed")

# Exclude queues containing a pattern
select(.name | contains("failed") | not)

# Exclude multiple queues
select(.name != "failed" and .name != "dead_letter")

Available RabbitMQ metrics:

messages - Total messages in queue
messages_ready - Messages ready to be delivered
messages_unacknowledged - Messages delivered but not acknowledged
consumers - Number of active consumers
memory - Queue memory usage in bytes

Note: Port 15672 is the standard RabbitMQ Management API port. Adjust the relationship key (.queue[0]) to match your actual relationship name.

General Notes

SSH commands should output a single numeric value. The parser extracts the first number it finds.

Collection Duration Tracking

Automatic Performance Monitoring:

Every metric automatically tracks how long it takes to collect, and this duration is written to InfluxDB as a separate metric named {metric_name}_collection_duration.

How it works:

No configuration needed - duration is tracked automatically for ALL metrics
Duration is measured in milliseconds
Written to InfluxDB with the same tags as the original metric
Includes an additional source_metric tag pointing to the original metric

Visualizing collection times (optional):

To show collection duration in Grafana dashboards, add it to your metrics-config.yaml:

metrics:
  # Original metric - duration tracked automatically
  - name: "s3_total_objects"
    type: "ssh"
    app_name: "pimcore"
    description: "Total number of objects in S3 storage"
    command: |
      AWS_ACCESS_KEY_ID='fake' ... | awk '{print $3}'

  # Duration metric - add this ONLY if you want to graph it
  - name: "s3_total_objects_collection_duration"
    description: "Time to collect S3 objects count"
    unit: "ms"
    # No type/command needed - data is already in InfluxDB

Benefits:

Identify slow-running queries or commands
Monitor performance degradation over time
Alert when collection time exceeds thresholds
Zero overhead - opt-in visualization only

Example use cases:

Track if S3 bucket listing gets slower as files increase
Monitor SQL query performance
Detect network latency issues

Adjusting Collection Frequency

Edit the cron schedule in .upsun/config.yaml:

crons:
  collect_metrics:
    spec: '*/5 * * * *'  # Every 5 minutes
    # spec: '*/15 * * * *'  # Every 15 minutes
    # spec: '0 * * * *'     # Every hour

Troubleshooting

Metrics not appearing in Grafana

Check if the cron job is running:

upsun ssh -A metrics-collector
cat /var/log/cron.log

Check InfluxDB has data:

upsun sql -A metrics-collector
# In InfluxDB CLI:
# from(bucket: "metrics") |> range(start: -1h) |> filter(fn: (r) => r._measurement != "")

Check Grafana datasource connection:
- Visit Grafana → Configuration → Data Sources
- Test the InfluxDB connection

Authentication errors

Make sure your API token is valid:

upsun auth:info

Testing Metrics

Test SQL queries directly (the collector uses --raw for easier parsing):

pimcore sql -p PROJECT_ID -e main --app pimcore --raw -- "SELECT COUNT(*) FROM your_table;"

Test SSH commands:

pimcore ssh -p PROJECT_ID -e main --app pimcore -- "ls -1 /var/log/*.log | wc -l"

Grafana not starting

Check logs:

upsun ssh -A grafana
cat /var/lib/grafana/logs/grafana.log

Why InfluxDB?

InfluxDB + Grafana (Current Choice)

✅ InfluxDB is a managed Upsun service - no container needed
✅ Purpose-built for time-series data
✅ Powerful Flux query language
✅ Simple Line Protocol for writing data
✅ Excellent retention policies
✅ Great Grafana integration

Alternatives Considered

Prometheus + Pushgateway

❌ Not available as Upsun managed services
❌ Would require self-hosting both services

StatsD + Graphite

❌ Not available as Upsun managed services
❌ More complex architecture
❌ Less flexible query language

License

MIT

Navigation Menu

Repository files navigation

Upsun Metrics Collection System

Architecture

Features

Setup

1. Configure API Token

2. Configure Projects and Metrics

3. Generate Dashboards

4. Deploy to Upsun

5. Access Grafana

Services Used

InfluxDB (Managed Service)

Grafana (Self-Hosted)

Metric Types & Examples

SQL Metrics

SSH Metrics

Multiple Outputs from One Command

PHP-FPM Worker Monitoring

Filesystem Monitoring

RabbitMQ Queue Monitoring

General Notes

Collection Duration Tracking

Adjusting Collection Frequency

Troubleshooting

Metrics not appearing in Grafana

Authentication errors

Testing Metrics

Grafana not starting

Why InfluxDB?

InfluxDB + Grafana (Current Choice)

Alternatives Considered

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.upsun		.upsun
grafana		grafana
.gitignore		.gitignore
README.md		README.md
collect_metrics.py		collect_metrics.py
dashboard-example.png		dashboard-example.png
generate_dashboards.py		generate_dashboards.py
metrics-config.yaml		metrics-config.yaml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

Upsun Metrics Collection System

Architecture

Features

Setup

1. Configure API Token

2. Configure Projects and Metrics

3. Generate Dashboards

4. Deploy to Upsun

5. Access Grafana

Services Used

InfluxDB (Managed Service)

Grafana (Self-Hosted)

Metric Types & Examples

SQL Metrics

SSH Metrics

Multiple Outputs from One Command

PHP-FPM Worker Monitoring

Filesystem Monitoring

RabbitMQ Queue Monitoring

General Notes

Collection Duration Tracking

Adjusting Collection Frequency

Troubleshooting

Metrics not appearing in Grafana

Authentication errors

Testing Metrics

Grafana not starting

Why InfluxDB?

InfluxDB + Grafana (Current Choice)

Alternatives Considered

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages