Merge pull request #30 from red-hat-data-services/upgrade-langfuse-v3

feat: Upgrade Langfuse v2 to v3

Author: tarun-etikala

README.md

@@ -23,6 +23,7 @@ Agents are organized by framework. Pick one and follow its README:
 | **CrewAI** | [WebSearch Agent](./agents/crewai/websearch_agent/) | CrewAI-based agent with a web search tool to query the internet and answer user questions. |
 | **Vanilla Python** | [OpenAI Responses Agent](./agents/vanilla_python/openai_responses_agent/) | Minimal agent with no framework: only the OpenAI Python client and an Action/Observation loop with tools. Use with OpenAI or any compatible API. |
 | **AutoGen** | [MCP Agent](./agents/autogen/mcp_agent/) | AutoGen AssistantAgent with MCP tools over SSE (e.g. churn prediction, math tools), FastAPI `/chat/completions`. |
+| **Langflow** | [Simple Tool Calling Agent](./agents/langflow/simple_tool_calling_agent/) | Tool-calling agent built with Langflow's visual flow builder. Calls external APIs as tools and reasons over results. Includes Langfuse v3 tracing. Runs locally via `podman-compose`. |
 
 ## Deployment Options
 
@@ -57,8 +58,10 @@ agentic-starter-kits/
 │   │   └── websearch_agent/         # LlamaIndex web search agent
 │   ├── vanilla_python/
 │   │   └── openai_responses_agent/  # OpenAI Responses API (no framework)
-│   └── autogen/
-│       └── mcp_agent/               # AutoGen + MCP (SSE)
+│   ├── autogen/
+│   │   └── mcp_agent/               # AutoGen + MCP (SSE)
+│   └── langflow/
+│       └── simple_tool_calling_agent/ # Langflow tool-calling agent
 └── README.md
 ```
 

agents/langflow/simple_tool_calling_agent/README.md

@@ -8,9 +8,16 @@
 
 ## What this agent does
 
-Outdoor activity planning agent built with Langflow. It recommends the best day and time for outdoor activities by reasoning across weather forecasts, air quality, and National Park Service data.
+A tool-calling agent built with Langflow's visual flow builder. It calls external APIs as tools (weather forecasts, national park data) and reasons over the results to answer user questions. Includes Langfuse v3 tracing out of the box. Runs locally via `podman-compose`.
 
-**Example query:** *"I want to go hiking near Denver this weekend. What day is best?"*
+### Included demo flow
+
+The shipped flow is an outdoor activity assistant — it checks weather conditions and national park alerts to help decide if conditions are good for outdoor activities.
+
+**Example queries:**
+- *"Can I go walking in Boston tomorrow at 3 PM?"*
+- *"I want to go hiking near Denver this weekend. What day is best?"*
+- *"Is it a good day for a picnic in San Francisco?"*
 
 ### Tools
 | Tool | API | Description |
@@ -52,9 +59,9 @@ chmod +x deploy-local.sh cleanup-local.sh
 
 This starts:
 - **Langflow** on http://localhost:7860 — the agent UI
-- **PostgreSQL** — shared database server. Hosts two databases: `langflow` (flows, users, settings) and `langfuse` (traces). The `langflow` database is created automatically by PostgreSQL; the `langfuse` database is created by `local/init-db.sh` on first startup
+- **PostgreSQL** — shared database server. Hosts two databases: `langflow` (flows, users, settings) and `langfuse` (metadata). The `langflow` database is created automatically by PostgreSQL; the `langfuse` database is created by `local/init-db.sh` on first startup
 - **Ollama** on http://localhost:11434 — local LLM (qwen2.5:7b), runs natively on host for GPU acceleration
-- **Langfuse v2** on http://localhost:3000 — tracing (admin@langflow.local / admin123)
+- **Langfuse v3** on http://localhost:3000 — tracing (admin@langflow.local / password auto-generated in `local/.env`), backed by ClickHouse, MinIO, and Redis
 
 ### Import and configure the flow
 
@@ -102,8 +109,9 @@ On the cluster, replace `localhost:7860` with your cluster's Langflow route URL.
 |------|---------------------|----------------------------|
 | Downloaded Ollama models (e.g., qwen2.5:7b) | Kept | Kept (stored on host, not in containers) |
 | Imported Langflow flows | Kept | **Deleted** (re-import needed) |
-| Langfuse traces | Kept | **Deleted** |
+| Langfuse traces (ClickHouse + MinIO) | Kept | **Deleted** |
 | PostgreSQL data | Kept | **Deleted** |
+| Redis cache | Kept | **Deleted** |
 | `.env` and `.ollama-enabled` config | Kept | **Deleted** |
 
 ---
@@ -202,7 +210,7 @@ Then update the **KServe vLLM** component in the Langflow UI to point to your re
 After running the agent, traces are automatically sent to Langfuse.
 
 **Locally:**
-1. Open http://localhost:3000 (login: admin@langflow.local / admin123)
+1. Open http://localhost:3000 (login: admin@langflow.local / password auto-generated in `local/.env`)
 2. Select the **Langflow Agent** project
 3. Click **Traces** in the left sidebar
 4. Click on any trace to see the full agent execution — LLM calls, tool invocations, inputs, and outputs

agents/langflow/simple_tool_calling_agent/deploy-local.sh

@@ -23,7 +23,22 @@ fi
 # Copy .env if it doesn't exist
 if [ ! -f "$LOCAL_DIR/.env" ]; then
   cp "$LOCAL_DIR/.env.example" "$LOCAL_DIR/.env"
-  echo "Created .env from .env.example — edit it if needed."
+  echo "Created .env from .env.example"
+fi
+
+# Generate Langfuse secrets if not already set
+if ! grep -q '^LANGFUSE_ADMIN_PASSWORD=.\+' "$LOCAL_DIR/.env" 2>/dev/null; then
+  GENERATED_PWD=$(openssl rand -base64 12)
+  sed -i.bak "s/^LANGFUSE_ADMIN_PASSWORD=.*/LANGFUSE_ADMIN_PASSWORD=$GENERATED_PWD/" "$LOCAL_DIR/.env"
+  rm -f "$LOCAL_DIR/.env.bak"
+  echo "Generated Langfuse admin password."
+fi
+
+if ! grep -q '^LANGFUSE_ENCRYPTION_KEY=.\+' "$LOCAL_DIR/.env" 2>/dev/null; then
+  GENERATED_KEY=$(openssl rand -hex 32)
+  sed -i.bak "s/^LANGFUSE_ENCRYPTION_KEY=.*/LANGFUSE_ENCRYPTION_KEY=$GENERATED_KEY/" "$LOCAL_DIR/.env"
+  rm -f "$LOCAL_DIR/.env.bak"
+  echo "Generated Langfuse encryption key."
 fi
 
 # Ask about Ollama on first run
@@ -81,11 +96,11 @@ cd "$LOCAL_DIR" || { echo "ERROR: Directory $LOCAL_DIR not found."; exit 1; }
 
 # Start containerized services (Langflow, PostgreSQL, Langfuse)
 echo ""
-echo "Starting local stack (Langflow + PostgreSQL + Langfuse)..."
+echo "Starting local stack (Langflow + PostgreSQL + Langfuse v3 + ClickHouse + MinIO + Redis)..."
 podman-compose up -d
 
 echo ""
-echo "Waiting for Langflow to start (this may take a minute)..."
+echo "Waiting for Langflow to start (this may take a few minutes)..."
 LANGFLOW_READY=false
 for i in $(seq 1 60); do
   if curl -s http://localhost:7860/health >/dev/null 2>&1; then
@@ -102,11 +117,28 @@ if [ "$LANGFLOW_READY" = false ]; then
   exit 1
 fi
 
+echo ""
+echo "Waiting for Langfuse to start (this may take several minutes)..."
+LANGFUSE_READY=false
+for i in $(seq 1 60); do
+  if curl -s http://localhost:3000/api/public/health >/dev/null 2>&1; then
+    LANGFUSE_READY=true
+    echo "Langfuse is ready!"
+    break
+  fi
+  sleep 10
+done
+
+if [ "$LANGFUSE_READY" = false ]; then
+  echo "WARNING: Langfuse did not start within 10 minutes. Tracing and logging will not work until it is up."
+  echo "Check logs: podman logs local_langfuse-web_1"
+fi
+
 echo ""
 echo "=== Local environment is ready ==="
 echo ""
 echo "  Langflow UI:  http://localhost:7860"
-echo "  Langfuse:     http://localhost:3000  (login: admin@langflow.local / admin123)"
+echo "  Langfuse:     http://localhost:3000  (login: admin@langflow.local / password auto-generated in local/.env)"
 if [ "$USE_OLLAMA" = "yes" ]; then
   echo "  Ollama API:   http://localhost:11434  (running natively on host)"
 fi

agents/langflow/simple_tool_calling_agent/flows/outdoor-activity-agent.json

@@ -995,7 +995,7 @@
                 "trace_as_input": true,
                 "trace_as_metadata": true,
                 "type": "str",
-                "value": "You are a national park trip planner and weather assistant. You MUST use the available tools to get real data for every question - never guess or make up information about alerts, weather, or park details.\n\nIMPORTANT: For every question about a park, call the appropriate tool. Do not answer from memory.\n\nTools:\nNPS Search Parks \u2014 find parks by state code and activity keyword\nNPS Park Alerts \u2014 get active closures/hazards for a park (uses four-letter park code)\nOpen-Meteo Forecast \u2014 get weather forecast for coordinates (latitude, longitude)\n\nWorkflow:\n\nParse the request. Identify: state(s), activity/interest, travel dates (if given), and group needs (e.g., kids, accessibility).\n\nSearch parks. Call NPS Search Parks with the state code and activity. Extract the park code, coordinates, and description from results. If the user mentions a specific park by name, still search to confirm its park code and coordinates.\n\nCheck alerts for each candidate park. Call NPS Park Alerts using the park code. Classify alerts as:\n\nBlocking (closure, danger): disqualifies the park unless the closure is partial and doesn't affect the user's activity.\nInformational (caution, info): mention but don't disqualify.\nCheck weather. Call Open-Meteo Forecast using the park's latitude and longitude. Evaluate conditions against the user's planned activity \u2014 e.g., rain matters more for hiking than museum visits. If the user provided travel dates, focus on that date range.\n\nRecommend. Pick the best park considering all three factors. If the top choice has blocking alerts or severe weather, suggest an alternative from the search results.\n\nShortcuts:\n\nWeather-only question \u2192 skip NPS tools, forecast directly.\nAlert-only question \u2192 skip weather tool.\nUser names a specific park \u2192 still check alerts and weather, skip search only if you already have the park code and coordinates.\n\nResponse format:\n\nProvide your recommendation in this structure:\n\nPark: name, location, and why it fits the activity\n\nAlerts: any active alerts (or \"No active alerts\")\n\nWeather Outlook: conditions for the travel dates, with a note on suitability for the activity\n\nTips: 1-2 practical tips (best trails, gear, timing, alternatives if weather turns)"
+                "value": "You are an outdoor activity assistant. You help people decide whether conditions are good for outdoor activities — walking, hiking, cycling, picnics, park visits, or anything else outside. You MUST use the available tools to get real data — never guess or make up weather, alerts, or park details.\n\nIMPORTANT: Always call the appropriate tool. Do not answer from memory.\n\nTools:\nOpen-Meteo Forecast — get weather forecast for coordinates (latitude, longitude)\nNPS Search Parks — find national parks by state code and activity keyword\nNPS Park Alerts — get active closures/hazards for a park (uses four-letter park code)\n\nHow to decide which tools to use:\n\n1. Every question gets a weather check. Look up the coordinates for the location the user mentioned (city, park, area) and call Open-Meteo Forecast.\n\n2. Only use NPS tools when the user asks about national parks, hiking trails, or outdoor activities near nature areas. If someone asks about walking in a city, cycling to work, or having a picnic in a city park, skip the NPS tools — they only cover national parks.\n\n3. When you do use NPS tools: search for parks first, then check alerts for each candidate. Classify alerts as blocking (closure, danger) or informational (caution, info).\n\nWorkflow:\n\nParse the request. Identify: location, activity, and dates/times (if given).\n\nCheck weather. Call Open-Meteo Forecast for the location. Evaluate conditions for the specific activity — rain matters more for a picnic than for a museum visit, wind matters for cycling, UV matters for a beach day.\n\nIf relevant, search parks. Only if the user is asking about hiking, national parks, or nature activities near a region. Call NPS Search Parks, then NPS Park Alerts for candidates.\n\nGive a clear answer. Lead with a direct yes/no/maybe recommendation, then explain why based on the data.\n\nResponse format:\n\nAdapt your response to the question. For simple weather checks, keep it short and direct. For park trip planning, include more detail. Always include:\n\n- A clear recommendation (good day for it, not ideal, or avoid)\n- Weather conditions that matter for the activity\n- Alerts or park info only if NPS tools were used\n- 1-2 practical tips if helpful"
               },
               "tools": {
                 "_input_type": "HandleInput",

agents/langflow/simple_tool_calling_agent/local/.env.example

@@ -3,6 +3,9 @@ POSTGRES_USER=langflow
 POSTGRES_PASSWORD=langflow
 POSTGRES_DB=langflow
 
+# Langfuse — admin password and encryption key are auto-generated on first run by deploy-local.sh
+LANGFUSE_ADMIN_PASSWORD=
+LANGFUSE_ENCRYPTION_KEY=
+
 # Ollama — model to pull on first run
 OLLAMA_MODEL=qwen2.5:7b
-

agents/langflow/simple_tool_calling_agent/local/podman-compose.yml

@@ -15,15 +15,15 @@ services:
       OPENAI_API_KEY: "not-needed"
       OLLAMA_BASE_URL: "http://host.containers.internal:11434"
       # Langfuse tracing
-      LANGFUSE_HOST: "http://langfuse:3000"
+      LANGFUSE_HOST: "http://langfuse-web:3000"
       LANGFUSE_PUBLIC_KEY: "pk-lf-local-dev"
       LANGFUSE_SECRET_KEY: "sk-lf-local-dev"
     volumes:
       - langflow_data:/var/lib/langflow:Z
     depends_on:
       postgres:
         condition: service_healthy
-      langfuse:
+      langfuse-web:
         condition: service_started
 
   # ── PostgreSQL (shared by Langflow and Langfuse) ────────────────────
@@ -42,16 +42,54 @@ services:
       timeout: 5s
       retries: 5
 
-  # ── Langfuse v2 (Tracing) ──────────────────────────────────────────
-  langfuse:
-    image: docker.io/langfuse/langfuse:2
+  # ── Langfuse v3 (Tracing) ────────────────────────────────────────
+
+  langfuse-web:
+    image: docker.io/langfuse/langfuse:3.162.0
+    restart: always
     ports:
-      - "3000:3000"
-    environment:
+      - "127.0.0.1:3000:3000"
+    depends_on: &langfuse-depends-on
+      postgres:
+        condition: service_healthy
+      clickhouse:
+        condition: service_healthy
+      minio:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+    environment: &langfuse-env
       DATABASE_URL: "postgresql://${POSTGRES_USER:-langflow}:${POSTGRES_PASSWORD:-langflow}@postgres:5432/langfuse"
       NEXTAUTH_URL: "http://localhost:3000"
       NEXTAUTH_SECRET: "local-dev-secret"
       SALT: "local-dev-salt"
+      ENCRYPTION_KEY: "${LANGFUSE_ENCRYPTION_KEY}"
+      TELEMETRY_ENABLED: "false"
+      # ClickHouse
+      CLICKHOUSE_MIGRATION_URL: "clickhouse://clickhouse:9000"
+      CLICKHOUSE_URL: "http://clickhouse:8123"
+      CLICKHOUSE_USER: "clickhouse"
+      CLICKHOUSE_PASSWORD: "clickhouse"
+      CLICKHOUSE_CLUSTER_ENABLED: "false"
+      # Redis
+      REDIS_HOST: "redis"
+      REDIS_PORT: "6379"
+      REDIS_AUTH: "localredis"
+      # MinIO (S3-compatible object storage)
+      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: "langfuse"
+      LANGFUSE_S3_EVENT_UPLOAD_REGION: "auto"
+      LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: "minio"
+      LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: "miniosecret"
+      LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: "http://minio:9000"
+      LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: "true"
+      LANGFUSE_S3_EVENT_UPLOAD_PREFIX: "events/"
+      LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: "langfuse"
+      LANGFUSE_S3_MEDIA_UPLOAD_REGION: "auto"
+      LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: "minio"
+      LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: "miniosecret"
+      LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: "http://minio:9000"
+      LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: "true"
+      LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: "media/"
       # Headless init — auto-creates org, project, user, and API keys
       LANGFUSE_INIT_ORG_ID: "langflow-org"
       LANGFUSE_INIT_ORG_NAME: "Langflow Agent"
@@ -61,11 +99,72 @@ services:
       LANGFUSE_INIT_PROJECT_SECRET_KEY: "sk-lf-local-dev"
       LANGFUSE_INIT_USER_EMAIL: "admin@langflow.local"
       LANGFUSE_INIT_USER_NAME: "admin"
-      LANGFUSE_INIT_USER_PASSWORD: "admin123"
-    depends_on:
-      postgres:
-        condition: service_healthy
+      LANGFUSE_INIT_USER_PASSWORD: "${LANGFUSE_ADMIN_PASSWORD}"
+
+  langfuse-worker:
+    image: docker.io/langfuse/langfuse-worker:3.162.0
+    restart: always
+    depends_on: *langfuse-depends-on
+    environment:
+      <<: *langfuse-env
+
+  # ── ClickHouse (Langfuse analytics DB) ──────────────────────────────
+  clickhouse:
+    image: docker.io/clickhouse/clickhouse-server:26.2
+    restart: always
+    user: "101:101"
+    environment:
+      CLICKHOUSE_DB: default
+      CLICKHOUSE_USER: clickhouse
+      CLICKHOUSE_PASSWORD: clickhouse
+    volumes:
+      - clickhouse_data:/var/lib/clickhouse:Z
+      - clickhouse_logs:/var/log/clickhouse-server:Z
+    healthcheck:
+      test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
+      interval: 5s
+      timeout: 5s
+      retries: 10
+      start_period: 30s
+
+  # ── MinIO (Langfuse S3-compatible object storage) ───────────────────
+  minio:
+    # MinIO uses date-based release tags instead of semver
+    image: docker.io/minio/minio:RELEASE.2025-09-07T16-13-09Z
+    restart: always
+    entrypoint: sh
+    command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
+    environment:
+      MINIO_ROOT_USER: minio
+      MINIO_ROOT_PASSWORD: miniosecret
+    volumes:
+      - minio_data:/data:Z
+    healthcheck:
+      test: ["CMD", "mc", "ready", "local"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+
+  # ── Redis (Langfuse caching + job queue) ────────────────────────────
+  redis:
+    image: docker.io/redis:7.4.8
+    restart: always
+    command: >
+      --requirepass localredis
+      --maxmemory-policy noeviction
+    volumes:
+      - redis_data:/data:Z
+    healthcheck:
+      test: ["CMD", "redis-cli", "-a", "localredis", "ping"]
+      interval: 3s
+      timeout: 10s
+      retries: 10
 
 volumes:
   langflow_data:
   postgres_data:
+  clickhouse_data:
+  clickhouse_logs:
+  minio_data:
+  redis_data: