feat: restructure Open Pulse stack and environment configuration

- Consolidated the entire Open Pulse stack under `infra/open-pulse-stack/`, including main compose files, CLI orchestrator overlay, and GrimoireLab assets.
- Introduced a single-file environment model with `infra/.env` as the authoritative deployment environment, while `<repo>/.env` is designated for the open-pulse CLI when interacting with external infrastructure.
- Updated `.gitignore` to recursively ignore `data/` directories to prevent accidental pollution of the working tree.
- Added `infra/.env.example` as a comprehensive template for deployment configuration.
- Simplified default authentication credentials across the stack to `openpulse` / `replace-me`, with stronger placeholders where necessary.
- Enhanced documentation to reflect the new structure and configuration processes, including a new `.env` wizard for easier setup.

Author: caviri

.env.example

@@ -0,0 +1,34 @@
+# ── Open Pulse — tool / client config ─────────────────────────────────────
+# Copy to `<repo>/.env` and fill in.
+#
+# This file is consumed by the open-pulse Python CLI / hub when running on
+# the host as a CLIENT against EXTERNAL infrastructure (Neo4j / SPARQL /
+# crawler living elsewhere). Compose never loads it — when you launch the
+# stack from `infra/`, all deployment env lives in `infra/.env`.
+#
+# So:
+#   - Bringing local infra up?    → fill `infra/.env`. Ignore this file.
+#   - Pointing the CLI at remote? → fill this file. `infra/.env` is unused.
+
+
+# ── Service endpoints ────────────────────────────────────────────────────
+# Override these to point at the remote services you want to talk to. When
+# left commented out, the CLI uses the in-network defaults (`neo4j:7687`,
+# `sparql-proxy:7878`, …) — those only resolve from inside the compose
+# network, so leaving them commented is appropriate for in-stack hub usage
+# but not for host-side CLI talking to remote infra.
+# NEO4J_BOLT_ENDPOINT=bolt://neo4j.example.com:7687
+# NEO4J_HTTP_ENDPOINT=http://neo4j.example.com:7474
+# SPARQL_ENDPOINT=https://sparql.example.com
+# CRAWLER_ENDPOINT=https://crawler.example.com
+# GIT_METADATA_EXTRACTOR_ENDPOINT=https://gme.example.com
+
+
+# ── Auth for the endpoints above ─────────────────────────────────────────
+# Whatever credentials the remote services expect. `replace-me` placeholders
+# match the open-pulse local-default convention; rotate before any remote
+# use.
+# NEO4J_AUTH=neo4j/replace-me
+# CRAWLER_API_TOKEN=replace-me
+# SPARQL_AUTH=replace-me
+# APPLIER_AUTH=replace-me

.github/dependabot.yml

@@ -66,9 +66,8 @@ updates:
   # Compose-managed images for the local services stack.
   - package-ecosystem: docker-compose
     directories:
-      - "/infra/compose"
+      - "/infra/open-pulse-stack"
       - "/infra/services/oxigraph"
-      - "/infra/services/grimoirelab"
       - "/infra/services/sparql-proxy"
       - "/infra/services/neo4j"
       - "/infra/services/portainer"

.github/workflows/docker-publish.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - develop
     paths:
       - "src/**"
       - "pyproject.toml"
@@ -28,7 +29,7 @@ permissions:
 
 env:
   REGISTRY: ghcr.io
-  # Path matches the OPEN_PULSE_IMAGE default in infra/compose/docker-compose.yml.
+  # Path matches the OPEN_PULSE_IMAGE default in infra/open-pulse-stack/docker-compose.yml.
   # Org is sdsc-ordes; package_name within the org is "open-pulse".
   IMAGE_NAME: sdsc-ordes/open-pulse
   PACKAGE_NAME: open-pulse

.github/workflows/docker-validate.yml

@@ -3,14 +3,14 @@ name: Compose Validate
 on:
   pull_request:
     paths:
-      - "infra/compose/**"
+      - "infra/open-pulse-stack/**"
       - ".github/workflows/docker-validate.yml"
   push:
     branches:
       - main
       - "release/**"
     paths:
-      - "infra/compose/**"
+      - "infra/open-pulse-stack/**"
       - ".github/workflows/docker-validate.yml"
 
 permissions:
@@ -33,8 +33,9 @@ jobs:
       fail-fast: false
       matrix:
         compose_files:
-          - "-f infra/compose/docker-compose.yml"
-          - "-f infra/compose/docker-compose.yml -f infra/compose/docker-compose.cli.yml"
+          - "-f infra/open-pulse-stack/docker-compose.yml"
+          - "-f infra/open-pulse-stack/docker-compose.yml -f infra/open-pulse-stack/docker-compose.cli.yml"
+          - "-f infra/open-pulse-stack/docker-compose.yml -f infra/open-pulse-stack/docker-compose.grimoirelab.yml"
     steps:
       - name: Checkout repository
         uses: actions/checkout@v5

.gitignore

@@ -17,12 +17,12 @@ infra/services/airflow/src/env_variables.json
 **/airflow/plugins/
 **/airflow/logs/
 
-# Docker/database runtime data
-/data/
-infra/services/graphdb/data/
+# Docker/database runtime data — ignore any `data/` dir at any level so an
+# accidental relative-path resolution (e.g. `infra/open-pulse-stack/data/`)
+# can't pollute the working tree. Per-service subdirs under the configured
+# OPEN_PULSE_DATA_DIR are also covered by this single rule.
+data/
 infra/services/graphdb/graphdb-data/
-infra/services/neo4j/data/
-infra/services/portainer/data/
 
 # Tentris local data and licenses
 infra/services/tentris-server/data/

AGENTS.md

@@ -129,19 +129,20 @@ overrides:
                                    identity-mapped, so nested `docker compose`
                                    resolves bind paths the same on both sides)
 
-  infra/compose/docker-compose.yml   ┐
-   ├── neo4j  oxigraph  sparql-proxy │ image-only refs; OPEN_PULSE_IMAGE
-   ├── crawler  extractor  selenium  │ pulls from GHCR by default
-   ├── hub  (--profile hub)          │ HUB_AUTH gate
-   └── grimoirelab-db  portainer …  ─┘
+  infra/open-pulse-stack/docker-compose.yml   ┐
+   ├── neo4j  oxigraph  sparql-proxy           │ image-only refs;
+   ├── crawler  extractor  selenium            │ OPEN_PULSE_IMAGE pulls
+   ├── hub  (--profile hub)                    │ from GHCR by default;
+   └── grimoirelab-db  portainer …            ─┘ HUB_AUTH gates the hub
 
-  infra/compose/docker-compose.cli.yml
+  infra/open-pulse-stack/docker-compose.cli.yml
    └── open-pulse-cli (overlay; auto-included when CLI runs inside it)
 
-  infra/services/grimoirelab/docker-compose.yml
+  infra/open-pulse-stack/docker-compose.grimoirelab.yml
    └── full GrimoireLab stack (mariadb, valkey, opensearch, mordred,
        sortinghat, nginx, projects-applier sidecar) — opt in via
-       `open-pulse deploy up --with-grimoire`
+       `open-pulse deploy up --with-grimoire`. Supporting assets at
+       `infra/open-pulse-stack/grimoirelab/`.
 
   data/                                ← single root for all bind mounts
    ├── neo4j/  oxigraph/  …            ← main stack writes here
@@ -158,15 +159,17 @@ uv run pytest -q
 
 # Build the unified image (one image for CLI / orchestrator / hub)
 docker build -f tools/images/Dockerfile-open-pulse -t open-pulse:local .
-echo "OPEN_PULSE_IMAGE=open-pulse:local" >> .env       # otherwise pulls GHCR
+echo "OPEN_PULSE_IMAGE=open-pulse:local" >> infra/.env  # otherwise pulls GHCR
 
 # Bring up the stack (interactive profile picker if no --profile flags)
 ./scripts/op deploy up --profile crawler --profile extractor --profile sparql --profile hub
-# or directly via the host's docker compose:
-docker compose -f infra/compose/docker-compose.yml -f infra/compose/docker-compose.cli.yml \
-  --env-file .env --profile hub up -d
+# or directly via the host's docker compose (only infra/.env is loaded):
+docker compose -f infra/open-pulse-stack/docker-compose.yml \
+               -f infra/open-pulse-stack/docker-compose.cli.yml \
+               --env-file infra/.env \
+               --profile hub up -d
 
-# Hub at http://localhost:9090 — log in with admin / $HUB_AUTH
+# Hub at http://localhost:9090 — log in with openpulse / $HUB_AUTH
 
 # Talk to the cli orchestrator from the host (handles git-bash path mangling):
 ./scripts/op deploy ps
@@ -181,13 +184,12 @@ docker compose -f infra/compose/docker-compose.yml -f infra/compose/docker-compo
 
 | Sub-command | Description |
 | --- | --- |
-| `open-pulse deploy up` | Deploy services via Docker Compose. Without `--profile` flags, opens an interactive selector. Creates `.env` from `infra/env/.env.example` if absent. |
+| `open-pulse deploy up` | Deploy services via Docker Compose. Without `--profile` flags, opens an interactive selector. Compose loads only `<repo>/infra/.env` (auto-seeded from `infra/.env.example` if missing). The tool/client `<repo>/.env` is for the open-pulse Python CLI / hub against external infra and is not a compose input. |
 | `open-pulse deploy down` | Tear down deployed services. `--volumes` / `-v` also removes named volumes. |
 | `open-pulse deploy ps` | Show the status of deployed containers. |
 
 **Profiles:**
 - `default` — Core services only (Neo4j)
-- `analysis` — Core + analysis notebook
 - `crawler` — Open Pulse Crawler API
 - `extractor` — GME extractor + Selenium
 - `sparql` — Oxigraph + sparql-proxy
@@ -196,10 +198,11 @@ docker compose -f infra/compose/docker-compose.yml -f infra/compose/docker-compo
 - `orchestration` — Portainer
 
 **Flags applicable to `up` / `down` / `ps`:**
-- `--with-cli` — Include `infra/compose/docker-compose.cli.yml` (auto-included
-  when the CLI itself runs inside the cli container, via the
-  `OPEN_PULSE_RUNNING_IN_CLI_CONTAINER=1` marker).
-- `--with-grimoire` — Also include `infra/services/grimoirelab/docker-compose.yml`.
+- `--with-cli` — Include `infra/open-pulse-stack/docker-compose.cli.yml`
+  (auto-included when the CLI itself runs inside the cli container, via
+  the `OPEN_PULSE_RUNNING_IN_CLI_CONTAINER=1` marker).
+- `--with-grimoire` — Also include
+  `infra/open-pulse-stack/docker-compose.grimoirelab.yml`.
 
 **Project root resolution** (`_find_project_root`):
 1. `$OPEN_PULSE_PROJECT_ROOT` if set
@@ -310,7 +313,7 @@ read-only at `/data` inside the hub so DuckDB can read other services' files.
 - **CI triggers** (path-scoped):
   - `src/**`, `tests/**`, `pyproject.toml`, `uv.lock` → Python CI
   - `tools/images/**`, `.devcontainer/**`, `pyproject.toml`, `uv.lock`,
-    `infra/compose/docker-compose*.yml` → Docker validation
+    `infra/open-pulse-stack/docker-compose*.yml` → Docker validation
   - `docs-site/**` → docs build
 - **Pre-commit**: `.pre-commit-config.yaml`, Ruff scoped to `src/`
 

CHANGELOG.md

@@ -9,6 +9,11 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 
 ### Changed
 
+- Consolidated the entire Open Pulse stack under `infra/open-pulse-stack/`. The main compose, the CLI orchestrator overlay, and the full GrimoireLab compose now live alongside each other as `docker-compose.yml`, `docker-compose.cli.yml`, and `docker-compose.grimoirelab.yml`; GrimoireLab supporting assets (applier source, config templates, sigils, one-shot scripts) moved to `infra/open-pulse-stack/grimoirelab/`. Updated `deploy.py` path constants, `health.py` `_COMPOSE_FILE`, `justfile` `regen-grimoire-config`, `.github/dependabot.yml`, the docker-validate workflow (with a third matrix entry exercising the grimoirelab compose), and every doc that referenced the old paths.
+- Split `.env` along the principle "when launching from `infra/`, all env lives in `infra/`; otherwise `<repo>/.env` is just for the open-pulse tool acting as a client". `<repo>/infra/.env` is the AUTHORITATIVE deployment env: image refs, ports, resource limits, storage paths, ALL container-side credentials, per-service knobs (HUB_AUTH, CRAWLER_*, EXTRACTOR_*, GrimoireLab block, V2/RAG flags). Compose loads only this file (`--env-file infra/.env`); every service additionally `env_file:`-pulls it so any key set there reaches the container without an explicit `environment:` mapping. `<repo>/.env` is the tool/client env, consumed only by the open-pulse Python CLI / hub when running on the host against EXTERNAL infrastructure — compose never reads it. Both files are gitignored; `infra/.env` auto-seeds from `infra/.env.example` on first `op deploy up`, while `<repo>/.env` is a manual copy from `<repo>/.env.example`. `deploy up` / `down` / `ps` and the cli + grimoirelab compose env_file directives all dropped the second `--env-file` / `env_file:` entry.
+- Simplified default auth across the stack to `openpulse` / `replace-me`. Where the underlying service forces a specific username (Neo4j → `neo4j`, OpenSearch admin → `admin`, MariaDB init → `root`) only the password changes; everywhere else (GrimoireLab Postgres user, SortingHat superuser, hub login UI) the default username is `openpulse`. Compose-baked fallbacks (`${NEO4J_AUTH:-...}`, `${GRIMOIRELAB_DB_USER:-...}`, `${GRIMOIRELAB_DB_PASSWORD:-...}`) and the runtime fallback in `gui/hub/routes/stats.py` now match. `replace-me` is a placeholder; rotate before any non-local deployment.
+- Pinned `OPEN_PULSE_DATA_DIR` to an absolute path during `.env` seeding. The previous default `./data` resolved differently between `op deploy …` (relative to repo root via `--project-directory`) and raw `docker compose -f infra/open-pulse-stack/…` (relative to the compose file), silently splitting state into two locations. `_ensure_env_files` now substitutes the line with the absolute repo-root data path when seeding `infra/.env` from the template; `OPEN_PULSE_HOST_PATH` is filled the same way. `GRIMOIRE_DATA_DIR` derives from `${OPEN_PULSE_DATA_DIR}` so GrimoireLab data lands alongside Neo4j et al. under a single root.
+- Hardened `.gitignore` to ignore `data/` recursively (any nesting level) so an accidental relative-path resolution can't pollute the working tree. Removed the per-path `infra/services/{neo4j,portainer}/data/` entries that the recursive rule now covers.
 - Added explicit `mem_limit`, `cpus`, and `restart` policies to every service in `infra/compose/docker-compose.yml` and `infra/services/grimoirelab/docker-compose.yml`. Each cap reads from a per-service env var (e.g. `OPENSEARCH_MEM_LIMIT`, `NEO4J_MEM_LIMIT`, `MORDRED_MEM_LIMIT`, `EXTRACTOR_MEM_LIMIT`) so production deploys override the dev defaults without editing the compose. Neo4j heap and page cache are now explicit (`NEO4J_HEAP`, `NEO4J_PAGECACHE`) and sized to fit under the new cap. Mordred default cap dropped from 4g to 2g (it idled at 1.78 GiB; the previous 4g was generous and contributed to the host-OS pressure that OOM-killed opensearch). See `dev/advise/2026-05-05-resource-caps-and-oom-diagnosis.md` for the full diagnosis, budget math, and how to apply the new caps to running services without downtime.
 - Added healthchecks to `opensearch-node1` (`_cluster/health?wait_for_status=yellow` with auth — `yellow` is the success state on a single-node deploy because replicas can't be allocated) and `opensearch-dashboards` (liveness via `GET /` — *not* `/api/status`, which requires auth in v3 and would always report unhealthy).
 - Simplified Compose topology to a two-file model under `infra/compose/`: `docker-compose.yml` for infra services and `docker-compose.cli.yml` as an optional CLI overlay.
@@ -28,8 +33,14 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 - Split monolithic `tests/test_cli.py` into per-domain test modules: `test_cli.py` (entry point), `test_deploy.py`, `test_quest.py`, `test_grimoire.py`, `test_health.py`, and `test_orchestrator.py`. Added `conftest.py` with shared fixtures.
 - Added new test cases: `deploy down --volumes` flag pass-through, `deploy down`/`ps` Docker-unavailable guards, `deploy up --file` compose override, `quest start --resume` flag forwarding, `quest start --config` custom path, pipeline failure propagation, grimoire `install-watcher --clone-dir`, mixed-state container health check, orchestrator checkpoint persistence on success and failure, and empty task list handling.
 
+### Removed
+
+- Removed `infra/env/` and the old `infra/services/grimoirelab/` directories. Their contents either moved to the new structure (`infra/services/grimoirelab/docker-compose.yml` → `infra/open-pulse-stack/docker-compose.grimoirelab.yml`; the `applier/`, `config/`, `python-scripts/`, `scripts/`, and `README.md` siblings → `infra/open-pulse-stack/grimoirelab/`) or were superseded (`infra/env/.env.example` → `<repo>/.env.example` + `infra/.env.example`; `infra/services/grimoirelab/.env.dist` → unified `infra/.env.example`).
+
 ### Added
 
+- Added `infra/.env.example` (deployment template — comprehensive: every container-side knob the local stack needs) and `<repo>/.env.example` (tool/client template — slim: endpoint overrides + auth for talking to remote infra). The CLI auto-substitutes `OPEN_PULSE_DATA_DIR` and `OPEN_PULSE_HOST_PATH` to absolute paths when seeding `infra/.env`.
+- Added `env_file: ../.env` directives to the crawler, extractor, and hub services in `infra/open-pulse-stack/docker-compose.yml` so any per-service knob set in `infra/.env` reaches the container automatically. This is what makes the extractor's V2 / RAG / agent-runtime knobs (V2_*, RCP_TOKEN, OPENALEX_MAILTO, HF_TOKEN, EPFL_GRAPH_*, ORCID_*, INDEX_QDRANT_URL, GRIMOIRE_GITHUB_TOKEN) reach the container without per-key `environment:` plumbing.
 - Added new command groups `services` and `gui` with grimoire subcommands split by domain (`services grimoire prepare-config`, `services grimoire install-watcher`, `gui grimoire`).
 - Added `open_pulse.utils.grimoire` package (`sparql_config.py`, `cronjob.py`) and `open_pulse.gui.grimoire_streamlit`.
 - Added new `open_pulse.services` modules:
@@ -135,6 +146,6 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 - Updated `.gitignore` with docs tooling artifacts (`node_modules/`, `docs-site/build/`).
 - Rewrote root `README.md` for onboarding with project purpose, architecture overview, DB stack quick start, `uv`-based analysis quick start, documentation navigation links, and release/contribution references.
 - Refactored root `docker-compose.yml` into a profile-aware topology with default Neo4j plus opt-in `analysis`, `grimoirelab`, and `orchestration` services.
-- Added healthchecks and dependency readiness gates for key profile services (`neo4j`, `analysis-notebook`, and `grimoirelab-db`).
+- Added healthchecks and dependency readiness gates for key profile services (`neo4j` and `grimoirelab-db`).
 - Expanded `analysis/README.md` with sequential orchestration usage and checkpoint resume guidance.
 - Expanded `analysis/README.md` with container build/smoke/non-root checks and devcontainer setup guidance.