pipeline refinement

Author: EvaMart

data/integration/runs/latest

@@ -1 +1 @@
-20260420T214128Z-c1e60c32
+20260421T111602Z-8d84134d-manual_group_correction

docs/cli.md

@@ -4,27 +4,29 @@
 
 ### `rsetl run`
 
-Run the integration pipeline, either as a full run, a partial run, or by resuming an existing run.
+Run the full ETL/integration pipeline, a partial pipeline, a single stage, or resume an existing run.
 
 **Options**
 
-* `--tag TEXT` — append a tag to the run ID.
+* `--tag TEXT` — append a tag to the generated run ID.
 * `--resume-run TEXT` — resume an existing run by run ID or run directory path.
 * `--no-merge` — skip the `merge` stage.
 * `--no-human-updates` — skip the `human_updates` stage.
-* `--remove-opeb-metrics` — enable the `remove_opeb_metrics` stage.
-* `--from-stage {transformation,grouping,remove_opeb_metrics,conflict_detection,simplify_blocks,json_to_jsonl,disambiguation,human_updates,merge,stats}` — start the pipeline from this stage.
-* `--until {transformation,grouping,remove_opeb_metrics,conflict_detection,simplify_blocks,json_to_jsonl,disambiguation,human_updates,merge,stats}` — run until this stage, inclusive.
-* `--only {transformation,grouping,remove_opeb_metrics,conflict_detection,simplify_blocks,json_to_jsonl,disambiguation,human_updates,merge,stats}` — run only one stage.
-* `--python-exe PATH` — Python executable for subprocesses (default: `python`).
-* `--workdir PATH` — working directory (default: `.`).
-* `--runs-root PATH` — root folder for versioned outputs (default: `data/integration/runs`).
+* `--remove-opeb-metrics` — disable removal of OEB metrics.
+* `--dry-run-disambiguation` — run disambiguation without creating conflict files or GitHub issues.
+* `--from-stage STAGE` — start the pipeline from this stage.
+* `--until STAGE` — run the pipeline until this stage, inclusive.
+* `--only STAGE` — run only one stage.
+* `--python-exe PATH` — Python executable for subprocesses. Default: `python`.
+* `--workdir PATH` — working directory. Default: `.`.
+* `--runs-root PATH` — root folder for run outputs. Default: `data/integration/runs`.
 
 **Stages**
 
-The pipeline supports the following stage names:
+The pipeline supports the following stage names, in order:
 
 * `transformation`
+* `license-normalization`
 * `grouping`
 * `remove_opeb_metrics`
 * `conflict_detection`
@@ -34,85 +36,155 @@ The pipeline supports the following stage names:
 * `human_updates`
 * `merge`
 * `stats`
+* `fairsoft`
 
 **Examples**
 
 ```bash
-# standard run
+# Standard full run
 rsetl run
 
-# run with tag and skip database merge
-rsetl run --tag 2025Q4 --no-merge
+# Run with a tag
+rsetl run --tag 2026Q2
 
-# remove OEB metrics and change output directory
-rsetl run --remove-opeb-metrics --runs-root /data/rso/runs
+# Run without merging results into the database
+rsetl run --tag test-run --no-merge
 
-# stop after disambiguation, before manual annotations are applied
-rsetl run --until disambiguation --remove-opeb-metrics --tag pre-annotation
+# Run until disambiguation
+rsetl run --until disambiguation --tag pre-human-review
 
-# resume an existing run from human updates onward
-rsetl run --resume-run 20260325T103000Z-ab12cd-pre-annotation --from-stage human_updates
+# Run disambiguation in dry-run mode
+rsetl run --only disambiguation \
+  --resume-run 20260428T090000Z-ab12cd-pre-human-review \
+  --dry-run-disambiguation
 
-# resume an existing run and execute only stats
-rsetl run --resume-run 20260325T103000Z-ab12cd-pre-annotation --only stats
+# Resume an existing run from human updates onward
+rsetl run \
+  --resume-run 20260428T090000Z-ab12cd-pre-human-review \
+  --from-stage human_updates
+
+# Resume an existing run and execute only FAIRsoft scoring
+rsetl run \
+  --resume-run 20260428T090000Z-ab12cd-pre-human-review \
+  --only fairsoft
 ```
 
 **Notes**
 
 * `--tag` cannot be used together with `--resume-run`.
 * When `--resume-run` is used, the existing run directory and run ID are reused.
-* Resumed runs append execution information to the existing `manifest.json`.
+* Resumed runs append a new execution record to the existing `manifest.json`.
+* When resuming, required input files for the selected stages must already exist.
+* The `remove_opeb_metrics`, `human_updates`, and `merge` stages can be skipped with their corresponding options.
+* `--dry-run-disambiguation` only affects the `disambiguation` stage.
 
 ---
 
 ### `rsetl run-transformation`
 
 Run only the transformation step.
 
-This command executes the transformation stage independently and creates a versioned run directory with provenance metadata, similarly to the full pipeline.
+This command executes transformation independently and creates a versioned run directory with provenance metadata.
 
 **Options**
 
-* `--tag TEXT` — append a tag to the run ID.
-* `--sources TEXT` — sources passed to the transformation step (default: `all`).
-* `--python-exe PATH` — Python executable for subprocesses (default: `python`).
-* `--workdir PATH` — working directory (default: `.`).
-* `--runs-root PATH` — root folder for versioned outputs (default: `data/integration/runs`).
+* `--tag TEXT` — append a tag to the generated run ID.
+* `--sources TEXT` — sources passed to the transformation step. Default: `all`.
+* `--python-exe PATH` — Python executable for subprocesses. Default: `python`.
+* `--workdir PATH` — working directory. Default: `.`.
+* `--runs-root PATH` — root folder for run outputs. Default: `data/integration/runs`.
 
 **Examples**
 
 ```bash
-# run only transformation with default sources
+# Run transformation with default sources
 rsetl run-transformation
 
-# run only transformation with a custom tag
+# Run transformation with a custom tag
 rsetl run-transformation --tag test1
 
-# run only transformation with a custom output directory
+# Run transformation for selected sources
+rsetl run-transformation --sources biotools
+
+# Use a custom output directory
 rsetl run-transformation --runs-root /data/rso/runs
 ```
 
 ---
 
+### `rsetl enrich-publications`
+
+Enrich publication metadata and citation counts using Europe PMC.
+
+This command delegates to the publication enrichment adapter and can update MongoDB, write a JSONL cache, or run in inspection/dry-run modes depending on the options passed.
+
+**Options**
+
+* `--collection TEXT` — MongoDB collection name. Default: `publicationsMetadataDev`.
+* `--jsonl-path PATH` — path to the JSONL cache/output file. Default: `data/cache/publications_enrichment.jsonl`.
+* `--progress-every N` — print progress every N processed documents. Default: `1000`.
+* `--limit N` — maximum number of documents to process.
+* `--no-skip-seen` — do not skip DOIs already present in the JSONL file.
+* `--no-skip-existing-europe-pmc-citations` — process records even if they already contain Europe PMC citations.
+* `--no-write-cache` — do not append results to the JSONL file.
+* `--no-update-db` — do not update MongoDB.
+* `--dry-run` — show configuration and exit without running enrichment.
+
+**Examples**
+
+```bash
+# Run publication enrichment with default settings
+rsetl enrich-publications
+
+# Process only 100 records
+rsetl enrich-publications --limit 100
+
+# Print progress more frequently
+rsetl enrich-publications --progress-every 100
+
+# Run enrichment without updating MongoDB
+rsetl enrich-publications --no-update-db
+
+# Reprocess records even if Europe PMC citations already exist
+rsetl enrich-publications --no-skip-existing-europe-pmc-citations
+
+# Check the effective configuration without running
+rsetl enrich-publications --dry-run
+```
+
+---
+
+### `rsetl run-webavailability`
+
+Run the daily web availability update and ensure ToolsDev URLs exist.
+
+Additional arguments are passed through to the web availability job.
+
+**Example**
+
+```bash
+rsetl run-webavailability
+```
+
+---
+
 ### `rsetl runs list`
 
 List available pipeline runs.
 
-Shows a compact summary of existing run directories, including whether they contain a manifest, whether disambiguation output exists, and the latest executed stages.
+Shows a compact summary including run ID, update time, manifest availability, disambiguation output availability, resumability, execution count, and latest executed stages.
 
 **Options**
 
-* `--workdir PATH` — working directory (default: `.`).
-* `--runs-root PATH` — root folder for versioned outputs (default: `data/integration/runs`).
+* `--workdir PATH` — working directory. Default: `.`.
+* `--runs-root PATH` — root folder for run outputs. Default: `data/integration/runs`.
 * `--json` — output the run list as JSON.
 
 **Examples**
 
 ```bash
-# list runs in table format
 rsetl runs list
 
-# list runs as JSON
 rsetl runs list --json
 ```
 
@@ -122,60 +194,53 @@ rsetl runs list --json
 
 Show details for a specific run.
 
-By default this prints a human-readable summary including run metadata, latest execution information, paths, and latest options.
+By default this prints a human-readable summary with run metadata, latest execution information, paths, and latest options.
 
 **Arguments**
 
 * `run_ref` — run ID or full run directory path.
 
 **Options**
 
-* `--workdir PATH` — working directory (default: `.`).
-* `--runs-root PATH` — root folder for versioned outputs (default: `data/integration/runs`).
+* `--workdir PATH` — working directory. Default: `.`.
+* `--runs-root PATH` — root folder for run outputs. Default: `data/integration/runs`.
 * `--json` — output the full manifest as JSON.
 
 **Examples**
 
 ```bash
-# show one run by run ID
-rsetl runs show 20260325T103000Z-ab12cd-pre-annotation
+rsetl runs show 20260428T090000Z-ab12cd-pre-human-review
 
-# show one run by full path
-rsetl runs show /data/rso/runs/20260325T103000Z-ab12cd-pre-annotation
+rsetl runs show /data/rso/runs/20260428T090000Z-ab12cd-pre-human-review
 
-# show manifest as JSON
-rsetl runs show 20260325T103000Z-ab12cd-pre-annotation --json
+rsetl runs show 20260428T090000Z-ab12cd-pre-human-review --json
 ```
 
 ---
 
 ### `rsetl runs latest`
 
-Show the most recent run.
-
-By default this prints a human-readable summary of the latest run.
+Show the latest run.
 
 **Options**
 
-* `--workdir PATH` — working directory (default: `.`).
-* `--runs-root PATH` — root folder for versioned outputs (default: `data/integration/runs`).
+* `--workdir PATH` — working directory. Default: `.`.
+* `--runs-root PATH` — root folder for run outputs. Default: `data/integration/runs`.
 * `--json` — output the full manifest as JSON.
 
 **Examples**
 
 ```bash
-# show latest run summary
 rsetl runs latest
 
-# show latest run as JSON
 rsetl runs latest --json
 ```
 
 ---
 
 ### `rsetl check-env`
 
-Check environment variables and connectivity to MongoDB, external APIs, and tokens.
+Check environment variables and API connectivity.
 
 **Example**
 
@@ -187,7 +252,8 @@ rsetl check-env
 
 ## Environment configuration
 
-You can store environment variables in a `.env` file instead of exporting them manually.
+The pipeline loads environment variables from a `.env` file if present.
+
 Example `.env`:
 
 ```env
@@ -199,24 +265,49 @@ MONGO_PWD=pass
 MONGO_AUTH_SRC=admin
 MONGO_DB=observatory
 
-# Disambiguation tokens
+# Disambiguation
 GITHUB_TOKEN=ghp_...
+GITLAB_TOKEN=...
 OPENROUTER_API_KEY=...
 HUGGINGFACE_API_KEY=...
-# GITLAB_TOKEN=...
+```
+
+## Run outputs
 
-# Optional APIs (useful for development)
-OBSERVATORY_API_URL=https://observatory.openebench.bsc.es/api
-LICENSES_API_URL=https://licenses-mapping/api
+Each full pipeline run creates a versioned directory under:
+
+```text
+data/integration/runs/<timestamp>-<gitsha>(-tag)/
 ```
 
-The pipeline automatically loads this file if present.
+A `latest` symlink points to the most recent run.
+
+Full pipeline runs write a `manifest.json` file containing:
 
-## Notes
+* run ID and run directory;
+* git short SHA;
+* created and last-updated timestamps;
+* paths to generated files;
+* latest execution options;
+* selected and executed stages;
+* masked environment configuration;
+* execution history.
 
-* `rsetl --help`, `rsetl run --help`, `rsetl run-transformation --help`, `rsetl runs --help`, `rsetl runs list --help`, `rsetl runs show --help`, `rsetl runs latest --help`, and `rsetl check-env --help` show contextual usage.
-* Each run creates a versioned directory under `data/integration/runs/<timestamp>-<gitsha>(-tag)/`.
-* A `latest` symlink points to the most recent run.
-* Full pipeline runs write a `manifest.json` file with provenance metadata.
-* Resumed runs update the existing `manifest.json` and append a new execution record to `execution_history`.
-* Transformation-only runs write a `manifest.transformation.json` file with provenance metadata.
+Transformation-only runs write their own transformation manifest.
+
+## Help
+
+Use the built-in help commands for contextual usage:
+
+```bash
+rsetl --help
+rsetl run --help
+rsetl run-transformation --help
+rsetl enrich-publications --help
+rsetl run-webavailability --help
+rsetl runs --help
+rsetl runs list --help
+rsetl runs show --help
+rsetl runs latest --help
+rsetl check-env --help
+```

docs/installation.md

@@ -21,7 +21,6 @@ Some stages call external services (APIs and model providers); make sure credent
 ??? info "Other services used"
     The following services are also accessed in some steps but require no credentials: 
 
-    - [Observatory REST API](https://observatory.openebench.bsc.es/api/docs): for FAIRness evaluation  
     - [Licenses-mapping API](https://observatory.openebench.bsc.es/licenses-mapping/docs): for SPDX license normalization  
     - [Europe PMC](https://europepmc.org/RestfulWebService) and [Semantic Scholar](https://www.semanticscholar.org/product/api) APIs: for publication enrichment