Merge pull request #127 from seqeralabs/cursor/open-prs-dev-alignment-6d6c

fix(ci): restore lint workflow on dev; document private CA for Platform

Author: FloWuenne

.github/workflows/linting.yml

@@ -1,7 +1,5 @@
-name: nf-core linting
-# This workflow is triggered on pushes and PRs to the repository.
-# It runs the `nf-core pipelines lint` and markdown lint tests to ensure
-# that the code meets the nf-core guidelines.
+name: Pipeline linting
+# Runs pre-commit (Prettier, editorconfig, etc.) on pushes to dev and on pull requests.
 on:
   push:
     branches:
@@ -26,58 +24,3 @@ jobs:
 
       - name: Run pre-commit
         run: pre-commit run --all-files
-
-  nf-core:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Check out pipeline code
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
-
-      - name: Install Nextflow
-        uses: nf-core/setup-nextflow@v2
-
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-          architecture: "x64"
-
-      - name: read .nf-core.yml
-        uses: pietrobolcato/action-read-yaml@1.1.0
-        id: read_yml
-        with:
-          config: ${{ github.workspace }}/.nf-core.yml
-
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install nf-core==${{ steps.read_yml.outputs['nf_core_version'] }}
-
-      - name: Run nf-core pipelines lint
-        if: ${{ github.base_ref != 'master' }}
-        env:
-          GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
-        run: nf-core -l lint_log.txt pipelines lint --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
-
-      - name: Run nf-core pipelines lint --release
-        if: ${{ github.base_ref == 'master' }}
-        env:
-          GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
-        run: nf-core -l lint_log.txt pipelines lint --release --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
-
-      - name: Save PR number
-        if: ${{ always() }}
-        run: echo ${{ github.event.pull_request.number }} > PR_number.txt
-
-      - name: Upload linting log file artifact
-        if: ${{ always() }}
-        uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4
-        with:
-          name: linting-logs
-          path: |
-            lint_log.txt
-            lint_results.md
-            PR_number.txt

.prettierignore

@@ -7,3 +7,5 @@ testing/
 testing*
 *.pyc
 bin/
+# Large single-line fixture JSON (Prettier would expand to thousands of lines)
+workflows/nf_aggregate/assets/log_dirs/**/*.json

AGENTS.md

@@ -66,10 +66,10 @@ uv run --with typer --with pyyaml \
 
 ### Services overview
 
-| Service | Purpose | Run command |
-|---------|---------|-------------|
-| Nextflow pipeline | Core product — aggregates metrics from Seqera Platform runs | `nextflow run . --input <csv> --outdir results -profile docker` |
-| Python benchmark_report.py | Normalizes JSON -> JSONL, aggregates report data, then renders HTML | See "Rebuild Command" section above |
+| Service                    | Purpose                                                             | Run command                                                     |
+| -------------------------- | ------------------------------------------------------------------- | --------------------------------------------------------------- |
+| Nextflow pipeline          | Core product — aggregates metrics from Seqera Platform runs         | `nextflow run . --input <csv> --outdir results -profile docker` |
+| Python benchmark_report.py | Normalizes JSON -> JSONL, aggregates report data, then renders HTML | See "Rebuild Command" section above                             |
 
 ### Running tests
 

README.md

@@ -19,7 +19,7 @@ The pipeline fetches run data from the Seqera Platform API and generates benchma
 
 - [Nextflow](https://www.nextflow.io/docs/latest/getstarted.html#installation) >=25.10.0
 - Account in [Seqera Platform](https://seqera.io/platform/)
-- [Access token](https://docs.seqera.io/platform/23.3.0/api/overview#authentication) which is your personal authorization token for the Seqera Platform CLI. This can be created in the user menu under **Your tokens**. Export the token as a shell variable directly into your terminal if running the pipelie locally. You will not need to set this if running the pipeline within the Seqera Platform as it will automatically be inherited from the executing environment.
+- [Access token](https://docs.seqera.io/platform/23.3.0/api/overview#authentication) which is your personal authorization token for the Seqera Platform CLI. This can be created in the user menu under **Your tokens**. Export the token as a shell variable directly into your terminal if running the pipeline locally. You will not need to set this if running the pipeline within the Seqera Platform as it will automatically be inherited from the executing environment.
 
   ```bash
   export TOWER_ACCESS_TOKEN=<your access token>
@@ -46,7 +46,25 @@ nextflow run seqeralabs/nf-aggregate \
     -profile docker
 ```
 
-If you are using a Seqera Platform Enterprise instance that is secured with a private CA SSL certificate not recognized by default Java certificate authorities, you can specify a custom `cacerts` store path through the `--java_truststore_path` parameter and optionally, a password with the `--java_truststore_password`. This certificate will be used to achieve connectivity with your Seqera Platform instance through API and CLI.
+If you are using a Seqera Platform Enterprise instance that is secured with a private CA SSL certificate not recognized by default Java certificate authorities, you can specify a custom `cacerts` store path through the `--java_truststore_path` parameter and optionally, a password with the `--java_truststore_password`. This configures the Nextflow JVM used for Seqera Platform API access (see `lib/SeqeraApi.groovy`).
+
+### Seqera Platform Enterprise with a private CA (containers)
+
+For API access from Nextflow, `--java_truststore_path` / `--java_truststore_password` are usually sufficient. Task containers may still lack your private CA when they open TLS connections. As a workaround, add the following under **Advanced options → Nextflow config** in Seqera Platform (replace `tower-server-url` with your Seqera host name only, without `https://`):
+
+```groovy
+process {
+   withName: /NORMALIZE_BENCHMARK_JSONL|AGGREGATE_BENCHMARK_REPORT_DATA|RENDER_BENCHMARK_REPORT|EXTRACT_TARBALL/ {
+     beforeScript = '''
+keytool -printcert -rfc -sslserver tower-server-url:443 > PRIVATE_CERT.pem
+keytool -importcert -alias seqera-ca -file PRIVATE_CERT.pem -keystore truststore.jks -storepass changeit -noprompt
+export JAVA_TOOL_OPTIONS="-Djavax.net.ssl.trustStore=$(pwd)/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
+'''
+   }
+}
+```
+
+This downloads the server certificate, builds a small JKS truststore in the task work directory, and points the JVM inside the task at it.
 
 ### Benchmark reports
 

docs/DESIGN.md

@@ -36,6 +36,7 @@ python bin/benchmark_report.py normalize-jsonl --data-dir <run_json_dir> --outpu
 ```
 
 Responsibilities:
+
 - read run JSON payloads
 - normalize runs/tasks/metrics rows
 - optionally normalize CUR parquet into `costs.jsonl`
@@ -54,6 +55,7 @@ python bin/benchmark_report.py aggregate-report-data --jsonl-dir <jsonl_bundle>
 ```
 
 Responsibilities:
+
 - stream JSONL rows
 - compute report sections:
   - `benchmark_overview`
@@ -75,13 +77,15 @@ python bin/benchmark_report.py render-html --data <report_data.json> --output <b
 ```
 
 Responsibilities:
+
 - load report JSON
 - apply brand/logo overrides when provided
 - render self-contained HTML from the Jinja template
 
 ## Benchmark CLI surface
 
 `bin/benchmark_report.py` provides:
+
 - `normalize-jsonl`
 - `aggregate-report-data`
 - `render-html`

main.nf

@@ -13,17 +13,9 @@
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 */
 
-include { NF_AGGREGATE         } from './workflows/nf_aggregate'
-include {
-    checkCondaChannels ;
-    dumpParametersToJSON ;
-    getWorkflowVersion
-} from 'plugin/nf-core-utils'
-include {
-    paramsSummaryLog ;
-    samplesheetToList ;
-    validateParameters
-} from 'plugin/nf-schema'
+include { NF_AGGREGATE } from './workflows/nf_aggregate'
+include { checkCondaChannels ; dumpParametersToJSON ; getWorkflowVersion } from 'plugin/nf-core-utils'
+include { paramsSummaryLog ; samplesheetToList ; validateParameters } from 'plugin/nf-schema'
 
 /*
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

modules/local/AGENTS.md

@@ -2,13 +2,14 @@
 
 Each module should keep one clear responsibility and its own terse `AGENTS.md`.
 
-| Module                                  | Status     | Purpose                                        |
-| --------------------------------------- | ---------- | ---------------------------------------------- |
-| `normalize_benchmark_jsonl/`            | **active** | Raw run JSON (+ optional CUR parquet) → JSONL |
-| `aggregate_benchmark_report_data/`      | **active** | JSONL bundle → `report_data.json`              |
-| `render_benchmark_report/`              | **active** | `report_data.json` + branding → HTML           |
-| `extract_tarball/`                      | active     | Extract run-data tarballs for external runs    |
+| Module                             | Status     | Purpose                                       |
+| ---------------------------------- | ---------- | --------------------------------------------- |
+| `normalize_benchmark_jsonl/`       | **active** | Raw run JSON (+ optional CUR parquet) → JSONL |
+| `aggregate_benchmark_report_data/` | **active** | JSONL bundle → `report_data.json`             |
+| `render_benchmark_report/`         | **active** | `report_data.json` + branding → HTML          |
+| `extract_tarball/`                 | active     | Extract run-data tarballs for external runs   |
 
 Testing
+
 - Keep stage-specific pytest tests beside the module under `tests/`.
 - Keep pipeline routing/integration scenarios under top-level `tests/`.

modules/local/aggregate_benchmark_report_data/AGENTS.md

@@ -1,21 +1,26 @@
 # aggregate_benchmark_report_data
 
 Purpose
+
 - Convert the JSONL bundle into a single `report_data.json` document for rendering.
 
 Owns
+
 - `AGGREGATE_BENCHMARK_REPORT_DATA` in `main.nf`
 - Python aggregation logic in `bin/benchmark_report_aggregate.py`
 - Stage-scoped tests under `tests/`
 
 Inputs
+
 - `jsonl_bundle/` from `normalize_benchmark_jsonl`
 
 Outputs
+
 - `report_data.json`
 - `versions.yml`
 
 Invariants
+
 - This is the boundary between streaming records and presentation data.
 - Prefer streaming iteration over JSONL inputs; avoid full-file eager loads unless clearly bounded.
 - Cost joins must key by `(run_id, process, hash)` to avoid cross-process collisions.
@@ -31,8 +36,10 @@ Invariants
   - `cost_overview`
 
 Edit guidance
+
 - Keep rendering/template logic out of this stage.
 - If output schema changes, update render tests and any HTML assertions.
 
 Tests
+
 - `pytest modules/local/aggregate_benchmark_report_data/tests/test_aggregate.py -q`

modules/local/extract_tarball/AGENTS.md

@@ -1,21 +1,27 @@
 # extract_tarball
 
 Purpose
+
 - Extract external run-data tarballs into directories of JSON artifacts.
 
 Owns
+
 - `EXTRACT_TARBALL` in `main.nf`
 
 Inputs
+
 - external tarball paths from the workflow routing layer
 
 Outputs
+
 - extracted directory per tarball
 - `versions.yml`
 
 Invariants
+
 - Extraction only. Do not mix benchmark parsing or aggregation into this module.
 - Downstream workflow code is responsible for collecting JSON files from extracted directories.
 
 Edit guidance
+
 - If tarball layout assumptions change, update the pipeline tarball scenario tests.

modules/local/normalize_benchmark_jsonl/AGENTS.md

@@ -1,36 +1,43 @@
 # normalize_benchmark_jsonl
 
 Purpose
+
 - Normalize collected run JSON into a stream-friendly `jsonl_bundle/`.
 
 Owns
+
 - `NORMALIZE_BENCHMARK_JSONL` in `main.nf`
 - Python stage logic in `bin/benchmark_report_normalize.py`
 - Stage-scoped tests under `tests/`
 
 Inputs
+
 - `data_dir/` of per-run JSON payloads
 - optional CUR parquet
 
 Outputs
+
 - `jsonl_bundle/runs.jsonl`
 - `jsonl_bundle/tasks.jsonl`
 - `jsonl_bundle/metrics.jsonl`
 - optional `jsonl_bundle/costs.jsonl`
 - `versions.yml`
 
 Invariants
+
 - JSONL is the handoff format for Fusion-friendly streaming.
 - One JSON object per line.
 - Task rows should already include derived fields needed downstream (`process_short`, `wait_ms`, `staging_ms`).
 - Failed tasks are filtered here so downstream stages stay simple.
 
 Edit guidance
+
 - Keep this stage about normalization only; no report aggregation or HTML concerns.
 - If you change row shape, update:
   - `bin/benchmark_report_normalize.py`
   - `modules/local/aggregate_benchmark_report_data/tests/test_aggregate.py`
   - any CLI compatibility tests affected
 
 Tests
+
 - `pytest modules/local/normalize_benchmark_jsonl/tests/test_normalize.py -q`