release 1.6.0 (#374)

Author: gordonkoehn

.github/workflows/test.yml

@@ -23,9 +23,6 @@ jobs:
           python-version: '3.11'
           cache: 'pip'
 
-      - name: Run black formatting check
-        uses: psf/black@stable
-
       - name: Set up Conda
         uses: conda-incubator/setup-miniconda@v3
         with:
@@ -49,12 +46,18 @@ jobs:
           conda activate sr2silo-dev
           poetry run pytest
 
-      - name: Run Ruff
+      - name: Run Ruff check
         run: |
           source $(conda info --base)/etc/profile.d/conda.sh
           conda activate sr2silo-dev
           poetry run ruff check .
 
+      - name: Run Ruff format check
+        run: |
+          source $(conda info --base)/etc/profile.d/conda.sh
+          conda activate sr2silo-dev
+          poetry run ruff format --check .
+
       - name: Run interrogate
         run: |
           source $(conda info --base)/etc/profile.d/conda.sh

.pre-commit-config.yaml

@@ -31,15 +31,12 @@ repos:
     hooks:
       - id: poetry-check
       - id: poetry-lock
-  - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: 'v0.9.9'
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: 'v0.14.2'
     hooks:
-      - id: ruff
-        args: [--fix, --exit-non-zero-on-fix]
-  - repo: https://github.com/psf/black
-    rev: 25.1.0
-    hooks:
-      - id: black
+      - id: ruff-check
+        args: [--fix]
+      - id: ruff-format
   - repo: https://github.com/econchick/interrogate
     rev: 1.7.0
     hooks:

README.md

@@ -12,13 +12,12 @@
 
 [![Project Status: POC – This project is currently under active development.](https://www.repostatus.org/badges/latest/concept.svg)](https://www.repostatus.org/#concept)
 [![CI/CD](https://github.com/cbg-ethz/sr2silo/actions/workflows/test.yml/badge.svg)](https://github.com/cbg-ethz/sr2silo/actions/workflows/test.yml)
-[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![Pytest](https://img.shields.io/badge/tested%20with-pytest-0A9EDC.svg)](https://docs.pytest.org/en/stable/)
 [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json)](https://github.com/charliermarsh/ruff)
 [![Pyright](https://img.shields.io/badge/type%20checked-pyright-blue.svg)](https://github.com/microsoft/pyright)
 
 ### General Use: Convert Nucleotide Alignment Reads - CIGAR in .BAM to Cleartext JSON
-sr2silo can convert millions of Short-Read nucleotide reads in the form of .bam CIGAR
+sr2silo can convert millions of Short-Read nucleotide reads in the form of `.bam` CIGAR
 alignments to cleartext alignments compatible with LAPIS-SILO v0.8.0+. It gracefully extracts insertions
 and deletions. Optionally, sr2silo can translate and align each read using [diamond / blastX](https://github.com/bbuchfink/diamond), handling insertions and deletions in amino acid sequences as well.
 
@@ -31,19 +30,18 @@ sr2silo outputs per read a JSON (compatible with LAPIS-SILO v0.8.0+):
 
 ```json
 {
-  "read_id": "AV233803:AV044:2411515907:1:10805:5199:3294",
-  "sample_id": "A1_05_2024_10_08",
-  "batch_id": "20241024_2411515907",
-  "sampling_date": "2024-10-08",
-  "location_name": "Lugano (TI)",
-  "read_length": "250",
-  "location_code": "05",
+  "readId": "AV233803:AV044:2411515907:1:10805:5199:3294",
+  "sampleId": "A1_05_2024_10_08",
+  "batchId": "20241024_2411515907",
+  "samplingDate": "2024-10-08",
+  "locationName": "Lugano (TI)",
+  "locationCode": "5",
+  "sr2siloVersion": "1.3.0",
   "main": {
     "sequence": "CGGTTTCGTCCGTGTTGCAGCCG...GTGTCAACATCTTAAAGATGGCACTTGTG",
     "insertions": ["10:ACTG", "456:TACG"],
     "offset": 4545
   },
-  "unaligned_main": "CGGTTTCGTCCGTGTTGCAGCCGATCATCTAGGT...TACAGGTTCGCGACGTGCTCGTGTGAAAGATGGCACTTGTG",
   "S": {
     "sequence": "MESLVPGFNEKTHVQLSLPVLQVRVRGFGDSVEEVLSEARQHLKDGTCGLVEVEKGV",
     "insertions": ["23:A", "145:KLM"],
@@ -85,24 +83,34 @@ Originally this was started for wrangling short-read genomic alignments from was
 
 sr2silo is designed to process nucleotide alignments from `.bam` files with metadata, translate and align reads in amino acids, gracefully handling all insertions and deletions and upload the results to the backend [LAPIS-SILO](https://github.com/GenSpectrum/LAPIS-SILO) v0.8.0+.
 
-**New Output Format for LAPIS-SILO v0.8.0+:**
-- Metadata fields are now at the root level (no nested "metadata" object)
+**Output Format for LAPIS-SILO v0.8.0+:**
+- Metadata fields use camelCase naming (e.g., `readId`, `sampleId`, `batchId`) to align with Loculus standards
+- Metadata fields are at the root level (no nested "metadata" object)
 - Genomic segments use a structured format with `sequence`, `insertions`, and `offset` fields
 - The main nucleotide segment is required and contains the primary alignment
 - Gene segments (S, ORF1a, etc.) contain amino acid sequences or `null` if empty
 - Insertions use the format `"position:sequence"` (e.g., `"123:ACGT"`)
-- Unaligned sequences are prefixed with `unaligned_` (e.g., `unaligned_main`)
+
+**Output Schema Configuration:**
+
+The output schema is defined in `src/sr2silo/silo_read_schema.py` using Pydantic models with field aliases for camelCase output. To modify the metadata fields:
+
+1. Edit `src/sr2silo/silo_read_schema.py` - Add/modify fields in `ReadMetadata` class
+2. Update `resources/silo/database_config.yaml` - Ensure field names match the Pydantic aliases
+3. Run validation: `python tests/test_database_config_validation.py`
+
+The validation ensures your Pydantic schema matches the SILO database configuration.
 
 For the V-Pipe to Silo implementation we include the following metadata fields at the root level:
 ```json
 {
-  "read_id": "AV233803:AV044:2411515907:1:10805:5199:3294",
-  "sample_id": "A1_05_2024_10_08",
-  "batch_id": "20241024_2411515907",
-  "sampling_date": "2024-10-08",
-  "location_name": "Lugano (TI)",
-  "read_length": "250",
-  "location_code": "05"
+  "readId": "AV233803:AV044:2411515907:1:10805:5199:3294",
+  "sampleId": "A1_05_2024_10_08",
+  "batchId": "20241024_2411515907",
+  "samplingDate": "2024-10-08",
+  "locationName": "Lugano (TI)",
+  "locationCode": "5",
+  "sr2siloVersion": "1.3.0"
 }
 ```
 
@@ -202,7 +210,7 @@ sr2silo process-from-vpipe \
 
 # Example: Submit to Loculus (use environment variables for credentials)
 export KEYCLOAK_TOKEN_URL=https://auth.example.com/token
-export SUBMISSION_URL=https://api.example.com/submit
+export BACKEND_URL=https://api.example.com/submit
 export GROUP_ID=123
 export USERNAME=your-username
 export PASSWORD=your-password
@@ -218,15 +226,13 @@ sr2silo submit-to-loculus --processed-file output.ndjson.zst
 
 sr2silo supports flexible configuration through environment variables, making it easy to use in different deployment scenarios including conda packages and pip installations.
 
-**Key features:**
-- CLI parameters override environment variables
-- **Recommended for credentials to avoid exposing sensitive information in command history**
+**Note:** CLI parameters override environment variables
 
 **Common configuration via environment variables:**
 ```bash
 # Authentication credentials (recommended approach for security)
 export KEYCLOAK_TOKEN_URL=https://auth.example.com/token
-export SUBMISSION_URL=https://backend.example.com/api
+export BACKEND_URL=https://backend.example.com/api
 export GROUP_ID=123
 export USERNAME=your-username
 export PASSWORD=your-password
@@ -242,20 +248,3 @@ sr2silo process-from-vpipe \
 sr2silo submit-to-loculus \
     --processed-file output.ndjson.zst
 ```
-
-### Tool Sections
-The code quality checks run on GitHub can be seen in
- - ``.github/workflows/test.yml`` for the python package CI/CD,
-
-We are using:
-
-  * [Ruff](https://github.com/charliermarsh/ruff) to lint the code.
-  * [Black](https://github.com/psf/black) to format the code.
-  * [Pyright](https://github.com/microsoft/pyright) to check the types.
-  * [Pytest](https://docs.pytest.org/) to run the unit tests code and workflows.
-  * [Interrogate](https://interrogate.readthedocs.io/) to check the documentation.
-
-
-## Contributing
-
-This project welcomes contributions and suggestions. For details, visit the repository's [Contributor License Agreement (CLA)](https://cla.opensource.microsoft.com) and [Code of Conduct](https://opensource.microsoft.com/codeofconduct/) pages.

conda-recipe/meta.yaml

@@ -1,6 +1,6 @@
 # conda recipe
 {% set name = "sr2silo" %}
-{% set version = "1.5.0" %}
+{% set version = "1.6.0" %}
 
 package:
   name: {{ name|lower }}