Merge pull request #561 from BU-ISCIII/develop

Develop merge for 2.2.11 release

Author: victor5lm

CHANGELOG.md

@@ -4,6 +4,45 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.11] - 2025-07-11 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.2.11
+
+### Credits
+
+- [Juan Ledesma](https://github.com/juanledesma78)
+- [Victor Lopez](https://github.com/victor5lm)
+- [Jaime Ozáez](https://github.com/jaimeozaez)
+
+### Template fixes and updates
+
+- Added micromamba environment PikaVirus_dev_2.6 to file hpc_slurm_pikavirus.config [#535](https://github.com/BU-ISCIII/buisciii-tools/pull/535).
+- Changed analysis_date in create_summary_report.sh to take properly dates from RSV services when using viralrecon [#540](https://github.com/BU-ISCIII/buisciii-tools/pull/540).
+- Fixed date formats for IRMA's template and excel_generator.py [#549](https://github.com/BU-ISCIII/buisciii-tools/pull/549).
+- Fixed the way whether samples are paired or single-end is detected [#550](https://github.com/BU-ISCIII/buisciii-tools/pull/550).
+- Removed pseudo_aligner parameter from RNASeq's lablog and added all missing symlinks in its RESULTS's lablog [#552](https://github.com/BU-ISCIII/buisciii-tools/pull/552).
+- Updated scratch.py and __main__.py to properly handle custom paths and temporary directories [#555](https://github.com/BU-ISCIII/buisciii-tools/pull/555).
+- Updated create_summary_report.sh to transform negative values into 0 [#556](https://github.com/BU-ISCIII/buisciii-tools/pull/556).
+- Updated the assembly stats script to handle files in RAW properly and when quast results are not available for any sample [#557](https://github.com/BU-ISCIII/buisciii-tools/pull/557).
+
+### Modules
+
+- Fixed clean module to handle subpaths stated in services.json [#543](https://github.com/BU-ISCIII/buisciii-tools/pull/543).
+- Fixed bioinfo_doc module to be able to indicate type (service_info or delivery) via CLI [#558](https://github.com/BU-ISCIII/buisciii-tools/pull/558).
+- Fixed the bioinfo_doc module to properly ask for email text notes and the scratch module to use proper scratch_tmp_path [#559](https://github.com/BU-ISCIII/buisciii-tools/pull/559).
+
+#### Added enhancements
+
+- Added new script to download multiple SRA entries in fastq format when necessary [#551](https://github.com/BU-ISCIII/buisciii-tools/pull/551).
+
+#### Fixes
+
+#### Changed
+
+- Modified bu-isciii > buisciii for all commands in README text [#548](https://github.com/BU-ISCIII/buisciii-tools/pull/548).
+
+#### Removed
+
+### Requirements
+
 ## [2.2.10] - 2025-05-21 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.2.10
 
 ### Credits

README.md

@@ -69,13 +69,13 @@ pip install .
 Run bu-isciii tools:
 
 ```bash
-bu-isciii --help
+buisciii --help
 ```
 
 Outputs the following:
 
 ```bash
-Usage: bu-isciii [OPTIONS] COMMAND [ARGS]...
+Usage: buisciii [OPTIONS] COMMAND [ARGS]...
 
 Options:
   --version                  Show the version and exit.
@@ -110,7 +110,7 @@ bu-isciii list
 Help:
 
 ```bash
-Usage: bu-isciii list [OPTIONS] <service>
+Usage: buisciii list [OPTIONS] <service>
 
   List available bu-isciii services.
 
@@ -163,13 +163,13 @@ Output:
 Example of usage:
 
 ```bash
-bu-isciii new-service <resolution_id>
+buisciii new-service <resolution_id>
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii new-service [OPTIONS] <resolution id>
+Usage: buisciii new-service [OPTIONS] <resolution id>
 
   Create new service, it will create folder and copy template depending on
   selected service.
@@ -186,13 +186,13 @@ Options:
 Example of usage:
 
 ```bash
-bu-isciii scratch --direction service_to_scratch <resolution_id>
+buisciii scratch --direction service_to_scratch <resolution_id>
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii scratch [OPTIONS] <resolution id>
+Usage: buisciii scratch [OPTIONS] <resolution id>
 
   Copy service folder to scratch directory for execution.
 
@@ -217,13 +217,13 @@ Options:
 Example of usage:
 
 ```bash
-bu-isciii finish <resolution_id>
+buisciii finish <resolution_id>
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii finish [OPTIONS] <resolution id>
+Usage: buisciii finish [OPTIONS] <resolution id>
 
   Service cleaning, remove big files, rename folders before copy and copy
   resolution FOLDER to sftp.
@@ -246,13 +246,13 @@ Finish module performs de following modules at onin this order, at once:
 Example of usage:
 
 ```bash
-bu-isciii clean <resolution_id>
+buisciii clean <resolution_id>
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii clean [OPTIONS] <resolution id>
+Usage: buisciii clean [OPTIONS] <resolution id>
 
   Service cleaning. It will either remove big files, rename folders before
   copy, revert this renaming, show removable files or show folders for no
@@ -278,21 +278,21 @@ Options:
 ##### scratch back
 
 ```bash
-bu-isciii scratch --direction scratch_to_service <resolution_id>
+buisciii scratch --direction scratch_to_service <resolution_id>
 ```
 
 ##### copy_sftp
 
 Example of usage:
 
 ```bash
-bu-isciii copy-sftp <resolution_id>
+buisciii copy-sftp <resolution_id>
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii copy-sftp [OPTIONS] <resolution id>
+Usage: buisciii copy-sftp [OPTIONS] <resolution id>
 
   Copy resolution FOLDER to sftp, change status of resolution in iskylims and
   generate md, pdf, html.
@@ -311,13 +311,13 @@ Options:
 Example of usage:
 
 ```bash
-bu-isciii bioinfo-doc <resolution_id>
+buisciii bioinfo-doc <resolution_id>
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii bioinfo-doc [OPTIONS] <resolution id>
+Usage: buisciii bioinfo-doc [OPTIONS] <resolution id>
 
   Create the folder documentation structure in bioinfo_doc server
 
@@ -342,13 +342,13 @@ Options:
 Example of usage:
 
 ```bash
-bu-isciii archive --date_from 2022-01-01 --date_until 2023-01-01
+buisciii archive --date_from 2022-01-01 --date_until 2023-01-01
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii archive [OPTIONS]
+Usage: buisciii archive [OPTIONS]
 
   Archive services or retrieve services from archive
 
@@ -375,13 +375,13 @@ Options:
 Example of usage:
 
 ```bash
-bu-isciii autoclean-sftp
+buisciii autoclean-sftp
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii autoclean-sftp [OPTIONS]
+Usage: buisciii autoclean-sftp [OPTIONS]
 
   Clean old sftp services
 
@@ -397,13 +397,13 @@ Options:
 Example of usage:
 
 ```bash
-bu-isciii fix-permissions -d /data/bi
+buisciii fix-permissions -d /data/bi
 ```
 
 Help:
 
 ```bash
-Usage: bu-isciii fix-permissions [OPTIONS]
+Usage: buisciii fix-permissions [OPTIONS]
 
   Fix permissions
 

buisciii/__main__.py

@@ -57,7 +57,7 @@ def run_buisciii():
     )
 
     # stderr.print("[green]                                          `._,._,'\n", highlight=False)
-    __version__ = "2.2.10"
+    __version__ = "2.2.11"
     stderr.print(
         "[grey39]    BU-ISCIII-tools version {}".format(__version__), highlight=False
     )
@@ -252,7 +252,7 @@ def new_service(ctx, resolution, path, no_create_folder, ask_path):
     "-t",
     "--tmp_dir",
     type=click.Path(),
-    default="/data/ucct/bi/scratch_tmp/bi/",
+    default="/scratch/bi/",
     help="Directory to which the files will be transfered for execution. Default: /data/ucct/bi/scratch_tmp/bi/",
 )
 @click.option(
@@ -411,7 +411,7 @@ def copy_sftp(ctx, resolution, path, ask_path, sftp_folder):
     "-t",
     "--tmp_dir",
     type=click.Path(),
-    default="/data/ucct/bi/scratch_tmp/bi/",
+    default="/scratch/bi/",
     help="Absolute path to the scratch directory containing the service.",
 )
 @click.pass_context

buisciii/assets/utils/lablog_SRA_download

@@ -0,0 +1,11 @@
+# This script is supposed to be run when having to download .fastq files from multiple SRA entries.
+# To run this script, please copy it to the corresponding RAW folder from the service.
+
+#module load SRA-Toolkit/3.0.5-gompi-2021a
+mkdir logs
+
+while IFS= read -r in || [ -n "$in" ]; do
+    echo "srun --partition short_idx --mem 4G --cpus-per-task 2 --time 02:00:00 --job-name ${in}_SRA_DOWNLOAD --output logs/SRA_DOWNLOAD.${in}.%j.log fastq-dump --split-files --gzip ${in} &" >> _01_SRA_fastq_download.sh
+done < ../ANALYSIS/samples_id.txt
+
+echo 'for f in *_1.fastq.gz *_2.fastq.gz; do [ -e "$f" ] || continue; mv "$f" "$(echo "$f" | sed -E '\''s/_([12])\.fastq\.gz$/_R\1.fastq.gz/'\'')"; done' > _02_fix_filenames.sh

buisciii/bioinfo_doc.py

@@ -51,11 +51,19 @@ def __init__(
         conf=None,
         email_psswd=None,
     ):
+        # Type validation
+        valid_types = ["service_info", "delivery"]
         if type is None:
             self.type = buisciii.utils.prompt_selection(
                 msg="Select the documentation type you want to create",
-                choices=["service_info", "delivery"],
+                choices=valid_types,
             )
+        else:
+            if type not in valid_types:
+                raise ValueError(
+                    f"Invalid type: '{type}'. It must be one of the following: {', '.join(valid_types)}."
+                )
+            self.type = type
         self.conf = conf.get_configuration("bioinfo_doc")
         if path is None:
             if ask_path:
@@ -647,9 +655,38 @@ def email_creation(self):
                         "\n", "<br />"
                     )
                 else:
-                    email_data["email_notes"] = buisciii.utils.ask_for_some_text(
-                        msg="Write email notes"
-                    ).replace("\n", "<br />")
+                    if buisciii.utils.prompt_yn_question(
+                        msg="Do you wish to provide a text file for email notes?",
+                        dflt=False,
+                    ):
+                        for i in range(3, -1, -1):
+                            email_data["email_notes"] = buisciii.utils.prompt_path(
+                                msg="Write the path to the file with RAW text as email notes"
+                            )
+                            if not os.path.isfile(
+                                os.path.expanduser(email_data["email_notes"])
+                            ):
+                                stderr.print(
+                                    f"Provided file doesn't exist. Attempts left: {i}"
+                                )
+                            else:
+                                stderr.print(
+                                    f"File selected: {email_data['email_notes']}"
+                                )
+                                break
+                        else:
+                            stderr.print(
+                                "No more attempts. Email notes will be given by prompt"
+                            )
+                            email_data["email_notes"] = (
+                                buisciii.utils.ask_for_some_text(
+                                    msg="Write email notes"
+                                ).replace("\n", "<br />")
+                            )
+                    else:
+                        email_data["email_notes"] = buisciii.utils.ask_for_some_text(
+                            msg="Write email notes"
+                        ).replace("\n", "<br />")
             else:
                 if buisciii.utils.prompt_yn_question(
                     msg="Do you wish to provide a text file for email notes?",

buisciii/clean.py

@@ -214,32 +214,37 @@ def scan_dirs(self, to_find):
         """
         self.check_path_exists()
         pathlist = []
-        # key: root, values: [[files inside], [dirs inside]]
         found = []
-        # TODO: This has to be revisite if it takes to long.
-        # I've tried to continue if found, but I guess there could be several work folders in the project.. Let's see how it goes
-        for root, dirs, files in os.walk(self.full_path):
-            for item_to_be_found in to_find:
-                if os.path.basename(root) == item_to_be_found:
-                    pathlist.append(root)
-                    found.append(item_to_be_found)
-                for file in files:
-                    path = os.path.join(root, file)
-                    if file == item_to_be_found:
-                        pathlist.append(path)
-                        found.append(item_to_be_found)
+
+        for item_to_find in to_find:
+            # If there are subpaths in to_find, split them into different components (e.g., ["virus_coverage", "plots"], from virus_coverage/plots)
+            path_parts = item_to_find.split(os.sep)
+            target_name = path_parts[-1]
+            parent_dirs = path_parts[:-1]
+
+            for root, dirs, files in os.walk(self.full_path):
+                if root.endswith(os.sep.join(parent_dirs)):
+                    # Check for matching directories, if any
+                    if target_name in dirs:
+                        full_path = os.path.join(root, target_name)
+                        pathlist.append(full_path)
+                        found.append(item_to_find)
+                    # Check for matching files, if any
+                    if target_name in files:
+                        full_path = os.path.join(root, target_name)
+                        pathlist.append(full_path)
+                        found.append(item_to_find)
 
         # Check found list without duplicates
-        if not list(dict.fromkeys(found)).sort() == to_find.sort():
+        if not sorted(list(dict.fromkeys(found))) == sorted(to_find):
             stderr.print(
-                "[orange]WARNING:Some files/dir to delete/rename have not been found"
+                "[orange]WARNING: Some files/dirs to delete/rename were not found"
             )
             for item in to_find:
                 if item not in found:
-                    stderr.print("[orange] %s" % item)
-            return pathlist
-        else:
-            return pathlist
+                    stderr.print(f"[orange] {item}")
+
+        return pathlist
 
     def find_work(self):
         """

buisciii/conf/configuration.json

@@ -56,7 +56,6 @@
             "'service_info.txt'",
             "'work'"
         ],
-        "scratch_path": "/scratch/bi/",
         "srun_settings": {
             "--partition": "middle_idx",
             "--time": "24:00:00",