Skip to content

Moved all singularity-related download functions into a new file #3509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 31 commits into
base: dev
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
ae0d4dd
Moved all singularity-related download functions into a new file
muffato Mar 25, 2025
bbd297f
Added a helper method to use a temporary file as intermediate output
muffato Mar 25, 2025
a9b2b00
First round of fixing the tests
muffato Mar 25, 2025
dc333b3
downloads.singularity functions should rely on proper parameters, no …
muffato Mar 25, 2025
fbc09ac
Duplicated with the line above
muffato Mar 25, 2025
fee7182
Added tests for utils.intermediate_file and fixed it !
muffato Mar 25, 2025
c507d9e
Simpler implementation
muffato Mar 25, 2025
aba5233
Added helper functions to manage the main task so that callers don't …
muffato Mar 26, 2025
0403a37
No need to build a temp path here
muffato Mar 26, 2025
159c34e
More tests
muffato Mar 26, 2025
693da64
Fixed the symlink_registries tests
muffato Mar 26, 2025
c5379fb
Rewrote to support older versions of Python
muffato Mar 26, 2025
8ee8eed
Added a dest for the main_task-management methods added to DownloadPr…
muffato Mar 26, 2025
71ab538
Added some tests for the DownloadProgress object
muffato Mar 26, 2025
a141820
Turned two methods into functions, simplified testing
muffato Mar 26, 2025
4133c90
Test comments
muffato Mar 26, 2025
2b8fbd6
Theoretically it could be None, so check it really is a string
muffato Mar 26, 2025
7813f30
More type hints
muffato Mar 27, 2025
cb71d42
More tests for get_container_filename
muffato Mar 27, 2025
2d8cb4d
No need to keep a wraper as a method
muffato Mar 27, 2025
4bfb45e
Better comments
muffato Mar 27, 2025
d6b8f6c
New class to support the downloads
muffato Mar 27, 2025
4abd736
Introduced a context manager that creates a new sub task. Useful to g…
muffato Mar 27, 2025
31f5627
Forgot that pytest.raises exists !
muffato Mar 27, 2025
06d3760
Moved the download class to utils
muffato Mar 27, 2025
3857fae
The output path cannot be a directory or a symbolic link
muffato Mar 27, 2025
31fe914
Basic test for FileDownloader.download_file
muffato Mar 27, 2025
02cc2ca
Empty files are not allowed
muffato Mar 27, 2025
b648bc1
Test log.debug too
muffato Mar 27, 2025
467ba3f
Test the kill_with_fire flag
muffato Mar 27, 2025
ea4bc89
Refactored the FileDownloader
muffato Mar 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion nf_core/pipelines/downloads/singularity.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ def symlink_registries(self, image_path: str) -> None:
The base image, e.g. ./nf-core-gatk-4.4.0.0.img will thus be symlinked as for example ./quay.io-nf-core-gatk-4.4.0.0.img
by prepending all registries in registry_set to the image name.

Unfortunately, out output image name may contain a registry definition (Singularity image pulled from depot.galaxyproject.org
Unfortunately, the output image name may contain a registry definition (Singularity image pulled from depot.galaxyproject.org
or older pipeline version, where the docker registry was part of the image name in the modules). Hence, it must be stripped
before to ensure that it is really the base name.
"""
Expand Down
32 changes: 10 additions & 22 deletions tests/pipelines/test_download.py
Original file line number Diff line number Diff line change
Expand Up @@ -632,12 +632,12 @@ def test_symlink_singularity_images(

# Call the method
singularity_fetcher = SingularityFetcher(
[],
(
"quay.io",
"community-cr-prod.seqera.io/docker/registry/v2",
"depot.galaxyproject.org/singularity",
),
[],
mock_rich_progress,
)
singularity_fetcher.symlink_registries(f"{tmp_path}/path/to/singularity-image.img")
Expand Down Expand Up @@ -696,41 +696,29 @@ def test_symlink_singularity_images_registry(
mock_open.return_value = 12 # file descriptor
mock_close.return_value = 12 # file descriptor

download_obj = DownloadWorkflow(
pipeline="dummy",
outdir=tmp_path,
container_library=("quay.io", "community-cr-prod.seqera.io/docker/registry/v2"),
)

download_obj.registry_set = {"quay.io", "community-cr-prod.seqera.io/docker/registry/v2"}

# Call the method with registry - should not happen, but preserve it then.
# Call the method with registry name included - should not happen, but preserve it then.
singularity_fetcher = SingularityFetcher(
[],
(
"quay.io",
"quay.io", # Same as in the filename
"community-cr-prod.seqera.io/docker/registry/v2",
"depot.galaxyproject.org/singularity",
),
[],
mock_rich_progress,
)
singularity_fetcher.symlink_registries(f"{tmp_path}/path/to/quay.io-singularity-image.img")
print(mock_resub.call_args)

# Check that os.makedirs was called with the correct arguments
# mock_makedirs.assert_any_call(f"{tmp_path}/path/to", exist_ok=True)
mock_makedirs.assert_called_once_with(f"{tmp_path}/path/to", exist_ok=True)

# Check that os.symlink was called with the correct arguments
mock_symlink.assert_called_with(
# assert_called_once_with also tells us that there was no attempt to
# - symlink to itself
# - symlink to the same registry
mock_symlink.assert_called_once_with(
"./quay.io-singularity-image.img",
"./community-cr-prod.seqera.io-docker-registry-v2-singularity-image.img",
"./community-cr-prod.seqera.io-docker-registry-v2-singularity-image.img", # "quay.io-" has been trimmed
dir_fd=12,
)
# Check that there is no attempt to symlink to itself (test parameters would result in that behavior if not checked in the function)
assert (
unittest.mock.call("./quay.io-singularity-image.img", "./quay.io-singularity-image.img", dir_fd=12)
not in mock_symlink.call_args_list
)

# Normally it would be called for each registry, but since quay.io is part of the name, it
# will only be called once, as no symlink to itself must be created.
Expand Down
Loading