Merge pull request #62 from ScilifelabDataCentre/divbase-cli-files-improvements

SQ-768: Improve divbase-cli files subcommand

Author: RMCrean

docs/cli/_auto_generated/files.md

@@ -16,35 +16,74 @@ $ divbase-cli files [OPTIONS] COMMAND [ARGS]...
 
 **Commands**:
 
-* `list`: list all files in the project's DivBase...
+* `ls`: list all currently available files in the...
+* `info`: Get detailed information about a specific...
 * `download`: Download files from the project's store on...
+* `stream`: Stream a file's content to standard output.
 * `upload`: Upload files to your project's store on...
-* `remove`: Remove files from the project's store on...
+* `rm`: Soft delete files from the project's store...
+* `restore`: Restore soft deleted files from the...
 
-## `divbase-cli files list`
+## `divbase-cli files ls`
 
-list all files in the project's DivBase store.
+list all currently available files in the project's DivBase store.
 
-To see files at a user specified project version (controlled by the 'divbase-cli version' subcommand),
-you can instead use the 'divbase-cli version info [VERSION_NAME]' command.
+You can optionally filter the listed files by providing a prefix.
+By default, DivBase query results files are hidden from the listing. Use the --include-results-files option to include them.
+To see information about the versions of each file, use the 'divbase-cli files info [FILE_NAME]' command instead
 
 **Usage**:
 
 ```console
-$ divbase-cli files list [OPTIONS]
+$ divbase-cli files ls [OPTIONS]
 ```
 
 **Options**:
 
+* `--tsv`: If set, will print the output in .TSV format for easier programmatic parsing.
+* `-p, --prefix TEXT`: Optional prefix to filter the listed files by name (only list files starting with this prefix).
+* `-r, --include-results-files`: If set, will also show DivBase query results files which are hidden by default.
+* `--project TEXT`: Name of the DivBase project, if not provided uses the default in your DivBase config file
+* `-c, --config PATH`: Path to your user configuration file. If you didn't specify a custom path when you created it, you don't need to set this.  [default: /home/roryc/.config/divbase/config.yaml]
+* `--help`: Show this message and exit.
+
+## `divbase-cli files info`
+
+Get detailed information about a specific file in the project's DivBase store.
+
+This includes all versions of the file and whether the file is currently marked as soft deleted.
+
+**Usage**:
+
+```console
+$ divbase-cli files info [OPTIONS] FILE_NAME
+```
+
+**Arguments**:
+
+* `FILE_NAME`: Name of the file to get information about.  [required]
+
+**Options**:
+
+* `--tsv`: If set, will print the output in .TSV format for easier programmatic parsing.
 * `--project TEXT`: Name of the DivBase project, if not provided uses the default in your DivBase config file
 * `-c, --config PATH`: Path to your user configuration file. If you didn't specify a custom path when you created it, you don't need to set this.  [default: /home/roryc/.config/divbase/config.yaml]
 * `--help`: Show this message and exit.
 
 ## `divbase-cli files download`
 
-Download files from the project's store on DivBase. This can be done by either:
+Download files from the project's store on DivBase.
+
+This can be done by either:
     1. providing a list of files paths directly in the command line
-    2. providing a directory to download the files to.
+    2. providing a text file with a list of files to download (new file on each line).
+
+To download the latest version of a file, just provide its name. "file1" "file2" etc.
+To download a specific/older version of a file, use the format: "file_name:version_id"
+You can get a file's version id using the 'divbase-cli file info [FILE_NAME]' command.
+You can mix and match latest and specific versions in the same command.
+E.g. to download the latest version of file1 and version "3xcdsdsdiw829x"
+of file2: 'divbase-cli files download file1 file2:3xcdsdsdiw829x'
 
 **Usage**:
 
@@ -69,12 +108,42 @@ You can also specify "." to download to the current directory.
 * `-c, --config PATH`: Path to your user configuration file. If you didn't specify a custom path when you created it, you don't need to set this.  [default: /home/roryc/.config/divbase/config.yaml]
 * `--help`: Show this message and exit.
 
+## `divbase-cli files stream`
+
+Stream a file's content to standard output.
+
+This allows your to pipe the output to other tools like 'less', 'head', 'zcat' and 'bcftools'.
+
+Examples:
+- View a file: divbase-cli files stream my_file.tsv | less
+- View a gzipped file: divbase-cli files stream my_file.vcf.gz | zcat | less
+- Run a bcftools command: divbase-cli files stream my_file.vcf.gz | bcftools view -h -  # The "-" tells bcftools to read from standard input
+
+**Usage**:
+
+```console
+$ divbase-cli files stream [OPTIONS] FILE_NAME
+```
+
+**Arguments**:
+
+* `FILE_NAME`: Name of the file you want to stream.  [required]
+
+**Options**:
+
+* `--version-id TEXT`: Specify this if you want to look at an older/specific version of the file. If not provided, the latest version of the file is used. To get a file's version ids, use the 'divbase-cli file info [FILE_NAME]' command.
+* `--project TEXT`: Name of the DivBase project, if not provided uses the default in your DivBase config file
+* `-c, --config PATH`: Path to your user configuration file. If you didn't specify a custom path when you created it, you don't need to set this.  [default: /home/roryc/.config/divbase/config.yaml]
+* `--help`: Show this message and exit.
+
 ## `divbase-cli files upload`
 
-Upload files to your project's store on DivBase by either:
-    1. providing a list of files paths directly in the command line
-    2. providing a directory to upload
-    3. providing a text file with or a file list.
+Upload files to your project's store on DivBase:
+
+To provide files to upload you can either:
+    1. provide a list of files paths directly in the command line
+    2. provide a directory to upload
+    3. provide a text file with or a file list.
 
 **Usage**:
 
@@ -84,7 +153,7 @@ $ divbase-cli files upload [OPTIONS] [FILES]...
 
 **Arguments**:
 
-* `[FILES]...`: Space seperated list of files to upload.
+* `[FILES]...`: Space separated list of files to upload.
 
 **Options**:
 
@@ -95,18 +164,20 @@ $ divbase-cli files upload [OPTIONS] [FILES]...
 * `-c, --config PATH`: Path to your user configuration file. If you didn't specify a custom path when you created it, you don't need to set this.  [default: /home/roryc/.config/divbase/config.yaml]
 * `--help`: Show this message and exit.
 
-## `divbase-cli files remove`
+## `divbase-cli files rm`
 
-Remove files from the project's store on DivBase by either:
-    1. providing a list of files paths directly in the command line
-    2. providing a text file with or a file list.
+Soft delete files from the project's store on DivBase
 
-'dry_run' mode will not actually delete the files, just print what would be deleted.
+To provide files to delete you can either:
+    1. provide a list of file names directly in the command line
+    2. provide a text file with a list of files to delete.
+
+Note that deleting a non existent file will be treated as a successful deletion.
 
 **Usage**:
 
 ```console
-$ divbase-cli files remove [OPTIONS] [FILES]...
+$ divbase-cli files rm [OPTIONS] [FILES]...
 ```
 
 **Arguments**:
@@ -115,8 +186,35 @@ $ divbase-cli files remove [OPTIONS] [FILES]...
 
 **Options**:
 
-* `--file-list PATH`: Text file with list of files to upload.
+* `--file-list PATH`: Text file with list of files to delete.
 * `--dry-run`: If set, will not actually delete the files, just print what would be deleted.
 * `--project TEXT`: Name of the DivBase project, if not provided uses the default in your DivBase config file
 * `-c, --config PATH`: Path to your user configuration file. If you didn't specify a custom path when you created it, you don't need to set this.  [default: /home/roryc/.config/divbase/config.yaml]
 * `--help`: Show this message and exit.
+
+## `divbase-cli files restore`
+
+Restore soft deleted files from the project's store on DivBase
+
+To provide files to restore you can either:
+    1. provide a list of files directly in the command line.
+    2. provide a text file with a list of files to restore (new file on each line).
+
+NOTE: Attempts to restore a file that is not soft deleted will be considered successful and the file will remain live. This means you can repeatedly run this command on the same file and get the same response.
+
+**Usage**:
+
+```console
+$ divbase-cli files restore [OPTIONS] [FILES]...
+```
+
+**Arguments**:
+
+* `[FILES]...`: Space seperated list of files/objects in the project's store on DivBase to restore.
+
+**Options**:
+
+* `--file-list PATH`: Text file with list of files to restore.
+* `--project TEXT`: Name of the DivBase project, if not provided uses the default in your DivBase config file
+* `-c, --config PATH`: Path to your user configuration file. If you didn't specify a custom path when you created it, you don't need to set this.  [default: /home/roryc/.config/divbase/config.yaml]
+* `--help`: Show this message and exit.

docs/user-guides/quick-start.md

@@ -84,7 +84,7 @@ divbase-cli files upload --upload-dir /path/to/directory/
 Check your uploaded files:
 
 ```bash
-divbase-cli files list
+divbase-cli files ls
 ```
 
 ## Step 7: Upload sample metadata

docs/user-guides/setup_divbase_cli.md

@@ -49,7 +49,7 @@ divbase-cli auth login [YOUR_EMAIL] # you'll be prompted for your password
 You will now be able to run commands against your DivBase projects. Here are some examples:
 
 ```bash
-divbase-cli files list # runs against the default project
+divbase-cli files ls # runs against the default project
 divbase-cli files download file1.vcf.gz file2.vcf.gz --project my-non-default-project
 divbase-cli files upload file1.vcf.gz file2.vcf.gz --project my-non-default-project
 ```

packages/divbase-api/src/divbase_api/routes/s3.py

@@ -14,7 +14,7 @@
 import logging
 from typing import Annotated
 
-from fastapi import APIRouter, Depends, HTTPException, status
+from fastapi import APIRouter, Body, Depends, HTTPException, status
 from fastapi.concurrency import run_in_threadpool
 
 from divbase_api.api_config import settings
@@ -36,9 +36,13 @@
     DownloadObjectRequest,
     FileChecksumResponse,
     GetPresignedPartUrlsRequest,
+    ListObjectsRequest,
+    ListObjectsResponse,
+    ObjectInfoResponse,
     PreSignedDownloadResponse,
     PreSignedSinglePartUploadResponse,
     PresignedUploadPartUrlResponse,
+    RestoreObjectsResponse,
     UploadSinglePartObjectRequest,
 )
 from divbase_lib.divbase_constants import MAX_S3_API_BATCH_SIZE
@@ -68,7 +72,7 @@ def check_too_many_objects_in_request(
 @s3_router.post("/download", status_code=status.HTTP_200_OK, response_model=list[PreSignedDownloadResponse])
 async def generate_download_urls(
     project_name: str,
-    objects_to_download: list[DownloadObjectRequest],
+    objects_to_download: Annotated[list[DownloadObjectRequest], Body(min_length=1, max_length=MAX_S3_API_BATCH_SIZE)],
     s3_signer_service: Annotated[S3PreSignedService, Depends(get_pre_signed_service)],
     project_and_user_and_role: tuple[ProjectDB, UserDB, ProjectRoles] = Depends(get_project_member),
 ):
@@ -89,12 +93,46 @@ async def generate_download_urls(
     return response
 
 
-@s3_router.get("/", status_code=status.HTTP_200_OK, response_model=list[str])
-async def list_files(
+# Post request instead of GET as GET doesn't support/encourage body content.
+@s3_router.post("/list", status_code=status.HTTP_200_OK, response_model=ListObjectsResponse)
+async def list_file_details(
+    project_name: str,
+    list_request: ListObjectsRequest,
+    project_and_user_and_role: tuple[ProjectDB, UserDB, ProjectRoles] = Depends(get_project_member),
+):
+    """
+    List all files in the project's bucket
+
+    You can provide a prefix to only list files that start with that prefix.
+    The response object includes a next_token if there are more files to list.
+    Which you can append to the next request to get the next page of results.
+    1000 files are returned at most per request.
+    """
+    project, current_user, role = project_and_user_and_role
+    if not has_required_role(role, ProjectRoles.READ):
+        raise AuthorizationError("You don't have permission to list files in this project.")
+
+    s3_file_manager = S3FileManager(
+        url=settings.s3.endpoint_url,
+        access_key=settings.s3.access_key.get_secret_value(),
+        secret_key=settings.s3.secret_key.get_secret_value(),
+    )
+
+    return await run_in_threadpool(
+        s3_file_manager.list_files_detailed,
+        bucket_name=project.bucket_name,
+        prefix=list_request.prefix,
+        next_token=list_request.next_token,
+    )
+
+
+@s3_router.get("/info", status_code=status.HTTP_200_OK, response_model=ObjectInfoResponse)
+async def get_object_info(
     project_name: str,
+    object_name: str,
     project_and_user_and_role: tuple[ProjectDB, UserDB, ProjectRoles] = Depends(get_project_member),
 ):
-    """List all files in the project's bucket"""
+    """Get details about all versions of a specific object/file in the project's store."""
     project, current_user, role = project_and_user_and_role
     if not has_required_role(role, ProjectRoles.READ):
         raise AuthorizationError("You don't have permission to list files in this project.")
@@ -105,15 +143,19 @@ async def list_files(
         secret_key=settings.s3.secret_key.get_secret_value(),
     )
 
-    return await run_in_threadpool(s3_file_manager.list_files, bucket_name=project.bucket_name)
+    return await run_in_threadpool(
+        s3_file_manager.get_detailed_object_info,
+        bucket_name=project.bucket_name,
+        object_name=object_name,
+    )
 
 
 @s3_router.post(
     "/upload/single-part", status_code=status.HTTP_200_OK, response_model=list[PreSignedSinglePartUploadResponse]
 )
 async def generate_single_part_upload_urls(
     project_name: str,
-    objects: list[UploadSinglePartObjectRequest],
+    objects: Annotated[list[UploadSinglePartObjectRequest], Body(min_length=1, max_length=MAX_S3_API_BATCH_SIZE)],
     s3_signer_service: Annotated[S3PreSignedService, Depends(get_pre_signed_service)],
     project_and_user_and_role: tuple[ProjectDB, UserDB, ProjectRoles] = Depends(get_project_member),
 ):
@@ -243,11 +285,15 @@ async def abort_multipart_upload(
 
 @s3_router.delete("/", status_code=status.HTTP_200_OK, response_model=list[str])
 async def soft_delete_files(
-    objects: list[str],
     project_name: str,
+    objects: Annotated[list[str], Body(min_length=1, max_length=MAX_S3_API_BATCH_SIZE)],
     project_and_user_and_role: tuple[ProjectDB, UserDB, ProjectRoles] = Depends(get_project_member),
 ):
-    """Soft delete files in the project's bucket. This adds a deletion marker to the files, but does not actually delete them."""
+    """
+    Soft delete files in the projects store.
+
+    This adds a deletion marker to the files, but does not actually delete them.
+    """
     project, current_user, role = project_and_user_and_role
     if not has_required_role(role, ProjectRoles.EDIT):
         raise AuthorizationError("You don't have permission to soft delete files in this project.")
@@ -265,11 +311,42 @@ async def soft_delete_files(
     )
 
 
+@s3_router.post("/restore", status_code=status.HTTP_200_OK, response_model=RestoreObjectsResponse)
+async def restore_soft_deleted_files(
+    project_name: str,
+    objects: Annotated[list[str], Body(min_length=1, max_length=MAX_S3_API_BATCH_SIZE)],
+    project_and_user_and_role: tuple[ProjectDB, UserDB, ProjectRoles] = Depends(get_project_member),
+):
+    """
+    Restore soft-deleted files from a project's store by removing the soft delete marker.
+
+    This finds the latest version of each file provided, and if it has a delete marker it removes it,
+    effectively restoring the previous version of the file.
+
+    NOTE: If you want to restore an older version of a file, you need to either:
+        1. download and upload the file again.
+        2. Hard delete the versions after the version you want to restore (making the latest version, the version you want).
+    """
+    project, current_user, role = project_and_user_and_role
+    if not has_required_role(role, ProjectRoles.EDIT):
+        raise AuthorizationError("You don't have permission to restore files in this project.")
+
+    check_too_many_objects_in_request(numb_objects=len(objects))
+
+    s3_file_manager = S3FileManager(
+        url=settings.s3.endpoint_url,
+        access_key=settings.s3.access_key.get_secret_value(),
+        secret_key=settings.s3.secret_key.get_secret_value(),
+    )
+
+    return await run_in_threadpool(s3_file_manager.restore_objects, objects=objects, bucket_name=project.bucket_name)
+
+
 # using POST as GET with body is not considered good practice
 @s3_router.post("/checksums", status_code=status.HTTP_200_OK, response_model=list[FileChecksumResponse])
 async def get_files_checksums(
-    object_names: list[str],
     project_name: str,
+    object_names: Annotated[list[str], Body(min_length=1, max_length=MAX_S3_API_BATCH_SIZE)],
     project_and_user_and_role: tuple[ProjectDB, UserDB, ProjectRoles] = Depends(get_project_member),
 ):
     """