4 - AWS-ML-healthcare-pipeline (#11)

* renames unit tests #4
* adds ref to solve docker login AWS #4
* save model under `dataset/models` path with timestamp to version control model; passed pre-commit hooks #4
* updates aws steps to setup workflow #4
* renaming project to automated-multimodal-medical-image-reporting
* adds bash scripts to unit testing model and datasets; tidies content of the project #4
* udpates .devcontainer/Dockerfile, README and scripts  #4
* adds Dockerfile with scripts/entrypoint.bash #4
* adds sections to .devcontainer/aws/README.md #4
* created job and adds its config file #4
* adds batch details to create job for aws batch workflow #4
* reorganise aws-services; adds aws-s3-bucket; adds aws-batch; adds scripts and udpates READMEs #4
* adds /docs/aws-services/aws-batch/ details and config file example #4
* tidies READMEs with figs #4
* adds links for AWS access portal and main fig in aws-services #4
* moving fig placement #4
* adds configs file for aws-servies #4
* adds aws-workflow.svg fig based on kumar-et-al-2022 #4
* fixing some hyperlinks #4
* adds Copyright (c) University College London to scripts; exclude check-yaml for docs/aws-services/*; and other minor updates #4
* adds SPDX which `defines a standardised way to share copyright and licensing information between projects and people` #4
* adds unit test for test_gpu_availability #4
* updates aws-batch workflow adding bash for stop local container and docker files workspace name #4
* tidied up main README with collapse section for train/eval and optional app #4
* adds Workflow to setup AWS Batch with input from @Tomasz-Grzybowski-AWS #4
* adds docs/resources.md solves #20
* removes comments and tidies bash scripts and adds hyperlinks to README #4
* minor changes
* fixing Parameter validation failed for `register-job-definition` from @Tomasz-Grzybowski-AWS #4
* adds trus-policy.json file; bash scripts for create/attach roles; and creates log group @Tomasz-Grzybowski-AWS #4
* adds batch-describe-jobs.bash but needs  --jobs <job-id> #4
* adds bashes update-job-queue.bash describe-job-queues.bash wih the help of @Tomasz-Grzybowski-AWS; and other minor udpates to docs #4
* updates banner and aim of the repo
* fixing links for resources
* new path `tutorials/aws-services/` and related updates #4
* tidies up links for aws-services and other cosmetic changes #4
* cosmetic changes to aws-batch/readme and runs and tests job monitoring scripts from the suggestions of @Tomasz-Grzybowski-AWS #4
* adds hyperlink
* updates tutorials/aws-services/aws-workflow.svg #4
* passed pre-commit hooks #4

Author: mxochicale

README.md

@@ -1,31 +1,27 @@
 <div style="text-align: center;" align="center">
-  <img src="docs/figs/cdi-hub-banner.svg" alt="real-time ai diagnosis for nystagmus" width="600"/>
-  <h1> :robot: CDI-HUB Reference Applications </h1>
+  <img src="docs/figs/cdi-hub-banner.svg" alt="cdi-hub" width="600"/>
+  <h1> :robot: CDI-HUB: Resources, Applications, and More  </h1>
 </div>
 
-This is central repository for CDI Applications, featuring templates, resources, documentation, and more. 
+This is the `cdi-hub` repository for the UCL Centre for Digital Innovation (UCL CDI), showcasing applications, use cases, tutorials, templates, resources, documentation, and more. It is designed to support researchers in seamlessly transitioning from idea to research to production with a focus on AWS services.
 
 ## Table of contents
-* [Overview](#overview)
 * [Prerequisites](#prerequisites)
 * [Tutorials](#tutorials)
 * [Contributing](#contributing)
 * [Glossary and references](#glossary-and-references)
 
-# Overview
-This repository is a comprehensive collection of CDI applications, offering templates, tutorials, and resources tailored for CDI cohorts, with a focus on AWS services.
-
 # Prerequisites 
 Github account, AWS account, and UCL account. 
 
 # Tutorials
-Explore [tutorials](tutorials) that showcase examples of best practices in machine learning and software development, such as working with multimodal medical data to train and test large language model pipelines and more.
+Explore [tutorials](tutorials) that showcase examples of best practices in machine learning and software development using AWS services, such a workflow to setup AWS Batch, multimodal medical data to train and test large language model pipelines and more.
 
 # Contributing
 See [CONTRIBUTING](CONTRIBUTING.md)
 
-# Glossary and references
-See [glossary](docs/glossary.md) and [references](docs/references.md).
+# Resources
+See [resources](docs/resources.md) and [glossary](docs/glossary.md).
 
 ## :octocat: Cloning repository
 * Generate your SSH keys as suggested [here](https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent)

docs/README.md

@@ -1,5 +1,5 @@
 # Documents
 
 * [Glossary](glossary.md)
-* [References](references.md)
+* [Resources](resources.md)
 

docs/figs/cdi-hub-banner.svg

@@ -0,0 +1,46 @@
+# Resources
+
+## AWS Support
+> You can change your AWS Support Plans for your account based on your business needs.
+[AWS Support Plans](https://docs.aws.amazon.com/awssupport/latest/user/aws-support-plans.html)
+
+## Security
+> Automate AWS security checks and centralize security alerts
+[AWS Security Hub](https://aws.amazon.com/security-hub/)
+
+>The AWS Startup Security Baseline (AWS SSB) is a set of controls that create a minimum foundation for businesses to build securely on AWS without decreasing their agility. These controls form the basis of your security posture and are focused on securing credentials, enabling logging and visibility, managing contact information, and implementing basic data boundaries.
+[AWS startup security baseline](https://docs.aws.amazon.com/prescriptive-guidance/latest/aws-startup-security-baseline/welcome.html)
+
+## Code Assistant
+> The most capable generative AI–powered assistant for software development [Amazon Q Developer](https://aws.amazon.com/q/developer/) (former [AWS Codewhisperer](https://docs.aws.amazon.com/codewhisperer/latest/userguide/what-is-cwspr.html))
+
+## AWS CDK
+> If your stack operation fails, you don't have to roll back resources that were already successfully provisioned and start over from the beginning every time. Instead, you can troubleshoot resources in a CREATE_FAILED or UPDATE_FAILED status, and then resume provisioning from the point where the problem occurred.  [CDK debugging: Stack failure options](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stack-failure-options.html)
+
+## Workshops 
+
+### AWS Amplify
+> Follow step-by-step instructions to create a simple full-stack web application
+* [Create a simple web application using AWS Amplify](https://aws.amazon.com/getting-started/hands-on/build-react-app-amplify-graphql/module-one/)
+
+### AWS Cognito
+> You will deep dive into Cognito and build out an Auth solution for a mythical Pet Store. You will be working with: [Amazon Cognito Workshop](https://catalog.workshops.aws/wyld-pets-cognito/en-US)
+
+### AWS SageMaker
+> This workshop takes you through a sequence of Jupyter notebooks demonstrating how to move from an ML idea to a production-ready solution by using Amazon SageMaker. [Amazon SageMaker MLOps](https://catalog.workshops.aws/mlops-from-idea-to-production/en-US). [Fine-tune a Foundation LLM on Trainium and deploy it for inference on Inferentia](https://catalog.us-east-1.prod.workshops.aws/workshops/1150f506-004e-4905-972e-61705427adb8/en-US)
+
+### AWS Bedrock
+> This workshop is designed to help introduce generative AI concepts through dozens of hands-on exercises. [Building with Amazon Bedrock and LangChain](https://catalog.workshops.aws/building-with-amazon-bedrock/en-US)
+
+## Tutorials
+
+### AWS Billing and Cost Management
+> AWS Billing and Cost Management provides a suite of features to help you set up your billing, retrieve and pay invoices, and analyze, organize, plan, and optimize your costs. [What is AWS Billing and Cost Management?](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-what-is.html#budgeting-planning-features)
+
+## Other resources
+* https://github.com/UCL-ARC/aws-playpen-docs
+* https://github.com/aws-samples
+* https://github.com/aws-samples/aws-research-workshops
+* https://github.com/UCL-ARC/terraform-template
+* https://skillbuilder.aws/ 
+* https://github.com/warmchang/techlead-1/blob/master/README_EN.md

docs/references.md

@@ -4,27 +4,15 @@
 The following tutorials provide both local and AWS setups to implement best practices in software and machine learning development. Key features include:
 1. pyproject: Tracks dependencies for streamlined project management.
 2. Pre-commit hooks: Ensures high-quality code with automated checks.
-3. Unit tests with pytest: Enables robust testing for data processing, model selection, training, inference, and evaluation, aligning with the FDA's Good Machine Learning Practices (FDA 2021).
+3. Unit tests with pytest: Enables robust testing for data processing, model selection, training, inference, and evaluation, aligning with the FDA's Good Machine Learning Practices [(FDA 2021)](https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles).
 4. Documentation (docs, CONTRIBUTING, CODE_OF_CONDUCT): Provides guidance for setup and requirements, outlines contribution protocols, and establishes a code of conduct to foster effective collaboration.
 5. Datasets (datasets): Manages data policies, preparation, and preprocessing needs.
-6. Source folder (src): Organizes APIs, models, and utilities for clarity and scalability.
+6. Source folder (src): Organises APIs, models, and utilities for clarity and scalability.
 
 This structured approach fosters efficient, maintainable, and scalable workflows!
 
-## Automatic medical image reporting (amir)
-This project leverages multimodal data, combining X-rays and doctors' reports, to predict diseases in unseen X-ray datasets. It is designed for both local execution and integration with AWS services. For more details, see [automatic-medical-image-reporting](automatic-medical-image-reporting).
-
-## References
-
-<details>
-  <summary>Click to see references</summary>
-
-* US Food and Drug Administration (FDA), Health Canada, and United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA). "Good machine learning practice for medical device development: guiding principles." FDA (2021).
-
-* Luzón, M. Victoria, Nuria Rodríguez-Barroso, Alberto Argente-Garrido, Daniel Jiménez-López, Jose M. Moyano, Javier Del Ser, Weiping Ding, and Francisco Herrera. "A tutorial on federated learning from theory to practice: Foundations, software frameworks, exemplary use cases, and selected trends." IEEE/CAA Journal of Automatica Sinica 11, no. 4 (2024): 824-850. https://scholar.google.com/scholar?cites=5891903286652706549
-
-* Qi, Pian, Diletta Chiaro, Antonella Guzzo, Michele Ianni, Giancarlo Fortino, and Francesco Piccialli. "Model aggregation techniques in federated learning: A comprehensive survey." Future Generation Computer Systems 150 (2024): 272-293. https://scholar.google.com/scholar?cites=2973562302699519092
-
-
-</details>
+## End-to-End AI Workflow for Automated Multimodal Medical Image Reporting
+This project leverages multimodal data, combining X-rays and doctors' reports, to predict diseases in unseen X-ray datasets. It is designed for both local execution and integration with AWS services. See further details [here](automated-multimodal-medical-image-reporting).
 
+## AWS services
+This tutorial provides a hands-on introduction to AWS services, including setting up, creating resources, configuring architecture, estimating resource costs, and benchmarking results. For more details, see [here](aws-services).

docs/resources.md

@@ -0,0 +1,24 @@
+FROM ubuntu:24.04
+FROM python:3.12
+#FROM nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04
+
+# ARGs passed by docker compose, ENV defined for local use
+ARG USER
+ENV USERNAME=${USER}
+
+ENV DEBIAN_FRONTEND noninteractive
+
+SHELL ["/bin/bash", "-cu"]
+
+RUN apt-get update && apt-get install -y python3 python3-pip
+RUN apt install -y vim vim-gtk3
+
+WORKDIR /ammir
+COPY . /ammir
+RUN pip install --upgrade pip
+RUN pip install --editable ".[test, learning]"
+
+ENTRYPOINT ["tail", "-f", "/dev/null"]
+
+# Set default shell to /bin/bash
+SHELL ["/bin/bash", "-cu"]

tutorials/README.md

@@ -0,0 +1,37 @@
+## Docker and data management
+
+### Build
+```
+docker compose -f docker-compose.yml build #Building estimated time
+
+#$docker images
+#REPOSITORY   TAG     IMAGE ID      CREATED         SIZE
+#ammir        latest  <ID>          current_time    16.8GB
+```
+
+### Launch and test image
+```
+bash launch_and_test_docker_image_locally.bash
+```
+
+
+### Commands
+```
+docker images
+docker ps
+docker attach <ID>
+docker stop <ID>
+docker stop $(docker ps -a -q) # stop all containers
+docker rename keen_einstein mycontainer
+docker rmi --force <ID>
+docker image prune -a #clean unused images
+docker system prune -f --volumes #clean unused systems
+docker inspect <container-name> (or <container-id>) 
+docker volume ls
+docker volume rm  <ID>
+```
+
+### References
+* [python-docker-image-build-install-required-packages](https://dev.to/behainguyen/python-docker-image-build-install-required-packages-via-requirementstxt-vs-editable-install-572j)
+* [speed-up-your-docker-builds-with-cache-from](https://lipanski.com/posts/speed-up-your-docker-builds-with-cache-from)
+* [docker-cheatsheets](https://github.com/cheat/cheatsheets/blob/master/docker)

tutorials/automated-multimodal-medical-image-reporting/.devcontainer/Dockerfile

@@ -0,0 +1,14 @@
+services:
+  main_container:
+    image: ammir:latest
+    container_name: ammir_v0.0.1
+    build:
+      context: ..
+      # cache_from:
+      #   - ammir:latest
+      dockerfile: .devcontainer/Dockerfile
+      args:
+        USER: ${USER}
+    platform: linux/x86_64
+    volumes:
+      - ${PWD}/..:/home/${USER}/workspace

tutorials/automated-multimodal-medical-image-reporting/.devcontainer/README.md

@@ -0,0 +1,5 @@
+docker run --name ammir_container --detach ammir:latest # -v $HOME/datasets/chest-xrays-indiana-university:datasets
+docker exec -it ammir_container bash
+
+# docker run -d -v <local path>:<container-path> <docker-image-name>
+# docker exec -it <> bash <docker-container-id> bash #or <NAME> bash