Merge pull request #3 from magdasmat/sync-ega-main

Author: magdasmat

.github/workflows/check_comment_coverage.yml

@@ -0,0 +1,24 @@
+name: Code comment coverage
+
+on:
+  pull_request:
+    branches:
+      - main
+      - develop
+
+jobs:
+  comment-coverage:
+    runs-on: ubuntu-latest
+
+    env:
+      TARGET_DIR: beacon
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Check comment coverage
+        run: python scripts/check_comment_coverage.py "$TARGET_DIR"

.github/workflows/check_comment_language.yml

@@ -0,0 +1,28 @@
+name: Language for comments needs to be English
+
+on:
+  pull_request:
+    branches:
+      - main
+      - develop
+
+jobs:
+  check-comments-language:
+    runs-on: ubuntu-latest
+
+    env:
+      TARGET_DIR: beacon
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          pip install pyspellchecker
+
+      - name: Check comment language
+        run: python scripts/check_comment_language.py "$TARGET_DIR"

.github/workflows/check_dead_code.yml

@@ -0,0 +1,28 @@
+name: Dead code checks
+
+on:
+  pull_request:
+    branches:
+      - main
+      - develop
+
+jobs:
+  dead-code-detection:
+    runs-on: ubuntu-latest
+
+    env:
+      TARGET_DIR: beacon
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          pip install vulture
+
+      - name: Check dead code
+        run: python scripts/check_dead_code.py "$TARGET_DIR"

.github/workflows/push_docker_image.yml

@@ -34,13 +34,16 @@ jobs:
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
           tags: |
+            # Accept the Git tag
+            type=ref,event=tag
+            # Extras that are optional
             type=ref,event=branch
             type=ref,event=pr
+            # Only applies if the tag is valid semver
             type=semver,pattern={{version}}
             type=semver,pattern={{major}}.{{minor}}
-            # set latest tag for master branch
-            type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'develop') }}
-            type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
+            # latest only on main branch
+            type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'main') }}
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
       - name: Build and push Docker image

.github/workflows/update_api_version.yml

@@ -4,35 +4,119 @@ Welcome to Beacon v2 Production Implementation (B2PI). This is an application th
 
 ## Documentation
 
-Please, go to [B2RI/B2PI docs website](https://b2ri-documentation-demo.ega-archive.org/) to know how to use Beacon v2 Production Implementation.
-
-## New release beacon v2.0-d4012a4 features added
-
-* Models plug in. Beacon PI now accepts different beacon flavours, based on different model specifications. Kicking off with two models: ga4gh beacon v2 default model and EUCAIM.
-* Conf now is not affected by further releases. Use your conf and keep it forever.
-* Cross queries between collections and non collections now are ready to be performed at full power.
-* Schema request now working: feel free to request any schema you'd like for beacon to return.
-* Validation on the fly per framework and model(s).
-* Configuration of the entities of each entry type now done by .yml files.
-* Restart of the app when conf files or generic conf is modified (no need to rebuild).
-* OR Filters (in test approach, as it is still not approved officially by GA4GH).
-* Other bug fixes.
-* Unit tests expanded, with a total of 313 now.
-
-## Main changes from B2RI
-
-* Handlers of the endpoints are classes, not functions
-* Unit testing has been developed for the application, starting with 108 unit tests that cover 4000 lines of code approximately (100%)
-* Concurrency testing has been applied for this new beacon instance, showing results of responses for more than 3 million genomic variants splitted in different datasets in less than 100 millisecs, for a total of 1000 requests made by 10 users per second at the same time.
-* Linking ids to a dataset in a yaml file is not needed anymore
-* A couple more indexes for mongoDB have been applied, that, in addition to the restructuration of the code, have improved the quickness of the responses
-* Authentication/Authorization is now applied as a decorator, not as a different container
-* LOGS now show more relevant information about the different processes (from request to response) including transaction id, the time of execution of each function and the initial call and the return call
-* Exceptions now are raised from the lower layer to the top layer, with information and status for the origin of the exception
-* Architecture of the code is not dependent on a particular database, meaning that different types of databases (and more than one) can be potentially applied to this instance (although now only MongoDB is the one developed)
-* Parameters are sanitized
-* Users can manage what entry types want their beacon to show by editing a manage conf file inside source
-* Admin-ui to manage all the configuration settings from a UI is in development.
+Please, go to [CRG Beacon docs website](https://b2ri-documentation-demo.ega-archive.org/) to know how to use Beacon v2 Production Implementation.
+
+## Upgrading mongoDB (New release v2.2)
+
+As mongoDB version 5 is [end of life](https://github.com/EGA-archive/beacon2-pi-api/issues/161), two new images for versions 6 and 7 of mongoDB are now available. Just comment and uncomment the image you prefer. **The mongoDB image that is recommended for production is the one at version 7 (default one).**
+
+### Upgrading MongoDB to version 6 from a container with an existing data for version 5
+
+This version is compatible with previous version 5. Just build the container with the version 6 uncommented and you will be ready to use it without further actions on your side.
+
+### Upgrading MongoDB to version 7 from a container with an existing data for version 5
+
+If your container you want to upgrade is version 5 and you want version 7, first you will have to rebuild the container with the version 6. After that you will need to go to the mongoshell of the container:
+
+```bash
+docker exec -it mongoprod mongosh
+```
+
+And go to admin database:
+
+```bash
+use admin
+```
+
+Once authenticated, you will need to make your mongodb instance upgrades compatible with version 6 by executing the following command:
+
+```bash
+db.adminCommand({ setFeatureCompatibilityVersion: "6.0" })
+```
+
+After that, stop the container, comment version 6 and uncomment image for version 7 and rebuild the container and your mongoDB will be upgraded to version 7.
+
+### Upgrading MongoDB to version 8 from a container with an existing data for version 7
+
+If your container you want to upgrade is version 7 and you want version 8, the procedure is a bit more complex than for other mongo version updates.
+
+First of all, you will need to dump your database executing the following command:
+
+```bash
+docker exec mongoprod mongodump \                  
+  -u root \
+  -p example \
+  --authenticationDatabase admin \
+  --out /data/db/dump
+```
+
+After that, you will need to copy out the dumped files from the volume to your root local filesystem:
+```bash
+cp -r ./beacon/connections/mongo/data/db/dump ./dump
+```
+
+After that, delete completely the mongo instance with version 7:
+```bash
+docker stop mongoprod
+docker rm mongoprod
+```
+
+And the persistent data remaining:
+```bash
+rm -r beacon/connections/mongo/data/db
+```
+
+If all steps before have completed successfully, comment version 7 and uncomment version 8 for db service (mongo) at `docker-compose.yml` file. Then, build the mongo instance with version 8:
+```bash
+docker compose up -d --build db  
+```
+
+Copy the dumped files to the persistent folder in your new mongo version 8 instance:
+```bash
+cp -r ./dump ./beacon/connections/mongo/data/db 
+```
+
+And restore the dumped files to be compatible with version 8 of mongo instance:
+```bash
+docker exec mongoprod mongorestore \            
+  -u root \
+  -p example \
+  --authenticationDatabase admin \
+  --drop \
+  /data/db/dump
+```
+
+Then remove the dumped files that have already been inserted to your new mongo instance with version 8:
+```bash
+rm -r dump 
+```
+
+And reindex the data:
+```bash
+docker exec beaconprod python -m beacon.connections.mongo.reindex
+```
+
+### Downgrading MongoDB to version 5 from an exising mongodb container with a greater version
+
+First, you will have to build the mongodb container using version 6. When up and running, execute the next commands:
+
+```bash
+docker exec -it mongoprod mongosh
+```
+
+And go to admin database:
+
+```bash
+use admin
+```
+
+Once authenticated, you will need to make your mongodb instance upgrades compatible with version 6 by executing the following command:
+
+```bash
+db.adminCommand({ setFeatureCompatibilityVersion: "5.0" })
+```
+
+After that, stop the container, comment version 6 and uncomment image for version 5 and rebuild the container and your mongoDB will be downgraded to version 5.
 
 ### TLS configuration
 
@@ -67,6 +151,14 @@ docker compose up -d --build
 Note: If you have an Apple Silicon Mac and use [Colima](https://github.com/abiosoft/colima) as your container runtime, you may have to change the default Colima settings for the Mongo docker container to start correctly.
 See [Fredrik Mørstad](https://stackoverflow.com/users/11494958/fredrik-m%c3%b8rstad)'s answer to [this stackoverflow post](https://stackoverflow.com/questions/67498836/docker-chown-changing-ownership-of-data-db-operation-not-permitted) for guidance on how to resolve this issue.
 
+Alternatively. if you are updating the BeaconPI instance from a previous version, it is recommended to use the next commands:
+
+```bash
+docker stop beaconprod db
+docker compose build --no-cache beaconprod db
+docker compose up -d --force-recreate beaconprod db
+```
+
 #### Up the containers (with services in independent servers)
 
 If you wish to have each service (or some of them) in different servers, you will need to use the remote version of the docker compose file, and deploy the remote services you need by selecting them individually in the build. Example:
@@ -324,7 +416,7 @@ AV_Dataset:
 
 ### Generic configuration
 
-The beacon needs some configuration in order to show the correct mappings or information. In order to do that, the next variables inside [conf.py](https://github.com/EGA-archive/beacon-production-prototype/tree/main/beacon/conf/conf.py) can be modified for that purpose, being **uri** a critical one for showing the correct domain in the mappings of your beacon. The **uri_subpath** will be added behind this **uri** variable, in case there is an extension of the domain for your beacon.
+The beacon needs some configuration in order to show the correct mappings or information. In order to do that, the next variables inside [conf.py](https://github.com/EGA-archive/beacon-production-prototype/tree/main/beacon/conf/conf.py) can be modified for that purpose, being **uri** a critical one for showing the correct domain in the mappings of your beacon. The **uri_subpath** will be added behind this **uri** variable, in case there is an extension of the domain for your beacon. See next paragraph: Tips for configuring a nginx proxy compatible with Beacon PI.
 
 ```bash
 beacon_id = 'org.ega-archive.beacon-ri-demo'  # ID of the Beacon
@@ -361,6 +453,28 @@ org_logo_url = 'https://legacy.ega-archive.org/images/logo.png'
 org_info = ''
 ``` 
 
+#### Tips for configuring a nginx proxy compatible with BeaconPI conf.py uri and uri_subpath vars
+
+If you are building a nginx proxy on top of beacon PI instance, the configuration of your nginx proxy can be a bit tricky if you don't have in mind what do uri and uri_subpath do. First of all, uri sets the root url of your beacon, and uri_subpath adds an extension to each of the endpoints' routes.
+This means, that if you want to add a nginx proxy with an extension between the root url and the /api (uri_subpath), you will need to set the extension to the root url of the localhost, like this:
+```nginx
+location /extension/api/ {
+    proxy_pass http://localhost:5050;
+
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto $scheme;
+}
+```
+
+And your conf.py variables will need to look like:
+```bash
+uri = 'https://<yourdomain>'
+uri_subpath = '/extension/api'
+complete_url = uri + uri_subpath
+```
+
 ### Models configuration
 
 #### Enable/Disable model
@@ -636,6 +750,33 @@ After editing any comfiguration variable, save the file and restart the API to a
 docker compose restart beaconprod
 ```
 
+### State checks
+
+Now state checks are available through `/health` endpoint. The implemented checks and their flow are the ones that are shown in the diagram below:
+
+![MongoDB vulnerabilities](https://github.com/EGA-archive/beacon-production-prototype/blob/main/ri-tools/files/Machine_State_v3.jpg)
+
+## Fix for MongoDB exploit (CVE-2025-14847)
+
+Beacon PI repository has been updated so the exploit for MongoDB (CVE-2025-14847) is not an issue anymore. In order to do that, the following points have been implemented:
+* Removed exposing ports in docker-compose.yml file
+* Built done from a mongod.conf file
+* Mongo image for major version 5 adjusted to 5.0.32, not allowing prior versions with the vulnerability to be built.
+
+**Please, make sure you update your mongoDB instance and rebuild the mongoDB container after this update.**
+
+The steps to reproduce this exploit and check that your instance is not vulnerable anymore is to download this [repo](https://github.com/Security-Phoenix-demo/mongobleed-exploit-CVE-2025-14847) and insert it in beacon folder.
+
+Then build the beaconprod conainer and execute the next command:
+```bash
+docker exec -it beaconprod python beacon/mongobleed-exploit-CVE-2025-14847-main/exploit/mongobleed.py --host mongoprod
+```
+If the message is something like: 
+![MongoDB no vulnerabilities](https://github.com/EGA-archive/beacon-production-prototype/blob/main/ri-tools/files/mongobleed_ok.png)
+Then it means the instance is safe.
+Otherwise, you would get a message like:
+![MongoDB vulnerabilities](https://github.com/EGA-archive/beacon-production-prototype/blob/main/ri-tools/files/mongobleed_vuln.png)
+
 ## Tests report
 
 ![Beacon prod concurrency test](https://github.com/EGA-archive/beacon-production-prototype/blob/main/ri-tools/files/concurrencytest.png)