Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,22 @@ Only triggers the `lint`, `tests`, `build-container`, `determine-changes` and `i

## automate_truststore.yml

This workflow will update the Genome nexus truststor file on a schedule and then run the mutation processing AND consortium release steps of the pipeline to make sure the pipeline is workfing with the new truststore.
This workflow will update the Genome nexus truststore file on a schedule and then run the mutation processing AND consortium release steps of the pipeline to make sure the pipeline is working with the new truststore.

The truststore used in our pipeline to run the genome nexus annotator on MAF data.

For more background on the truststore and troubleshooting, please see:
[Updating Genome Nexus Annotator and Dependencies](https://sagebionetworks.jira.com/wiki/spaces/APGD/pages/3016687662/Updating+Genome+Nexus+Annotator+and+Dependencies#Updating-the-trust-ssl-file)
[Updating Genome Nexus Annotator and Dependencies](https://sagebionetworks.jira.com/wiki/spaces/APGD/pages/3016687662/Updating+Genome+Nexus+Annotator+and+Dependencies#Updating-the-trust-ssl-file)


## build_genome_nexus_annotator.yml

This workflow will update and build the Genome Nexus `annotator.jar` file from a
user inputted param: `commit_hash`. This `commit_hash` value comes from pulling from a specific commit in the [genome-nexus-annotation-pipeline](github.com/genome-nexus/genome-nexus-annotation-pipeline).

This workflow runs **on demand** [through triggering the workflow via a pull request to either modify the commit hash or updating something else](https://github.com/Sage-Bionetworks/Genie/actions/workflows/build_genome_nexus_annotator.yml).

After a successful build and upload to the [Genome Nexus Testing folder on synapse](https://www.synapse.org/Synapse:syn70781006), it will run the mutation processing step of the pipeline to annotate the test data. The manual part will be checking the annotated test data after this to make sure the results are expected.

For more background on the `annotator.jar`, validating the annotated results after the build and troubleshooting, please see:
[Updating Genome Nexus Annotator and Dependencies](https://sagebionetworks.jira.com/wiki/spaces/APGD/pages/3016687662/Updating+Genome+Nexus+Annotator+and+Dependencies#Updating-the-annotator.jar)
145 changes: 145 additions & 0 deletions .github/workflows/build_genome_nexus_annotator.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
name: Build and Upload Genome Nexus Annotator

on:
push:
paths:
- '.github/workflows/build_genome_nexus_annotator.yml'

env:
# versions for dependencies used in annotator.jar build
MAVEN_VERSION: '3.9.9'
COMMIT_HASH: "2e67ebd08cf7c26bf1f55f2baf4b73ac36531119"
JAVA_VERSION: "21"
JAVA_DISTRIBUTION: "corretto"
# synapse id of the Genome Nexus Testing folder
OUTPUT_FOLDER_SYNAPSE_ID: syn70781006
# env vars related to testing the annotator.jar
PROD_DOCKER: ghcr.io/sage-bionetworks/genie:main
SYNAPSE_AUTH_TOKEN: ${{ secrets.SYNAPSE_AUTH_TOKEN }}
TEST_PROJECT_SYNID: syn7208886
jobs:
build-annotator:
runs-on: ubuntu-latest

steps:
- name: Checkout workflow repo
uses: actions/checkout@v4

- name: Set up Java
uses: actions/setup-java@v4
with:
java-version: ${{ env.JAVA_VERSION }}
distribution: ${{ env.JAVA_DISTRIBUTION }}

- name: Install Maven
run: |
wget https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.zip
unzip apache-maven-${MAVEN_VERSION}-bin.zip
echo "$PWD/apache-maven-${MAVEN_VERSION}/bin" >> $GITHUB_PATH

- name: Install Python & Synapse client
run: |
python3 -m pip install --upgrade pip
pip install synapseclient chardet

- name: Clone Genome Nexus Annotation Pipeline and check out commit hash
run: |
git clone https://github.com/genome-nexus/genome-nexus-annotation-pipeline.git
cd genome-nexus-annotation-pipeline
git checkout ${{ env.COMMIT_HASH }}

- name: Configure application.properties and log4j.properties to be GENIE-specific
run: |
cd genome-nexus-annotation-pipeline
CONFIG_PATH="annotationPipeline/src/main/resources"
mkdir -p "$CONFIG_PATH"

# Copy example configs first
cp "${CONFIG_PATH}/application.properties.EXAMPLE" "${CONFIG_PATH}/application.properties"
cp "${CONFIG_PATH}/log4j.properties.EXAMPLE" "${CONFIG_PATH}/log4j.properties"

# Modify application.properties lines in place
sed -i \
-e 's|^spring.batch.job.enabled=.*|spring.batch.job.enabled=false|' \
-e 's|^spring.jmx.enabled=.*|spring.jmx.enabled=false|' \
-e 's|^chunk=.*|chunk=100|' \
-e 's|^genomenexus.enrichment_fields=.*|genomenexus.enrichment_fields=annotation_summary,sift,polyphen,my_variant_info|' \
-e 's|^genomenexus.isoform_query_parameter=.*|genomenexus.isoform_query_parameter=isoformOverrideSource|' \
-e 's|^genomenexus.base=.*|genomenexus.base=https://genie.genomenexus.org/|' \
"${CONFIG_PATH}/application.properties"

# Modify log4j.properties line
sed -i \
-e 's|^log4j.appender.a.File.*|log4j.appender.a.File=/tmp/genomenexus-logfile.log|' \
"${CONFIG_PATH}/log4j.properties"

echo "[INFO] Application and log configs updated from EXAMPLE files."

- name: Build annotator.jar (skips tests)
run: |
cd genome-nexus-annotation-pipeline
mvn clean install -DskipTests
find . -name annotator.jar

- name: Upload annotator.jar, application.properties and log4j.properties to Synapse
run: |
cd genome-nexus-annotation-pipeline
# Locate the built JAR dynamically
FILE_PATH=$(ls -t annotationPipeline/target/annotationPipeline-*.jar | head -n 1)

if [ ! -f "$FILE_PATH" ]; then
echo "Could not find annotationPipeline JAR under annotationPipeline/target/"
echo "Available .jar files:"
find annotationPipeline/target -type f -name "*.jar" || true
exit 1
fi

echo "Found JAR: $FILE_PATH"

# Rename to annotator.jar
TARGET_PATH="annotationPipeline/target/annotator.jar"
cp "$FILE_PATH" "$TARGET_PATH"

echo "Renamed to: $TARGET_PATH"
ls -lh "$TARGET_PATH"

echo "Uploading to Synapse entity ${{ env.OUTPUT_FOLDER_SYNAPSE_ID }}..."

synapse login

# Upload annotator.jar file!
synapse store "$TARGET_PATH" --parentid ${{ env.OUTPUT_FOLDER_SYNAPSE_ID }}

# Upload application and log4j files
CONFIG_PATH="annotationPipeline/src/main/resources"
synapse store "${CONFIG_PATH}/application.properties" --parentid ${{ env.OUTPUT_FOLDER_SYNAPSE_ID }}
synapse store "${CONFIG_PATH}/log4j.properties" --parentid ${{ env.OUTPUT_FOLDER_SYNAPSE_ID }}

check-annotator-build:
needs: build-annotator
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Pull Public Docker Image from GHCR
run: |
docker pull ${{ env.PROD_DOCKER }}

- name: Start Docker Container
run: |
docker run -d --name genie-container \
-e SYNAPSE_AUTH_TOKEN="${{ env.SYNAPSE_AUTH_TOKEN }}" \
${{ env.PROD_DOCKER }} \
sh -c "while true; do sleep 1; done"

- name: Run processing on mutation data in test pipeline
run: |
docker exec genie-container \
python3 /root/Genie/bin/input_to_database.py mutation \
--project_id ${{ env.TEST_PROJECT_SYNID }} \
--genie_annotation_pkg /root/annotation-tools \
--createNewMafDatabase

- name: Stop and Remove Docker Container
run: docker stop genie-container && docker rm genie-container