Skip to content

About

Source code for the GWAS Diversity Monitor

Topics

Resources

Stars

16 stars

Watchers

4 watching

Forks

Repository files navigation

DOI Generic badge Generic badge Generic badge

Introduction

This is a repository to accompany the GWAS Diversity Monitor, currently maintained as part of the Leverhulme Centre for Demographic Science. The dashboard can be found at:


The frontend is written in D3 4.0 (a JavaScript library for visualizing data with HTML, SVG, and CSS) with the backend hosted in Flask (a lightweight WSGI web application framework written in Python). Grateful attributions regarding data are made to the EMBL-EBI GWAS Catalog. In summary, the dashboard visualizes a systematic interactive review of all GWAS published to date. This repo can be cloned and ran on-the-fly to generate a server on localhost as required. The dashboard and associated summary statistics check daily for updates from the Catalog and update with new releases. The cron adapts a couple of functions from our Scientometric Review of all GWAS

Prerequisites

As a pre-requisite to running this locally, you will need a working Python 3 installation with all of the necessary dependencies detailed in requirements.txt (generated by pipreqs). We strongly recommend virtual environments and Anaconda.

Running the Code

To run: clone this directory, cd into the directory, download the data with "python generate_data.py", and serve the project by calling "python gwasdiversitymonitor.py". The launcher starts the Flask application behind Gunicorn, using production WSGI workers rather than Flask's development server. By default it binds to port 8000; set GWAS_HOST, GWAS_PORT, GWAS_WORKERS, GWAS_THREADS, or GWAS_TIMEOUT to override the server settings.

Data generation is staged under data/.generate_data and published only after every required raw, wrangled, JSON, and download artifact passes validation. If generation is interrupted after downloading the raw Catalog files, the next run reuses that validated raw snapshot and resumes the wrangling. Publication is coordinated with application reads, and a complete previous runtime snapshot remains available if publication itself is interrupted. A run is skipped as unchanged only when the raw-input fingerprints, generation parameters, implementation, static bundle, and complete published artifact manifest all match.

To do this run "python -m venv virtualenv" from the root of the project. This will create a directory called "virtualenv". Navigate into virtualenv/bin and run "pip install -r requirements.txt" to install the requirements of the project inside your new virtual environment. Then run the project from the root of the project (above the virtualenv/) with ./virtualenv/bin/python gwasdiversitymonitor.py.

Structure

The root of the folder contains routes.py, which is the main file which runs the application. This file registers paths inside the application, creates appropriate variables, and passes them into templates (html files) which renders the page. This is our Controller in an MVC framework.

Each call to @app.route('path') defines a path and almost all of these just return templates for pages. The exception is @app.route("/getCSV/") which is handling the data download for all cases and returns a file response. The other exception in this file is, at the top, @app.context_processor. This injects some functionality into our flask templates for us. It is used to give us a user_agent checker (check what device/browser is accessing the application) check the state of the cookie policy, and inject the google analytics key from the config file.

The other important file here is DataLoader.py. This is a simple file containing a series of helper functions that route.py uses to load and reshape the data from the wrangled csv's into a shape that is workable with d3.

Data: pulls, wrangles and creates all data used in this project.

Static: This subfolder contains all of the assets for the application. CSS/Sass, fonts, images and js will all be found here. The one we care most about is the js. There is a script.js file which contains some global functions and then a file for each graph.

Each graph js file contains a function, which is then called from the global template, to instantiate the graph. The global filters recall these functions to redraw the graphs with the new filter state. "internal" filters, eg. year and parent term for each tile, are handled by event handlers inside each graph file.

Docker Deployment

To launch the app using the Docker deployment, you must first install Docker Compose.

The Docker deployment for this application uses three containers that are defined in docker-compose.yml:

  1. gwas_nginx: An nginx web server
  2. gwas_flask: The Flask web application running behind a gunicorn WSGI server (see ./deploy/flask.Dockerfile)
  3. gwas_cron: A cron scheduler to generate new data daily (see ./deploy/cron.Dockerfile)

To launch the Docker containers from the command line:

cd .../gwasdiversitymonitor
docker-compose up -d --build

You can check the status of active docker containers using:

docker ps
docker stats

While the containers are running, you can view the app from your web browser by navigating to http://localhost:80.

To stop the containers, use:

docker-compose down

Versioning

This dashboard is currently at Version 1.0.0 (with an article conditionally accepted Nature Genetics). During data generation, previously unseen combinations of known ancestry terms are classified automatically, recorded in /data/support/dict_replacer_broad.tsv, and synchronized into data_static.zip for the next run. The classifier can only use the existing Broader values from that dictionary. A combination containing a genuinely unknown component is left unclassified in /data/unmapped/unmapped_broader.txt for manual review rather than creating or guessing a new Broader type.

License

This work is free. You can redistribute it and/or modify it under the terms of the MIT license, subject to the conditions regarding the data imposed by the EMBL-EBI license. The dashboard comes without any warranty, to the extent permitted by applicable law.

Acknowledgements

We are grateful to the extensive help provided by Global Initiative (and in particular to Jamie, Nynke, Veatriki, Alex, Lea and Quentin). Additional help and specific comments came from Ian Knowles, Yi Liu, Jiani Yan, Molly Przeworski, Ben Domingue, Sam Trejo and the Sociogenome group more generally.

Future Versions

  1. Additional tabs regarding funders, network analysis, and so forth.

About

Source code for the GWAS Diversity Monitor

Topics

Resources

Stars

16 stars

Watchers

4 watching

Forks

Releases

Packages

Used by

Contributors

Languages