Skip to content

Commit e12d977

Browse files
alismxshanice-skylightmcmcgrath13EmmanuelNwa247
authored
QC VM code (#82)
* added code for application and db containers only * removed aidbox and added execution permissions for wizard script * added database option to wizard, added profile to compose, and removed aidbox from README * removed testing branch from scripts * removed testing comment * cleaned up readme * feat: update env handling and add orchestration support (#65) * refactor(shell): improve env var parsing and quoting logic, update docker-compose container names, and add GCP image upload script * fix: update orchestration env service URLs to use dibbs container names * fix: update wizard messages and clone branch in provision script - Remove extra spaces and add quotes to schema options in wizard prompts. - Clone dibbs-vm using the alis/gcp branch. - Adjust environment variable formatting in provision.sh. * chore(provision): remove deprecated docker compose vars export to .bashrc * feat: export DIBBS_SERVICE and DIBBS_VERSION in bashrc * feat: add macOS support for tar compression Use gtar on macOS to compress the raw disk image, ensuring compatibility across systems. * fix: remove branch specification from dibbs-vm clone command The change removes the hardcoded branch (-b alis/gcp) flag to clone the default branch instead. * removed aidbox and added execution permissions for wizard script * update to readme * feat: add support for NBS_API_PUB_KEY (#68) * Add GCP VM setup documentation (#70) * Add GCP VM setup documentation * update on GCP documentation * docs: update README with documentation and examples sections (#71) (docs) Restructure docs within the docs directory, update links within the mkdocs site. (docs) Reorganize docs and update section headers to improve navigation across the docs site. * feat: add gitsha to build scripts and packer configuration (#73) Add support for using the current git commit hash to uniquely tag build directories, image names, and output files. This update modifies build.sh, gcp_image.sh, and various packer configuration files to append the short gitsha to version identifiers. --------- Co-authored-by: shanice-skylight <shanice@skylight.digital> Co-authored-by: Mary McGrath <m.c.mcgrath13@gmail.com> Co-authored-by: Chukwuemeka Emmanuel Nwakire <emmanuel@skylight.digital>
1 parent 0db42f1 commit e12d977

21 files changed

Lines changed: 78252 additions & 3 deletions

‎.gitignore‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,6 @@ docker/dibbs-ecr-viewer/.env
66
docker/dibbs-ecr-viewer/ecr-viewer.wizard
77
docker/dibbs-ecr-viewer/ecr-viewer.bak
88

9-
docker/dibbs-query-connectory/.env
10-
docker/dibbs-query-connectory/query-connectory.wizard.env
11-
docker/dibbs-query-connectory/query-connectory.env.bak
9+
dibbs-query-connector/.env
10+
dibbs-query-connector/query-connector.wizard.env
11+
dibbs-query-connector/query-connector.env.bak

‎dibbs-query-connector/README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
### Development Database Disclaimer
2+
3+
This virtual machine image includes a database intended **for development purposes only**.
4+
It is **not configured for production use** and should not be used to store sensitive or live data.
5+
Performance, security, and reliability settings may differ from production environments.
6+
7+
**Use this database only for testing and development.**
8+
9+
10+
### Query Connector Data for Query Building
11+
12+
When initializing the backend database for the first time, the Query Connector makes the value sets associated with 200+ reportable conditions available to users tasked with building queries for their jurisdiction. To run this seeding script, you'll need to obtain the UMLS and eRSD API key's using the instructions below.
13+
14+
To group value sets by condition and to group the conditions by type, the Query Connector obtains and organizes data from the eRSD and the VSAC in the following way:
15+
16+
1. The Query Connector retrieves the 200+ reportable conditions from the eRSD as well as the value sets' associated IDs.
17+
2. Using the value set IDs from the eRSD, the Query Connector retrieves the value set's comprehensive information from the VSAC, i.e., the LOINC, SNOMED, etc. codes associated with each value set ID.
18+
3. The Query Connector then organizes these value sets according to the conditions with which they're associated, making the result available to users interested in building queries. The conditions are additionally organized by category, e.g., sexually transmitted diseases or respiratory conditions, using a mapping curated by HLN Consulting.
19+
20+
21+
#### Obtaining an eRSD API Key and UMLS API Key
22+
23+
To obtain the free API keys, please visit the following URLs and follow the sign up instructions.
24+
25+
- [https://ersd.aimsplatform.org/#/api-keys](https://ersd.aimsplatform.org/#/api-keys)
26+
- [https://uts.nlm.nih.gov/uts/login](https://uts.nlm.nih.gov/uts/login)
27+
28+
29+
30+
Save your API keys and license to input during the wizard script , the environment variabels are called `ERSD_API_KEY` and `UMLS_API_KEY` The wizard will assit you with updating these values for the `.env` file so that they can be accessed when running the Query Connector app.
31+
32+
Adjust your `DATABASE_URL` as needed.
33+
34+
### Setup wizard script for Query Connector Application
35+
1. Logon to the virtual machine
36+
2. Run the wizard script to setup the Query Connector application:
37+
```bash
38+
./dibbs-query-connector-wizard.sh
39+
```
40+
3. Navigate to following urls:
41+
- Query Connector at <IP.ADDRESS>:3000
42+
- Portainer at <IP.ADDRESS>:9000

‎dibbs-query-connector/database.env‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
POSTGRES_USER=postgres
2+
POSTGRES_PASSWORD=pw
3+
POSTGRES_DB=tefca_db

‎dibbs-query-connector/dibbs-query-connector-wizard.sh.home‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
#!/bin/bash
2+
3+
# This script is a setup wizard for the Query Connector application. It guides the user through the process of configuring
4+
# environment variables and setting up the necessary configurations for running the application using Docker Compose.
5+
6+
7+
project_dir=$HOME/dibbs-vm/dibbs-query-connector
8+
dibbs_query_connector_env=$project_dir/dibbs-query-connector.env
9+
dibbs_query_connector_bak=$project_dir/dibbs-query-connector.bak
10+
dibbs_query_connector_wizard=$project_dir/dibbs-query-connector.wizard
11+
12+
echo ""
13+
echo -e "m******************************************************"
14+
echo -e "WARNING: This virtual machine image includes a database intended for development purposes only. It is not configured for
15+
production use and should not be used to store sensitive or live data. Performance, security, and reliability settings may differ
16+
from production environments."
17+
echo -e "m******************************************************"
18+
19+
# Prompt function
20+
prompt() {
21+
local key="$1"
22+
local description="$2"
23+
local default="$3"
24+
local input
25+
26+
echo
27+
echo "$description"
28+
29+
# If the key looks like a secret, use silent prompt
30+
if [[ "$key" == *"PASSWORD"* || "$key" == *"SECRET"* || "$key" == *"KEY"* ]]; then
31+
read -s -p "$key [$default]: " input
32+
echo
33+
else
34+
read -p "$key [$default]: " input
35+
fi
36+
37+
# Fallback to default if input is empty
38+
input="${input:-$default}"
39+
40+
# Remove existing entry
41+
sed -i "/^$key=/d" "qc.env" 2>/dev/null
42+
# Append new value
43+
echo "$key=$input" >> "qc.env"
44+
echo -e "Updating your configuration."
45+
cat qc.env > dibbs-query-connector.env
46+
cp dibbs-query-connector.env "$dibbs_query_connector_env"
47+
export $(cat dibbs-query-connector.env | xargs)
48+
49+
}
50+
51+
prompt "DIBBS_SERVICE" "Name of DIBBS service that is being deployed (e.g. dibbs-query-connector). Default: dibbs-query-connector" "dibbs-query-connector"
52+
prompt "DIBBS_VERSION" "Name of DIBBS version that is being deployed (e.g. 0.1.0). Default: 0.9.1" "0.9.1"
53+
prompt "NODE_ENV" "Specify the environment (e.g., development, production, test). Default: production" "production"
54+
prompt "EXISTING_DATABASE" "Do you have a pre-existing database available to connect to Query Connector(e.g. yes, no) Default: yes" "yes"
55+
prompt "DATABASE_URL" "Enter the full database connection string (db connection string format: postgres://username:password@host:port/db_name) Default: postgres://postgres:pw@db:5432/tefca_db." "postgres://postgres:pw@db:5432/tefca_db"
56+
prompt "AUTH_DISABLED" "Enter 'true' to disable authentication, or 'false' to enable it. Default: true" "true"
57+
prompt "NODE_TLS_REJECT_UNAUTHORIZED" "Enter '0' to allow self-signed certs, '1' to reject them. Default: 0" "0"
58+
prompt "AUTH_SECRET" "Enter the secret key used to sign auth tokens for NEXTJS (keep it secure). " "Replace with auth token "
59+
prompt "AUTH_URL" "Enter the full URL of your authentication provider or auth endpoint. Default: http://query-connector:3000" "http://query-connector:3000"
60+
prompt "ERSD_API_KEY" "Enter the API key used to access the ERSD (Electronic Resources Service Directory). See README for detailed instructions" "Replace with ERSD API key"
61+
prompt "UMLS_API_KEY" "Enter your UMLS (Unified Medical Language System) API key. See README for detailed instructions" "Replace with UMLS API key"
62+
63+
64+
start_all_docker_containers() {
65+
echo "Starting all Docker containers..."
66+
cd "$project_dir"
67+
docker compose up -d
68+
docker compose up -d db
69+
}
70+
71+
start_docker_containers() {
72+
echo "Starting Docker containers..."
73+
cd "$project_dir"
74+
docker compose up -d
75+
}
76+
77+
if [[ "$EXISTING_DATABASE" == "no" ]]; then
78+
start_all_docker_containers
79+
else
80+
start_docker_containers
81+
fi

‎dibbs-query-connector/docker-compose.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
services:
2+
# PostgreSQL DB for custom query and value set storage
3+
db:
4+
image: "postgres:17.4"
5+
restart: always
6+
ports:
7+
- "5432:5432"
8+
env_file: "database.env"
9+
volumes:
10+
- ./vs_dump.sql:/docker-entrypoint-initdb.d/vs_dump.sql
11+
healthcheck:
12+
test: ["CMD-SHELL", "pg_isready -U postgres"]
13+
interval: 2s
14+
timeout: 30s
15+
retries: 20
16+
profiles: [deploy-db]
17+
18+
# Flyway migrations and DB version control
19+
flyway:
20+
image: flyway/flyway:11-alpine
21+
command: -configFiles=/flyway/conf/flyway.conf -schemas=public -connectRetries=60 migrate
22+
volumes:
23+
- ./flyway/sql:/flyway/sql
24+
- ./flyway/conf/flyway.conf:/flyway/conf/flyway.conf
25+
depends_on:
26+
db:
27+
condition: service_started
28+
profiles: [deploy-db]
29+
30+
# Next.js app
31+
query-connector:
32+
image: ghcr.io/cdcgov/$DIBBS_SERVICE/query-connector:$DIBBS_VERSION
33+
restart: always
34+
tty: true
35+
ports:
36+
- "3000:3000"
37+
env_file: "dibbs-query-connector.env"
38+
39+
40+
#Portainer
41+
portainer:
42+
image: portainer/portainer-ce
43+
restart: always
44+
ports:
45+
- "9000:9000"
46+
volumes:
47+
- /var/run/docker.sock:/var/run/docker.sock
48+
49+
docs:
50+
build:
51+
context: ./../docs
52+
dockerfile: Dockerfile
53+
ports:
54+
- "80:80"
55+
56+

‎dibbs-query-connector/flyway/conf/flyway.conf‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
flyway.url=jdbc:postgresql://db:5432/tefca_db
2+
flyway.locations=filesystem:/flyway/sql
3+
flyway.user=postgres
4+
flyway.password=pw
5+
flyway.baselineOnMigrate=false

‎dibbs-query-connector/flyway/sql/V01_01__tcr_custom_query_schema.sql‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
2+
3+
CREATE TABLE IF NOT EXISTS conditions (
4+
id TEXT PRIMARY KEY,
5+
system TEXT,
6+
name TEXT,
7+
version TEXT,
8+
category TEXT
9+
);
10+
11+
CREATE TABLE IF NOT EXISTS category_data (
12+
condition_name TEXT,
13+
condition_code TEXT PRIMARY KEY,
14+
category TEXT,
15+
FOREIGN KEY(condition_code) REFERENCES conditions(id)
16+
);
17+
18+
CREATE TABLE IF NOT EXISTS valuesets (
19+
id TEXT PRIMARY KEY,
20+
oid TEXT,
21+
version TEXT,
22+
name TEXT,
23+
author TEXT,
24+
type TEXT,
25+
dibbs_concept_type TEXT
26+
);
27+
28+
CREATE TABLE IF NOT EXISTS concepts (
29+
id TEXT PRIMARY KEY,
30+
code TEXT,
31+
code_system TEXT,
32+
display TEXT,
33+
gem_formatted_code TEXT,
34+
version TEXT
35+
);
36+
37+
CREATE TABLE IF NOT EXISTS condition_to_valueset (
38+
id TEXT PRIMARY KEY,
39+
condition_id TEXT,
40+
valueset_id TEXT,
41+
source TEXT,
42+
FOREIGN KEY (condition_id) REFERENCES conditions(id),
43+
FOREIGN KEY (valueset_id) REFERENCES valuesets(id)
44+
);
45+
46+
CREATE TABLE IF NOT EXISTS valueset_to_concept (
47+
id TEXT PRIMARY KEY,
48+
valueset_id TEXT,
49+
concept_id TEXT,
50+
FOREIGN KEY (valueset_id) REFERENCES valuesets(id),
51+
FOREIGN KEY (concept_id) REFERENCES concepts(id)
52+
);
53+
54+
CREATE TABLE IF NOT EXISTS query (
55+
id UUID DEFAULT uuid_generate_v4 (),
56+
query_name VARCHAR(255) UNIQUE,
57+
query_data JSON,
58+
conditions_list TEXT[],
59+
author VARCHAR(255),
60+
date_created TIMESTAMP,
61+
date_last_modified TIMESTAMP,
62+
time_window_number INT,
63+
time_window_unit VARCHAR(80),
64+
PRIMARY KEY (id));
65+
66+
-- Create indexes for all primary and foreign keys
67+
68+
CREATE INDEX IF NOT EXISTS conditions_id_index ON conditions (id);
69+
CREATE INDEX IF NOT EXISTS conditions_name_index ON conditions (name);
70+
71+
CREATE INDEX IF NOT EXISTS valuesets_id_index ON valuesets (id);
72+
73+
CREATE INDEX IF NOT EXISTS concepts_id_index ON concepts (id);
74+
75+
CREATE INDEX IF NOT EXISTS condition_to_valueset_id_index ON condition_to_valueset (id);
76+
CREATE INDEX IF NOT EXISTS condition_to_valueset_condition_id_index ON condition_to_valueset (condition_id);
77+
CREATE INDEX IF NOT EXISTS condition_to_valueset_valueset_id_index ON condition_to_valueset (valueset_id);
78+
79+
CREATE INDEX IF NOT EXISTS valueset_to_concept_id_index ON valueset_to_concept (id);
80+
CREATE INDEX IF NOT EXISTS valueset_to_concept_valueset_id_index ON valueset_to_concept (valueset_id);
81+
CREATE INDEX IF NOT EXISTS valueset_to_concept_concept_id_index ON valueset_to_concept (concept_id);
82+
83+
CREATE INDEX IF NOT EXISTS query_id_index ON query (id);
84+
CREATE INDEX IF NOT EXISTS query_name_index ON query (query_name);

‎dibbs-query-connector/flyway/sql/V02_01__add_fhir_servers.sql‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
CREATE TABLE IF NOT EXISTS fhir_servers (
2+
id UUID DEFAULT uuid_generate_v4 (),
3+
name TEXT,
4+
hostname TEXT,
5+
headers JSON DEFAULT NULL
6+
);
7+
8+
CREATE INDEX IF NOT EXISTS fhir_servers_id_index ON fhir_servers (id);
9+
CREATE INDEX IF NOT EXISTS fhir_servers_name_index ON fhir_servers (name);
10+
11+
INSERT INTO fhir_servers (name, hostname) VALUES ('Public HAPI: Direct','https://hapi.fhir.org/baseR4');

‎dibbs-query-connector/flyway/sql/V03_01__add_fhir_server_connection_columns.sql‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
-- Add columns for connection tracking
2+
ALTER TABLE fhir_servers
3+
ADD COLUMN last_connection_attempt TIMESTAMP WITH TIME ZONE,
4+
ADD COLUMN last_connection_successful BOOLEAN DEFAULT FALSE;
5+
6+
-- Update existing rows to have null values for new columns
7+
UPDATE fhir_servers
8+
SET last_connection_attempt = NULL,
9+
last_connection_successful = NULL;
10+
11+
-- Convert name index to unique constraint
12+
DROP INDEX fhir_servers_name_index;
13+
CREATE UNIQUE INDEX fhir_servers_name_index ON fhir_servers (name);
14+
15+
-- Truncate existing table
16+
TRUNCATE TABLE fhir_servers;
17+
18+
-- Direct FHIR servers
19+
INSERT INTO fhir_servers (name, hostname, last_connection_attempt, last_connection_successful)
20+
VALUES
21+
('Public HAPI: Direct', 'https://hapi.fhir.org/baseR4', current_timestamp, true),
22+
('HELIOS Meld: Direct', 'https://gw.interop.community/HeliosConnectathonSa/open', current_timestamp, true),
23+
('JMC Meld: Direct', 'https://gw.interop.community/JMCHeliosSTISandbox/open', current_timestamp, true),
24+
('Local e2e HAPI Server: Direct', 'http://hapi-fhir-server:8080/fhir', current_timestamp, true),
25+
('OPHDST Meld: Direct', 'https://gw.interop.community/CDCSepHL7Connectatho/open', current_timestamp, true);
26+
27+
-- eHealthExchange FHIR servers
28+
INSERT INTO fhir_servers (name, hostname, headers, last_connection_attempt, last_connection_successful)
29+
VALUES
30+
('HELIOS Meld: eHealthExchange',
31+
'https://concept01.ehealthexchange.org:52780/fhirproxy/r4/',
32+
'{
33+
"Accept": "application/json, application/*+json, */*",
34+
"Accept-Encoding": "gzip, deflate, br",
35+
"Content-Type": "application/fhir+json; charset=UTF-8",
36+
"X-DESTINATION": "MeldOpen",
37+
"X-POU": "PUBHLTH",
38+
"prefer": "return=representation",
39+
"Cache-Control": "no-cache"
40+
}',
41+
current_timestamp,
42+
true),
43+
('JMC Meld: eHealthExchange',
44+
'https://concept01.ehealthexchange.org:52780/fhirproxy/r4/',
45+
'{
46+
"Accept": "application/json, application/*+json, */*",
47+
"Accept-Encoding": "gzip, deflate, br",
48+
"Content-Type": "application/fhir+json; charset=UTF-8",
49+
"X-DESTINATION": "JMCHelios",
50+
"X-POU": "PUBHLTH",
51+
"prefer": "return=representation",
52+
"Cache-Control": "no-cache"
53+
}',
54+
current_timestamp,
55+
true),
56+
('OpenEpic: eHealthExchange',
57+
'https://concept01.ehealthexchange.org:52780/fhirproxy/r4/',
58+
'{
59+
"Accept": "application/json, application/*+json, */*",
60+
"Accept-Encoding": "gzip, deflate, br",
61+
"Content-Type": "application/fhir+json; charset=UTF-8",
62+
"X-DESTINATION": "OpenEpic",
63+
"X-POU": "PUBHLTH",
64+
"prefer": "return=representation",
65+
"Cache-Control": "no-cache"
66+
}',
67+
current_timestamp,
68+
true),
69+
('CernerHelios: eHealthExchange',
70+
'https://concept01.ehealthexchange.org:52780/fhirproxy/r4/',
71+
'{
72+
"Accept": "application/json, application/*+json, */*",
73+
"Accept-Encoding": "gzip, deflate, br",
74+
"Content-Type": "application/fhir+json; charset=UTF-8",
75+
"X-DESTINATION": "CernerHelios",
76+
"X-POU": "PUBHLTH",
77+
"prefer": "return=representation",
78+
"Cache-Control": "no-cache",
79+
"OAUTHSCOPES": "system/Condition.read system/Encounter.read system/Immunization.read system/MedicationRequest.read system/Observation.read system/Patient.read system/Procedure.read system/MedicationAdministration.read system/DiagnosticReport.read system/RelatedPerson.read"
80+
}',
81+
current_timestamp,
82+
true);

‎dibbs-query-connector/flyway/sql/V04_01__add_fhir_server_disable_cert_validation.sql‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
ALTER TABLE fhir_servers
2+
ADD COLUMN disable_cert_validation BOOLEAN DEFAULT FALSE;
3+
4+
-- Update rows with eHealthExchange in the server name to have disable_cert_validation set to true
5+
UPDATE fhir_servers
6+
SET disable_cert_validation = TRUE
7+
WHERE name LIKE '%eHealthExchange%';
8+
9+
-- Update rows with eHealthExchange in the server name to strip the trailing slash from the hostname
10+
UPDATE fhir_servers
11+
SET hostname = regexp_replace(hostname, '/$', '')
12+
WHERE name LIKE '%eHealthExchange%';

0 commit comments

Comments
 (0)