OLD Instructions On Deploying in a VM

Overview

Note Services that were found failing and have been disabled in vocms0231 include:

• postgres-11: Probably deprecated, since DBoD is used.
• httpd: Apache service, probably deprectated in favor of nginx.
• hdqm-warmup: Pointing to /data/hdqm/current/etc/warmup.sh, probably a deprecated warmup.sh script (I could not find it).
• hdqm-warmup.timer: Cron job, running hdqm-warmup on a schedule.

Deploying the application

An OpenStack virtual machine (vocms0231.cern.ch) is used to deploy HQDM. Both the Production and Test instances are deployed in the same machine.

An nginx instance takes care of load balancing and exposing the production and test apps to ports 80 and 81, correspondingly. Two routes are configured:

1. /: Serves the static site files in the frontend directory.
2. /api: Forwards requests to the gunicorn server, which serves the Flask app (api.py).

The dqm_extractor.py script is run as a service (hdqm-extractor.service). The API server is also run as a service (hdqm.service).

The complete architecture of the deployment can be seen in Architecture Overview.

Prerequisites

nginx

sudo yum install nginx
sudo systemctl enable nginx
sudo systemctl start nginx

Note If nginx complains that it can't bind to port, make sure to request the ports to be opened in puppet:
https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocms/merge_requests/72 And open them using SELinux: sudo semanage port -m -t http_port_t -p tcp 8081
Also important:

sudo firewall-cmd --zone=public --add-port=81/tcp --permanent 
sudo firewall-cmd --reload

Note Make sure to make root directory accessible in SELinux:
sudo chcon -Rt httpd_sys_content_t /data/hdqm-test/CentralHDQM/frontend/
sudo chcon -Rt httpd_sys_content_t /data/hdqm/

A kerberos keytab file

Note Extraction is performed on behalf of the cmsdqm user because we need a user that would be in CERN.CH domain to access EOS file system**

User credentials are stored in a keytab file. This file needs to be updated when the password changes. Below are the instructions on how to do that:

sudo su cmsdqm
ktutil
# Keep in mind the capital letters - they are important!
# add_entry -password -p [email protected] -k 1 -e aes256-cts-hmac-sha1-96  # This does not work anymore?
add_entry -password -p [email protected] -k 1 -e arcfour-hmac
write_kt /data/hdqm/.keytab
exit
# Get the kerberos token. This will grant access to EOS
kinit -kt /data/hdqm/.keytab cmsdqm
# Make EOS aware of the new kerberos token
/usr/bin/eosfusebind -g

# Verify
klist -kte /data/hdqm/.keytab

More info about kerberos: https://twiki.cern.ch/twiki/bin/view/Main/Kerberos

System packages

• python3-devel
• libpqxx-devel
• posgresql, so that the pg_config binary is available, necessary for the psycopg2-binary python package.
• gcc-c++, necessary for building greenlet.

Production

1. Make sure nginx is running and you have a keytab created.
2. Download the update.sh script in /data/hdqm
3. Create an .env file based on .env_sample in /data/hdqm, changing the secrets as needed.

sudo su cmsdqm
kinit  # Just in case
cd /data/hdqm
bash update.sh
exit  # Go back to your sudo-capable account

# Copy the service configuration
sudo cp deployment/service/hdqm.service /etc/systemd/system/
sudo cp deployment/service/hdqm-extract.service /etc/systemd/system/
sudo cp deployment/service/hdqm-extract.timer /etc/systemd/system/

# Reload the systemctl daemon and start the services
sudo systemctl daemon-reload

sudo systemctl enable hdqm.service
sudo systemctl start hdqm.service 

sudo systemctl enable hdqm-extract.service
# Do not start it, it will be done by the timer

sudo systemctl enable hdqm-extract.timer
sudo systemctl start hdqm-extract.timer 


# Copy the nginx configs, give SELinux permissions and restart nginx
sudo cp deployment/nginx/conf.d/nginx.conf /etc/nginx/
sudo cp deployment/nginx/conf.d/hdqm.conf /etc/nginx/conf.d/
sudo chcon -Rt httpd_sys_content_t /data/hdqm
sudo systemctl restart nginx

PaaS proxy

1. Go to Topology.
2. Right click --> Add to Project --> Helm Charts
3. Select the Cern Auth Proxy and Install Helm Chart.
4. Leave Upstream configuration as is.
5. Edit Routing Configuration --> Public Application Hostname and enter the URL to serve the app to (e.g. cms-hdqm.web.cern.ch).
6. Under Authentication options --> Extra Arguments enter: --upstream=http://vocms0231:80

How to roll back to an old version

In order to roll back to a previous version, set the current symlink to point to the required version folder (in the same directory) and restart the service:

cd /data/hdqm
ln -s -f -n <FOLDER_OF_THE_REQUIRED_VERSION> current
sudo systemctl restart hdqm.service

Development (a.k.a "Test")

1. Make sure nginx is running and you have a keytab created.

sudo su cmsdqm
cd /data/hdqm-test
git clone https://github.com/cms-DQM/CentralHDQM.git && cd CentralHDQM && git checkout dev
cp .env_sample backend/.env
# Edit the `.env` file to contain the secrets needed.
kinit
python3 -m venv venv
source venv/bin/activate
python -m pip install -U pip
python -m pip install -r requirements.txt
exit  # Go back to your sudo-capable account

# Copy the service configuration
sudo cp deployment/service/hdqm-test.service /etc/systemd/system/
sudo cp deployment/service/hdqm-extract-test.service /etc/systemd/system/
sudo cp deployment/service/hdqm-extract-test.timer /etc/systemd/system/

# Reload the systemctl daemon and start the services
sudo systemctl daemon-reload

sudo systemctl enable hdqm-test.service
sudo systemctl start hdqm-test.service 

sudo systemctl enable hdqm-extract-test.service
# Do not start it, it will be done by the timer

sudo systemctl enable hdqm-extract-test.timer
sudo systemctl start hdqm-extract-test.timer 


# Copy the nginx configs, give SELinux permissions and restart nginx
sudo cp deployment/nginx/conf.d/nginx.conf /etc/nginx/
sudo cp deployment/nginx/conf.d/hdqm-test.conf /etc/nginx/conf.d/
sudo chcon -Rt httpd_sys_content_t /data/hdqm-test/CentralHDQM/frontend/
sudo systemctl restart nginx

How to rollback to an old version

In order to rollback to the previous version set current symlink to point to the required version folder (in the same directory) and restart the service:

cd /data/hdqm
ln -s -f -n <FOLDER_OF_THE_REQUIRED_VERSION> current
sudo systemctl restart hdqm.service