Merge branch 'release/3.3.0'

Author: pbchase

CHANGELOG.md

@@ -23,7 +23,7 @@ This project provides a virtual machine wherein it hosts the local REDCap instan
 
 ### Packaging and Deployment
 
-The packaging and deployment tools are designed to deploy REDCap to Debian Linux hosts. They may or may not work with non-Debian REDCap hosts.  They cannot deploy REDCap to Windows hosts. The packaging and deployment tools are written using the [Fabric](http://www.fabfile.org/) system. Fabric is written in Python3, so both Python 3 and Fabric3 must be installed to do packaging and deployment.
+The packaging and deployment tools are designed to deploy REDCap to Debian Linux hosts. They may or may not work with non-Debian REDCap hosts.  They cannot deploy REDCap to Windows hosts. The packaging and deployment tools are written using the [Fabric](http://www.fabfile.org/) system. The tools use Fabric 1.x
 
 
 ## Installing dependencies
@@ -117,7 +117,7 @@ After about two minutes, the VM should be accessible at the value of the variabl
 
 ## (Re)deploying REDCap with Fabric Tools
 
-In addition to the REDCap deployed by the Vagrant provisioning scripts, this repository includes a suite of deployment and upgrade tools that can configure a host for deployment, package REDCap with numerous extensions, deploy a new REDCap instance and upgrade an existing one.  You can use these commands any host where you have sufficient privileges or against this vagrant-deployed VM.
+In addition to the REDCap deployed by the Vagrant provisioning scripts, this repository includes a suite of deployment and upgrade tools that can configure a host for deployment, package REDCap with numerous extensions, deploy a new REDCap instance and upgrade an existing one.  You can use these commands on any Linux host where you have sufficient privileges or against this vagrant-deployed VM.
 
 ### Fabric Prerequisites
 
@@ -127,6 +127,14 @@ This tool is written in Python 3 and uses Fabric 1.x methods. To use it, make su
 pip install 'fabric<2.0'
 ```
 
+If you make a mess of things before you get it right, you might have to uninstall the mess like this:
+
+```bash
+brew uninstall fabric
+pip uninstall 'fabric<2.0'
+pip install 'fabric<2.0'
+```
+
 
 ### Configure Fabric for the Virtual Machine
 
@@ -277,6 +285,54 @@ sudo a2enmod php8.1
 sudo systemctl restart apache2
 ```
 
+### Upgrade PHP from 7.4 to 8.2
+
+```bash
+# Upgrade PHP from 7.4 to 8.2
+sudo apt install -y libapache2-mod-php8.2 \
+  php8.2 \
+  php8.2-cli \
+  php8.2-common \
+  php8.2-curl \
+  php8.2-gd \
+  php8.2-imap \
+  php8.2-mbstring \
+  php8.2-mysql \
+  php8.2-odbc \
+  php8.2-opcache \
+  php8.2-readline \
+  php8.2-soap \
+  php8.2-xml \
+  php8.2-zip
+
+cd /etc
+sudo -E git add .
+sudo -E git commit -m "Commit PHP upgrades and other files"
+
+cd /etc/php
+grep -lr upload_max_filesize * | sudo xargs -i sed "s/upload_max_filesize.*/upload_max_filesize = 256M/;" -i {}
+grep -lr post_max_size * | sudo xargs -i sed "s/post_max_size.*/post_max_size = 256M/;" -i {}
+grep -lr max_input_vars * | sudo xargs -i sed "s/.*max_input_vars.*/max_input_vars = 100000/;" -i {}
+grep -lr session.cookie_secure * | sudo xargs -i sed "s/.*session.cookie_secure.*/session.cookie_secure = On/;" -i {}
+grep -lr date.timezone * | sudo xargs -i sed "s/.*date.timezone.*=.*/date.timezone = 'America\/New_York'/;" -i {}
+
+cd /etc
+sudo -E git add .
+sudo -E git commit -m "Commit PHP configuration changes"
+
+# fix imagick
+sudo apt install -y php-imagick
+sudo sed -i 's/policy domain="coder" rights="none" pattern="PDF"/policy domain="coder" rights="read" pattern="PDF"/;' /etc/ImageMagick-6/policy.xml
+cd /etc
+sudo -E git add .
+sudo -E git commit -m "Install php-imagick and adjust policy to REDCap requirements"
+
+# Switch to new PHP in Apache
+sudo a2dismod php7.4
+sudo a2enmod php8.2
+sudo systemctl restart apache2
+```
+
 ### Install specific PHP packages
 
 On some hosts, you might need to install a specific packages. At UF, we have one host we call "warrior" that needs the `mpdf` package. To install a custom package, first, install `composer`
@@ -296,8 +352,8 @@ Then run these steps in the current redcap Libraries directory as user deploy
 
 ```bash
 sudo su - deploy
-#cd /var/https/stage_c/redcap_v13.4.11/Libraries/
-cd /var/www/prod/redcap_v13.4.11/Libraries/
+#cd /var/https/stage_c/redcap_v14.3.14/Libraries/
+cd /var/www/prod/redcap_v14.6.2/Libraries/
 ```
 
 Then run `composer require` with the package you need install:
@@ -313,6 +369,45 @@ Restart apache when you are done
 sudo service apache2 restart
 ```
 
+## Failed SQL upgrades
+
+If an upgrade operation fails on a `.php` or a `.sql` file, the code will die like this:
+
+```
+[redcap.ctsi.ufl.edu] run: php /var/https/redcap/redcap_v14.1.3/generate_upgrade_sql_from_php.php /var/https/redcap/redcap_v14.1.3/Resources/sql/upgrade_14.01.01.php | mysql
+[redcap.ctsi.ufl.edu] out: ERROR 1553 (HY000) at line 5: Cannot drop index 'redcap_userid': needed in a foreign key constraint
+[redcap.ctsi.ufl.edu] out: 
+
+
+Fatal error: run() received nonzero return code 1 while executing!
+
+Requested: php /var/https/redcap/redcap_v14.1.3/generate_upgrade_sql_from_php.php /var/https/redcap/redcap_v14.1.3/Resources/sql/upgrade_14.01.01.php | mysql
+Executed: /bin/bash -l -c "php /var/https/redcap/redcap_v14.1.3/generate_upgrade_sql_from_php.php /var/https/redcap/redcap_v14.1.3/Resources/sql/upgrade_14.01.01.php | mysql"
+
+Aborting.
+Disconnecting from [email protected]... done.
+```
+
+At this point things get messy and you will have to get creative in your problem solving. What follows is not a recipe. It's more like a pile of ingredients from which you could make an omelette. It's still your responsibility cook them right and not burn the omelette, because you're gonna have to eat it.
+
+### Debugging
+
+You can see the output of the script/SQl file that caused the by running that PHP command or catting out that SQL file that failed. e.g.,
+
+```bash
+php /var/https/redcap/redcap_v14.1.3/generate_upgrade_sql_from_php.php /var/https/redcap/redcap_v14.1.3/Resources/sql/upgrade_14.01.01.php
+```
+
+Running the SQL lines in that file one by one might help you understand which one failed. It's challenging because some of those commands will fail because they _did_ run successfully the first time, but they will error the second time. This often happens when a SQL command added an object that cannot be "re-added". Think carefully about the responses you see when running each command. Read the REDCap community forum to see if others have experienced the errors you see and have a solution.
+
+Even if that task goes well and you can apply the file that failed there might be more files to run after that one. You can locate those by enumerating the files in the `redcap_vN.M.O/Resources/sql/` directory. Look for versioned file names that are higher than the version number on which the script failed. If this seems tedious, you might want to try the next procedure.
+
+
+### Ask REDCap to help you
+
+Another approach is to generate all of the SQL upgrade commands between the version you are at and the version you are upgrading to by accessing the Upgrade page in the Control Center. It will make the whole blob, running the PHP scripts, concatenating their output with the SQL files and appending the lines that document the upgrade. All that's pretty useful. Do be aware that every SQL line that succeeded will still be in the blob of SQL the Control Center Upgrade page generates for you. It's more content to wade through, but it's all in one file with the last three "upgrade event" statements.
+
+If you think you are done with the upgrade process, put the host back online via the Control Center _General Configuration_ page. Then check the Control Center _Configuration Check_ age for any SQL commands that still need to be applied. 
 
 ## Developer notes
 

README.md

@@ -1 +1 @@
-3.2.1
+3.3.0

VERSION

@@ -1,4 +1,7 @@
 {
+    "config": {
+        "allow-plugins": false
+    },
     "require": {
         "wikimedia/composer-merge-plugin": "dev-master"
     },

composer.json

@@ -10,8 +10,7 @@ cd $MYTARGETDIR
 php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
 
 # install composer
-# pin to version 1.latest due to 2.x conflict: https://github.com/wikimedia/composer-merge-plugin/issues/184
-php composer-setup.php --1
+php composer-setup.php
 
 # install dependencies
 php composer.phar install

deploy/composer/deploy.sh

@@ -97,6 +97,14 @@ def deploy_modules_into_build_space():
             local("cp -r %s/* %s/redcap/modules/%s_v%s" % (tempdir,env.builddir,module['name'],module['version']))
             local("rm -rf %s" % tempdir)
 
+def deploy_redcap_redirect_into_build_space():
+    """
+    Deploy redcap_redirect into build space
+    """
+    tempdir = local("mktemp -d 2>&1", capture = True)
+    local("git clone %s %s" % ("https://github.com/susom/redcap-redirect.git",tempdir))
+    local("cp -r %s/redcap_redirect.php %s/redcap/" % (tempdir,env.builddir))
+    local("rm -rf %s" % tempdir)
 
 def deploy_plugins_into_build_space(target_within_build_space="/redcap/plugins"):
     """
@@ -180,6 +188,7 @@ def package(redcap_zip="."):
     clean(env.builddir)
     make_builddir(env.builddir)
     redcap_version_and_package_type = extract_redcap(redcap_zip)
+    deploy_redcap_redirect_into_build_space()
     deploy_plugins_into_build_space()
     deploy_modules_into_build_space()
     # Deploy modules framework only if redcap major version is less than 8

package.py

@@ -88,17 +88,17 @@ def apply_incremental_db_changes(old, new):
     path_to_sql_generation = '/'.join([env.live_pre_path, env.project_path, 'redcap_v' + new, 'generate_upgrade_sql_from_php.php'])
     for file in files.splitlines():
         match = re.search(r"(upgrade_)(\d+.\d+.\d+)(.)(php|sql)", file)
-        version = match.group(2)
-        version = utility.convert_version_to_int(version)
-        if version > old and version <= new_as_an_int:
-            if fnmatch.fnmatch(file, "*.php"):
-                print((file + " is a php file!\n"))
-                with settings(user=env.deploy_user):
-                    run('php %s %s | mysql' % (path_to_sql_generation,file))
-            else:
-                print(("Executing sql file %s" % file))
-                with settings(user=env.deploy_user):
-                    run('mysql < %s' % file)
+        if match is not None:
+            version = utility.convert_version_to_int(match.group(2))
+            if version > old and version <= new_as_an_int:
+                if fnmatch.fnmatch(file, "*.php"):
+                    print((file + " is a php file!\n"))
+                    with settings(user=env.deploy_user):
+                        run('php %s %s | mysql' % (path_to_sql_generation,file))
+                else:
+                    print(("Executing sql file %s" % file))
+                    with settings(user=env.deploy_user):
+                        run('mysql < %s' % file)
     # Finalize upgrade
     utility_redcap.set_redcap_config('redcap_last_install_date', datetime.now().strftime("%Y-%m-%d"))
     utility_redcap.set_redcap_config('redcap_version', new)

upgrade.py

@@ -2,6 +2,7 @@
 from tempfile import mkstemp
 import os
 import utility
+import utility_test
 try:
     import configparser
 except:
@@ -108,6 +109,7 @@ def test(warn_only=False):
     utility.write_remote_my_cnf()
     version = get_current_redcap_version()
     utility.delete_remote_my_cnf()
+    utility_test.main()
     local("python tests/test.py %s/ redcap_v%s/" % (env.url_of_deployed_app,version))
     with settings(warn_only=True):
         if local("python tests/test.py %s/ redcap_v%s/" % (env.url_of_deployed_app,version)).failed:
@@ -118,3 +120,4 @@ def test(warn_only=False):
                 abort("One or more tests failed.")
         else:
             return(True)
+        

utility_redcap.py

@@ -0,0 +1,48 @@
+from fabric.api import *
+import unittest
+
+class CheckHostAccessibility(unittest.TestCase):
+    def setUp(self):
+        self.hosts = [
+            "https://api.reporter.nih.gov",
+            "https://eutils.ncbi.nlm.nih.gov",
+            "https://www.ncbi.nlm.nih.gov",
+            "https://icite.od.nih.gov",
+            "https://pub.orcid.org",
+            "https://redcap.vanderbilt.edu",
+            "https://api.altmetric.com",
+            "https://api.patentsview.org",
+            "https://api.nsf.gov",
+            "https://api.ies.ed.gov",
+            "https://ies.ed.gov",
+            "https://taggs.hhs.gov",
+            "https://api.elsevier.com",
+            "https://dev.elsevier.com",
+            "https://ws.isiknowledge.com"
+        ]
+        self.failed_hosts = []
+
+    def check_host_accessibility(self, host):
+        with settings(warn_only=True, user=env.deploy_user):
+            result = run(f"curl -s -o /dev/null -w '%{{http_code}}' -L {host}")
+            http_code = result.stdout.strip()
+            return http_code
+
+    def runTest(self):
+        for host in self.hosts:
+            http_code = self.check_host_accessibility(host)
+            if http_code != "200":
+                # Collect details about failed hosts for assertion later
+                self.failed_hosts.append(f"{host} - HTTP Status: {http_code}")
+        self.assertTrue(len(self.failed_hosts) == 0, "Some hosts were not accessible:\n" + "\n".join(self.failed_hosts))
+
+def suite():
+    suite = unittest.TestSuite()
+    suite.addTest(CheckHostAccessibility())
+    return suite
+
+def main():
+    runner = unittest.TextTestRunner()
+    test_suite = suite()
+    runner.run(test_suite)
+