rework devops workflow

[nicoalee]

Currently, our devops workflow works, but is quite vulnerable to breaking. As we progress in this project (and start releasing software) we need to start thinking about how we want our devops pipeline to reflect that.

Current devops process:

1. Create PRs to master branch
2. update staging server (optional): https://synth.neurostore.xyz/
   - manual git push to staging server from local computer
   - this is currently what we use as our playground
   - post-receive git hook on the staging server rebuilds the docker containers
3. update prod server: https://compose.neurosynth.org/
   - manual push to prod server from local computer
   - not sure how we currently deploy

This current process has a few drawbacks:

• the staging server is collecting commits and its git working directory is diverging from the main master branch as commits to the staging server are pushed and pile up
• there is a lot of clutter in the staging server as it has diverged from the master branch: there are lots of folders (like synth) that dont exist anymore.
• we do not have a separate release branch

Proposed devops process:

1. Create a release branch which would serve as our branch reflecting the prod server
2. Dev branches are created and pull requests are made to the master branch
   - dev github action workflow is triggered for all dev branches which includes build and unit/integration tests
   - instead of maintaining a git remote repo on dream, we could run something like docker save <image> | ssh -C deepdream docker load to push the built docker image itself via ssh. This can be done from any branch.
3. Merges into the master branch result in a docker image being built and deployed to our PROD machine
   - prod github action workflow is triggered which includes unit/integration tests as well as deployment to prod

Improvements:

• staging server does not need to hold the actual github repo - we just need to pass it our images. Commits do not pile up and diverge from the original master branch
• no clutter as staging server is not aware of any code - just the docker images passed to it
• a separate release branch is created and triggered automatically when we merge into it