Add versioning strategy

README.md

@@ -1,64 +1,102 @@
-## How to build and develop using the new dockerrized app.
+## How to build and develop using the new dockerized app.
+
+### Prerequisites
+
+If not already satisfied, add the following entry into your `/etc/hosts` file at the very top:
+
+```
+127.0.0.1       fakeservices.datajoint.io
+```
+
+This will create an alias to your `localhost` based on requests to `fakeservices.datajoint.io`.
+
+Make sure to also define a `.env` file as follows:
+
+``` sh
+# minimum
+DJ_PASS=db_password
+AWS_ACCESS_KEY_ID=aws_key
+AWS_SECRET_ACCESS_KEY=aws_secret
+DEMO_PASSWORD=ibl_navigator_password
+JWT_SECRET=secret
+# utilized for remote deployment
+SUBDOMAINS=sub
+URL=example.com
+# utilized for load testing
+TEST_DJ_HOST=test_db_host
+TEST_DJ_USER=test_db_user
+TEST_DJ_PASS=test_db_password
+```
+
+### Versioning
+
+Versioning is now handled in `version.env`. Make sure to increment the version as appropriate to service based on [semantic versioning](https://semver.org/) guidelines. For services specific to `public` development, version should indicate as such e.g. `vX.X.X-public`. To determine the global version for git tagging, utilize a vector summation between python backend, node backend, frontend, and nginx. If related to `public` development, make sure to additionally add the `-public` suffix. Additionally, version can be inspected within node backend by sending a request to `/version`. To properly handle versioning both for image tags and api requests, you will need to prepend your `docker-compose` commands with `env $(grep -hv '^#' version.env) `.
+
+### Build
 
 To be 100% sure of the new build to be reflected - use below
-`docker-compose -f docker-compose-dev.yml build --no-cache`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-dev.yml build --no-cache`
 
 Then,
-`docker-compose -f docker-compose-dev.yml up`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-dev.yml up`
 to begin the development in `ng serve` mode - go to
-localhost:9000 to see the site.
-`docker-compose -f docker-compose-dev.yml down`
-when done developing.
+fakeservices.datajoint.io:9000 to see the site.
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-dev.yml down`
+when done developing. **Note: When utilizing `dev` and `test` docker-compose files, `ibl-frontend/frontend-content/src/environments/*.ts` files are overwritten due to volume mount. Make sure not to commit these updates.**
 
 For casual re-build/up process
-`docker-compose -f docker-compose-dev.yml up --build`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-dev.yml up --build`
 
 For detached mode and to add log after the fact
-`docker-compose -f docker-compose-dev.yml up -d`
-`docker-compose -f docker-compose-dev.yml logs -f`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-dev.yml up -d`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-dev.yml logs -f`
 
-To see the production build using `ng build --prod`,
-do the regular docker-compose up then go to localhost:8080
-`docker-compose up --build`
+to do the regular docker-compose up then go to localhost:9000
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-build.yml up --build`
 
 to check inside docker 
-`docker-compose -f docker-compose-dev.yml exec ibl-node-server /bin/bash`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-dev.yml exec ibl-node-server /bin/bash`
 
 --------------------------------
 for deploy (general)
 
-Before building, make sure `build: ./ibl-frontend` is UNcommented in docker-compose.yml.
-`docker-compose build ibl-navigator` once that's built,
-`docker push registry.vathes.com/ibl-navigator/frontend:v0.0`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-build.yml build ibl-navigator` once that's built,
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-build.yml push ibl-navigator`
 
-commentout the `build: ./ibl-frontend`
-
-repeat for other 3 `iblapi` `ibl-node-server` `nginx` and push to appropriate directory. Update the tags accordingly as well.
+repeat for other 2 `iblapi` `ibl-node-server` and push to appropriate directory. Update the tags accordingly as well.
 
 for testdev deploy
 comment out test/* directory in `.dockerignore` (until proper storage solution is in place)
-for test dev mode, make sure `STAGING=true` for nginx > environment setting.
+
+make sure to update `SUBDOMAINS` key in `.env` file to `testdev`.
+make sure to update `URL` key in `.env` file to `datajoint.io`.
+
+for test dev mode, in `docker-compose-deploy.yml` make sure `STAGING=true` for `letsencrypt` > environment setting.
 
 `ssh testdev` go to `ibl-navigator`
-`docker-compose down` to stop what's already running
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-deploy.yml down` to stop what's already running
 `sudo rm -R letsencrypt-keys` to get rid of key folder that generated in the previous run.
-`git pull origin dev` to get the latest from `mahos/ibl-navigator` repo.
+`git pull origin dev` to get the latest from `vathes/ibl-navigator` repo.
 make sure to move over to the `dev` branch by `git checkout dev`
 `docker login registry.vathes.com` to docker to get access.
-`docker-compose pull` to get the ibl-navigator container
-`docker-compose up --build -d`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-deploy.yml pull` to get the ibl-navigator container
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-deploy.yml up -d`
 
 -----------------------------------
 
 for real deploy
-for client deploy mode, comment out `STAGING=true` for nginx > environment setting.
+
+make sure to update `SUBDOMAINS` key in `.env` file to `djcompute`.
+make sure to update `URL` key in `.env` file to `internationalbrainlab.org`.
+
+for client deploy mode, in `docker-compose-deploy.yml` make sure to comment out `STAGING=true` for `letsencrypt` > environment setting.
 
 `ssh djcompute` go to `nagivator-deployer/ibl-navigator`
-`docker-compose down` to stop what's already running
-`git pull origin master` to get the latest from `mahos/ibl-navigator` repo.
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-deploy.yml down` to stop what's already running
+`git pull origin master` to get the latest from `vathes/ibl-navigator` repo.
 make sure to move over to the `master` branch by `git checkout master`
 `docker login registry.vathes.com` to docker to get access.
-`docker-compose pull` to get the ibl-navigator container
-`docker-compose up --build -d`
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-deploy.yml pull` to get the ibl-navigator container
+`env $(grep -hv '^#' version.env) docker-compose -f docker-compose-deploy.yml up -d`
 
 -------------------------------------

backend/Dockerfile

@@ -1,22 +1,29 @@
-FROM datajoint/jupyter:python3.6
+FROM raphaelguzman/djlab:py3.6-debian
 
-# for production builds
-ADD . /src/iblapi
-
-#RUN pip uninstall -y datajoint && pip install git+https://github.com/dimitri-yatsenko/datajoint-python.git@dev#egg=datajoint-python
+# RUN \
+#     pip uninstall -y datajoint && \
+#     pip install \
+#         git+https://github.com/dimitri-yatsenko/datajoint-python.git@dev#egg=datajoint-python
 
 RUN pip install --upgrade --pre datajoint
 
-RUN \
-    pip install -e /src/iblapi && \
-    chmod +x /src/iblapi/run-ibl-api.prod.sh && \
-    chmod +x /src/iblapi/run-ibl-api.dev.sh
-
 HEALTHCHECK       \
     --timeout=3s \ 
     --retries=20  \
     CMD           \
-        curl --fail http://localhost:5000/v0/lab || exit 1
+        wget --quiet --tries=1 --spider http://localhost:5000/v0/lab > /dev/null 2>&1 || exit 1
 
 
 ENTRYPOINT ["/src/iblapi/run-ibl-api.prod.sh"]
+
+# for production builds
+RUN mkdir -p /src/iblapi
+COPY --chown=dja:anaconda ["notebooks", "/src/iblapi/notebooks"]
+COPY --chown=dja:anaconda ["./*.txt", "./*.sh", "./*.rst", "./*.py", "/src/iblapi/"]
+
+RUN \
+    pip install -e /src/iblapi && \
+    chmod +x /src/iblapi/run-ibl-api.prod.sh && \
+    chmod +x /src/iblapi/run-ibl-api.dev.sh
+
+# COPY --chown=dja:anaconda ["tests", "/src/iblapi/tests"]

backend/iblapi.py

@@ -45,6 +45,8 @@ def test_mkvmod(mod):
 ephys = mkvmod('ephys')
 histology = mkvmod('histology')
 test_histology = test_mkvmod('histology')
+original_max_join_size = dj.conn().query(
+    "show variables like 'max_join_size'").fetchall()[0][1]
 
 dj.config['stores'] = {
     'ephys': dict(
@@ -103,6 +105,7 @@ def dumps(cls, obj):
 
 reqmap = {
     '_q': None,
+    '_health': None,
     'lab': reference.Lab,
     'labmember': reference.LabMember,
     'labmembership': reference.LabMembership,
@@ -186,6 +189,8 @@ def do_req(subpath):
         abort(404)
     elif obj == '_q':
         return handle_q(pathparts[1], args, proj, **kwargs)
+    elif obj == '_health':
+        return dumps(dict(healthy=True))
     else:
         q = (reqmap[obj] & args)
         if proj:
@@ -215,7 +220,8 @@ def handle_q(subpath, args, proj, **kwargs):
     post_process = None
     if subpath == 'sessionpage':
         sess_proj = acquisition.Session().aggr(
-            acquisition.SessionProject().proj('session_project', dummy2='"x"') * dj.U('dummy2'),
+            acquisition.SessionProject().proj('session_project', dummy2='"x"')
+            * dj.U('dummy2'),
             session_project='IFNULL(session_project, "unassigned")',
             keep_all_rows=True
         )
@@ -229,12 +235,19 @@ def handle_q(subpath, args, proj, **kwargs):
             ephys.ProbeInsertion().proj(dummy2='"x"') * dj.U('dummy2'),
             nprobe='count(dummy2)',
             keep_all_rows=True)
-        # training_status = acquisition.Session.aggr(analyses_behavior.SessionTrainingStatus.proj(dummy3='"x"') * dj.U('dummy3'), nstatus='count(dummy3)', keep_all_rows=True) 
-        q = (acquisition.Session() * sess_proj * psych_curve * ephys_data * subject.Subject() * subject.SubjectLab() * subject.SubjectUser() * analyses_behavior.SessionTrainingStatus()
-             & ((reference.Lab() * reference.LabMember())
+        training_status = acquisition.Session.aggr(
+            analyses_behavior.SessionTrainingStatus.proj(dummy3='"x"') * dj.U('dummy3'),
+            nstatus='count(dummy3)',
+            keep_all_rows=True)
+
+        q = (acquisition.Session() * sess_proj * psych_curve * ephys_data * training_status
+             * subject.Subject() * subject.SubjectLab()
+             & (reference.Lab() * reference.LabMember()
                 & reference.LabMembership().proj('lab_name', 'user_name'))
-            & args)
-        # q = acquisition.Session() * sess_proj * psych_curve * ephys_data * training_status * subject.Subject() * subject.SubjectLab() & ((reference.Lab() * reference.LabMember() & reference.LabMembership().proj('lab_name', 'user_name')))
+             & args)
+        dj.conn().query("SET SESSION max_join_size={}".format('18446744073709551615'))
+        q = q.proj(*proj).fetch(**kwargs) if proj else q.fetch(**kwargs)
+        dj.conn().query("SET SESSION max_join_size={}".format(original_max_join_size))
     elif subpath == 'subjpage':
         print('Args are:', args)
         proj_restr = None
@@ -412,10 +425,8 @@ def post_process(ret):
     else:
         abort(404)
 
-    if proj:
-        ret = q.proj(*proj).fetch(**kwargs)
-    else:
-        ret = q.fetch(**kwargs)
+    ret = q if isinstance(q, (list, dict)) else (q.proj(*proj).fetch(**kwargs)
+                                                 if proj else q.fetch(**kwargs))
 
     # print('D type', ret.dtype)
     # print(ret)

docker-compose-base.yml

@@ -0,0 +1,97 @@
+# Do NOT `docker-compose up` this file.
+# This is simply a base that other docker-compose files extend based on needs.
+version: '2.4'
+x-net: &net
+  networks:
+      - main
+services:
+  iblapi:
+    <<: *net
+    image: registry.vathes.com/ibl-navigator/iblapi:${DJ_API_VERSION}
+    environment:
+    - DJ_USER=maho
+    - DJ_HOST=datajoint.internationalbrainlab.org
+    # for public demo
+    # - DJ_USER=ibl-navigator
+    # - DJ_HOST=datajoint-public.internationalbrainlab.org
+    - AWS_DEFAULT_REGION=us-east-1
+    - WORKER_COUNT=4
+    env_file: ./.env
+    healthcheck:
+      test: >
+        wget --quiet --tries=1 --spider \
+          http://localhost:5000/v0/_health > /dev/null 2>&1 || exit 1
+      timeout: 3s
+      retries: 20
+    entrypoint: ["/src/iblapi/run-ibl-api.prod.sh"]
+    # cpus: 0.65
+    #cpu_quota: 100000
+    #cpu_period: 100ms
+  ibl-navigator:
+    <<: *net
+    image: registry.vathes.com/ibl-navigator/frontend:${FRONTEND_VERSION}
+    healthcheck:
+      test: curl --fail http://localhost:9000 || exit 1
+      timeout: 3s
+      retries: 20
+  ibl-node-server:
+    <<: *net
+    image: registry.vathes.com/ibl-navigator/node-server:${NODE_API_VERSION}
+    environment:
+    - NODE_ENV=development
+    - DEMO_USERNAME=ibluser
+    - PY_BACKEND=http://iblapi:5000
+    env_file:
+      - .env
+      - version.env
+    healthcheck:
+      test: curl --fail http://localhost:3333/version || exit 1
+      timeout: 3s
+      retries: 20
+      # interval: 1s
+  letsencrypt:
+    <<: *net
+    image: linuxserver/letsencrypt:amd64-0.38.0-ls62
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - [email protected]
+      - TZ=Europe/London
+      - ONLY_SUBDOMAINS=true
+      - VALIDATION=http
+    healthcheck:
+      test: /healthcheck.sh
+      timeout: 5s
+      retries: 300
+      interval: 1s
+    # depends_on:
+    #   nginx:
+    #     condition: service_healthy
+  nginx:
+    <<: *net
+    image: raphaelguzman/nginx:${DJ_NGINX_VERSION}
+    environment:
+      - ADD_frontend_TYPE=REST
+      - ADD_frontend_ENDPOINT=ibl-navigator:9000
+      - ADD_frontend_PREFIX=/
+      - ADD_nodebackend_TYPE=REST
+      - ADD_nodebackend_ENDPOINT=ibl-node-server:3333
+      - ADD_nodebackend_PREFIX=/api
+      - HTTPS_PASSTHRU=TRUE
+      - CERTBOT_HOST=letsencrypt:80
+    ports:
+      - "80:80"
+      - "443:443"
+    # depends_on:
+    #   iblapi:
+    #     condition: service_healthy
+    #   ibl-node-server:
+    #     condition: service_healthy
+    #   ibl-navigator:
+    #     condition: service_healthy
+networks:
+  main:
+    ipam:
+      driver: default
+      config:
+        - subnet: 10.28.0.0/16

docker-compose-build.yml

@@ -0,0 +1,43 @@
+# Use: Production build and manual test. Provided things look good, you may follow with a push.
+# env $(grep -hv '^#' version.env) docker-compose -f docker-compose-build.yml up --build
+# env $(grep -hv '^#' version.env) docker-compose -f docker-compose-build.yml push
+version: '2.4'
+x-net: &net
+  networks:
+      - main
+services:
+  iblapi:
+    extends:
+      file: ./docker-compose-base.yml
+      service: iblapi
+    build: ./backend
+    <<: *net
+  ibl-navigator:
+    extends:
+      file: ./docker-compose-base.yml
+      service: ibl-navigator
+    build: ./ibl-frontend
+    environment:
+    - PROD_NODE_BACKEND=https://fakeservices.datajoint.io/api
+    <<: *net
+  ibl-node-server:
+    extends:
+      file: ./docker-compose-base.yml
+      service: ibl-node-server
+    build: ./node_server
+    <<: *net
+  nginx:
+    extends:
+      file: ./docker-compose-base.yml
+      service: nginx
+    environment:
+    - SUBDOMAINS=fakeservices
+    - URL=datajoint.io
+    - CERTBOT_HOST=
+    <<: *net
+networks:
+  main:
+    ipam:
+      driver: default
+      config:
+        - subnet: 10.28.0.0/16