Docs edits, CICD wf

Author: CBroz1

.github/workflows/development.yaml

@@ -37,3 +37,22 @@ jobs:
           COMPOSE_HTTP_TIMEOUT: "120"
         run: |
           docker-compose -f LNX-docker-compose.yaml up --build --exit-code-from app
+  publish-docs:
+    if: |
+      github.event_name == 'push' &&
+      startsWith(github.ref, 'refs/tags')
+    needs: test
+    runs-on: ubuntu-latest
+    env:
+      DOCKER_CLIENT_TIMEOUT: "120"
+      COMPOSE_HTTP_TIMEOUT: "120"
+    steps:
+      - uses: actions/checkout@v2
+      - name: Deploy docs
+        run: |
+          export MODE=BUILD
+          export PACKAGE=datajoint
+          export UPSTREAM_REPO=https://github.com/${GITHUB_REPOSITORY}.git
+          export HOST_UID=$(id -u)
+          docker compose -f docs/docker-compose.yaml up --exit-code-from docs --build
+          git push origin gh-pages

CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
+[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
+
+## [3.5.0] - 2022-03-21
+
++ Bugfix: Cascading delete for renamed foreign keys (#379) PR #386
++ Minor: Add renaming the same attribute multiple times within a single projection PR #386
++ Minor: Add config for reading values with 32-bit dimensions (datajoint/mym#86) PR #395
+
+[3.5.0]: https://github.com/datajoint/element-deeplabcut/releases/tag/3.5.0

README.md

@@ -4,26 +4,25 @@
 
 DataJoint for MATLAB is a high-level programming interface for relational databases designed to support data processing chains in science labs. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, and querying data.
 
-DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at Baylor College of Medicine for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers.
-Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com). Related resources are listed at https://datajoint.org.
-
-For more information, please visit our [documentation](https://datajoint.com/docs/core/datajoint-matlab/).
-
-## Config
-For help in utilizing `dj.config` (added in `3.4.0`), you may access the help via `help('dj.config')` or review it online [here](https://github.com/datajoint/datajoint-matlab/blob/c2bd6b3e195dfeef773d4e12bad5573c461193b0/%2Bdj/config.m#L2-L27). Formal documentation to follow.
+For more information, see our
+[general](https://datajoint.com/docs/welcome/) and
+[MATLAB](https://datajoint.com/docs/core/datajoint-matlab/) documentation pages.
 
 ## Citation
+
 + If your work uses DataJoint for MATLAB, please cite the following Research Resource Identifier (RRID) and manuscript.
 
 + DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for MATLAB (version `<Enter version number>`)
 
 + Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
 
-## Running Tests Locally
+## For Developers: Running Tests Locally
+
 <details>
 <summary>Click to expand details</summary>
 
-* Create an `.env` with desired development environment values e.g.
++ Create an `.env` with desired development environment values e.g.
+
 ``` sh
 MATLAB_USER=rguzman
 MATLAB_LICENSE=IyBCRUd... # For image usage instructions see https://github.com/guzman-raphael/matlab, https://hub.docker.com/r/raphaelguzman/matlab
@@ -34,11 +33,12 @@ MATLAB_GID=1000
 MYSQL_TAG=5.7
 MINIO_VER=RELEASE.2022-01-03T18-22-58Z
 ```
-* `cp local-docker-compose.yaml docker-compose.yaml`
-* `docker-compose up` (Note configured `JUPYTER_PASSWORD`)
-* Select a means of running MATLAB e.g. Jupyter Notebook, GUI, or Terminal (see bottom)
-* Add `tests` directory to path e.g. in MATLAB, `addpath('tests')`
-* Run desired tests. Some examples are as follows:
+
++ `cp local-docker-compose.yaml docker-compose.yaml`
++ `docker-compose up` (Note configured `JUPYTER_PASSWORD`)
++ Select a means of running MATLAB e.g. Jupyter Notebook, GUI, or Terminal (see bottom)
++ Add `tests` directory to path e.g. in MATLAB, `addpath('tests')`
++ Run desired tests. Some examples are as follows:
 
 | Use Case                     | MATLAB Code                                                                    |
 | ---------------------------- | ------------------------------------------------------------------------------ |
@@ -47,23 +47,20 @@ MINIO_VER=RELEASE.2022-01-03T18-22-58Z
 | Run one specific test        | `runtests('TestTls/TestTls_testInsecureConn')`                                   |
 | Run tests based on test name | `import matlab.unittest.TestSuite;`<br>`import matlab.unittest.selectors.HasName;`<br>`import matlab.unittest.constraints.ContainsSubstring;`<br>`suite = TestSuite.fromClass(?Main, ... `<br><code>&nbsp;&nbsp;&nbsp;&nbsp;</code>`HasName(ContainsSubstring('Conn')));`<br>`run(suite)`|
 
-
 ### Launch Jupyter Notebook
 
-* Navigate to `localhost:8888`
-* Input Jupyter password
-* Launch a notebook i.e. `New > MATLAB`
-
++ Navigate to `localhost:8888`
++ Input Jupyter password
++ Launch a notebook i.e. `New > MATLAB`
 
 ### Launch MATLAB GUI (supports remote interactive debugger)
 
-* Shell into `datajoint-matlab_app_1` i.e. `docker exec -it datajoint-matlab_app_1 bash`
-* Launch Matlab by runnning command `matlab`
-
++ Shell into `datajoint-matlab_app_1` i.e. `docker exec -it datajoint-matlab_app_1 bash`
++ Launch Matlab by running command `matlab`
 
 ### Launch MATLAB Terminal
 
-* Shell into `datajoint-matlab_app_1` i.e. `docker exec -it datajoint-matlab_app_1 bash`
-* Launch Matlab with no GUI by runnning command `matlab -nodisplay`
++ Shell into `datajoint-matlab_app_1` i.e. `docker exec -it datajoint-matlab_app_1 bash`
++ Launch Matlab with no GUI by running command `matlab -nodisplay`
 
 </details>

docs/.docker/Dockerfile

@@ -9,7 +9,7 @@ RUN \
     git config --global user.email "[email protected]"&& \
     git config --global pull.rebase false && \
     git init
-COPY --chown=anaconda:anaconda ./${PACKAGE} /main/${PACKAGE}
+# COPY --chown=anaconda:anaconda ./${PACKAGE} /main/${PACKAGE}
 COPY --chown=anaconda:anaconda ./docs/mkdocs.yaml /main/docs/mkdocs.yaml
 COPY --chown=anaconda:anaconda ./docs/src /main/docs/src
 COPY --chown=anaconda:anaconda ./CHANGELOG.md /main/docs/src/about/changelog.md

docs/cspell.json

@@ -15,9 +15,16 @@
         "src/archive"
     ],
     "words": [
-        "conda",
+        "ghtb",
+        "disp",
+        "mltbx",
+        "setenv",
+        "isempty",
+        "fetchn",
+        "struct",
         "linenums",
         "repr",
+        "conda",
         "numpy",
         "sess",
         "cond",

docs/mkdocs.yaml

@@ -11,12 +11,11 @@ nav:
     - Common Commands:  query-lang/common-commands.md
     - Operators: query-lang/operators.md
     - Iteration: query-lang/iteration.md
-    - Query Caching: query-lang/query-caching.md
   - Reproducibility:
     - Table Tiers: reproduce/table-tiers.md
     - Make Method: reproduce/make-method.md
-  - Tutorials: tutorials.md
-  - Changelog: about/changelog.md
+  # - Tutorials: tutorials.md # Commented out pending viable draft
+  - Changelog: changelog.md
 
 # ---------------------------- STANDARD -----------------------------
 
@@ -50,9 +49,6 @@ plugins:
   - redirects:
       redirect_maps:
         "index.md": "getting-started/index.md"
-  - gen-files:
-      scripts:
-      - ./src/api/make_pages.py
   - literate-nav:
       nav_file: navigation.md
   - exclude-search:

docs/src/changelog.md

@@ -0,0 +1 @@
+../../CHANGELOG.md

docs/src/concepts/existing-pipelines.md

@@ -21,7 +21,9 @@ experiment_schema = experiment.getSchema();
 experiment_schema.v.Session() & 'session_id=1234';
 ```
 
-Note: You can view the available tables in a schema by using tab completion on
-the `schema.v` property.
+???+ Note
+
+    You can view the available tables in a schema by using tab completion on
+    the `schema.v` property.
 
 To visualize an unfamiliar schema, see commands for generating [diagrams](../../getting-started/#diagram).

docs/src/getting-started/index.md

@@ -10,11 +10,6 @@ First, please install DataJoint via one of the following:
     2. Search and Select `DataJoint`
     3. Select *Add from GitHub*
 
-=== "GHToolbox"
-
-    1. Install *GHToolbox* using using an appropriate method in https://github.com/datajoint/GHToolbox
-    2. run: `ghtb.install('datajoint/datajoint-matlab')`
-
 === "Matlab < R2016"
 
     1. Utilize MATLAB built-in GUI i.e. *Top Ribbon -> Add-Ons -> Get Add-Ons*
@@ -25,6 +20,11 @@ First, please install DataJoint via one of the following:
     6. Right-Click and Select *Install*
     7. Select *Install*
 
+=== "GHToolbox"
+
+    1. Install *GHToolbox* using using an appropriate method in https://github.com/datajoint/GHToolbox
+    2. run: `ghtb.install('datajoint/datajoint-matlab')`
+
 === "From source"
 
     1. Download `DataJoint.mltbx` locally
@@ -48,15 +48,15 @@ server `tutorial-db.datajoint.io` with username `alice` and password
 
 ```matlab
 setenv DJ_USER alice
-setenv DJ_HOST alicelab.datajoint.io
+setenv DJ_HOST tutorial-db.datajoint.io
 setenv DJ_PASS 'fake-password'
 ```
 
 !!! note
 
     Although you may connect to any MySQL server of your choice, the DataJoint company
     offers an online tutorial environment at `tutorial-db.datajoint.io`. Simply sign up
-    for a free[DataJoint account](https://accounts.datajoint.io). You will be granted
+    for a free [DataJoint account](https://accounts.datajoint.io). You will be granted
     privileges to create schemas that are prefixed as `{user}_`.
 
 You will need to execute these commands at the beginning of each DataJoint work session.
@@ -71,11 +71,15 @@ password when connecting to the database.
 To change the database password, use the following command
 
 ```matlab
->> dj.setPassword('my#cool!new*psswrd')
+>> dj.setPassword('my#cool!new*password')
 ```
 
 And update your credentials in your startup script for the next session.
 
+For more information on various settings, access help via `help('dj.config')` or review
+it online
+[here](https://github.com/datajoint/datajoint-matlab/blob/c2bd6b3e195dfeef773d4e12bad5573c461193b0/%2Bdj/config.m#L2-L27).
+
 ## Creating Schemas
 
 A schema can be created either automatically using the `dj.createSchema`
@@ -90,7 +94,7 @@ We can create the database schema using the following command:
 query(dj.conn, 'CREATE SCHEMA `{user}_my_schema`')
 ```
 
-!!! Note "Server privileges"
+??? Note "Server privileges"
 
     You must have create privileges for the schema name pattern. It is a common practice
     to grant all privileges to users for schemas that begin with the username, in
@@ -250,6 +254,9 @@ Or for individual or sets of tables:
 erd my_schema.Rectangle
 draw(dj.ERD(my_schema.Rectangle) + dj.ERD(my_schema.Area))
 ```
+
+![Shapes Pipeline](../images/shapes_pipeline.svg)
+
 ### Customize
 
 Adding or substracting a number to a diagram object adds nodes downstream or upstream,

docs/src/query-lang/common-commands.md

@@ -1,4 +1,3 @@
-XX - refer to existing table defintions
 # Common Commands
 
 <!-- ## Insert is present in the general docs here-->
@@ -39,9 +38,9 @@ For example, the two methods below are equivalent although the second
 method creates an extra variable.
 
 ``` matlab
-result = fetch(university.Student, '*');
+result = fetch(my_schema.Rectangle, '*');
 
-query = university.Student;
+query = my_schema.Rectangle;
 result = query.fetch()
 ```
 
@@ -53,7 +52,7 @@ the `struct`.
 
 ``` matlab
 keys = query.fetch;
-keys = fetch(university.Student & university.StudentMajor);
+keys = fetch(my_schema.Rectangle & my_schema.Area);
 ```
 
 Note that MATLAB allows calling functions without the parentheses `()`.
@@ -65,7 +64,7 @@ retrieves the entire result as a struct array.
 
 ``` matlab
 data = query.fetch('*');
-data = fetch(university.Student & university.StudentMajor, '*');
+data = fetch(my_schema.Rectangle & my_schema.Area, '*');
 ```
 
 ??? Note "For very large tables..."
@@ -92,8 +91,8 @@ returned in the form of cell arrays, even when `query` happens to contain a sing
 entity.
 
 ``` matlab
-[name, img] = query.fetch1('name', 'image'); % (1)
-[names, imgs] = query.fetchn('name', 'image'); % (2)
+[shape_id, height] = query.fetch1('shape_id', 'shape_height'); % (1)
+[shape_ids, heights] = query.fetchn('shape_id', 'shape_height'); % (2)
 ```
 
 1. When the table has exactly one entity.
@@ -106,20 +105,21 @@ retrieved by `fetchn`. This can be done by adding a special input argument indic
 the request and another output argument to receive the key values:
 
 ``` matlab
-[names, imgs, keys] = query.fetchn('name', 'image', 'KEY');
+[shape_ids, heights, keys] = query.fetchn('shape_id', 'shape_height', 'KEY');
 ```
 
 The resulting value of `keys` will be a column array of type `struct`.
 This mechanism is only implemented for `fetchn`.
 
 ### Rename and calculate
 
-In DataJoint for MATLAB, all `fetch` methods have all the same capability as the `proj
-<proj>` operator. For example, renaming an attribute can be accomplished using the
-syntax below.
+In DataJoint for MATLAB, all `fetch` methods have all the same capability as the 
+[`proj` operator](../operators#proj). For example, renaming an attribute can be 
+accomplished using the syntax below.
 
 ``` matlab
-[names, BMIs] = query.fetchn('name', 'weight/height/height -> bmi');
+[shape_ids, perimeter] = query.fetchn('shape_id', ...
+    '2*(shape_height+shape_width) -> perimeter');
 ```
 
 See [`proj`](../operators#proj) for an in-depth description of projection.
@@ -129,23 +129,23 @@ See [`proj`](../operators#proj) for an in-depth description of projection.
 To sort the result, add the additional `ORDER BY` argument in `fetch` and `fetchn`
 methods as the last argument. 
 
-The following command retrieves the field `course_name` from courses in the biology
-department, sorted by course number.
+The following command retrieves the field `shape_id` from rectangles with height greater
+than 2, sorted by width.
 
 ``` matlab
-notes = fetchn(university.Course & 'dept="BIOL"', 'course_name', ...
-     'ORDER BY course');
+notes = fetchn(my_schema.Rectangle & 'shape_height>2"', 'shape_id' ...
+     'ORDER BY shape_width');
 ```
 
 The ORDER BY argument is passed directly to SQL and follows the same syntax as the 
 [ORDER BY clause](https://dev.mysql.com/doc/refman/5.7/en/order-by-optimization.html)
 
 Similarly, the LIMIT and OFFSET clauses can be used to limit the result to a subset of
-entities. For example, to return the most advanced courses, one could do the
+entities. For example, to return the five rectangles with largest area, one could do the
 following:
 
 ``` matlab
-s = fetch(university.Course, '*', 'ORDER BY course DESC LIMIT 5')
+s = fetch(my_schema.Area, '*', 'ORDER BY shape_area DESC LIMIT 5')
 ```
 
 The limit clause is passed directly to SQL and follows the same