Merge pull request #658 from ixcat/v0.12-doc-update

[WIP] docs-parts updates for v0.12

Author: dimitri-yatsenko

README.md

@@ -22,6 +22,65 @@ If you already have an older version of DataJoint installed using `pip`, upgrade
 ```bash
 pip3 install --upgrade datajoint
 ```
+## Python Native Blobs
+
+For the v0.12 release, the variable `enable_python_native_blobs` can be
+safely enabled for improved blob support of python datatypes if the following
+are true:
+
+  * This is a new DataJoint installation / pipeline(s)
+  * You have not used DataJoint prior to v0.12 with your pipeline(s)
+  * You do not share blob data between Python and Matlab
+
+Otherwise, please read the following carefully:
+
+DataJoint v0.12 expands DataJoint's blob serialization mechanism with
+improved support for complex native python datatypes, such as dictionaries
+and lists of strings.
+
+Prior to DataJoint v0.12, certain python native datatypes such as
+dictionaries were 'squashed' into numpy structured arrays when saved into
+blob attributes. This facilitated easier data sharing between Matlab
+and Python for certain record types. However, this created a discrepancy
+between insert and fetch datatypes which could cause problems in other
+portions of users pipelines.
+
+For v0.12, it was decided to remove the type squashing behavior, instead
+creating a separate storage encoding which improves support for storing
+native python datatypes in blobs without squashing them into numpy
+structured arrays. However, this change creates a compatibility problem
+for pipelines which previously relied on the type squashing behavior
+since records saved via the old squashing format will continue to fetch
+as structured arrays, whereas new record inserted in DataJoint 0.12 with
+`enable_python_native_blobs` would result in records returned as the
+appropriate native python type (dict, etc).  Read support for python
+native blobs also not yet implemented in DataJoint for Matlab.
+
+To prevent data from being stored in mixed format within a table across
+upgrades from previous versions of DataJoint, the
+`enable_python_native_blobs` flag was added as a temporary guard measure
+for the 0.12 release. This flag will trigger an exception if any of the
+ambiguous cases are encountered during inserts in order to allow testing
+and migration of pre-0.12 pipelines to 0.11 in a safe manner.
+
+The exact process to update a specific pipeline will vary depending on
+the situation, but generally the following strategies may apply:
+
+  * Altering code to directly store numpy structured arrays or plain
+    multidimensional arrays. This strategy is likely best one for those 
+    tables requiring compatibility with Matlab.
+  * Adjust code to deal with both structured array and native fetched data.
+    In this case, insert logic is not adjusted, but downstream consumers
+    are adjusted to handle records saved under the old and new schemes.
+  * Manually convert data using fetch/insert into a fresh schema.
+    In this approach, DataJoint's create_virtual_module functionality would 
+    be used in conjunction with a a fetch/convert/insert loop to update 
+    the data to the new native_blob functionality.
+  * Drop/Recompute imported/computed tables to ensure they are in the new
+    format.
+
+As always, be sure that your data is safely backed up before modifying any
+important DataJoint schema or records.
 
 ## Documentation and Tutorials
 A number of labs are currently adopting DataJoint and we are quickly getting the documentation in shape in February 2017.

docs-parts/admin/5-blob-config_lang1.rst

@@ -1,20 +1,17 @@
 .. code-block:: python
 
-   # default external storage
-   dj.config['external'] = dict(
-                 protocol='s3',
-                 endpoint='https://s3.amazonaws.com',
-                 bucket = 'testbucket',
-                 location = '/datajoint-projects/myschema',
-                 access_key='1234567',
-                 secret_key='foaf1234')
+  dj.config['stores'] = {
+    'external': dict(  # 'regular' external storage for this pipeline
+                  protocol='s3',
+                  endpoint='https://s3.amazonaws.com',
+                  bucket = 'testbucket',
+                  location = '/datajoint-projects/myschema',
+                  access_key='1234567',
+                  secret_key='foaf1234'),
+    'external-raw'] = dict( # 'raw' storage for this pipeline
+                  protocol='file',
+                  location='/net/djblobs/myschema')
+  }
+  # external object cache - see fetch operation below for details.
+  dj.config['cache'] = '/net/djcache'
 
-   # raw data storage
-   dj.config['extnernal-raw'] = dict(
-                 protocol='file',
-                 location='/net/djblobs/myschema')
-
-   # external object cache - see fetch operation below for details.
-   dj.config['cache'] = dict(
-                 protocol='file',
-                 location='/net/djcache')

docs-parts/admin/5-blob-config_lang4.rst

@@ -1,3 +1,29 @@
+
+To remove only the tracking entries in the external table, call `delete`
+on the external table for the external configuration with the argument 
+`delete_external_files=False`.
+
+.. note::
+
+  Currently, cleanup operations on a schema's external table are not 100% 
+  transaction safe and so must be run when there is no write activity occurring 
+  in tables which use a given schema / external store pairing.
+
 .. code-block:: python
 
-    >>> schema.external_table.delete_garbage()
+    >>> schema.external['external_raw'].delete(delete_external_files=False)
+
+To remove the tracking entries as well as the underlying files, call `delete`
+on the external table for the external configuration with the argument
+`delete_external_files=True`.
+
+.. code-block:: python
+
+    >>> schema.external['external_raw'].delete(delete_external_files=True)
+
+.. note::
+
+  Setting `delete_external_files=True` will always attempt to delete
+  the underlying data file, and so should not typically be used with 
+  the `filepath` datatype.
+

docs-parts/admin/5-blob-config_lang5.rst

@@ -1,3 +1,29 @@
+0.12.0 -- October 1, 2019
+-------------------------
+* Dropped support for Python 3.4
+* Support secure connections with TLS (aka SSL) PR #620
+* Convert numpy array from python object to appropriate data type if all elements are of the same type (#587) PR #608
+* Remove expression requirement to have additional attributes (#604) PR #604
+* Support for filepath datatype (#481) PR #603, #659
+* Support file attachment datatype (#480, #592, #637) PR #659
+* Fetch return a dict array when specifying `as_dict=True` for specified attributes. (#595) PR #593
+* Support of ellipsis in `proj`:  `query_expression.proj(.., '-movie')` (#499) PR #578
+* Expand support of blob serialization (#572, #520, #427, #392, #244, #594) PR #577
+* Support for alter (#110) PR #573
+* Support for `conda install datajoint` via `conda-forge` channel (#293)
+* `dj.conn()` accepts a `port` keyword argument (#563) PR #571
+* Support for UUID datatype (#562) PR #567
+* `query_expr.fetch("KEY", as_dict=False)` returns results as `np.recarray`(#414) PR #574
+* `dj.ERD` is now called `dj.Diagram` (#255, #546) PR #565
+* `dj.Diagram` underlines "distinguished" classes (#378) PR #557
+* Accept alias for supported MySQL datatypes (#544) PR #545
+* Support for pandas in `fetch` (#459, #537) PR #534
+* Support for ordering by "KEY" in `fetch` (#541) PR #534
+* Improved external storage - a migration script needed from version 0.11  (#467, #475, #480, #497) PR #532
+* Increase default display rows (#523) PR #526
+* Bugfixes (#521, #205, #279, #477, #570, #581, #597, #596, #618, #633, #643, #644, #647, #648, #650, #656)
+* Minor improvements (#538)
+
 0.11.1 -- Nov 15, 2018
 ----------------------
 * Fix ordering of attributes in proj (#483 and #516)