Fix merge conflicts.

Author: guzman-raphael

CHANGELOG.md

@@ -1,6 +1,13 @@
 ## Release notes
 
-### 0.14.1 -- TBD
+### Upcoming
+- Fixed - Fix altering a part table that uses the "master" keyword - PR [#991](https://github.com/datajoint/datajoint-python/pull/991)
+- Fixed - `.ipynb` output in tutorials is not visible in dark mode ([#1078](https://github.com/datajoint/datajoint-python/issues/1078)) PR [#1080](https://github.com/datajoint/datajoint-python/pull/1080)
+- Changed - Readme to update links and include example pipeline image
+- Changed - Docs to add landing page and update navigation
+- Changed - `.data` method to `.stream` in the `get()` method for S3 (external) objects PR [#1085](https://github.com/datajoint/datajoint-python/pull/1085)
+- Fixed - Docs to rename `create_virtual_module` to `VirtualModule`
+- Added - Skeleton from `datajoint-company/datajoint-docs` repository for docs migration
 - Added - Initial `pytest` for `test_connection`
 
 ### 0.14.0 -- Feb 13, 2023
@@ -16,7 +23,7 @@
 - Deprecated - `table._update()` PR [#1073](https://github.com/datajoint/datajoint-python/pull/1073)
 - Deprecated - old-style foreign key syntax PR [#1073](https://github.com/datajoint/datajoint-python/pull/1073)
 - Deprecated - `dj.migrate_dj011_external_blob_storage_to_dj012()` PR [#1073](https://github.com/datajoint/datajoint-python/pull/1073)
-* Added - Method to set job keys to "ignore" status - PR [#1068](https://github.com/datajoint/datajoint-python/pull/1068)
+- Added - Method to set job keys to "ignore" status - PR [#1068](https://github.com/datajoint/datajoint-python/pull/1068)
 
 ### 0.13.8 -- Sep 21, 2022
 - Added - New documentation structure based on markdown PR [#1052](https://github.com/datajoint/datajoint-python/pull/1052)

LNX-docker-compose.yml

@@ -44,7 +44,7 @@ services:
       interval: 15s
   fakeservices.datajoint.io:
     <<: *net
-    image: datajoint/nginx:v0.2.4
+    image: datajoint/nginx:v0.2.5
     environment:
       - ADD_db_TYPE=DATABASE
       - ADD_db_ENDPOINT=db:3306

README.md

@@ -1,8 +1,6 @@
 [![DOI](https://zenodo.org/badge/16774/datajoint/datajoint-python.svg)](https://zenodo.org/badge/latestdoi/16774/datajoint/datajoint-python)
-[![Build Status](https://travis-ci.org/datajoint/datajoint-python.svg?branch=master)](https://travis-ci.org/datajoint/datajoint-python)
 [![Coverage Status](https://coveralls.io/repos/datajoint/datajoint-python/badge.svg?branch=master&service=github)](https://coveralls.io/github/datajoint/datajoint-python?branch=master)
 [![PyPI version](https://badge.fury.io/py/datajoint.svg)](http://badge.fury.io/py/datajoint)
-[![Requirements Status](https://requires.io/github/datajoint/datajoint-python/requirements.svg?branch=master)](https://requires.io/github/datajoint/datajoint-python/requirements/?branch=master)
 [![Slack](https://img.shields.io/badge/slack-chat-green.svg)](https://datajoint.slack.com/)
 
 # Welcome to DataJoint for Python!
@@ -12,22 +10,32 @@ DataJoint for Python is a framework for scientific workflow management based on
 DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at Baylor College of Medicine for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers.
 Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com).
 
-- [Getting Started](https://datajoint.com/docs/core/datajoint-python/latest/getting-started/)
-- [DataJoint Elements](https://datajoint.com/docs/elements/) - Catalog of example pipelines
-- [DataJoint CodeBook](https://codebook.datajoint.io) - Interactive online tutorials
-- Contribute
+## Data Pipeline Example
+
+![pipeline](https://raw.githubusercontent.com/datajoint/datajoint-python/master/images/pipeline.png)
+
+[Yatsenko et al., bioRxiv 2021](https://doi.org/10.1101/2021.03.30.437358)
+
+## Getting Started
+
+- Install from PyPI
+
+     ```bash
+     pip install datajoint
+     ```
+
+- [Documentation & Tutorials](https://datajoint.com/docs/core/datajoint-python/)
+
+- [Interactive Tutorials](https://github.com/datajoint/datajoint-tutorials) on GitHub Codespaces
 
+- [DataJoint Elements](https://datajoint.com/docs/elements/) - Catalog of example pipelines for neuroscience experiments
+
+- Contribute
   - [Development Environment](https://datajoint.com/docs/core/datajoint-python/latest/develop/)
+
   - [Guidelines](https://datajoint.com/docs/community/contribute/)
 
 - Legacy Resources (To be replaced by above)
   - [Documentation](https://docs.datajoint.org)
-  - [Tutorials](https://tutorials.datajoint.org)
 
-## Citation
-
-- If your work uses DataJoint for Python, please cite the following Research Resource Identifier (RRID) and manuscript.
-
-- DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for Python (version `<Enter version number>`)
-
-- Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
+  - [Tutorials](https://tutorials.datajoint.org)

datajoint/s3.py

@@ -68,7 +68,9 @@ def fput(self, local_file, name, metadata=None):
     def get(self, name):
         logger.debug("get: {}:{}".format(self.bucket, name))
         try:
-            return self.client.get_object(self.bucket, str(name)).data
+            with self.client.get_object(self.bucket, str(name)) as result:
+                data = [d for d in result.stream()]
+            return b"".join(data)
         except minio.error.S3Error as e:
             if e.code == "NoSuchKey":
                 raise errors.MissingExternalFile("Missing s3 key %s" % name)

datajoint/user_tables.py

@@ -238,3 +238,7 @@ def drop(self, force=False):
             raise DataJointError(
                 "Cannot drop a Part directly.  Delete from master instead"
             )
+
+    def alter(self, prompt=True, context=None):
+        # without context, use declaration context which maps master keyword to master table
+        super().alter(prompt=prompt, context=context or self.declaration_context)

docs/docker-compose.yaml

@@ -7,7 +7,7 @@ services:
       context: ../
       args:
         - PACKAGE
-    image: ${PACKAGE}-docs
+    image: ${PACKAGE}_python-docs
     environment:
       - PACKAGE
       - UPSTREAM_REPO
@@ -16,7 +16,7 @@ services:
       - ..:/main
     user: ${HOST_UID}:anaconda
     ports:
-      - 8080:80
+      - 80:80
     command:
       - sh
       - -c

docs/mkdocs.yaml

@@ -1,24 +1,77 @@
 # ---------------------- PROJECT SPECIFIC ---------------------------
 
-site_name: DataJoint Python
+site_name: DataJoint Documentation
 repo_url: https://github.com/datajoint/datajoint-python
 repo_name: datajoint/datajoint-python
 nav:
-  - DataJoint Python: getting-started/index.md
+  - DataJoint Python: index.md
   - Getting Started: getting-started/index.md
-  - Existing Pipelines: concepts/existing-pipelines.md
-  - Query Language:
-      - Common Commands: query-lang/common-commands.md
-      - Operators: query-lang/operators.md
-      - Iteration: query-lang/iteration.md
-      - Query Caching: query-lang/query-caching.md
+  - Concepts: 
+    - Principles: concepts/principles.md
+    - Glossary: concepts/glossary.md
+  - System Administration: 
+    - Database Administration: sysadmin/dba.md
+    - File Storage: sysadmin/filestore.md
+  - Client Configuration: 
+    - Install: client/install.md
+    - Credentials: client/creds.md
+    - Settings: client/settings.md
+    - File Stores: client/stores.md
+  - Schema Design:
+    - Schema Creation: design/schema.md
+    - Table Definition:
+      - Table Tiers: design/tables/tiers.md
+      - Declaration Syntax: design/tables/declare.md
+      - Primary Key: design/tables/primary.md
+      - Attributes: design/tables/attributes.md
+      - Lookup Tables: design/tables/lookup.md
+      - Blobs: design/tables/blobs.md
+      - Attachments: design/tables/attach.md
+      - Filepaths: design/tables/filepath.md
+      - Custom Datatypes: design/tables/customtype.md
+      - Dependencies: design/tables/dependencies.md
+      - Indexes: design/tables/indexes.md
+      - Master-Part Relationships: design/tables/master-part.md
+    - Schema Diagrams: design/diagrams.md
+    - Entity Normalization: design/normalization.md 
+    - Data Integrity: design/integrity.md
+    - Schema Recall: design/recall.md
+    - Schema Drop: design/drop.md
+    - Schema Modification: design/alter.md
+  - Data Manipulations:
+    - Insert: manipulation/insert.md
+    - Delete: manipulation/delete.md
+    - Update: manipulation/update.md
+    - Transactions: manipulation/transactions.md
+  - Data Queries:
+    - Common Commands: query/common-commands.md
+    - Fetch: query/fetch.md
+    - Iteration: query/iteration.md
+    - Operators: query/operators.md
+    - Restrict: query/restrict.md
+    - Projection: query/project.md
+    - Join: query/join.md
+    - Aggregation: query/aggregation.md
+    - Union: query/union.md
+    - Universal Sets: query/universals.md
+    - Query Caching: query/query-caching.md
+  - Computations:
+    - Make Method: compute/make.md
+    - Populate: compute/populate.md
+    - Key Source: compute/key-source.md
+    - Distributed Computing: compute/distributed.md
+  - Internals:
+    - SQL Transpilation: internal/transpilation.md
   - Reproducibility:
       - Table Tiers: reproduce/table-tiers.md
       - Make Method: reproduce/make-method.md
+  - Existing Pipelines: existing-pipelines.md
   - Tutorials:
       - tutorials/json.ipynb
+  - FAQ: faq.md
   - Develop: develop.md
-  - Changelog: about/changelog.md
+  - Citation: citation.md
+  - Changelog: changelog.md
   - API: api/ # defer to gen-files + literate-nav
 
 # ---------------------------- STANDARD -----------------------------
@@ -50,9 +103,6 @@ theme:
         name: Switch to light mode
 plugins:
   - search
-  - redirects:
-      redirect_maps:
-        "index.md": "getting-started/index.md"
   - mkdocstrings:
       default_handler: python
       handlers:
@@ -95,11 +145,11 @@ markdown_extensions:
         - name: mermaid
           class: mermaid
           format: !!python/name:pymdownx.superfences.fence_code_format
+  - pymdownx.magiclink # Displays bare URLs as links
+  - pymdownx.tasklist: # Renders check boxes in tasks lists
+      custom_checkbox: true
 extra:
   generator: false # Disable watermark
-  analytics:
-    provider: google
-    property: !ENV GOOGLE_ANALYTICS_KEY
   version:
     provider: mike
   social:

docs/src/.overrides/assets/stylesheets/extra.css

@@ -15,27 +15,35 @@
 html a[title="DataJoint"].md-social__link svg {
     color: var(--dj-primary);
 }
+
 html a[title="Slack"].md-social__link svg {
     color: var(--dj-primary);
 }
+
 html a[title="LinkedIn"].md-social__link svg {
     color: var(--dj-primary);
 }
+
 html a[title="Twitter"].md-social__link svg {
     color: var(--dj-primary);
 }
+
 html a[title="GitHub"].md-social__link svg {
     color: var(--dj-primary);
 }
+
 html a[title="DockerHub"].md-social__link svg {
     color: var(--dj-primary);
 }
+
 html a[title="PyPI"].md-social__link svg {
     color: var(--dj-primary);
 }
+
 html a[title="StackOverflow"].md-social__link svg {
     color: var(--dj-primary);
 }
+
 html a[title="YouTube"].md-social__link svg {
     color: var(--dj-primary);
 }
@@ -91,3 +99,7 @@ html a[title="YouTube"].md-social__link svg {
     /* previous/next text */
     /* --md-footer-fg-color: var(--dj-white); */
 }
+
+[data-md-color-scheme="slate"] .jupyter-wrapper .Table Td {
+    color: var(--dj-black)
+}

docs/src/.overrides/partials/nav.html

@@ -14,7 +14,7 @@
         {#-
         Add DataJoint home link to navigation header, otherwise unchanged
         -#}
-        <a href="https://datajoint.com/docs/" title="DataJoint">
+        <a href="https://datajoint.com/docs/core/" title="DataJoint">
             ⬅ Home
         </a>
     </label>