Merge pull request #88 from guzman-raphael/delete_quick

Allow cascade option for delete and other fixes

Author: Synicix

CHANGELOG.md

@@ -3,18 +3,32 @@
 Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
 
 ## [Unreleased]
+
+## Security
+- Documentation with detail regarding warning on bearer token. (#83) PR #88
+
+## Fixed
+- Incorrect virtual module reference of `schema_virtual_module` in table metadata. (#85) PR #88
+
 ### Added
-- Docker `dev` environment that supports hot reloading.
-- Documentation on setting up environments within `docker-compose` header.
+- Docker `dev` environment that supports hot reloading. PR #79
+- Documentation on setting up environments within `docker-compose` header. PR #79
+- `cascade` option for `/delete_tuple` route. (#86) PR #88
+- When delete with `cascade=False` fails due to foreign key relations, returns a HTTP error code of `409 Conflict` with a JSON body containing specifics of 1st child. (#86) PR #88
+
+### Changed
+- Replaced `DJConnector.snake_to_camel_case` usage with `datajoint.utils.to_camel_case`. PR #88
+- Default behavior for `/delete_tuple` now deletes without cascading. (#86) PR #88
+- Consolidated `pytest` fixtures into `__init__.py` to facilitate reuse. PR #88
 
 ### Removed
-- Docker `base` environment to simplify dependencies.
+- Docker `base` environment to simplify dependencies. PR #79
 
 ## [0.1.0a5] - 2021-02-18
 ### Added
 - List schemas method.
 - List tables method.
-- Create, Read, Update, Delete (CRUD) operations for DataJoint table tiers: `dj.Manual`, `dj.Lookup`.
+- Data entry, update, delete, and view operations for DataJoint table tiers: `dj.Manual`, `dj.Lookup`.
 - Read table records with proper paging and compounding restrictions (i.e. filters).
 - Read table definition method.
 - Support for DataJoint attribute types: `varchar`, `int`, `float`, `datetime`, `date`, `time`, `decimal`, `uuid`.

README.md

@@ -44,7 +44,7 @@ AS_SCRIPT=    # If 'TRUE', will not keep container alive but run tests and exit
   ```shell
   PKG_DIR=/opt/conda/lib/python3.8/site-packages/pharus # path to pharus installation
   TEST_DB_SERVER=example.com:3306 # testing db server address
-  TEST_DB_USER=root # testing db server user (needs CRUD on schemas, tables, users)
+  TEST_DB_USER=root # testing db server user (needs DDL privilege)
   TEST_DB_PASS=unsecure # testing db server password
   ```
 - For syntax tests, run `flake8 ${PKG_DIR} --count --select=E9,F63,F7,F82 --show-source --statistics`

docker-compose-test.yaml

@@ -36,7 +36,7 @@ services:
           PKG_DIR=/opt/conda/lib/python3.8/site-packages/pharus
           flake8 $${PKG_DIR} --count --select=E9,F63,F7,F82 --show-source --statistics
           echo "------ UNIT TESTS ------"
-          pytest -sv --cov-report term-missing --cov=$${PKG_DIR} /main/tests
+          pytest -sv --cov-report term-missing --cov=pharus /main/tests
           echo "------ STYLE TESTS ------"
           flake8 $${PKG_DIR} --count --max-complexity=20 --max-line-length=95 --statistics
         else

pharus/interface.py

@@ -1,5 +1,6 @@
 """Library for interfaces into DataJoint pipelines."""
 import datajoint as dj
+from datajoint.utils import to_camel_case
 import datetime
 import numpy as np
 from functools import reduce
@@ -94,9 +95,9 @@ def list_tables(jwt_payload: dict, schema_name: str):
                 tables_dict_list['imported_tables'].append(dj.utils.to_camel_case(table_name))
             elif table_type == 'Part':
                 table_name_parts = table_name.split('__')
-                tables_dict_list['part_tables'].append(DJConnector.snake_to_camel_case(
-                    table_name_parts[-2]) + '.' + DJConnector.snake_to_camel_case(
-                        table_name_parts[-1]))
+                tables_dict_list['part_tables'].append(
+                    to_camel_case(table_name_parts[-2]) + '.' +
+                    to_camel_case(table_name_parts[-1]))
             else:
                 raise UnsupportedTableType(table_name + ' is of unknown table type')
 
@@ -262,8 +263,9 @@ def get_table_definition(jwt_payload: dict, schema_name: str, table_name: str):
         """
         DJConnector.set_datajoint_config(jwt_payload)
 
-        schema_virtual_module = dj.create_virtual_module(schema_name, schema_name)
-        return getattr(schema_virtual_module, table_name).describe()
+        local_values = locals()
+        local_values[schema_name] = dj.VirtualModule(schema_name, schema_name)
+        return getattr(local_values[schema_name], table_name).describe()
 
     @staticmethod
     def insert_tuple(jwt_payload: dict, schema_name: str, table_name: str,
@@ -344,7 +346,7 @@ def update_tuple(jwt_payload: dict, schema_name: str, table_name: str,
 
     @staticmethod
     def delete_tuple(jwt_payload: dict, schema_name: str, table_name: str,
-                     tuple_to_restrict_by: dict):
+                     tuple_to_restrict_by: dict, cascade: bool = False):
         """
         Delete a specific record based on the restriction given (Can only delete 1 at a time)
         :param jwt_payload: Dictionary containing databaseAddress, username and password
@@ -356,6 +358,8 @@ def delete_tuple(jwt_payload: dict, schema_name: str, table_name: str,
         :type table_name: str
         :param tuple_to_restrict_by: Record to restrict the table by to delete
         :type tuple_to_restrict_by: dict
+        :param cascade: Allow for cascading delete, defaults to False
+        :type cascade: bool
         """
         DJConnector.set_datajoint_config(jwt_payload)
 
@@ -382,7 +386,7 @@ def delete_tuple(jwt_payload: dict, schema_name: str, table_name: str,
             raise InvalidDeleteRequest('Nothing to delete')
 
         # All check pass thus proceed to delete
-        tuple_to_delete.delete(safemode=False)
+        tuple_to_delete.delete(safemode=False) if cascade else tuple_to_delete.delete_quick()
 
     @staticmethod
     def get_table_object(schema_virtual_module, table_name: str):
@@ -413,14 +417,3 @@ def set_datajoint_config(jwt_payload: dict):
         dj.config['database.user'] = jwt_payload['username']
         dj.config['database.password'] = jwt_payload['password']
         dj.conn(reset=True)
-
-    @staticmethod
-    def snake_to_camel_case(string: str):
-        """
-        Helper method for converting snake to camel case
-        :param string: String in snake format to convert to camel case
-        :type string: str
-        :return: String formated in CamelCase notation
-        :rtype: str
-        """
-        return ''.join(string_component.title() for string_component in string.split('_'))

pharus/server.py

@@ -13,6 +13,9 @@
 import jwt
 from json import loads
 from base64 import b64decode
+from datajoint.errors import IntegrityError
+from datajoint.table import foregn_key_error_regexp
+from datajoint.utils import to_camel_case
 
 app = Flask(__name__)
 # Check if PRIVATE_KEY and PUBIC_KEY is set, if not generate them.
@@ -65,6 +68,22 @@ def api_version():
 @app.route(f"{environ.get('PHARUS_PREFIX', '')}/login", methods=['POST'])
 def login():
     """
+    *WARNING*: Currently, this implementation exposes user database credentials as plain text
+        in POST body once and stores it within a bearer token as Base64 encoded for subsequent
+        requests. That is how the server is able to submit queries on user's behalf. Due to
+        this, it is required that remote hosts expose the server only under HTTPS to ensure
+        end-to-end encryption. Sending passwords in plain text over HTTPS in POST request body
+        is common and utilized by companies such as GitHub (2021) and Chase Bank (2021). On
+        server side, there is no caching, logging, or storage of received passwords or tokens
+        and thus available only briefly in memory. This means the primary vulnerable point is
+        client side. Users should be responsible with their passwords and bearer tokens
+        treating them as one-in-the-same. Be aware that if your client system happens to be
+        compromised, a bad actor could monitor your outgoing network requests and capture/log
+        your credentials. However, in such a terrible scenario, a bad actor would not only
+        collect credentials for your DataJoint database but also other sites such as
+        github.com, chase.com, etc. Please be responsible and vigilant with credentials and
+        tokens on client side systems. Improvements to the above strategy is currently being
+        tracked in https://github.com/datajoint/pharus/issues/82.
     Login route which uses DataJoint database server login. Expects:
         (html:POST:body): json with keys
             {databaseAddress: string, username: string, password: string}
@@ -301,8 +320,17 @@ def delete_tuple(jwt_payload: dict):
         DJConnector.delete_tuple(jwt_payload,
                                  request.json["schemaName"],
                                  request.json["tableName"],
-                                 request.json["restrictionTuple"])
+                                 request.json["restrictionTuple"],
+                                 **{k: v.lower() == 'true'
+                                    for k, v in request.args.items() if k == 'cascade'},)
         return "Delete Sucessful"
+    except IntegrityError as e:
+        match = foregn_key_error_regexp.match(e.args[0])
+        return dict(error=e.__class__.__name__,
+                    error_msg=str(e),
+                    child_schema=match.group('child').split('.')[0][1:-1],
+                    child_table=to_camel_case(match.group('child').split('.')[1][1:-1]),
+                    ), 409
     except Exception as e:
         return str(e), 500