Apiv2 (#1128)

* Add compression support

JSON:API responses can be potentially be quite large, especially
since no minification to the JSON output is performed.

Adding deflate compression with reduce transfer sizes.

* WIP: Add extra development tooling

* WIP: Start conversion of hashtopolis to new JSON:API standard

* WIP: Start of GET of JSON:API implementation

Focus on  working on Users, GlobalPermissionGroups and AccessGroups

* Fix ensure consistent composer packages get installed.

This file was never commited and missed, due to the wildcard *.lock pattern set in .gitigore

https://getcomposer.org/doc/01-basic-usage.md#commit-your-composer-lock-file-to-version-control

* Fix expansion parameter not working.

Rename to reflect new naming (include)

* Cosmetic: Rename variables to match new meaning

It is confusing to mix expand with include. Thus rename all
 variables in  python test code from expand to include.

* Add interactive debugging of pytest unit files

The hashtopolis.py cannot be debugged with default Interactive
debugging of VScode debugging of pytests.

Thus make a small workaround based of the idea:
https://stackoverflow.com/questions/62419998/how-can-i-get-pytest-to-not-catch-exceptions/62563106#62563106

* WIP: Start working on delete and creation actions

* Fix 500 returned when page is not found

Should be 404, how-ever this trow was not captured, due to incorrect ordering.

* WIP: Innitial support for PATCH/CREATE operations

* WIP: All unit tests complete. GET mostly implemented

- PATCH/DELETE requires mapping to new format.
- Helpers need some thinking on request/response.
- Relation updates needs to be implemented.

* WIP: Need ER relations for CREATE

When creating an object the ER relations are sent via JSON:API
how-ever without an formal definition, there is no way of knowing
whether something is considered an relation.

* WIP: Example model for CREATE in client

Add PoC conversion using Django tooling for reference purposes.
This could be used as base for creating objects within the python libary
(also to do some client-side checking as well).

* Updated PHP dependencies

* Refactor pagination to adhere to JSON:API standards

- Updated pagination parameters from 'page[after]' and 'page[size]' to 'page[offset] and 'page[limit']
- This change aligns with the JSON:API specification for pagination (https://jsonapi.org/format/#fetching-pagination)

* Refactored links object to adhere to JSON:API standard
-changed 'page[after]' and 'page[size] to 'page[offset]' and
'page[limit]'  in the link objects

* feat: Add support for limit queries in ORM

* Made pagination and sorting working by fixing some logic bugs

* Added more input filtering for the Limit filter

* Fixed the location header to comply to the JSON API standard in case of an POST request

* WIP: Made start to link first and last attribute in json response to GET requests for pagination

* Added page size exceeded error handling and started implementing link next and previous for pagination

* FEAT: Implemented first and last in paginated json response

* FEAT: Implemented previous and next within JSON pagination response

* FEAT: Made PATCH request compliant to JSON API 1.1

* FEAT made patch To-One relationship compliant to JSON API standard

* FEAT made get request to relationship working also throught intermediate tables

* FEAT made pagination in to many relationships work

* FEAT implemented patching to many relationships

* FEAT added to many relationship from cracker to task in API

* FEAT added to many relationship from crackerType to task in API

* FEAT implemented delete to many relationship link and fixed bug in patch to many relationship link

* FEAT fixed bug in PATCH to many relation and implemented POST to many relation

* FEAT made patchOne compliant to JSON API 1.1 standard

* FEAT fix a bug withing pagination and ordering and made Post one compliant to JSON API 1.1

* FEAT updated the tests to adhere to JSON API spec

* FEAT added tests for pagination

* clean up pagination test

* Added range validation to validating data

* Added a json response moduel for the responses

* Added function to validate if its allowed to mutate DBA fields

* Fixed inconsitency in color len in task feautures

* Added test that will verify database size constraint

* FEAT: made helper API endpoints compliant to json API standard, by putting no resource record responses in the metadata

---------

Co-authored-by: Rick van der Zwet <[email protected]>
Co-authored-by: Rick van der Zwet <[email protected]>
Co-authored-by: Jesse van Zutphen (DBS) <[email protected]>

Author: 4 people

.devcontainer/devcontainer.json

@@ -13,6 +13,7 @@
         "xdebug.php-debug",
         "bmewburn.vscode-intelephense-client",
         "editorconfig.editorconfig",
+        "eamodio.gitlens",
         "github.vscode-pull-request-github",
         "ms-python.python",
         "ms-python.flake8",

.editorconfig

@@ -63,11 +63,38 @@
             }
         },
         {
-            "name": "Python: File",
-            "type": "python",
-            "request": "launch",
-            "program": "${file}",
-            "justMyCode": true
+          "name": "Python: htcli (list Users with Include)",
+          "type": "python",
+          "request": "launch",
+          "program": "./ci/apiv2/htcli.py",
+          "args": ["list", "Users", "-v", "DEBUG", "--include", "globalPermissionGroup"],
+          "justMyCode": true
+        },
+        {
+          "name": "Python: htcli run delete-test-data",
+          "type": "python",
+          "request": "launch",
+          "program": "./ci/apiv2/htcli.py",
+          "args": ["run", "delete-test-data", "--commit"],
+          "justMyCode": true
+        },
+        {
+          "name": "Python: Debug pytest file",
+          "type": "python",
+          "request": "launch",
+          "module": "pytest",
+          "args": ["${file}", "--exitfirst"],
+          "justMyCode": true,
+          "env": {
+            "_PYTEST_RAISE": "1"
+          },
+        },
+        {
+          "name": "Python: File",
+          "type": "python",
+          "request": "launch",
+          "program": "${file}",
+          "justMyCode": true
         }
     ],
     "inputs": [

.vscode/launch.json

@@ -9,7 +9,7 @@ TOKEN=$(curl -X POST --user admin:hashtopolis http://localhost:8080/api/v2/auth/
 
 Fetch object:
 ```
-curl --header "Content-Type: application/json" -X GET --header "Authorization: Bearer $TOKEN" 'http://localhost:8080/api/v2/ui/hashlists/1?expand=hashes' -d '{}'
+curl --compressed --header "Authorization: Bearer $TOKEN" -g 'http://localhost:8080/api/v2/ui/hashtypes?page[size]=5'
 ```
 
 Access database:
@@ -23,6 +23,12 @@ docker exec $(docker ps -aqf "ancestor=mysql:8.0") mysql -u root -phashtopolis -
 docker exec $(docker ps -aqf "ancestor=mysql:8.0") tail -f /tmp/mysql_all.log
 ```
 
+Shortcut for testing within development setup:
+```
+cd ~/src/hashtopolis/server/ci/apiv2
+pytest --exitfirst --last-failed
+```
+
 ### paper flipchart scribbles
 
 #### v2 beta

ci/apiv2/HACKING.md

@@ -0,0 +1,12 @@
+import os
+import pytest
+
+if os.getenv('_PYTEST_RAISE', "0") != "0":
+
+    @pytest.hookimpl(tryfirst=True)
+    def pytest_exception_interact(call):
+        raise call.excinfo.value
+
+    @pytest.hookimpl(tryfirst=True)
+    def pytest_internalerror(excinfo):
+        raise excinfo.value

ci/apiv2/conftest.py

@@ -1,4 +1,4 @@
 {
-            "typeName": "generic",
+            "typeName": "generic2",
             "isChunkingAvailable": true
  }

ci/apiv2/create_crackertype_001.json

@@ -0,0 +1,14 @@
+components:
+  schemas:
+      Hash:
+        type: object
+        properties:
+          id:
+            type: integer
+            minimum: 1
+            readOnly: true
+          userMembers:
+            type: array
+            items:
+              $ref: #/components/schemas/User
+            uniqueItems: true