Deserializing JSON to Python Node

[weizhongzou]

What does node dictionary look like after loading from json? Does the loading take one-node only or it can handle a series of nodes?

[InnocentBug]

It will deserialize any standard python structure of nodes.

So, if you have nodes, that have nodes as children, the whole graph is deserialized.

If your json describes a list of nodes, you will obtain a list of deserialized nodes.
Same with a dictionary, if that is how your JSON looks like.

[nh916]

Request sent to API

This is pretty similar the JSON request that we send to the API, but we are using the requests python library to send HTTP POST requests

{
  "headers": {
    "Content-Type": "application/json",
    "Authorization": "Bearer YOUR_JWT_TOKEN"
  },
  "body": {
    "node": [
      "Project"
    ],
    "uid": "_:9b0a7eea-ef6d-4006-b6c7-c99d786a8c2f",
    "uuid": "9b0a7eea-ef6d-4006-b6c7-c99d786a8c2f",
    "name": "test_integration_experiment_project_name_4e9eec35ed264de4abe255392a5b6674",
    "collection": [
      {
        "node": [
          "Collection"
        ],
        "uid": "_:5514803b-10fb-48fe-820a-577a69fa22e3",
        "uuid": "5514803b-10fb-48fe-820a-577a69fa22e3",
        "name": "my collection name",
        "experiment": [
          {
            "node": [
              "Experiment"
            ],
            "uid": "_:74c9b3ac-392c-48bf-9c6b-3a3b6650f625",
            "uuid": "74c9b3ac-392c-48bf-9c6b-3a3b6650f625",
            "name": "my experiment name"
          }
        ],
        "inventory": [],
        "citation": []
      }
    ]
  }
}

Response Gotten From API

This is pretty much the response that we'll get from the API

{
   "admin":[
      {
         "created_at":"2023-04-28T21:19:10.518348Z",
         "model_version":"1.0.0",
         "node":[
            "User"
         ],
         "picture":"e52ca0a1-d7a8-40b3-b3f7-a0c752029be6.jpg",
         "uid":"0x15f901",
         "updated_at":"2023-05-12T21:38:43.91429Z",
         "username":"Navid Hariri",
         "uuid":"e52ca0a1-d7a8-40b3-b3f7-a0c752029be6"
      }
   ],
   "admin_count":1,
   "collection":[
      {
         "admin_count":0,
         "created_at":"2023-06-27T17:38:43.431595Z",
         "experiment":[
            {
               "admin_count":0,
               "computation_count":0,
               "computation_process_count":0,
               "created_at":"2023-06-27T17:38:43.431551Z",
               "data_count":0,
               "locked":false,
               "member_count":0,
               "model_version":"1.0.0",
               "name":"my experiment name",
               "node":[
                  "Experiment"
               ],
               "notes":"",
               "process_count":0,
               "public":false,
               "uid":"0x195710",
               "updated_at":"2023-06-27T17:38:43.431565Z",
               "uuid":"74c9b3ac-392c-48bf-9c6b-3a3b6650f625"
            }
         ],
         "experiment_count":1,
         "inventory_count":0,
         "locked":false,
         "member_count":0,
         "model_version":"1.0.0",
         "name":"my collection name",
         "node":[
            "Collection"
         ],
         "notes":"",
         "public":false,
         "uid":"0x19570f",
         "updated_at":"2023-06-27T17:38:43.431598Z",
         "uuid":"5514803b-10fb-48fe-820a-577a69fa22e3"
      }
   ],
   "collection_count":1,
   "created_at":"2023-06-27T17:38:43.431627Z",
   "locked":false,
   "material_count":0,
   "member_count":0,
   "model_version":"1.0.0",
   "name":"test_integration_experiment_project_name_4e9eec35ed264de4abe255392a5b6674",
   "node":[
      "Project"
   ],
   "notes":"",
   "public":false,
   "uid":"0x19570e",
   "updated_at":"2023-06-27T17:38:43.43163Z",
   "uuid":"9b0a7eea-ef6d-4006-b6c7-c99d786a8c2f"
}

Deserialized JSON To Project Node (Python Object)

Converting JSON to Project Node

This is pretty much how we deserialize and how it will look

my_project_from_api = cript.load_nodes_from_json(json.dumps(my_project_from_api_dict))

print(my_project_from_api .json)

{
   "node":[
      "Project"
   ],
   "uid":"_:dcd67e75-98dc-472a-83ff-5e3f1013f0b7",
   "uuid":"9b0a7eea-ef6d-4006-b6c7-c99d786a8c2f",
   "created_at":"2023-06-27T17:38:43.431627Z",
   "updated_at":"2023-06-27T17:38:43.43163Z",
   "model_version":"1.0.0",
   "name":"test_integration_experiment_project_name_4e9eec35ed264de4abe255392a5b6674",
   "admin":[
      {
         "uuid":"e52ca0a1-d7a8-40b3-b3f7-a0c752029be6"
      }
   ],
   "collection":[
      {
         "node":[
            "Collection"
         ],
         "uid":"_:774a674b-cacf-47ee-9818-5d05f8996ed9",
         "uuid":"5514803b-10fb-48fe-820a-577a69fa22e3",
         "created_at":"2023-06-27T17:38:43.431595Z",
         "updated_at":"2023-06-27T17:38:43.431598Z",
         "model_version":"1.0.0",
         "name":"my collection name",
         "experiment":[
            {
               "node":[
                  "Experiment"
               ],
               "uid":"_:8da1fb1e-e24b-45f2-9032-a5e72a8839a2",
               "uuid":"74c9b3ac-392c-48bf-9c6b-3a3b6650f625",
               "created_at":"2023-06-27T17:38:43.431551Z",
               "updated_at":"2023-06-27T17:38:43.431565Z",
               "model_version":"1.0.0",
               "name":"my experiment name"
            }
         ],
         "inventory":[
            
         ],
         "citation":[
            
         ]
      }
   ]
}

You may have realized that the JSON request that we send to the API has a decent amount of data, but when the API returns a response to us it has much more information, that is because the API can figure out who is sending request from the JWT and responds with more information for us to use.

The JWT token has many fields of data encoded it into it and signed with the API secret cryptographic key

Then when we deserialize the API response to Python Project Node we have more data than we originally sent, but less data than the API gave us, this is because we do not need/use all the data that the API responds with. However, in the future we might want to consider using more or less depending on different things.

The serialization and deserialization was mostly the great work of @InnocentBug!

Postman

For interacting with the API, I think the most standard tool and the one I use is the Postman software, which essentially is a nice way to send cURL requests to the API.

Resources

• cURL docs
• YouTube video tutorial of Postman

[InnocentBug]

Maybe one thing to note for the deserialization and serialization.

Currently, we do not support some situations for deserialization.

1. If an object is represented as a UID only (because it is fully represented elsewhere in the graph), the order inside the JSON is relevant. First an object has to represented fully, and after that definition, a second occurrence can be abbreviated to a UID only. Our deserialization code, does not support a random order.
2. When deserializing, all nodes have to be either fully specified, or abbreviated as UID (see 1). We do not support an abbreviation as a UUID yet. It should be possible to support this at a later time, by requesting this node from the back end, but at this point it is not supported.

The second case, should not occur with API responses, but might occur when an SDK serialized JSON is stored locally to disk.
For the serialization, some nodes are required to be UUID edges only (for example, Inventory.material). This is the default, but it would not be possible to deserialize this JSON subsequently (see 2.).
To avoid this situation, it is possible to obtain a full serialization, that can be fully deserialized like so.

with open("project.json", "w")  as json_file:
  json_file.write(project.get_json(condense_to_uuid={}).json)

Here we manually disable the UUID edge condensation, that would cause trouble for a subsequent deserialization.
Other kwargs can also be passed through the get_json function, for example indent=2 to generated a more human readable JSON.

[nh916]

Deserializing JSON to Python Node

The Serialize Deserialize part of the SDK was mostly handled by Ludwig, so he is more of an expert on that part. It uses the native JSON module from Python to do serialization and deserialization.

Deserialization

You can essentially give it a huge JSON and it can take the JSON and convert it to all the respective Python node objects, and nest them correctly inside of each other.
For example, if you have a JSON of a project with collection and experiment nested inside of each other, it will create a project node object, and inside of it put a collection node objects, and inside of it put an experiment node objects correctly. You can work with your Python nodes happily never knowing you converted them from JSON to Python node objects.
For deserialization we give the node class kwargs instead of arguments and it uses that to create the node. Every node and sub-object I believe accepts kwargs.

Serialization

For serializing from Python node objects, it essentially recursively walks down from the project node object to whatever it finds that is nested inside of it and converts all the data classes to JSON as it goes and nests them inside of each other.

DRY JSON Using UUID

The serialization is a bit tricky because there are some places that nodes have to repeat for the data model. For example, a material is part of both a process, inventory, and project, and in the beginning, we wrote the nodes out fully and had them repeat within the JSON. This resulted in a JSON that was over 8 million lines long. At some point Ludwig started to optimize the JSON and just created the nodes once and then referred to them by UUID in other parts of the JSON where it needed to repeat, making the JSON smaller and DRY.
Some of this resulted in tricky code that we may need to clean up/upgrade later. We may look into Pydantic, when v2 is stable, for serialization, deserialization. Pydantic has a feature to enforce types as well giving a layer of protection so the user never enters a string when it should be an int and catching those errors for us.

Material Identifier Sub-Object

Another tricky part is that the material node has identifier sub-objects in the data model, but the API does not have identifiers as sub-objects and instead has it as fields on the material node. If we put every identifier as an attribute with getters and setters in the material class, then our SDK code would be very rigid and any new identifier vocabulary would break the SDK, so we engineered a solution for this.
The SDK creates a list of dictionaries for the user to input their identifier type and value, essentially creating an identifier sub-object, but without the dedicated class. However, we serialize the list of dictionaries to fields on the material node before sending it to the API. Making our software more flexible, scalable, and easier to maintain. We may want to implement identifier sub-objects at some point in the future as a dedicated class and focus efforts on serializing it to whatever the API wants and gives, but for the first version I think this is sufficient.

Serialization and Deserialization Code

The code for serialization and deserialization is mostly handled within src/cript/nodes/core.py and src/cript/nodes/utils.py
We should have comments and docstrings written for every part of the code, and we rigorously enforce clean code, so it should be understandable, but if you don’t understand please feel free to let us know and maybe we can add some comments/docstrings/documentation to make the code easier to understand.

Saving Projects

We only allow users to save Project nodes to the API because everything is connected to the Project node, this way we don’t have any orphaned nodes within the SDK or API. However, in the future if we see the need, we might look into adding a feature of allowing users to save any nodes. Plans for this are uncertain as I currently know. I do not have plans to add this feature currently, nor do I know if we’ll need this, nor do I know the timeline for it.