Define a JSON format to input into the compliance report

[andrewyatz]

Related to #14 there is a need to define the JSON structure to pass into the compliance report generator. Currently refget has a format, which was defined as a logical output of the current refget compliance toolkit. The JSON format should be capable of

• present metadata about the implementation being tested
• record metadata about the test run e.g. time executed at, time finished perhaps
• record the results of all the tests
• group these tests by categories as appropriate

Once a basic JSON format is defined we can define a JSON schema to describe the format and formalise the expected input into the site. You can see some example JSON schemas from here.

[andrewyatz]

Should also review the current refget and rnaget API compliance reports for metadata to report on each test run

[rkaushik15]

Please review the following draft format and let me know if any changes are required
https://pastebin.com/aFmRX0cX

[andrewyatz]

I've sent this through a beautifier to better visualise what's going on.

[
  {
    "server_name": "",
    "base_url": "",
    "date_time": "",
    "implemented": 3,
    "total_tests": 3,
    "total_tests_passed": 1,
    "total_tests_failed": 1,
    "total_tests_skipped": 1,
    "test_results": {
      "sequence": [
        {
          "test_name": "",
          "result": 1,
          "description": "",
          "text": "",
          "parent": "",
          "api_query_info": "",
          "warning": "",
          "cases": ""
        }
      ],
      "sequence_range": [
        {
          "test_name": "",
          "result": 1,
          "description": "",
          "text": "",
          "parent": "",
          "api_query_info": "",
          "warning": "",
          "cases": ""
        }
      ],
      "info": [
        {
          "test_name": "",
          "result": 1,
          "description": "",
          "text": "",
          "parent": "",
          "api_query_info": "",
          "warning": "",
          "cases": ""
        }
      ],
      "metadata": [
        {
          "test_name": "",
          "result": 1,
          "description": "",
          "text": "",
          "parent": "",
          "api_query_info": "",
          "warning": "",
          "cases": ""
        }
      ]
    },
    "total_warnings": 1,
    "high_level_summary": {
      "sequence": {
        "name": "",
        "result": 0
      },
      "sequence_range": {
        "name": "",
        "result": -1
      },
      "info": {
        "name": "",
        "result": 1
      },
      "metadata": {
        "name": "",
        "result": 1
      }
    }
  }
]

[andrewyatz]

Few comments from my 1st pass review

• what is the difference between implemented and total_tests?
• what does total_warnings convey?
• what is the difference between description and text?
• do we need a parent field if we have already grouped tests e.g. would we set a parent of metadata if it is already in the metadata test array
• could we rename high_level_summary to summary?
• do we want a way for a test to record any kind of headers used e.g. in a hash?

[rkaushik15]

• "implemented" field was intended to give an overview of the functionalities' test that we have written. An example of the usage would be
  "implemented":{
  "sequence":true,
  "sequence_range":true,
  "metadata":true,
  "info":true
  }

• total_warings convey the total number of tests that have thrown a warning. A warning is different than a failed test, so it might be a useful metric to have.

• "description" field describes what we are trying to do, that is what the test case is, whereas the "text" field is sort of like a description of the RESULT. For eg., "the server supports circular sequence and successfully returns a circular sequence on querying" or "the test is skipped as the server does not support circular sequence". We can change these to "test_description" and "result_text".

• I thought of the same thing and kept the parent field as it might be used to get the result of the "overview" page. This is something that I will find out once I start coding and would depend upon the type of approach we use to get that overview count.

• The "api_query_info" field would be an array that stores the Request, Response, and other info. We can easily add header info into this field as well, it would simply be adding one more line of code. So I think we should store the info.

[rkaushik15]

I missed one point.
Sure, we can change it to "summary"

[andrewyatz]

Okay all of this sounds really good. Let's go with the model and can you generate a real example from the test suite and let's go from there

[rkaushik15]

Cool, I’ll start and push the code asap

[rkaushik15]

Draft JSON Schema document -> https://drive.google.com/file/d/1ohNThHdwjjU2CXlXkCaiUXv-4F98B3B9/view?usp=sharing

Please review and let me know if any changes or additions are needed.

[andrewyatz]

Thanks @rkaushik15 I've had a quick look and here's my first set of comments

• xxxxx_object is used in multiple places. I'm not sure it adds to the legibility of the schema.
  • We know the element is an object because type says so and it has properties
  • Consider switching test_object to something more descriptive e.g. test or test_result
• I am not sure what the use of $id gives us
  • e.g. when you say #/total_failed_tests why give it an $id?
  • Consider only using $id when the object will be referenced again
• There is no format attribute set for date_time or base_url
  • JSON schema has a number of format hints which are shortcut parameters used to check data integrity primarily in string fields
  • https://json-schema.org/understanding-json-schema/reference/string.html#format
  • I would add formats to all string fields where there is an imposed format expected
  • If there is not a default format available then consider writing a regular expression
• #service_object/result should be a boolean because there is are two states described; pass or fail.
  • If there is a third type then that should be mentioned in the schema
  • #test_object/result has a similar data structure and does have three states with 0 being skip.
  • Should this have been copied into the #service_object/result?
• Should api_query_info be another object or a string?
  • Pairwise key/value does mean we can control parameter rendering
  • There's also no way to differentiate between a request and response header

Also have a look at https://www.jsonschemavalidator.net/ and paste in your JSON test response and check that the schema does validate it. The JSON responses should also, once live, use a $schema tag to refer to this JSON schema you're writing up now

[rkaushik15]

• Sure, changing it to 'test', etc.

• I honestly only added the $id field because I saw the field in the few examples I went through and lack knowledge about the conventions and utility of the tag. But I agree, we don't need to refer to each and every property, so I'll remove it where it isn't necessary.

• Thanks for linking the format options, I will go through it.

• It depends on what we want the overview section of the report to say in these cases. For eg., currently, I've only added 'pass' or 'fail' logic to the endpoints (i.e., even if some tests are skipped for a valid reason, the endpoint overview says 'pass' and not 'skipped'). So the #service_object/result is different than #test_object/result which supports for 'skip' also. We can convert this field to boolean, or should we change the logic a bit to also mention skips in the overview section?

• I left api_query_info as an array of string to make this section optional, i.e., developers are free to add request, response or header info if they wish but also they can add any other info (or leave out one of the above) such as response headers, as a simple string to this array. Would making it a defined object restrict the usage to only the 'keys' that we would add?

• I did test the output report against this schema and it was validated. I will add the $schema tag to the JSON report.