Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ These are:

1. `optional/`: Contains tests that are considered optional. Note that this subdirectory currently conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (in which case they are not truly optional in such a language). In the future this directory structure will be made richer to reflect these differences more clearly.

2. `proposals/`: Contains a subfolder for each active proposal to the specification. If the proposal is a keyword (generally the case), then the subfolder will bear the name of that keyword. Inside the proposal subfolder is a series of test files that would contain amendments to the required test suite should the proposal be incorporated into the specification. These test should be considered volitile while the proposal is in development, however implementations claiming to support the proposal are expected to pass its tests.
2. `proposals/`: Contains a subfolder for each active proposal to the specification. If the proposal is a keyword (generally the case), then the subfolder will bear the name of that keyword. Inside the proposal subfolder is a series of test files that would contain amendments to the required test suite should the proposal be incorporated into the specification. These tests should be considered volatile while the proposal is in development; however, implementations claiming to support the proposal are expected to pass its tests.

## Using the Suite to Test a Validator Implementation

Expand Down
8 changes: 4 additions & 4 deletions annotations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on which values of an instance. These tests are agnostic of any output format.

## Supported Dialects

Although the annotation terminology of didn't appear in the spec until 2019-09,
Although annotation terminology didn't appear in the spec until 2019-09,
the concept is compatible with every version of JSON Schema. Test Cases in this
Test Suite are designed to be compatible with as many releases of JSON Schema as
possible. They do not include `$schema` or `$id`/`id` keywords so
Expand All @@ -25,7 +25,7 @@ A short description of what behavior the Test Case is covering.

The `compatibility` option allows you to set which dialects the Test Case is
compatible with. Test Runners can use this value to filter out Test Cases that
don't apply the to dialect currently under test. The terminology for annotations
don't apply to the dialect currently under test. The terminology for annotations
didn't appear in the spec until 2019-09, but the concept is compatible with
older releases as well. When setting `compatibility`, test authors should take
into account dialects before 2019-09 for implementations that chose to support
Expand All @@ -42,7 +42,7 @@ Case is compatible with draft-07 and up.
**Example**: `"compatibility": "7"`

You can use a `<=` operator to indicate that the Test Case is compatible with
releases less then or equal to the given release. This example indicates that
releases less than or equal to the given release. This example indicates that
the Test Case is compatible with 2019-09 and under.

**Example**: `"compatibility": "<=2019"`
Expand Down Expand Up @@ -111,6 +111,6 @@ could contribute annotations for a single keyword.
An empty object is an assertion that the annotation must not appear at the
`location` for the `keyword`.

As a convention for this Test Suite, the `expected` array should be sorted such
As a convention for this Test Suite, the `expected` object should be sorted such
that the most recently encountered value for an annotation given top-down
evaluation of the schema comes before previously encountered values.
2 changes: 1 addition & 1 deletion annotations/test-case.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"pattern": "^(<=|=)?([123467]|2019|2020)(,(<=|=)?([123467]|2019|2020))*$"
},
"schema": {
"markdownDescription": "This schema shouldn't include `$schema` or `id`/`$id` unless necesary for the test because Test Cases should be designed to work with as many releases as possible.",
"markdownDescription": "This schema shouldn't include `$schema` or `id`/`$id` unless necessary for the test because Test Cases should be designed to work with as many releases as possible.",
"type": ["boolean", "object"]
},
"externalSchemas": {
Expand Down
4 changes: 2 additions & 2 deletions output-tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ _**NOTE** Although the formats didn't change much between 2019-09 and 2020-12, t

The tests are organized by specification release and then into two categories: content and structure.

Content tests verify that the keywords are producing the correct annotations and/or error messages. Since there are no requirements on the content of error messages, there's not much that can be verified for them, but it is possible to identify when a error message _could_ be present. Primarily, these tests need to extensively cover the annotation behaviors of each keyword. The only output format needed for these tests is `basic` for 2019-09/2020-12 and `list` for later versions.
Content tests verify that the keywords are producing the correct annotations and/or error messages. Since there are no requirements on the content of error messages, there's not much that can be verified for them, but it is possible to identify when an error message _could_ be present. Primarily, these tests need to extensively cover the annotation behaviors of each keyword. The only output format needed for these tests is `basic` for 2019-09/2020-12 and `list` for later versions.

Structure tests verify that the structures of the various formats (i.e. `flag`, `basic`, `detailed`, `verbose` for 2019-09/2020-12 and `flag`, `list`, `hierarchical` for later versions) are correct. These tests don't need to cover each keyword; rather they need to sufficiently cover the various aspects of building the output structures by using whatever keywords are necessary to do so.

Expand All @@ -22,7 +22,7 @@ The `output` property itself has a property for each of the output formats where

## Other notes

### Ambiguity around 2020-09/2020-12 `basic`
### Ambiguity around 2019-09/2020-12 `basic`

The 2019-09/2020-12 specs don't define the structure of `basic` very thoroughly. Specifically there is a nuance where if the list contains a single output node, there are two possible structures, given the text:

Expand Down
4 changes: 2 additions & 2 deletions tests/v1/proposals/README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# JSON Schema Proposals Tests

The subfolders herein represent tests for each active proposal to the specification. If the proposal is a keyword (generally the case), then the subfolder will bear the name of that keyword. Inside the proposal subfolder is a series of test files that would contain amendments to the required test suite should the proposal be incorporated into the specification. These test should be considered volitile while the proposal is in development, however implementations claiming to support the proposal are expected to pass its tests.
The subfolders herein represent tests for each active proposal to the specification. If the proposal is a keyword (generally the case), then the subfolder will bear the name of that keyword. Inside the proposal subfolder is a series of test files that would contain amendments to the required test suite should the proposal be incorporated into the specification. These tests should be considered volatile while the proposal is in development; however, implementations claiming to support the proposal are expected to pass its tests.

> [!IMPORTANT]
> This folder is NOT intended for proposed additions to the test suite or the specification.

Proposals to the test suite can be made by [opening an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues/new) in this repository.

Proposals to the specification should be made by following the process defined by our [Feature Life Cycle](https://github.com/json-schema-org/json-schema-spec/blob/main/PROCESS.md#feature-life-cycle).
Proposals to the specification should be made by following the process defined by our [Feature Life Cycle](https://github.com/json-schema-org/json-schema-spec/blob/main/PROCESS.md#feature-life-cycle).
Loading