Some suggestions for future development

[keksipurkki]

Hi,

I have given this library a test run in our current project. I'm using the 0.4.0 stable version, but I am aware that you are working on a 1.0.0 release.

After some initial hurdles, I have been able to fully automate the implementation of API smoke tests, and I think this library fits quite well in our overall testing setup. At the very least, it makes for an excellent way to test that the published API docs do not contain bugs.

However, I'd like to point out three areas where the library could be improved:

1. Allow the use of documented example objects

The OpenAPI spec allows to specify examples of schema objects (as JSON). It would be handy if the library would be able to take advantage of the example objects when generating payloads rather than relying on autogenerated data. (Naturally, there should be a mechanism for adjusting for dynamic properties like timestamp fields)

2. More ergonomic API for testing individual endpoints

Often we are focusing on a single OpenAPI operation when developing. The exclude/include API of the 0.4.0 version a bit difficult to use to filter out everything except one particular endpoint.

3. Prefer operationId to path/method pairs

IMO, the OpenAPI operationId is a more natural way of thinking about OpenAPI specs. We also use operationIds when we talk about the API rather than the path/method pairs. The way I would go about writing the test suite is one robot file per operation and one test per documented API response.

[robinmackaij]

Thanks for your remarks! Please find my feedback below.

Hi,

I have given this library a test run in our current project. I'm using the 0.4.0 stable version, but I am aware that you are working on a 1.0.0 release.

After some initial hurdles, I have been able to fully automate the implementation of API smoke tests, and I think this library fits quite well in our overall testing setup. At the very least, it makes for an excellent way to test that the published API docs do not contain bugs.

However, I'd like to point out three areas where the library could be improved:

1. Allow the use of documented example objects

The OpenAPI spec allows to specify examples of schema objects (as JSON). It would be handy if the library would be able to take advantage of the example objects when generating payloads rather than relying on autogenerated data. (Naturally, there should be a mechanism for adjusting for dynamic properties like timestamp fields)

I've in the past considered using the example field in data generation, but discarded the idea for a number of reasons.
The main reason is that it's a free-form field:

A free-form property to include an example of an instance for this schema. To represent examples that cannot be naturally represented in JSON or YAML, a string value can be used to contain the example with escaping where necessary.

Deprecated: The example property has been deprecated in favor of the JSON Schema examples keyword. Use of example is discouraged, and later versions of this specification may remove it.

So the best I could do, from a data generation point of view, is try to cast whatever is in the examples field to the type of the schema and use the current mechanics if that fails.
There's a catch to that however: if the example value is valid only once, any subsequent use of that example value would cause a failed request. In addition, there would be no variations in data for any field that has an example defined. I believe that that would reduce the possibility of finding implementation errors in the target API.

Now if you'd really like to use the example from the spec in data generation, that's possible using the PropertyValueConstraint from the mappings_path file. Note that this also allows you to generate the dynamic properties in any (Python) way you want.

Considering all these points, using examples in data generation is not something I'm planning to support.

2. More ergonomic API for testing individual endpoints

Often we are focusing on a single OpenAPI operation when developing. The exclude/include API of the 0.4.0 version a bit difficult to use to filter out everything except one particular endpoint.

Are you using a combination of included_paths and excluded_paths with the use of the wildcard *? We settled on this combination of include / exclude and filtering at the client project where this was needed.

Now if you're really focusing on a single endpoint, I'd encourage you to take a look at the v1.0.0 beta; the library generator would allow you to generate a library for your API and you can then use the desired endpoint keyword to flesh things out before enabling the endpoint in the OpenApiDriver suite.
You could take a look at how that could look in the tests on the beta branch

3. Prefer operationId to path/method pairs

IMO, the OpenAPI operationId is a more natural way of thinking about OpenAPI specs. We also use operationIds when we talk about the API rather than the path/method pairs. The way I would go about writing the test suite is one robot file per operation and one test per documented API response.

The library generator coming in v1.0.0 does use the operationId to generate the keywords but there's a flag to use the summary value instead and the the fallback is {method}_{path}.
There's a few reasons for this:

1. The operationId is optional. There is a uniqueness requirement, but e.g. an OpenAPI spec with operationIds only defined for POST methods would be perfectly fine (as long as each one is unique within the scope of that API).
2. Some API frameworks generate an operationId that can be overridden (but often are not) and, depending on the path and path parameter names, can be usable or completely unreadable.

What's important to realize here is the at the core, this is http communication which is a method on a url (which follows from the path). The operationId is a "label for humans" for each unique path/method pair, it plays no role in actual http requests. This is the reason OpenApiDriver generates the test cases like it does: each path can have one or more methods that it support and each path/method pair can result in one or more responses.

Now if you're in a situation where the API has nice, usable operationIds defined, I would (again =) ) urge you to take a look at the library generator since I believe it aligns very well with your one robot file per operation and one test per documented API response. You'd essentially have one suite per generated keyword in the library.

[keksipurkki]

Thanks for your reply.

The points you highlight make sense from a library maintainer point of view. However, I think your points boil down to being 'opinionated' versus 'general'. I work in finance and when it comes down to actually writing a test, practically none of the request body properties can be randomly generated. In fact, the APIs we write try their best to prevent random data from passing thru! The latest "victim" is the name of the payee in a payment order. Pretty soon, it too is verified against the credit account number (https://www.europeanpaymentscouncil.eu/what-we-do/other-schemes/verification-payee).

Now, the challenge would be how to specify that the credit account must be owned by the payee using OpenAPI, an offspring of JSON schema technology. This is where the example object escape hatch comes in handy.

I would therefore argue in favor of the Open-Closed principle. If your library users want to use their example objects, the library API should allow it with the caveat that the responsibility of code correctness is now on the user.

I did not actually know that the operationId is optional. I would nonetheless argue that the whole point of OpenAPI is to create an abstraction on top of the HTTP protocol. Since it's the development that costs, not the electricity bill, it pays to cater to developers. At least I would find it completely OK if the library was 'opinionated' and required the use of operationIds.

We will give the 1.0.0 version a try!

[robinmackaij]

What I have to consider is that not all users of the libraries have a say in whether or not certain fields in an OpenAPI spec are there and / or correct. For this reason, I cannot make using examples for data generation standard behavior. It could be added as a flag to able attempting using the example value, where a Dto mapping would override any possible examples.

When I read the OpenAPI Specification Introduction I don't feel like the idea behind the OAS is to create an abstraction on top of the HTTP protocol; the goal seems to be to document the API's interface and behavior in a consistent way. The available paths and the methods on each path are crucial in understanding how an API works, so hiding that information is, in my opinion, not the right approach.
My personal opinion aside; as long as the operationId is optional in the specification and considering the users of these libraries often don't have a say in whether or not the API they have to test / work with does define the operationId, I will not change the libraries to require the operationId.

Also note that using the operationId as primary "entrypoint" would also mean that the included_paths and excluded_paths filtering (especially the wildcard support) would no longer be possible since test case generation would no longer be path-based. For some APIs I've worked on, that'd be a huge step back.

Now what would be a small change is to add the operationId (if available) to the tags of the generated test cases. Would that be helpful for you?