Feature Request: Option to Fail Validation on Undocumented Properties with runners: openapi3:

[yuki2006]

thank you for maintaining this very useful library,

Problem/Motivation:

When the backend implementation isn't strictly based on the OpenAPI specification, or when evolving an API, discrepancies can arise between the actual response and the documented contract. Currently, when using runners: openapi3:, runn does not fail validation if the API response contains properties not explicitly defined in the referenced OpenAPI schema (e.g., properties added to the backend but not the spec, or properties removed from the spec but forgotten in the backend implementation).

This makes it difficult to detect when the actual API behavior drifts from the documented specification. We need a way to easily identify responses that include these undocumented properties, ensuring the API aligns with its stated contract and helping catch implementation oversights or forgotten cleanup tasks.

Proposed Solution:

Introduce an option, perhaps within the runners: definition for the http runner, or as a global flag, to enable stricter schema validation against the provided OpenAPI specification.

When this option is enabled:

1. runn should validate the response body against the schema specified in openapi3:.
2. If the response body contains any properties not defined in the corresponding schema definition (and additionalProperties is not explicitly set to true or a schema in the OpenAPI document itself), the validation should fail.
3. The test step should fail, and the output should clearly indicate that the failure was due to the presence of undocumented properties in the response body, listing the specific properties found.

Expected Behavior:

We would like the MRE provided below to fail when the proposed strict validation option is enabled, reporting an error because the undocumented id property exists in the response but is not defined in the openapi.yaml schema for the /item endpoint's 200 response.

Minimal Reproducible Example (MRE):

This MRE demonstrates the current behavior where the test passes despite an undocumented property in the response.

. openapi.yaml (Defines only the name property)

openapi: 3.0.3
info:
  title: API
  version: 1.0.0
paths:
  /item:
    get:
      summary: Get an item
      operationId: getItem
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                properties:
                  name: # 'id' property is NOT defined here
                    type: string
                    example: "Example Item"
                required:
                  - name

server.go (Returns id and name)

package main

import (
    "encoding/json"
    "log"
    "net/http"
)

// Item struct includes 'id' which is not in openapi.yaml
type Item struct {
    ID   int    `json:"id"`
    Name string `json:"name"`
}

func itemHandler(w http.ResponseWriter, r *http.Request) {
    // Response includes the 'id' field
    item := Item{ID: 1, Name: "Sample"}
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)
    err := json.NewEncoder(w).Encode(item)
    if err!= nil {
        log.Printf("Error encoding JSON response: %v", err)
        http.Error(w, "Internal Server Error", http.StatusInternalServerError)
    }
}

func main() {
    http.HandleFunc("/item", itemHandler)
    port := "8080"
    log.Printf("Starting server on port %s", port)
    if err := http.ListenAndServe(":"+port, nil); err!= nil {
        log.Fatalf("Could not start server: %s\n", err)
    }
}

test.runn.yaml

desc: Test OpenAPI
runners:
  req:
    endpoint: http://localhost:8080
    openapi3: openapi.yaml

steps:
  - desc: ""
    req:
      /item:
        get:
          body: null
    test: |
       current.res.status == 200

Result Explanation:

Current Behavior: The runn run test.runn.yaml command SUCCEEDS with the current implementation. Reason: Even with runners: openapi3:, the undocumented id property in the response is ignored by default.

Expected Behavior (with proposed feature enabled):

We request an option that would cause the runn run test.runn.yaml command to FAIL. Reason: The response contains the id property, which is not defined in the schema, and the requested strict validation option should report this as an undocumented property.

[k1LoW]

@yuki2006 Thank you for your request.

In the OpenAPI Spec, the default value for additionalProperties is true.

As runn, we should follow that specification,
and we believe it is appropriate to specify additionalProperties: false in the OpenAPI documentation.