Add `$self` for self-identifying documents #4389

handrews · 2025-02-28T15:07:53Z

See the proposal for background information.

This is a rather minimal approach, as @karenetheridge and I plan to work together on a more thorough revamp of the document parsing / reference resolution sections.

This adds $self as a way for a document to define its own URI for use in reference targets, and as the base URI for relative URI references in the document.

This does not impact the resolution of relative API URLs. [NOTE: I'm not entirely sure about this, but it seems more useful this way to allow multiple deployed locations of an OAD to correspond to multiple deployments of the API.)

Tick one of the following options:

schema changes are included in this pull request
schema changes are needed for this pull request but not done yet
no schema changes are needed for this pull request

lornajane

This looks pretty solid, but could we introduce use of $self in one or two examples so people realise it's there?

ralfhandl · 2025-03-04T21:26:58Z

Looks good. An example would be nice.

handrews · 2025-03-06T05:07:05Z

@lornajane @ralfhandl I thought I did include one but apparently I did that on an earlier attempt on a different branch. I'll port it over in the morning.

karenetheridge

A few comments/questions throughout.

karenetheridge · 2025-03-06T16:27:53Z

src/oas.md

 In practice, this is usually the retrieval URI of the document, which MAY be determined based on either its current actual location or a user-supplied expected location.
+The document's base URI MUST also be used to resolve a relative `$self` URI reference.


add "...which is then used to resolve relative URI references in other Objects, as above" ?

An example to use lower down could be: the document has "$self": "/api" in it, but at runtime the base URI is provided as as "https://dev.example.com" -- therefore the effective value of $self for this document (the URI to use as the base for all future relative resolutions) is "https://dev.example.com/api".

I'm confused about this addition since in the description of the pull request there's a mention that the $self value does NOT affect how relative URLs are resolved. They should still be relative to wherever-you-found-this, as I understood it.

"wherever-you-found-this" is the URI in $self, is it not?

@lornajane I think there might be some terminology muddiness here- URIs (meaning OpenAPI Description URIs) and URLs (meaning API URLs) are resolved differently, and have been since OAS v3.1.0.

@karenetheridge no, "wherever-you-found-this" is the retrieval URI per RFC3986 §5.1.3. $self is the URI defined in content per RFC3986 §5.1.1. Since $self is a URI, it may not be a sort of URI that indicates any sort of "where", while the retrieval URI is more-or-less by definition a URL (the "more-or-less" is because there are some URI schemes that do not inherently indicate a location, but have a well-defined process to map to a location that might change over time, e.g. the doi: scheme).

karenetheridge · 2025-03-06T16:29:48Z

src/oas.md

@@ -322,7 +324,7 @@ Relative references in CommonMark hyperlinks are resolved in their rendered cont
 API endpoints are by definition accessed as locations, and are described by this specification as **_URLs_**.

 Unless specified otherwise, all fields that are URLs MAY be relative references as defined by [RFC3986](https://tools.ietf.org/html/rfc3986#section-4.2).
-Unless specified otherwise, relative references are resolved using the URLs defined in the [Server Object](#server-object) as a Base URL. Note that these themselves MAY be relative to the referring document.
+Unless specified otherwise, relative references are resolved using the URLs defined in the [Server Object](#server-object) as a Base URL. Note that these themselves MAY be relative to the referring document (**NOT** the [OpenAPI Object's](#openapi-object) `$self` field).


add "..., which may itself be relative)" to the end of this sentence?

@karenetheridge I'm not sure what you mean, we already say that the Server Object URLs may be relative, and the point of the parenthetical aside is that $self URI is irrelevant here, so adding more information about $self would just be a distraction and make it feel relevant.

Okay. I originally actually overlooked the "$self is not used here" part entirely as it wasn't what I was expecting.

That means that the referring URL of the document (if there is one) holds relevance to more than just $self - other things are resolved against it as well.

I'll take up the rest of this in your new examples.

src/oas.md

karenetheridge · 2025-03-06T16:33:08Z

src/oas.md

@@ -475,7 +478,7 @@ An object representing a Server.

 | Field Name | Type | Description |
 | ---- | :----: | ---- |
-| <a name="server-url"></a>url | `string` | **REQUIRED**. A URL to the target host. This URL supports Server Variables and MAY be relative, to indicate that the host location is relative to the location where the document containing the Server Object is being served. Variable substitutions will be made when a variable is named in `{`braces`}`. |
+| <a name="server-url"></a>url | `string` | **REQUIRED**. A URL to the target host. This URL supports Server Variables and MAY be relative, to indicate that the host location is relative to the location where the document containing the Server Object is being served. Variable substitutions will be made when a variable is named in `{`braces`}`.   Note that the [OpenAPI Object's](#openapi-object) `$self` field is **NOT** used for relative URL reference resolution. |


But we do use the document's URI itself -- so should we mention that explicitly here?

@karenetheridge

to indicate that the host location is relative to the location where the document containing the Server Object is being served.

This indicates that we use the document's URL, but not its $self-defined URI. I'm uncertain if a.) this is the right approach, or b.) it is explained well.

I did this to maintain a philosophical split between API URLs and OAD URIs. For OADs, you want to separate location and identity. The OAD should be the same thing no matter where it is.

But for APIs, particularly multiple deployments of the "same" API (e.g. the management interface present on every physical instance of some network device), the location is what matters. I want to make a call to the API on box foo.example.com, not on bar.example.com, even though the OAD for both boxes is the same.

So it seemed reasonable to tie relative API URLs to the document URL, and not to its identity.

Again, I'm not sure that's the right idea. It's weird. And it gets slippery when you include RFC3986 §5.1.2 (base URI from encapsulating entity), as that can be a location or more of an identity. Meh.

Opinions welcome here. As you can see, I realized there was a problem and picked an answer, but I'm not at all sure it's correct, and I'm not at all attached to it.

I'm not sure if what you're thinking is the same as what I'm thinking. Can we come up with some examples? In so doing, this might help improve the wording here so it's clearer.

@karenetheridge yes, see other comments on how I did write examples, they just got lost on another branch. I will update as soon as the house-moving tasks let up, which will probably be next week sometime.

This indicates that we use the document's URL, but not its $self-defined URI. I'm uncertain if a.) this is the right approach, or b.) it is explained well.

(b) was indeed the case above. Your examples helped, as per my new comments now.

src/schemas/validation/schema.yaml

handrews · 2025-03-19T19:30:28Z

@lornajane @ralfhandl @karenetheridge I ended up reworking this a bit as I added examples.

ralfhandl

+1, after correcting the resolved URLs in the example.

src/oas.md

handrews · 2025-03-21T17:38:29Z

@ralfhandl please don't merge things into my PRs on my behalf. I prefer rebasing and make a point to do so. I seem to have forgotten this time, but just ask and I will do it.

handrews · 2025-03-21T17:40:02Z

@ralfhandl this is also because of how I use git and how I sometimes structure commits for review, and sometimes rework that structure which is extremely hard to do when there are merges.

ralfhandl · 2025-03-21T18:18:39Z

Fair point, will do.

We seem to use git and GitHub somewhat differently. I for example dislike force-pushes to PRs that I already have reviewed. Which is why I usually only review PRs that are set to „ready for review“ unless explicitly asked to review a draft PR.

handrews · 2025-03-23T15:10:53Z

@ralfhandl I never use force-pushes to change existing commits, just to rebase. Any further changes are always new commits.

This adds `$self` as a way for a document to define its own URI for use in reference targets, and as the base URI for relative URI references in the document. This does not impact the resolution of relative API URLs.

Co-authored-by: Ralf Handl <[email protected]>

Co-authored by: Karen Etheridge <[email protected]>

handrews · 2025-03-23T15:17:29Z

@ralfhandl also I'm not sure what "ready to review" has to do with this situation. The PR was ready for review, then I got feedback and added more commits to address it.

handrews · 2025-03-23T15:37:06Z

@ralfhandl I should explain this:

sometimes rework that structure which is extremely hard to do when there are merges

I don't do it often on this project, but sometimes if the set of commits is lengthy and complex I will git rebase-i and clean it up before merging. I've also just been working this way for years, including on the OAS, and suddenly having my git usage patterns changed on me without warning is disconcerting.

karenetheridge · 2025-03-24T20:09:23Z

I am also in favour of squashing and rebasing (just before merging, not during an active review) rather than using merge commits, as a lot of merge commits make a mess of the commit topology.

karenetheridge

I think that's all my comments for this pass.

They will read a bit confusingly though because the order in which they appear in the document is nearly the reverse of that in which I wrote them -- as I had to get down to your examples before I really understood what you're going for, and then I had to go back and drop in some questions higher up where the concepts are first mentioned! (sorry!)

Actually never mind, it looks like github has changed how it does things since I last had a similar complaint -- and the comments are added here in chronological order, not in the order in which they appear in the file! yay github.

karenetheridge · 2025-03-25T22:19:30Z

src/oas.md

+
+In this example, both Schema Objects use `https://example.com/openapi` as their base URI for resolving their relative `$id` values to `https://example.com/schemas/foo` and `https://example.com/schemas/bar`.  The `$ref` under `properties` is resolved against the `$id`-provided base URI `https://example.com/schemas/foo`, producing `https://example.com/schemas/bar`, which is the `$id`-assigned URI of the Bar schema component.
+
+Note that using embedded `$id` keywords prevents using `$ref: "#/components/schemas/Bar"` in the `properties` field's `$ref` because the base URI for such fragments is set by the `$id`.  Therefore, a `$ref: "#/components/schemas/Bar"` would resolve to `"https://example.com/schemas/foo#/components/schemas/Bar"`, which is not useful.


Suggested change

Note that using embedded `$id` keywords prevents using `$ref: "#/components/schemas/Bar"` in the `properties` field's `$ref` because the base URI for such fragments is set by the `$id`. Therefore, a `$ref: "#/components/schemas/Bar"` would resolve to `"https://example.com/schemas/foo#/components/schemas/Bar"`, which is not useful.

Note that using embedded `$id` keywords would prevent using `$ref: "#/components/schemas/Bar"` in the `properties` field's `$ref` because the base URI for such fragments is set by the `$id`. Therefore, a `$ref: "#/components/schemas/Bar"` would resolve to `"https://example.com/schemas/foo#/components/schemas/Bar"`, which is not the presumably-intended location.

karenetheridge · 2025-03-25T22:19:40Z

src/oas.md

+```
+
+1. The base URI used to resolve `$self` is the retrieval URI, resulting in an effective `$self` value of `https://example.com/openapi` (OADs that use HTTP content negotiation to provide both JSON and YAML representations typically do not use a file extension in their self-assigned URI, to avoid needing to change URIs based on the format).
+2. The base URI for the `$ref` comes from `$self` (`https://example.com/openapi`), producing a resolved URI of `https://example.com/shared#/components/pathItems/Foo`


Suggested change

2. The base URI for the `$ref` comes from `$self` (`https://example.com/openapi`), producing a resolved URI of `https://example.com/shared#/components/pathItems/Foo`

2. The base URI for the `$ref` comes from `$self` (`https://example.com/openapi`), producing a resolved URI of `https://example.com/shared#/components/pathItems/Foo`.

karenetheridge · 2025-03-25T22:34:11Z

src/oas.md

@@ -322,7 +324,7 @@ Relative references in CommonMark hyperlinks are resolved in their rendered cont
 API endpoints are by definition accessed as locations, and are described by this specification as **_URLs_**.

 Unless specified otherwise, all fields that are URLs MAY be relative references as defined by [RFC3986](https://tools.ietf.org/html/rfc3986#section-4.2).
-Unless specified otherwise, relative references are resolved using the URLs defined in the [Server Object](#server-object) as a Base URL. Note that these themselves MAY be relative to the referring document.
+Unless specified otherwise, relative references are resolved using the URLs defined in the [Server Object](#server-object) as a Base URL. Note that these themselves MAY be relative to the referring document (**NOT** the [OpenAPI Object's](#openapi-object) `$self` field).


Okay. I originally actually overlooked the "$self is not used here" part entirely as it wasn't what I was expecting.

That means that the referring URL of the document (if there is one) holds relevance to more than just $self - other things are resolved against it as well.

I'll take up the rest of this in your new examples.

karenetheridge · 2025-03-25T22:40:05Z

src/oas.md

+  description: The test API on this device
+```
+
+For API URLs, the `$self` field, which identifies the OpenAPI Document, is ignored, and the retrieval URL is used instead.  This produces a normalized production URL of `https://device1.example.com`, and a normalized test URL of `https://device1.example.com/test`.


Thank you for this paragraph -- this actually provides a bit of clarity regarding how server urls are handled, which I had been meaning to raise in an issue as I was running into conflicts when implementing them as part of request matching.

karenetheridge · 2025-03-25T22:44:43Z

src/oas.md

@@ -475,7 +478,7 @@ An object representing a Server.

 | Field Name | Type | Description |
 | ---- | :----: | ---- |
-| <a name="server-url"></a>url | `string` | **REQUIRED**. A URL to the target host. This URL supports Server Variables and MAY be relative, to indicate that the host location is relative to the location where the document containing the Server Object is being served. Variable substitutions will be made when a variable is named in `{`braces`}`. |
+| <a name="server-url"></a>url | `string` | **REQUIRED**. A URL to the target host. This URL supports Server Variables and MAY be relative, to indicate that the host location is relative to the location where the document containing the Server Object is being served. Variable substitutions will be made when a variable is named in `{`braces`}`.   Note that the [OpenAPI Object's](#openapi-object) `$self` field is **NOT** used for relative URL reference resolution. |


This indicates that we use the document's URL, but not its $self-defined URI. I'm uncertain if a.) this is the right approach, or b.) it is explained well.

(b) was indeed the case above. Your examples helped, as per my new comments now.

karenetheridge · 2025-03-25T22:46:16Z

src/oas.md

@@ -342,6 +416,7 @@ This is the root object of the [OpenAPI Description](#openapi-description).
 | Field Name | Type | Description |
 | ---- | :----: | ---- |
 | <a name="oas-version"></a>openapi | `string` | **REQUIRED**. This string MUST be the [version number](#versions) of the OpenAPI Specification that the OpenAPI Document uses. The `openapi` field SHOULD be used by tooling to interpret the OpenAPI Document. This is _not_ related to the API [`info.version`](#info-version) string. |
+| <a name-"oas-self"></a>$self | `URI-reference` (without a fragment) | Sets the URI of this document, which also serves as its base URI in accordance with [RFC 3986 §5.1.1](https://www.rfc-editor.org/rfc/rfc3986#section-5.1.1); the value MUST NOT be the empty string and MUST NOT contain a fragment (even if the fragment is empty).  Implementations MUST support referencing a document by the resolved URI defined by this field. |


I'm not sure of the exact wording, but can we somehow make it clear that "referencing" here means "via a $ref keyword", not "Locatable through an HTTP request at this location"?

karenetheridge · 2025-03-25T22:48:51Z

src/oas.md

+See [JSON Schema draft 2020-12 Section 8.2](https://www.ietf.org/archive/id/draft-bhutton-json-schema-01.html#section-8.2) for more information about base URIs in Schema Objects.
+
+The most common base URI source in the absence of `$self` or `$id` is the retrieval URI, in accordance with RFC3986 Section 5.1.3.
+Since not all tools support direct retrieval, and because direct retrieval is sometimes prevented by network conditions or security policies, tools SHOULD allow users to provide the intended retrieval URI along with the document.


Do we need to say here that if such a retrieval URI is provided by the user, that the document MUST actually be reachable at that location (i.e. making this a URL not just a URI)?

However, that would mean that $refs within the document are only actually reachable at their resolved URI if and only if the (resolved) $self URI is reachable.

handrews added enhancement re-use: ref/id resolution how $ref, operationId, or anything else is resolved labels Feb 28, 2025

handrews added this to the v3.2.0 milestone Feb 28, 2025

handrews requested review from a team as code owners February 28, 2025 15:07

lornajane reviewed Mar 4, 2025

View reviewed changes

karenetheridge reviewed Mar 6, 2025

View reviewed changes

karenetheridge mentioned this pull request Mar 8, 2025

Disallow fragments in uri-references that point elsewhere in the OpenAPI description #4423

Closed

3 tasks

ralfhandl linked an issue Mar 19, 2025 that may be closed by this pull request

OAS3 Tag/template to define external-components URL #1979

Open

handrews force-pushed the self2 branch from 2da9317 to 3364fc6 Compare March 19, 2025 19:29

handrews mentioned this pull request Mar 20, 2025

Open Community (TDC) Meeting, Thursday 20 March 2025 #4429

Open

ralfhandl requested changes Mar 21, 2025

View reviewed changes

src/oas.md Outdated Show resolved Hide resolved

ralfhandl previously approved these changes Mar 21, 2025

View reviewed changes

ralfhandl dismissed their stale review via 5013d41 March 21, 2025 17:34

ralfhandl previously approved these changes Mar 21, 2025

View reviewed changes

handrews force-pushed the self2 branch from 5013d41 to 97c8a45 Compare March 21, 2025 17:38

handrews and others added 4 commits March 23, 2025 08:12

Add $self for self-identifying documents

579e32f

This adds `$self` as a way for a document to define its own URI for use in reference targets, and as the base URI for relative URI references in the document. This does not impact the resolution of relative API URLs.

Rework and include examples.

df793f4

Fix example

10bec93

Co-authored-by: Ralf Handl <[email protected]>

Make schema check more precise for $self

ff315fa

Co-authored by: Karen Etheridge <[email protected]>

handrews dismissed ralfhandl’s stale review via ff315fa March 23, 2025 15:15

handrews force-pushed the self2 branch from 97c8a45 to ff315fa Compare March 23, 2025 15:15

karenetheridge reviewed Mar 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `$self` for self-identifying documents #4389

Add `$self` for self-identifying documents #4389

handrews commented Feb 28, 2025 •

edited by karenetheridge

Loading

lornajane left a comment

ralfhandl commented Mar 4, 2025

handrews commented Mar 6, 2025 •

edited

Loading

karenetheridge left a comment

karenetheridge Mar 6, 2025

lornajane Mar 17, 2025

karenetheridge Mar 24, 2025

handrews Mar 24, 2025

karenetheridge Mar 6, 2025

handrews Mar 21, 2025

karenetheridge Mar 25, 2025

karenetheridge Mar 6, 2025

handrews Mar 6, 2025

karenetheridge Mar 7, 2025 •

edited

Loading

handrews Mar 7, 2025

karenetheridge Mar 25, 2025

handrews commented Mar 19, 2025

ralfhandl left a comment

handrews commented Mar 21, 2025

handrews commented Mar 21, 2025

ralfhandl commented Mar 21, 2025

handrews commented Mar 23, 2025

handrews commented Mar 23, 2025

handrews commented Mar 23, 2025

karenetheridge commented Mar 24, 2025

karenetheridge left a comment •

edited

Loading

karenetheridge Mar 25, 2025

karenetheridge Mar 25, 2025

karenetheridge Mar 25, 2025

karenetheridge Mar 25, 2025

karenetheridge Mar 25, 2025

karenetheridge Mar 25, 2025

karenetheridge Mar 25, 2025

		In practice, this is usually the retrieval URI of the document, which MAY be determined based on either its current actual location or a user-supplied expected location.
		The document's base URI MUST also be used to resolve a relative `$self` URI reference.


		In this example, both Schema Objects use `https://example.com/openapi` as their base URI for resolving their relative `$id` values to `https://example.com/schemas/foo` and `https://example.com/schemas/bar`. The `$ref` under `properties` is resolved against the `$id`-provided base URI `https://example.com/schemas/foo`, producing `https://example.com/schemas/bar`, which is the `$id`-assigned URI of the Bar schema component.

		Note that using embedded `$id` keywords prevents using `$ref: "#/components/schemas/Bar"` in the `properties` field's `$ref` because the base URI for such fragments is set by the `$id`. Therefore, a `$ref: "#/components/schemas/Bar"` would resolve to `"https://example.com/schemas/foo#/components/schemas/Bar"`, which is not useful.

	2. The base URI for the `$ref` comes from `$self` (`https://example.com/openapi`), producing a resolved URI of `https://example.com/shared#/components/pathItems/Foo`
	2. The base URI for the `$ref` comes from `$self` (`https://example.com/openapi`), producing a resolved URI of `https://example.com/shared#/components/pathItems/Foo`.

Add $self for self-identifying documents #4389

Are you sure you want to change the base?

Add $self for self-identifying documents #4389

Conversation

handrews commented Feb 28, 2025 • edited by karenetheridge Loading

lornajane left a comment

Choose a reason for hiding this comment

ralfhandl commented Mar 4, 2025

handrews commented Mar 6, 2025 • edited Loading

karenetheridge left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenetheridge Mar 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

handrews commented Mar 19, 2025

ralfhandl left a comment

Choose a reason for hiding this comment

handrews commented Mar 21, 2025

handrews commented Mar 21, 2025

ralfhandl commented Mar 21, 2025

handrews commented Mar 23, 2025

handrews commented Mar 23, 2025

handrews commented Mar 23, 2025

karenetheridge commented Mar 24, 2025

karenetheridge left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add `$self` for self-identifying documents #4389

Add `$self` for self-identifying documents #4389

handrews commented Feb 28, 2025 •

edited by karenetheridge

Loading

handrews commented Mar 6, 2025 •

edited

Loading

karenetheridge Mar 7, 2025 •

edited

Loading

karenetheridge left a comment •

edited

Loading