v3.2: `$self` field (Alternative Approach) #4556

handrews · 2025-04-29T04:17:31Z

This is an alternative to PR #4389. ~~The only functional difference is that $self is required to be absolute (a change that @karenetheridge and @mikekistler are debating).~~

The main difference is that I drastically reduced the normative specification text and instead gave examples of all of the different ways to establish a base URI. [EDIT: and put the examples in an appendix] This is probably more understandable than trying to explain the options in prose.

@notEthan I also dumped a lot of the use cases as I think I figured out a more concise way to get the point across.

This adds $self as a way for a document to define its own URI for use in reference targets, and as the base URI for relative URI references in the document.

This does not impact the resolution of relative API URLs.

schema changes are included in this pull request
schema changes are needed for this pull request but not done yet
no schema changes are needed for this pull request

handrews · 2025-04-30T03:36:31Z

After staring at the giant examples section a lot, I decided it works better as an appendix. (force push is just because I accidentally picked up a stray file in the commit and had to edit it to pull that out).

lornajane · 2025-05-01T15:37:39Z

This is the most readable of the lot, and I think I "get" the whole thing a lot more now. Thank you for doing this work and all the iterations!

Minor points/suggestions from me (I'm not ready to approve but I've only got small questions):

in the proposal we called it self but it looks like now it is $self. We have other URI fields with plain-text names, why does this one need the $? Since it's being added at the top level, we can be confident of not having name clashes. (why isn't it in info with the other document information like license? Am I asking too many questions?)
I'd like to propose we move the appendix to the learn site and link to it. Is the learn site really good/mature/stable enough to link to from a spec file? I think it is, and the information would be better there (but needs to be very well signposted since we haven't done this for other content).

lornajane · 2025-05-01T16:33:46Z

Good discussion in the meeting:

content does need to go to the learn site, but we're not going to block this change for it
$self makes sense because it's like $ref in being "special" and related to how references work

karenetheridge · 2025-05-01T17:00:08Z

I am in agreement with the shuffling of the examples to the learn site rather than in the main spec, but not with the change from relative to absolute uri.

mikeschinkel · 2025-05-02T00:49:28Z

@handrews — A big +1 from me for including examples in the spec. I would prefer them to be in-line so a reader does not have to jump around to follow them, but if those examples need to be in an appendix that is better than not having any or having a link to someplace that might eventually 404.

My biggest pet peeve about specs in general is that they do not include examples which make most specs all-but-inaccessible to those like me who need examples to understand the prose.

P.S. Moving to the learn site only works IMO if there is a guarantee that the learn site will never change those examples nor take them down which, I believe, is impossible to guarantee. FTW.

ralfhandl · 2025-05-02T08:24:19Z

Relative vs. absolute $self doesn't seem to make much of a difference.

A relative $ref is now relative to the retrieval URL. Making it relative to a relative $self which is itself relative to the retrieval URL only adds the convenience of making the $ref value shorter. It does not add a feature.
A relative $ref that is relative to an absolute $self also only adds the convenience of making the $ref value shorter (and relative instead of absolute). It does not add a feature.

Seems the only feature added by $self is an in-document identifier for an OpenAPI description, and that feature seems to benefit from an absolute URI value.

What am I missing?

mikekistler

I think there may be some errors in the examples of URI resolution in Appendix G. I pointed out two and I think the rest should probably be checked for accuracy before we merge this.

src/oas.md

mikekistler · 2025-05-02T21:56:53Z

src/oas.md

+For the relative `$ref` in the first document, it is resolved against `$self` to produce `https://example.com/shared/foo#/components/requestBodies/Foo`.
+The portion of that URI before the '#' matches the `$self` of the second document, so the reference target is resolved to `#/components/requestBodies/Foo` in that second document.
+
+In that document, the `$ref` in the Request Body Object is resolved using that document's `$self` as the base URI, producing `https://example.com/schemas/foo`.


This too does not jive with the resolver that Copilot built. It says that the target URI is actually "https://example.com/api/schemas/foo".

@mikekistler should be fixed. Really, "these look wrong" would have been sufficient here. The errors were just because I reworked the various paths at some point to make the examles cover more cases, and some things didn't get updated post-rework. Nothing obscure was going on.

Although it sounds like you produced something useful beyond this which is cool!

karenetheridge · 2025-05-06T17:55:54Z

Another reason to allow $self to be relative is that it is consistent with JSON Schema, where $id can be relative, so it would be easier to reason through how $ref resolution works in a document and apply RFC3986 consistently if both keywords worked the same way.

I see no gain from forcing $self to be absolute; it just restricts what users and implementations can do with the keyword. We could just as easily say that $self is not needed at all, because the user can always supply the base URI via the retrieval URI. Our goal should be to open up usecases, not restrict them.

ralfhandl · 2025-05-08T06:26:30Z

$self is not needed at all, because the user can always supply the base URI via the retrieval URI

The main feature $self adds is an in-document identifier allowing to determine whether documents retrieved from different URIs or locally stored without their retrieval URI are actually the same and not just textually identical by chance.

Using $self with a relative URI means opting out its main feature and can be considered an abuse of $self. This abuse would be prevented by requiring an absolute URI value.

ralfhandl · 2025-05-08T06:53:23Z

src/oas.md

@@ -354,6 +384,8 @@ This is the root object of the [OpenAPI Description](#openapi-description).

 This object MAY be extended with [Specification Extensions](#specification-extensions).

+Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI such as the retrieval URI, however this behavior is not interoperable and relying on it is NOT RECOMMENDED.


I don't understand this sentence.

If I want my OpenAPI Documents to be interoperable, what SHOULD I do when referencing (other) OpenAPI Documents that contain $self?

@ralfhandl As stated in RFC2119, "SHOULD" and "RECOMMENDED" are equivalent, as are their negations. So anything that is NOT RECOMMENDED is something that you SHOULD NOT do, and therefore you SHOULD do the other thing (in this case, use the $self URI as your reference target).

I can add a sentence making that explicit if you think it is needed.

This came from a discussion with @karenetheridge about other ambiguities with $id. Let's say that you have enabled auto-retrieval of references, treating them as URLs (because you have some reason to believe that will work, like the OAD author said so, out-of-band). If you successfully retrieve a referenced document from X because one $ref used X, and it has a $self URI of Y... you've already retrieved it from X. Can you keep referencing it as X or do you have to reference it as Y, even though you know X works because you just got it there?

This is a weird case. With JSON Schema, we made a mistake and didn't talk about it, as @karenetheridge pointed out. But intuitively, most people seem to think that if you can use X successfully, and particularly if you already used X as the retrieval URL (actually retrieved, not just supplied as the "yeah if you retrieved this you woudl have gotten it from X" simulated retrieval URI) once and it worked, then X should keep working.

You can come up with a similar situation where the "other" URI is from an encapsulated entity (multipart/related example) or even an application-supplied default. These get harder and harder to contrive, but it's possible.

I don't want to require (MUST) that implementations support these messy situations. Implementations really SHOULD NOT perform unrestricted retrieval automatically as it's a security nightmare. To say nothing of the weird contortions needed to justify the other possible sources.

But if you are supporting retrieval, you can easily get into this situation where someone uses a retrieval URL that does not match $self, and it works, and you have to decide whether to keep letting it work or not.

This sentence allows implementations to keep letting that work if they want to, without requiring it. Given the history of $ref, particularly in 2.0 (and arguably 3.0) where it is strictly a URL, many tools almost certainly behave this way already. This sentence gives them the flexibility to leave that in place (and keep compatibility for their existing 3.1 customers) while adding the new 3.2 functionality.

Does that help? Again, I can tack on "Therefore, for interoperability, OAD authors SHOULD (or even MUST? Techncially it's not interoperable otherwise?) ensure their reference target URIs align with $self." or something to that effect if you think it is needed.

Thanks for the explanation, that is what I guessed the text wants to express. Please add the explicit positive instruction "OAD authors SHOULD...".

@ralfhandl already done on line 390

karenetheridge · 2025-05-08T16:11:21Z

Using $self with a relative URI means opting out its main feature and can be considered an abuse of $self

Is there an example (for the learn site, appendix or wherever) that you can construct that would demonstrate this issue, as a way of cautioning against it?

lornajane · 2025-05-08T16:11:45Z

After many discussions about relative vs absolute, we agreed in TDC this week that supporting both would make a more open feature and since the retrieval URL is already used in other places, the additional burden for tools maintainers is not huge.

handrews · 2025-05-08T18:53:59Z

I have updated this with the outcome of today's TDC call. I have also:

@ralfhandl further clarified the consequences of using alternative URIs
Added an example (a random UUID URN) for the application-specific default option
Added an example of relative $self and $id

The new commits were added in one push, and then I force-pushed the rebased commits unchanged to pick up the latest syntax highlighting that supports this change building without warnings.

handrews · 2025-05-08T18:55:32Z

hmm... I thought I was only adding two new commits, looks like maybe I had not pushed some other local fixes? My apologies for any confusion, just be aware that I have never edited commits once pushed, so if a commit looks familiar it's the same as it was.

This adds `$self` as a way for a document to define its own URI for use in reference targets, and as the base URI for relative URI references in the document. This does not impact the resolution of relative API URLs.

Including fixing a bug in one of the URIs.

Never change your directory structure halfway through writing examples...

Fix the $self resolution examples to consistently use /api/, which is used in all of the relevant base URIs and is not impacted by any of the relative paths.

This also further clarifies the need to use `$self` in reference targets for interoperability even if other URIs might work in some cases.

handrews · 2025-05-11T15:35:19Z

One more new bugfix commit plus an unmodified rebase force-push.

ralfhandl

+1 to the content, suggestions for its representation

src/oas.md

ralfhandl · 2025-05-12T07:02:53Z

src/oas.md

Could we consistently use "retrieval URL"?

Currently we have a mix of "retrieval URI" and "retrieval URL", and the L-variant is more accurate.

When you actually retrieved something, it's the retrieval URL. When you supply it as a simulation of having retrieved it, it's more properly a retrieval URI, which is how RFC3986 states it.

So if we only use one, it should be retrieval URI, not retrieval URL.

ralfhandl · 2025-05-12T07:04:43Z

src/oas.md

@@ -388,6 +420,9 @@ This is the root object of the [OpenAPI Description](#openapi-description).

 This object MAY be extended with [Specification Extensions](#specification-extensions).

+Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI such as the retrieval URI, however this behavior is not interoperable and relying on it is NOT RECOMMENDED.


Requirement first, recommendation second. The next suggestion is part of this one.

Suggested change

Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI such as the retrieval URI, however this behavior is not interoperable and relying on it is NOT RECOMMENDED.

Interoperable OAD documents MUST reference an OpenAPI Document that contain `$self` by its `$self` URI. Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI (for example the retrieval URL), however this behavior is not interoperable and relying on it is NOT RECOMMENDED.

I see your point but this reworking doesn't quite feel right, and also as noted elsewhere, we should standardize on retrieval URI rather than URL. I will rework it.

ralfhandl · 2025-05-12T07:05:23Z

src/oas.md

@@ -388,6 +420,9 @@ This is the root object of the [OpenAPI Description](#openapi-description).

 This object MAY be extended with [Specification Extensions](#specification-extensions).

+Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI such as the retrieval URI, however this behavior is not interoperable and relying on it is NOT RECOMMENDED.
+OAD authors MUST write references using the target document's `$self` URI in order to have interoperable behavior.


Suggested change

OAD authors MUST write references using the target document's `$self` URI in order to have interoperable behavior.

Couldn't include this line in the preceding suggestion 😞

src/oas.md

ralfhandl · 2025-05-12T07:27:22Z

src/oas.md

+Note that referring to a schema with a JSON Pointer that crosses a Schema Object with a `$id` [is not interoperable](https://www.ietf.org/archive/id/draft-bhutton-json-schema-01.html#name-json-pointer-fragments-and-).
+The JSON Schema specification does not address the case of using a pointer _to_ a Schema Object containing an `$id` without crossing into that Schema Object.
+Therefore it is RECOMMENDED that OAD authors use `$id` values to reference such schemas rather than JSON Pointers.


Try to formulate this as a positive requirement:

Suggested change

Note that referring to a schema with a JSON Pointer that crosses a Schema Object with a `$id` [is not interoperable](https://www.ietf.org/archive/id/draft-bhutton-json-schema-01.html#name-json-pointer-fragments-and-).

The JSON Schema specification does not address the case of using a pointer _to_ a Schema Object containing an `$id` without crossing into that Schema Object.

Therefore it is RECOMMENDED that OAD authors use `$id` values to reference such schemas rather than JSON Pointers.

Note that a Schema Object with a `$id` SHOULD be referenced by its `$id` value because the JSON Schema specification does not address the case of using a [JSON Pointer to a Schema Object containing an `$id`](https://www.ietf.org/archive/id/draft-bhutton-json-schema-01.html#name-json-pointer-fragments-and-).

I don't think your restatement quite captures everything, but I will look at it and see what I can do.

ralfhandl · 2025-05-12T07:28:59Z

src/oas.md

+Note also that it is impossible for the reference at `#/components/schemas/Foo/properties/bar/$ref` to reference the schema at `#/components/schemas/Bar` using a JSON Pointer, as the JSON Pointer would be resolved relative to `https://example.com/api/schemas/foo`, not to the OpenAPI Document's base URI from `$self`.
+


Is this something that a lot of people try to do despite it being impossible? Maybe remove this paragraph.

Suggested change

Note also that it is impossible for the reference at `#/components/schemas/Foo/properties/bar/$ref` to reference the schema at `#/components/schemas/Bar` using a JSON Pointer, as the JSON Pointer would be resolved relative to `https://example.com/api/schemas/foo`, not to the OpenAPI Document's base URI from `$self`.

Yes, many people don't understand that the $id creates what's essentially a new scope, so I would prefer not to remove this.

ralfhandl · 2025-05-12T07:39:35Z

src/oas.md

+        bar:
+          $ref: schemas/bar
+--boundary-example
+Content-Type: application/schema+json; schema=https://spec.openapis.org/oas/3.2/schema-base/YYYY-MM-DD


The schema=... media type parameter is surprising and doesn't seem to be defined by JSON Schema - can we please remove it?

@ralfhandl that parameter has always been part of JSON Schema. I believe the situation in that draft is the result of an incomplete removal of the media type registration, which moved to draft-ietf-httpapi-rest-api-mediatypes. However, after draft-ietf-httpapi-rest-api-mediatypes-04, JSON Schema was split out, and due to the current JSON Schema team's abandonment of the IETF, has not be placed in a new draft.

But the schema media type parameter has been supported for over a decade and is correct, and this is the correct usage.

src/oas.md

Co-authored-by: Ralf Handl <[email protected]>

handrews · 2025-05-12T20:53:30Z

@ralfhandl I have tried to address everything that I did not accept as a commit.

Everything is now "retrieval URI"
I reworked the requirement-then-caveat ordering
I reworked the crossing-$id text and dropped the mention of the unclear "point too but not across $id" case; I think @karenetheridge had wanted that included? But it's tricky to word without just being more confusing, and idk how much we should try to compensate for JSON Schema's shortcomings here
I added "only" to the weird JSON Pointer case that you asked if people actually do, because I agree it read strangely, and I think the new wording is much more clear

Regarding the schema media type parameter, while that usage is the most correct usage, it is a bit confusing because of the need to put YYYY-MM-DD in the URI, so I might be willing to drop it on those grounds. Otherwise, we'll need to update that to the correct date so people don't copy-paste the placeholder.

...because the YYYY-MM-DD in the URI makes it more trouble than it's worth.

handrews · 2025-05-12T20:55:55Z

@ralfhandl went ahead and removed the parameter because of the YYYY-MM-DD problem. Which makes me sad because I would rather promote the long-established correct use of the media type.

handrews added the re-use: ref/id resolution how $ref, operationId, or anything else is resolved label Apr 29, 2025

handrews added this to the v3.2.0 milestone Apr 29, 2025

handrews requested review from a team as code owners April 29, 2025 04:17

This was referenced Apr 29, 2025

Add $self for self-identifying documents #4389

Closed

Open Community (TDC) Meeting, Thursday 01 May 2025 #4549

Open

handrews force-pushed the self3 branch from af1b721 to 47c3194 Compare April 30, 2025 03:38

mikekistler reviewed May 2, 2025

View reviewed changes

handrews mentioned this pull request May 4, 2025

Open Community (TDC) Meeting, Thursday 08 May 2025 #4561

Open

ralfhandl requested a review from mikekistler May 8, 2025 06:27

ralfhandl reviewed May 8, 2025

View reviewed changes

handrews force-pushed the self3 branch from 0d929e0 to fd03f5d Compare May 8, 2025 18:51

handrews added 6 commits May 11, 2025 08:34

Add $self for self-identifying documents

7bd8a15

This adds `$self` as a way for a document to define its own URI for use in reference targets, and as the base URI for relative URI references in the document. This does not impact the resolution of relative API URLs.

Fix missing multipart boundary

71ec162

Move base URI examples to appendix

db9f643

Improved examples

2832652

Including fixing a bug in one of the URIs.

Fix more example bugs

4435fd3

Never change your directory structure halfway through writing examples...

All example URI paths start with /api/...

182ca17

Fix the $self resolution examples to consistently use /api/, which is used in all of the relevant base URIs and is not impacted by any of the relative paths.

handrews added 2 commits May 11, 2025 08:34

Allow relative $self, include examples

7be51a4

This also further clarifies the need to use `$self` in reference targets for interoperability even if other URIs might work in some cases.

Fix another base URI example bug

4068c09

handrews force-pushed the self3 branch from bc0199d to 4068c09 Compare May 11, 2025 15:34

ralfhandl requested changes May 12, 2025

View reviewed changes

ralfhandl requested a review from a team May 12, 2025 07:48

handrews and others added 3 commits May 12, 2025 13:28

Apply suggestions from code review

966c0c6

Co-authored-by: Ralf Handl <[email protected]>

Better wording

eace593

Co-authored-by: Ralf Handl <[email protected]>

Review feedback

8709e12

Remove link to metaschema...

fc66d13

...because the YYYY-MM-DD in the URI makes it more trouble than it's worth.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v3.2: `$self` field (Alternative Approach) #4556

v3.2: `$self` field (Alternative Approach) #4556

handrews commented Apr 29, 2025 •

edited

Loading

handrews commented Apr 30, 2025 •

edited

Loading

lornajane commented May 1, 2025

lornajane commented May 1, 2025

karenetheridge commented May 1, 2025

mikeschinkel commented May 2, 2025

ralfhandl commented May 2, 2025

mikekistler left a comment

mikekistler May 2, 2025

handrews May 4, 2025

karenetheridge commented May 6, 2025

ralfhandl commented May 8, 2025 •

edited

Loading

ralfhandl May 8, 2025

handrews May 8, 2025

ralfhandl May 11, 2025

handrews May 11, 2025 •

edited

Loading

karenetheridge commented May 8, 2025

lornajane commented May 8, 2025

handrews commented May 8, 2025

handrews commented May 8, 2025

handrews commented May 11, 2025

ralfhandl left a comment

ralfhandl May 12, 2025

handrews May 12, 2025

ralfhandl May 12, 2025

handrews May 12, 2025

ralfhandl May 12, 2025

ralfhandl May 12, 2025

handrews May 12, 2025

ralfhandl May 12, 2025

handrews May 12, 2025

ralfhandl May 12, 2025

handrews May 12, 2025 •

edited

Loading

handrews commented May 12, 2025 •

edited

Loading

handrews commented May 12, 2025

		@@ -354,6 +384,8 @@ This is the root object of the [OpenAPI Description](#openapi-description).

		This object MAY be extended with [Specification Extensions](#specification-extensions).

		Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI such as the retrieval URI, however this behavior is not interoperable and relying on it is NOT RECOMMENDED.

		@@ -388,6 +420,9 @@ This is the root object of the [OpenAPI Description](#openapi-description).

		This object MAY be extended with [Specification Extensions](#specification-extensions).

		Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI such as the retrieval URI, however this behavior is not interoperable and relying on it is NOT RECOMMENDED.

	Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI such as the retrieval URI, however this behavior is not interoperable and relying on it is NOT RECOMMENDED.
	Interoperable OAD documents MUST reference an OpenAPI Document that contain `$self` by its `$self` URI. Implementations MAY choose to support referencing OpenAPI Documents that contain `$self` by another URI (for example the retrieval URL), however this behavior is not interoperable and relying on it is NOT RECOMMENDED.

		Note also that it is impossible for the reference at `#/components/schemas/Foo/properties/bar/$ref` to reference the schema at `#/components/schemas/Bar` using a JSON Pointer, as the JSON Pointer would be resolved relative to `https://example.com/api/schemas/foo`, not to the OpenAPI Document's base URI from `$self`.

v3.2: $self field (Alternative Approach) #4556

Are you sure you want to change the base?

v3.2: $self field (Alternative Approach) #4556

Conversation

handrews commented Apr 29, 2025 • edited Loading

handrews commented Apr 30, 2025 • edited Loading

lornajane commented May 1, 2025

lornajane commented May 1, 2025

karenetheridge commented May 1, 2025

mikeschinkel commented May 2, 2025

ralfhandl commented May 2, 2025

mikekistler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenetheridge commented May 6, 2025

ralfhandl commented May 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

handrews May 11, 2025 • edited Loading

Choose a reason for hiding this comment

karenetheridge commented May 8, 2025

lornajane commented May 8, 2025

handrews commented May 8, 2025

handrews commented May 8, 2025

handrews commented May 11, 2025

ralfhandl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

handrews May 12, 2025 • edited Loading

Choose a reason for hiding this comment

handrews commented May 12, 2025 • edited Loading

handrews commented May 12, 2025

v3.2: `$self` field (Alternative Approach) #4556

v3.2: `$self` field (Alternative Approach) #4556

handrews commented Apr 29, 2025 •

edited

Loading

handrews commented Apr 30, 2025 •

edited

Loading

ralfhandl commented May 8, 2025 •

edited

Loading

handrews May 11, 2025 •

edited

Loading

handrews May 12, 2025 •

edited

Loading

handrews commented May 12, 2025 •

edited

Loading