Conversation
|
I have a few high-level questions about how this service is intended to work. First, the server will need to identify which resources are indexed with which types. Does every resource get indexed? What is used as a type value? A link header (e.g., rel=type) or an RDF property (for RDF documents)? What happens if a client submits a JSON-LD document? MUST the server perform a full parse of that input (this has significant security implications). In the input document, a client decides which resources are indexed, but I do not see a mechanism by which a client can signal that certain resources are indexed and other are not. Here, the question is primarily: do clients populate the type index or do servers? Second, where does the Third, the Fourth, the Finally, should this feature be defined in a separate document, as is the case for the Access Request feature? |
The Solid client Type Index spec was trying to simulate type search in the only way they could as clients using containers. We should have a simple type search that has nothing to do with CRUD - that is covered elsewhere in the spec. How an implementation implements the search should not be specified. The discussion with respect to search at the last FTF meeting was to have type search rather than a more comprehensive search. The Type Index using containers as clients implementation doesn't make much sense when a server is involved. I think we should rename this Type Search (and not reference the client based Type Index implementation terminology) |
|
|
||
| <section> | ||
| <h4>Security</h4> | ||
| <p>Servers MUST enforce authorization per access grants. Responses to GET/POST on <code>TypeIndexService</code> or <code>searchEndpoint</code> include only types, defaults, locations, and resource URIs the authenticated client may read (via grants). Unauthorized entries omitted. Grant listing implicitly yields a client-specific type index.</p> |
There was a problem hiding this comment.
I think there should be some required consistency between Access Grants and type indexes. In SAI access grants also serve as type index which besides type and location also provides information about access modes and in the future can include usage policies. Currently there is no default equivalent but I see a need to incubate this future with focus on UX. As for consistency, Besides public read and indexed resources. For all other protected resources which are not public read and not publicly indexed. Is application expected to check both, access grants where it is the assignee and known type indexes? What if those two diverge? I understand that the intention is to have access grants and type indexes as two independent features but they should still reflect some common source of truth.
There was a problem hiding this comment.
It makes no sense to be putting things in containers by type - what if something has more than one type? (Most things do) etc. This is an implementation idea best left to implementations or to clients. To create containers and put things in them, that seems to already be covered by other parts of the spec.. What is not currently possible (solid spec) is search on a storage by type no matter where something is - this is only possible with a server type search. Search is the foundational feature for most apps, so having a simple type search will get a foot in the door to providing what all apps actually need.
There was a problem hiding this comment.
@gibsonf1 I'm not sure if your reply is specific to my comment, type index seems to assume some kind of partitioning by type, which is not a new idea https://www.w3.org/TR/void/#class-property-partitions
My point was mostly abuout consistency with AuthZ. It also isn't a new idea to use access control artifacts for the target resource(s) location discovery
There was a problem hiding this comment.
@gibsonf1 I'm not sure if your reply is specific to my comment, type index seems to assume some kind of partitioning by type, which is not a new idea https://www.w3.org/TR/void/#class-property-partitions
@elf-pavlik My point here is that the "Type Index" is a specific client side implementation of an attempt to do type search - which makes no sense in the LWS server spec. We should have a Type Search spec here instead which has no specifics on how an implementation might do things, as opposed to the case in the Solid Client Type Index implementation spec.
There was a problem hiding this comment.
I think my other comment could be releavant here. Without references to requirements, and through them to use cases, it may not be clear what this proposal intends to satisfy.
There was a problem hiding this comment.
I agree with @elf-pavlik. The field of search is extremely broad, and without clear reference to requirements, it will be challenging to define a proposal that satisfies the entire WG.
My understanding is that the specific requirements are based on a need for basic data discovery (distinct from service or capability discovery). In addition, this needs to be implementable by the widest range of servers in an interoperable way. Furthermore the design needs to draw on prior art so that we don't define something that no one will use.
In no way should the type index mechanism interfere with an implementation offering other query endpoints.
There was a problem hiding this comment.
I agree with @elf-pavlik. The field of search is extremely broad, and without clear reference to requirements, it will be challenging to define a proposal that satisfies the entire WG.
My understanding is that the specific requirements are based on a need for basic data discovery (distinct from service or capability discovery). In addition, this needs to be implementable by the widest range of servers in an interoperable way. Furthermore the design needs to draw on prior art so that we don't define something that no one will use.
In no way should the type index mechanism interfere with an implementation offering other query endpoints.
The agreed to requirement at the F2F was to limit search to a simple type search. This is far less precarious than trying to spec out a client implementation of a type index to try to simulate type search. And is there any in production server based "type index" anywhere? I think not.
| "next": "..." | ||
| } | ||
| </pre> | ||
| <p><code>searchEndpoint</code> REQUIRED.<br><code>default</code> OPTIONAL.<br><code>location</code> array of container URIs (always an array).</p> |
There was a problem hiding this comment.
Which use cases and requirements the default property intends to address?
I understand that it intends to indicate a default container for given storage. Are there any restrictions on how many default containers for given type can be indicated in a storage, I would expect just one.
When user has multiple storages, if application has access to discover indexes for given type across some of those storages, there still can be a friction with identifying default of which storage should be used. I think the design of default needs to be evaluated against use cases and requirements it intends to address.
|
@pchampin I just saw your presentation at Solid World, and the whole issue you are trying to solve with dealing with files to try to get query is solved with server based /search actually https://www.youtube.com/watch?v=0Vrqn1Ie3c0. (On TwinPod, we would just use search to do all you need there) This is specifically why search is a foundational feature for any app that needs it. |
remove "default" field add search endpoint to Storage description
instances can be found with [typeSearchService]
|
The "default" and "location" fields have been removed from the typeIndexService proposal. The filters for the typeIndexService are, therefore, no longer needed and have been removed. The POST operation on typeIndexService no longer made any sense and has been removed. The typeSearchService, because multiple filters can be added, can be used for finding both lws:Containers and/or lws:DataResource with pagination being handled. Other text clarifications made. |
|
How does this proposal address sharing resources via Capability URLs? |
Co-authored-by: Aaron Coburn <aaronc@inrupt.com>
|
|
||
| <section id="type-index-services"> | ||
| <h3>Search and Type Index Services</h3> | ||
| <p>The <code>TypeIndexService</code> provides a server-managed discovery mechanism to query the distinct resource types available within a storage, while the <code>TypeSearchService</code> allows clients to retrieve the specific resource URIs matching those types. |
There was a problem hiding this comment.
This should be named SearchService as it already searches for things other than types.
There was a problem hiding this comment.
Do you mean?
| <p>The <code>TypeIndexService</code> provides a server-managed discovery mechanism to query the distinct resource types available within a storage, while the <code>TypeSearchService</code> allows clients to retrieve the specific resource URIs matching those types. | |
| <p>The <code>TypeIndexService</code> provides a server-managed discovery | |
| mechanism to query the distinct resource types available within a storage, | |
| while the <code>SearchService</code> allows clients to retrieve the specific | |
| resource URIs matching those types. |
| <p> | ||
| To prevent inadvertent or malicious corruption by clients, these services are strictly populated and managed by the server. To mitigate security and performance risks associated with deep-parsing arbitrary resource bodies, servers are not required to parse resource content to discover types. Instead, servers SHOULD derive resource types from HTTP Link headers provided by the client during resource creation or modification, combined with the server's intrinsic knowledge of the resource's state (such as native LWS classes like <code>https://www.w3.org/ns/lws#Container</code>). | ||
| </p><p> | ||
| Servers MAY additionally derive types from the resource representation itself when they are able to parse it. Implementations are encouraged to do so where feasible, as richer type discovery improves the utility of the Type Index and Type Search services. When a server enriches the type index from resource content, the types it surfaces MUST be treated identically to those derived from Link headers for the purposes of indexing, search, and authorization filtering. |
There was a problem hiding this comment.
Do you mean?
| Servers MAY additionally derive types from the resource representation itself when they are able to parse it. Implementations are encouraged to do so where feasible, as richer type discovery improves the utility of the Type Index and Type Search services. When a server enriches the type index from resource content, the types it surfaces MUST be treated identically to those derived from Link headers for the purposes of indexing, search, and authorization filtering. | |
| Servers MAY additionally derive types from the resource representation itself | |
| when they are able to parse it. Implementations are encouraged to do so where | |
| feasible, as richer type discovery improves the utility of the Type Index and | |
| Search services. When a server enriches the type index from resource content, | |
| the types it surfaces MUST be treated identically to those derived from Link | |
| headers for the purposes of indexing, searching, and authorization filtering. |
| </section> | ||
|
|
||
| <section> | ||
| <h4>GET [TypeSearchService]</h4> |
There was a problem hiding this comment.
Do you mean?
| <h4>GET [TypeSearchService]</h4> | |
| <h4>GET [SearchService]</h4> |
| </p> | ||
| <p> | ||
| The <code>type</code> parameter is the mandatory baseline and MUST be supported by every server that implements the | ||
| <code>TypeSearchService</code>. A server MAY additionally index other <em>descriptive</em> link |
There was a problem hiding this comment.
Do you mean?
| <code>TypeSearchService</code>. A server MAY additionally index other <em>descriptive</em> link | |
| <code>SearchService</code>. A server MAY additionally index other <em>descriptive</em> link |
| </p> | ||
| <p class="note">Clients MUST URL-encode query parameter keys and values.</p> | ||
| <p>Returns <code>application/lws+json</code>: A paginated <code>ContainerPage</code> whose items describe the matching LWS resources. Aside from <code>type</code> and any additional relation parameters described above, no other parameters are defined.</p> | ||
| <p class="note">A <code>ContainerPage</code> returned by the <code>TypeSearchService</code> is a <em>synthetic</em> result set, not a representation of a container resource. The <code>ContainerPage</code> media type and pagination model are reused only for client convenience; the response does not identify a container, has no containment relationship to its members, is not retrievable or mutable as a resource, and its membership reflects the query rather than any stored hierarchy. Each item is a resource description carrying at least <code>id</code> and <code>type</code>, mirroring a container member; a server MAY include additional descriptive metadata (such as <code>mediaType</code>, <code>size</code>, or <code>modified</code>) but is not required to. The same applies to the <code>POST</code> form.</p> |
There was a problem hiding this comment.
Do you mean?
| <p class="note">A <code>ContainerPage</code> returned by the <code>TypeSearchService</code> is a <em>synthetic</em> result set, not a representation of a container resource. The <code>ContainerPage</code> media type and pagination model are reused only for client convenience; the response does not identify a container, has no containment relationship to its members, is not retrievable or mutable as a resource, and its membership reflects the query rather than any stored hierarchy. Each item is a resource description carrying at least <code>id</code> and <code>type</code>, mirroring a container member; a server MAY include additional descriptive metadata (such as <code>mediaType</code>, <code>size</code>, or <code>modified</code>) but is not required to. The same applies to the <code>POST</code> form.</p> | |
| <p class="note">A <code>ContainerPage</code> returned by the | |
| <code>SearchService</code> is a <em>synthetic</em> result set, not a | |
| representation of a container resource. The <code>ContainerPage</code> | |
| media type and pagination model are reused only for client convenience; | |
| the response does not identify a container, has no containment relationship | |
| to its members, is not retrievable or mutable as a resource, and its | |
| membership reflects the query rather than any stored hierarchy. Each item | |
| is a resource description carrying at least <code>id</code> and | |
| <code>type</code>, mirroring a container member; a server MAY include | |
| additional descriptive metadata (such as <code>mediaType</code>, | |
| <code>size</code>, or <code>modified</code>) but is not required to. The | |
| same applies to the <code>POST</code> form.</p> |
| </section> | ||
|
|
||
| <section> | ||
| <h4>POST [TypeSearchService]</h4> |
There was a problem hiding this comment.
Do you mean?
| <h4>POST [TypeSearchService]</h4> | |
| <h4>POST [SearchService]</h4> |
| <section> | ||
| <h4>Request Equivalence and Errors</h4> | ||
| <p> | ||
| A server implementing the <code>TypeSearchService</code> MUST support both the <code>GET</code> and |
There was a problem hiding this comment.
Do you mean?
| A server implementing the <code>TypeSearchService</code> MUST support both the <code>GET</code> and | |
| A server implementing the <code>SearchService</code> MUST support both the <code>GET</code> and |
| <section> | ||
| <h4>Security and Authorization</h4> | ||
| <p> | ||
| Servers MUST enforce authorization. Responses to GET/POST on <code>TypeIndexService</code> or <code>TypeSearchService</code> include only types and resource URIs that the authenticated client is explicitly authorized to read. |
There was a problem hiding this comment.
Do you mean?
| Servers MUST enforce authorization. Responses to GET/POST on <code>TypeIndexService</code> or <code>TypeSearchService</code> include only types and resource URIs that the authenticated client is explicitly authorized to read. | |
| Servers MUST enforce authorization. Responses to GET/POST on | |
| <code>TypeIndexService</code> or <code>SearchService</code> include | |
| only types and resource URIs that the authenticated client is | |
| explicitly authorized to read. |
|
|
||
| <pre class="example" title="GET Search for Instances (Using DataResource filter)"> | ||
| Request | ||
| GET /types/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource |
There was a problem hiding this comment.
Do you mean?
| GET /types/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource | |
| GET /search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource |
| Response | ||
| HTTP/1.1 200 OK | ||
| Content-Type: application/lws+json | ||
| Link: <https://example.org/types/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource&page=1>; rel="first" |
There was a problem hiding this comment.
Do you mean?
| Link: <https://example.org/types/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource&page=1>; rel="first" | |
| Link: <https://example.org/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource&page=1>; rel="first" |
| HTTP/1.1 200 OK | ||
| Content-Type: application/lws+json | ||
| Link: <https://example.org/types/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource&page=1>; rel="first" | ||
| Link: <https://example.org/types/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource&page=2>; rel="next" |
There was a problem hiding this comment.
Do you mean?
| Link: <https://example.org/types/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource&page=2>; rel="next" | |
| Link: <https://example.org/search?type=https://schema.org/Person&type=https://www.w3.org/ns/lws%23DataResource&page=2>; rel="next" |
|
|
||
| <pre class="example" title="GET Search for container resources (is-a)"> | ||
| Request | ||
| GET /types/search?type=https://www.w3.org/ns/lws%23Container |
There was a problem hiding this comment.
GET /search?type=https://www.w3.org/ns/lws%23Container
There was a problem hiding this comment.
Do you mean?
| GET /types/search?type=https://www.w3.org/ns/lws%23Container | |
| GET /search?type=https://www.w3.org/ns/lws%23Container |
|
|
||
| <pre class="example" title="POST Search with Multiple Type Filters"> | ||
| Request | ||
| POST /types/search |
There was a problem hiding this comment.
Do you mean?
| POST /types/search | |
| POST /search |
|
|
||
| <pre class="example" title="CNF Filter (OR within a group, AND across groups)"> | ||
| Request (GET form: comma = OR, repeated parameter = AND) | ||
| GET /types/search?type=https://schema.org/Person,http://xmlns.com/foaf/0.1/Person&type=https://www.w3.org/ns/lws%23DataResource |
There was a problem hiding this comment.
Do you mean?
| GET /types/search?type=https://schema.org/Person,http://xmlns.com/foaf/0.1/Person&type=https://www.w3.org/ns/lws%23DataResource | |
| GET /search?type=https://schema.org/Person,http://xmlns.com/foaf/0.1/Person&type=https://www.w3.org/ns/lws%23DataResource |
| GET /types/search?type=https://schema.org/Person,http://xmlns.com/foaf/0.1/Person&type=https://www.w3.org/ns/lws%23DataResource | ||
|
|
||
| Equivalent Request (POST form: nested array = OR group, outer array = AND) | ||
| POST /types/search |
There was a problem hiding this comment.
Do you mean?
| POST /types/search | |
| POST /search |
|
|
||
| <pre class="example" title="Filtering on an indexed relation (describedby)"> | ||
| Request (GET: type AND a 'describedby' relation; relation name used as the parameter) | ||
| GET /types/search?type=https://schema.org/Person&describedby=https://shapes.example/PersonShape |
There was a problem hiding this comment.
Do you mean?
| GET /types/search?type=https://schema.org/Person&describedby=https://shapes.example/PersonShape | |
| GET /search?type=https://schema.org/Person&describedby=https://shapes.example/PersonShape |
| GET /types/search?type=https://schema.org/Person&describedby=https://shapes.example/PersonShape | ||
|
|
||
| Equivalent Request (POST) | ||
| POST /types/search |
There was a problem hiding this comment.
Do you mean?
| POST /types/search | |
| POST /search |
|
|
||
| <section id="type-index-services"> | ||
| <h3>Search and Type Index Services</h3> | ||
| <p>The <code>TypeIndexService</code> provides a server-managed discovery mechanism to query the distinct resource types available within a storage, while the <code>TypeSearchService</code> allows clients to retrieve the specific resource URIs matching those types. |
There was a problem hiding this comment.
...SearchService allows clients to search on resources including matching on indexed types.
There was a problem hiding this comment.
Do you mean?
| <p>The <code>TypeIndexService</code> provides a server-managed discovery mechanism to query the distinct resource types available within a storage, while the <code>TypeSearchService</code> allows clients to retrieve the specific resource URIs matching those types. | |
| <p>The <code>TypeIndexService</code> provides a server-managed discovery mechanism | |
| to query the distinct resource types available within a storage, while the | |
| <code>SearchService</code> allows clients to search on resources including | |
| matching on indexed types. |
| "serviceEndpoint": "https://example.org/types/index" | ||
| }, | ||
| { | ||
| "type": "TypeSearchService", |
There was a problem hiding this comment.
Do you mean this?
| "type": "TypeSearchService", | |
| "type": "SearchService", |
| }, | ||
| { | ||
| "type": "TypeSearchService", | ||
| "serviceEndpoint": "https://example.org/types/search" |
There was a problem hiding this comment.
"serviceEndpoint": "https://example.org/search"
There was a problem hiding this comment.
Do you mean this?
| "serviceEndpoint": "https://example.org/types/search" | |
| "serviceEndpoint": "https://example.org/search" |
|
@gibsonf1 specific url for the serviceEndpoint in Storage Description can and most likely will be different for every server, that's why it needs to be discovered. https://pr-preview.s3.amazonaws.com/w3c/lws-protocol/pull/115.html Once WG agrees on using different naming than TypeIndex / TypeSearch snippets can be updated accordingly |
acoburn
left a comment
There was a problem hiding this comment.
June 8, 2026, WG meeting approved this PR for merging

The type index is the simplest, basic "query" mechanism.
It is a registry that advertises which types exist in storage and exactly which containers hold resources of each type. Clients use it to discover types and retrieve matching resource URIs, - all server-enforced for authorization.
Advanced queries are out of scope.
additional reference
Preview | Diff