Have separate auxiliary resource for containment index

[damooo]

Currently containment index of a container (contains statements + basic metadata of contained resources) is specified to be part of container's content. Solid experience shows that this will be painful and leads to compatibility issues, implementation difficulties and modelling difficulties. Following are issues.

1. Containers cannot have arbitrary primary representations, and are limited to lws+json, unlike other resources. Most applications and modelling paradigms expects uniform freedom with representations of resources in the holonymy. For example many wishes to have a arbitrary html hypermedia representation for container (Like book's cover page). Or just free rdf representation like other self describing rdf resources enabling HATEOS. In Solid's world, lack of consensus on this led to ad-hoc implementation assumptions and implementation divergence. See also: [UC] Authoring Arbitrary Hypermedia lws-ucs#33 , Specify/advise semantics of default Container resources (index.html/.ttl) solid/specification#69 , Specify container description solid/specification#227 , Alternative solutions to container HATEOAS solid/specification#525 .

2. Having server managed data and user defined data in same resource leads to implementation difficulties and non-determinism regarding which is which.

3. Due to containment metadata recording modified time of each child, the modified attribute and Last-Modified header change will be propagated upwords till storage root for each modification. This is immensely problematic. Solid had to specify hackish exception.

4. Container creation now requires empty representation payload and other quirks diverging from naturality and consistency with other resources, as specified in Concerns about container creation #77 by @pchampin . If containment index metadata is refactored into representation of an auxiliary resource, then container can have arbitrary representations at the creation time with a qualifier.

Due to such, it is requested to seriously consider having separate resource for containment metadata (Just like linkset resource for other relations), and link it to the container through a link in the linkset. It will make many things natural and less painful. As containment pagination is already mandated, clients have no special problem to follow a link to glean membership.

[pchampin]

@damoo rephrasing to be sure I understand your proposal:

you suggest that a Container would be

• a resource with an arbitrary media-type and client-controlled content (like a Data Resource), and
• with a specific server-managed auxiliary resource holding membership information (which is what makes it a Container rather than a Data Resource)

This is an interesting notion...

It raises some further questions:

• would it be possible to get the membership information directly via content-negotiation?
• would it be possible to turn a Data Resource into a Container after it was created?
• would it be possible to turn a (probably empty) Container into a Data Resource?

[damooo]

Yes. That's what I tried to express.

with a specific server-managed auxiliary resource holding membership information ( which is what makes it a Container rather than a Data Resource )

Probably we can also have a link with type relation with link target being lws:Container or such to qualify containers. This same link can also be used to qualify containers at creation time, as in Solid.

would it be possible to get the membership information directly via content-negotiation?

We just do HEAD on container, and get Link header linking it to index aux resource and retrieve the index resource. One extra HEAD is not much issue, as it will also give links to other linked resources. Essentially same control semantics as linkset resource, but for containment links alone.

would it be possible to turn a Data Resource into a Container after it was created?
would it be possible to turn a (probably empty) Container into a Data Resource?

These can be optional. As many servers may also want to follow uri restrictions (not uri semantics) to not break web affordances, they can't do such conversions. But it could be very useful when one want to hard paginate a big resource into a compound resource backed by a container.

Client controlled hypermedia is the fundamental, that will enable first class HATEOS sans hackish modelling and index.[ttl/html] files etc.

[damooo]

After analysing it further, I feel it could allow another important requirement. That of gradual adoption of LWS by existing websites without breaking their existing content and it's organization paradigm.

Many websites have resources with implicit semantics of containers (like /blogs/ etc.), and members. Those containers normally come as custom html hypermedia. If one wish to use LWS standard for such, they have to either create separate artificial resources for LWS containers, or make existing container-like resources as LWS containers. In such cases, this proposal will allow them to have existing content as it is, with just one more header to point containment index for container resources.

Thus it will make LWS as non breaking evolution of existing web. And uniform handling of all resource representations makes developers' work bit easy.

[TallTed]

Future searchers will thank us for changing the issue title from seperate to separate.

[damooo]

would it be possible to get the membership information directly via content-negotiation?

@pchampin Could you check alternate proposal at #95 that allows this, and aligns with current direction of the spec?

[gibsonf1]

I think its VERY important to separate talking about container representation(s) and the actual container resource as managed by a given implementation. In the TwinPod case, for example, there is only one resource ever at a given URI, but with multiple representations. So the idea of putting in a spec that you have to not only have the implementation store a resource in a certain way, but then have additional resources about the resource, makes no sense to me. In other words, the whole idea of "auxiliary resource" makes no sense to me from a spec perspective. If there is some subject URI in a request, the representation can manage things, however an implementation handles the backend, for example needing to have multiple files for it etc)

[acoburn]

@damooo Thank you for opening this issue. Some feedback on the points made above

1. Containers cannot have arbitrary primary representations, and are limited to lws+json

This is not my understanding. Containers MUST have an application/lws+json representation, but they are not limited to that. An implementation may always supply alternatives, such as text/turtle, but such content negotiation that changes the format of the HTTP payload is not required by LWS. It is, in fact, a goal of LWS to specify a particular format for container representations in order to help achieve greater interoperability. Supporting arbitrary formats and contents leads to less interoperability.

2. Having server managed data and user defined data in same resource leads to implementation difficulties and non-determinism regarding which is which.

I very much agree. At Inrupt, we have encountered many issues with the mixing of client- and server-managed data in a single resource body, as is allowed by the current Solid specification. I have also encountered similar issues with the Fedora specification in this regard. A goal of LWS is to have a mechanism to clearly separate the server-managed data from the client-managed data in any LWS resource.

3. Due to containment metadata recording modified time of each child, the modified attribute and Last-Modified header change will be propagated upwards.

I also agree here (in a strict sense). It is, however, worth noting that including the modified field is optional and if you are concerned about this, you could simply not include that field. Without the modified field, the issue disappears, but doing so also removes data that might be useful to clients. An alternative would be to allow clients to opt-in to retrieve this sort of extended metadata via Prefer: return=representation; ... headers combined with a judicious use of weak ETags.

4. Container creation now requires empty representation payload

This has been clarified. Creating a container involves sending a Link: <https://www.w3.org/ns/lws#Container>; rel="type" header along with no Content-Type header and no payload. That avoids the issue of, e.g. sending an empty JSON document, which is technically invalid per that media type.

[damooo]

Thanks a lot @acoburn for detailed reply.

This is not my understanding. Containers MUST have an application/lws+json representation, but they are not limited to that. An implementation may always supply alternatives, such as text/turtle, but such content negotiation that changes the format of the HTTP payload is not required by LWS.

Yes. This is the crux. Though an implementation can always support alternative representations, all implementations need not. Thus a general "client" cannot assume that it can associate some arbitrary hypermedia with a container across target server implementations. Effectively making it no go for general clients, and they have to resort to hackish models to enable seamless HATEOS.

Thus the problem arises because, spec mandates some server-managed information in the resource representation itself. I believe it would be wise to dedicate other auxiliary resource for such server-managed data, and leave the container resources open for arbitrary client chosen representations, just like other resources. All primary resources must be open for client chosen representations in a way that is interoperable across servers (Subject to any further server specific constraints, but not due to implementation difficulty due to spec requirement of bearing server managed data). This also makes interaction model uniform across resources. And aligns much with web. Allows for interoperable HATEOS. Solves above "modified" problem in clean way.

[acoburn]

@damooo I am not sure I follow your argument.

All primary resources must be open for client chosen representations in a way that is interoperable across servers

Are you arguing that if one client stores a JPEG image in a storage, another client MUST be able to perform content negotiation such that it can retrieve a PNG? The same question can be asked about different JSON-LD framing profiles. You express this as if content negotiation support is an axiom for all resources stored in LWS, and I do not believe that is the case.

By requiring arbitrary content negotiation of container representations, not only does that make server implementations more complicated but it results in less interoperability across clients. Note today how every Solid server has a very different way of representing the contents of a container, especially if that contain is requested as JSON-LD. Content negotiation is a feature that a server can add, but requiring a particular media type for containers will provide a baseline of interoperability across clients. That type of interoperability is the problem we are trying to solve with the current container definition.

Also, as I understand this issue, you are also arguing that, unlike how this is done with Solid and Fedora, you would like to see the container membership listing presented in an auxiliary resource while allowing arbitrary client metadata to be in the "primary" resource for the container. There are several issues with this:

1. Every container will have a listing of member resources. Even if that listing is empty, the payload itself will not be empty. Not every container will have attached metadata (e.g. in a describedby resource). This means that, for every container that does not need auxiliary description, there will be a spec-mandated but empty resource that then points to the actual container listing
2. This reversal is exactly opposite of how this works for DataResource objects. That lack of consistency makes the LWS protocol API less understandable
3. One of the central purposes of containers is to present a listing of its members. By reversing the positioning of Metadata and Container, a client will need to perform two separate HTTP requests to achieve this core use case

As mentioned above, this is also a major departure for how this is done in the specifications that form our input documents. I am not inherently opposed to making such significant changes, but there needs to be a high bar for doing so, and it needs to be solving a specific problem that has been adopted by the WG. In addition to the issues enumerated above, and because I do not see this change as solving a significant problem, I would be very wary of pursuing such a change.

[damooo]

@acoburn
That assumes that the arbitrary representation one will want is always containing "metadata" about container. Not at all. The argument is to allow arbitrary representations. Say an xml resource as a container, or a html describing some other world resource, or an rdf source describing arbitrary domain resources. Not just some self describing metadata resource. Just like other resources.

For example, I am modelling a TEI xml editing app. TEI files are so big for many works, that it is not possible at all to open one single xml resource and provide editing experience. So I am creating a nested resource hierarchy representing entire xml state. Each resource is an xml fragment package wrapping well-ballenced xml. Each container resource is also such fragment, but the fragment body root node will have child nodes included into it through <xi:xinclude> tags referring elements in each child fragment resource. Thus the entire hierarchy adheres to standard xml technologies, and uses nested containers from lws.

Just like other resources have some metadata resources, these container resources too will have. In above case, for example describing the xml fragment. But additionally the containment index metadata resource too.

[pchampin]

@damooo your example about TEI xml files is a bit confusing, because the user-managed representation is somehow coupled with the server-managed representation. Namely, whenever I post a new item in the TEI container, the server will update the membership metadata, but it would be the responsibility of the client to add the corresponding <xi:include> tag to the newly added resource to maintain consistency, right?
So in that scenario, it seems that what you need is rather a server able to content-negociate the membership metadata in TEI XML, rather than allowing user-managed content for the container...

To support the current proposal, a less confusing use-case, IMO, is the blog example you used earlier (I should say "blog API" -- I'm assuming that everything is represented as structured data, which will be rendered to HTML by a dedicated app). With your proposal, a blog could be represented represented by a LWS container, each post in the blog is represented by a resource inside that container.

• The user-managed content of the blog is general metadata about the blog (its title, its editor, a short text describing the blog as a whole).
• The server-managed content of the blog is the list of data resources that it contains, giving access to individual posts
• Each post is typically a data-resource, with (user-managed) content describing the individual post (title, date, author, body)
• But in fact, some posts could be containers themselves, that would contain for example images linked from the body of the post; since containers and data-resources alike "return" their user-managed content by default, this would cause no trouble to applications rendering the posts.

I think this proposal has merits.

To @acoburn's point: yes, this departs from the way Solid application work currently: when they deference a container's URI, they expect to get the membership metadata directly (while this proposal requires an additional hop). But to be fair, they expect it in a format that will not be the same in LWS, so there will be a breaking changed already (even though some implementations may content-negociate it to Solid-friendly TTL, that will not be required).

[damooo]

@pchampin

So in that scenario, it seems that what you need is rather a server able to content-negociate the membership metadata in TEI XML, rather than allowing user-managed content for the container...

Yes, that's true. It would be client's responsibility to maintain consistency for general servers. But at least it could do such organization of resource, as servers generally support arbitrary representation on containers.

But OTOH, there could be servers that support such an extension, where for each big xml file, they offer a side car container tree view till leaf nodes bearing manageble chunks, enabling "xml api". Also server can support paginated live view of large rdf datasets, paginated into lws container, also enabling efficient collections support with a thin extension layer. I am actually speccing those, and few other lightweight extensions for a server that is tuned for knowledge graph driven hypermedia systems, focussing on digital humanities. And many of those extensions assume this particular affordance. Hence I am literally pleading.

But in fact, some posts could be containers themselves, that would contain for example images linked from the body of the post; since containers and data-resources alike "return" their user-managed content by default, this would cause no trouble to applications rendering the posts.

Thanks for writing that. I would use the oppertunity to plug about an extension called "compound resources". Though WG need not consider it for the spec, I believe it is basic and general enough to be in the spec, and can guide ecosystem towards proper abstractions and models.

A compound resource is essentially a sub class of container. Where normal containers are there for organizing multiple **disparate, independent, and complete resources ** into a namespace, A compound resource represents one logical resource, but distributed into multiple lws resources in a container, as an implementation detail for convenience in representation and interaction. This is general model for many applications like blog posts, chats, etc. These all benefits from few more general affordences over that of normal container. Like:

• Recursive delete MUST be supported. Seen in conversations like "Delete that chat", "Delete that post", etc.

• "Last Modified" propagates to compound root. Thus, applications know when the compound resource is updated. Seen in "Chat was updated few minutes ago", "Post was updated".

• Allows to have other compound resources in it's internal hierarchy for compound sections.

• For servers supporting versioning, these forms an "archival group". Update to any internal resource will version entire compound resource. (Of course, forward deltas, etc. can be used for efficiency at implementation). See fedora's Archival group. Seen in "Post's last version", "Book's last version", etc.

• They SHOULD have a type, and their parent containers should include the type of these as part of their basic revealed containment metadata. This will allow us to provide rich listing views with special handling of typed compound resource members (like icons, etc). Google drive, etc already does that. Probably, they could also have a NAMASTE resource too.

• Leaf DataResources should be able to be converted into a compound resource, if and when needed, like above blog post.

As I am modelling my repo server and apps, I found that this notion of "compound resource" is a pit of success, and guides us towards better and better abstractions. And it is lightweight enough to be in the spec.

And they also illustrate the versatality this proposal would give to applications, by not artificially limiting capabilities of container resources, and allow us to upgrade resource model to containers as needs scale.

But to be fair, they expect it in a format that will not be the same in LWS, so there will be a breaking changed already.

Not only that. LWS mandates pagination. Thus clients must be prepared to traverse multiple hops to glean entire containment index, unlike in Solid. And thus having to traverse one extra initial link won't be much complication for such clients.