Add a property to capture LLM and agent input

[HolgerKnublauch]

I suggest we add a property similar to sh:description and rdfs:comment, with the explicit goal to be used to communicate the semantics of a shape with AI agents that understand natural language. Particular use cases are:

• constraints and rules that cannot be formally expressed in SHACL
• for users that are not familiar enough with SHACL to formally express them
• maybe as a textual starting point to formalize semantics later

Examples etc are in the PR #726

Rendered online: https://raw.githack.com/w3c/data-shapes/issue-725/shacl12-core/index.html#intent

[afs]

Is there deployed experience?

Maybe SHACL should have a way to label features as "experimental" to allow more such potential use case focused features while allowing for later revision or withdrawal.

[HolgerKnublauch]

Yes, there is deployed experience. We (among others) have algorithms that take SHACL shape definitions to instruct LLMs about what sort of entities exist in the domain of interest, and what constraints should be applied. For example we use this for entity extraction - generating valid instances of ontology classes from text. These algorithms take property labels, datatypes, classes and other constraints, but also the sh:description and rdfs:comments and send them to the LLM as background knowledge. The proposal here is to strengthen these use cases, because sh:description is a bit vague on what it should contain. With sh:intent we explicitly encourage developers to write down additional rules about the semantics of a shape.

In fact this is additional deployed experience. We are offering graphical editing tools (more coming in the 9.1 release of TopBraid), allowing users to do conceptual modeling. The idea is that domain experts (with no knowledge of SHACL) can create initial versions of an ontology, that get later refined by people with SHACL skills. All they need to do is write down the intended semantics of classes and properties in free text, and a property such as sh:intent will allow them to create multiple atomic rules that can later be "ticked off" when they are converted to formal SHACL definitions. Starting with free text is a good way of evolving domain ontologies. Individual values of sh:intent can be marked as converted via reification, which is easier done than with large sh:description blocks.

I also don't fully understand the question. If you are asking for deployed experience here, then we should also ask this for every other new feature including SHACL rules. I assume this WG is allowed to produce features for SHACL 1.2 based on what we collectively find sensible, to the best of our knowledge. Getting features out into the market of course takes time and foresight.

And surely such feature could be labeled as "At Risk" which is already an established W3C approach.

[afs]

And surely such feature could be labeled as "At Risk" which is already an established W3C approach.

"At risk" has a specific meaning in W3C. e.g. CR features that may be removed or may be accepted by the time REC is reached.

[robert-david]

We are also using literals to put LLM instructions or input into knowledge graphs. I think it will be a common use case in the future. So +1 to have a dedicated property for this.

However, i am not sure about the use case of informally specifying/adding constraints. I can imagine that in practice this might lead to misunderstandings and inconsistencies. Also, i am not sure what the use of informally specifying a constraint that you can not express with SHACL would be. So -1 for this.

[HolgerKnublauch]

We are also using literals to put LLM instructions or input into knowledge graphs. I think it will be a common use case in the future. So +1 to have a dedicated property for this.

@robert-david ok that's good to know. Could you share a couple of examples for LLM instructions and how they differ from general descriptions of the intended constraints?

However, i am not sure about the use case of informally specifying/adding constraints. I can imagine that in practice this might lead to misunderstandings and inconsistencies. Also, i am not sure what the use of informally specifying a constraint that you can not express with SHACL would be. So -1 for this.

Is your concern that the prose in the sh:intent may differ from what is formally expressed in pure SHACL? That situation already exists with rdfs:comment and sh:description and cannot be prevented.

The use for informal constraints includes scenarios that COULD be expressed in SHACL-SPARQL but not Core, and the user is not familiar with SPARQL.

Examples (from the current branch):

  # This can be expressed in SPARQL and through a path expression + sh:lessThan
  sh:intent "A parent must be strictly older than each of their children." ;

  # This can be expressed in SPARQL using NOW() or by changing the shapes graph once a day
  sh:intent "The birth date must not be in the future." ;

I think it will be good if we lower the barrier of entry for newcomers so that they can express as much as possible without leaving their SHACL shapes, even if they are not experts in SHACL.

But these sh:intents could also be conditions that not even available to the SHACL engine but could be available to an AI agent.

  # This is similar to what Eliana mentioned yesterday
  sh:intent "The maximum value of X is limited by the result of the JavaScript expression `...`" ;

  # This is an example of information that an AI agent could look up from other sources
  sh:intent "This shape assumes that Greenland is not part of the US and that US is still in NATO. Otherwise, ignore." ;

[HolgerKnublauch]

Thinking out loud.

One of the questions raised during last week's call was about the name of the property. I still like sh:intent, but I guess sh:instruction could also work if we regard this as a property to store agent instructions. This would however neglect the use of this property for knowledge capture from domain experts.

If there are other proposals for names, please write them down.

Another question was whether the property should be split to more cleanly distinguish the use case of agent instructions vs extra constraints. In that case, the properties

• sh:instruction (purely for agents)
• sh:textualConstraint (for knowledge capture AND as input to agents)

could make sense. What do people think?