Test if content creation with AI could stimulate the creation of reviewed content #1887

ksuess · 2025-03-04T09:16:17Z

Referencing #1882

I took empty chapter https://6.docs.plone.org/backend/subscribers.html as an example.

A first prompt:
Write a documentation chapter about subscribers (event handler) in Plone for Version Plone 6. Take the two chapters of Plone 5 documentation https://5.docs.plone.org/external/plone.app.dexterity/docs/advanced/event-handlers.html and https://5.docs.plone.org/develop/addons/components/events.html into account.

tool: ChatGPT https://chatgpt.com

The response to the above mentioned prompt listed the following sources:

Preview of this chapter: https://plone6--1887.org.readthedocs.build/backend/subscribers.html

📚 Documentation preview 📚: https://plone6--1887.org.readthedocs.build/

[EDIT] I added the used tool and the sources information of the response.

Write a documentation chapter about subscribers (event handler) in Plone for Version Plone 6. Take the two chapters of Plone 5 documentation https://5.docs.plone.org/external/plone.app.dexterity/docs/advanced/event-handlers.html and https://5.docs.plone.org/develop/addons/components/events.html into account.

acsr · 2025-03-04T11:05:22Z

@MrTango Is there a deeper reason the topic was renamed from Event Handlers to Subscribers by the documentation team? I could not find something related (a #LADR can help)

The current Subscribers (event handlers) chapter was inserted as todo by @MrTango , no further notes in the commit found…

acsr · 2025-03-04T11:53:34Z

I would suggest to put this example explicitly in draft mode.

In this particular case at first sight content was changed seriously and shifts focus. This can become too misleading to make this an easy peasy task. (my opinion) – LLMs are chatterboxes (like me).

A longer response is needed to dive into the technical backgrounds . It could explain some general issues with the approach "as is". On the other hand we can achieve a lot with some adjustments to the workflow, that adds complexity but is worth the effort.

First of all: AI can do a lot of things very well. We are using this with respect to actual needs

To create valueful results, you need to create proper system prompts or better meta-prompts in advance to setup the constraints. Without the results are too vague and always need review.

The amount of review is directly dependent on the quality of the prompts and workflow pipeline you use.

Beyond the creation of the basic system prompts, the setup of the boundaries the sources to be used, you also can create workflows and QS chains that can iterate over the content many times and do automated QA. It is also a clever idea to use another system/model in the pipeline to review the output of another.

In my opinion your attempt is a nice try but as you may see in the result there is a longer way to go than a oneshot prompt – in particular for longer content snippets, that are more intertwingled with other context (that is a real challenge for LLMs).

Just playing around with too simple LLM requests (aka prompts) ends up with more afterwork and spoils the benefits at the end.

Misleading interpretation of focus

The main thing is that the new content is putting focus on parts and omits others looking like an opinionated version, that was not requested by the prompt.

One simple example:

The simple and effective statement: "Zope’s event model is synchronous. " is completely omitted in the new output, it would help a lot having and keeping this upfront.
Instead synchronous is mentioned without introduction the performance context at the end and asynchronous is mentioned as a contradiction.

One Shot prompts are not the best approach suitable for our needs

As a one shot prompt the approach is a nice try, but there is a lot possible to improve the result by setting up an AI toolchain based on current serious experiences with this very fast evolving technology.

Topics are:

System Prompts (Meta-Prompting)
- Adjusting temperature to zero etc.
Existing Content as RAG
Feature Store and Feedback Management
Multistep Workflows (aka AI Agents)
Quality Assurance
- This should be split into a pipe of actions (not properly ordered:
  - Add a Link to the meta documentation of the Plone Documentation as requirements for new Docs for Plone 6 as a reference for the AI Meta-Prompt prerequisites
  - List all the code parts involved in the Plone 5 documentation mentioned
  - Compare the relevant related parts with the code in Plone 6
  - Was anything changed in the underlying code except the estimated renaming from Event Handlers to Subscribers
  - Is something available from the Roadmap of Plone6/7
  - is something relevant in the forum available (examples for subscribers are available)
    I stop here for now
- using a different AI to qa AI output

Make this not a blocker

On the other hand if the content is properly reviewed and corrected, one can accept the final PR.

Long term discussion on AI generated docs, based on existing docs

My concern is that we end up with fast created content and people missing the knowledge to fix the flaws. Like @stevepiercy can volunteer a lot but has to stop when the technical knowledge of the topic is missing.

acsr · 2025-03-04T13:13:41Z

For those who are interested: During Beethoven Sprint I created a meta-documentation for creation of more informal content like translated success stories using LLMs based on talk transscripts on Plone.de (it is hidden and not public). I can drop you the link to a GDocs version for discussions. Since that point in time a lot of water went down the river rhine as we say it in the AI business.

jensens

Actually a stunning good starting point, but its not correct in all places.

jensens · 2025-03-04T13:34:43Z

docs/backend/subscribers.md

-Move content from:
-   https://5.docs.plone.org/external/plone.app.dexterity/docs/advanced/event-handlers.html
-   https://5.docs.plone.org/develop/addons/components/events.html
+In Plone 6, the event system provides a robust mechanism for developers to execute custom actions in response to various occurrences within the system.


Suggested change

In Plone 6, the event system provides a robust mechanism for developers to execute custom actions in response to various occurrences within the system.

In Plone 6, the event system provides a mechanism for developers to execute custom actions in response to various occurrences within the system.

jensens · 2025-03-04T13:35:46Z

docs/backend/subscribers.md

+Plone's event system is built upon the Zope Component Architecture's `zope.event` package, enabling a publish-subscribe pattern where events are broadcasted, and subscribers (event handlers) respond accordingly.
+Key characteristics of this system include:
+
+- **Simplicity**: The event system is straightforward to implement and use.


I would remove this line

jensens · 2025-03-04T13:38:52Z

docs/backend/subscribers.md

+- **No Return Values**: Event handlers do not return values.
+- **Exception Handling**: Exceptions raised within an event handler can interrupt the request processing.
+
+Events are typically defined by interfaces and may carry pertinent information about the event.


Suggested change

Events are typically defined by interfaces and may carry pertinent information about the event.

Events are matching interfaces and may carry relevant information about the event.

defined by is not correct, they match an interface. This could be possibly expressed better.

Actually I had to look up pertinent, keep the language simple.

jensens · 2025-03-04T13:40:19Z

docs/backend/subscribers.md

+
+## Registering Event Handlers
+
+Event handlers can be registered in two primary ways: using ZCML (Zope Configuration Markup Language) or Python decorators.


Unless one uses Grok (discouraged) this is wrong. ZCML is always needed. The whole section needs adjustments.

stevepiercy · 2025-03-04T13:51:27Z

AI can be a useful tool, but ultimately content experts need to review the content for accuracy, and documentarians need to review it for broader contexts than AI can handle.

In other words, our volunteer jobs for which we are generously compensated are not threatened by AI. 😏

But seriously, there have been multiple incidents with AI being misused, primarily in Volto, once resulting in a copyright infringement issue that I escalated to the Board to manage, but also with shitty first-timer contributions that waste our time to review.

I recommended an AI policy to the Board, and I hope they publish its AI policy soon. IMO anytime AI is used in a contribution, then the contributor must disclose that fact, as @ksuess has responsibly done, giving us the prompts used to reproduce. However, I would add that the AI tool used must also be disclosed.

ksuess · 2025-03-04T15:49:46Z

I am amazed at the wealth of information and opinions on this topic.

For the sake of completion I added the used tool and the sources information of the response I posted with the first commit of this PR.

ksuess · 2025-03-04T15:54:32Z

Copyright: I think this is a sensitive topic.
But the case in Volto is somehow an example, simple to avoid. The person lazily posted swedish translations including the authors name, which is not himself. I mean, really? The first rule is to read the response of the AI tool. The second is to check the sources. Before these checks it is lazy to post something. And it's lazy not to even mention that an AI tool has been used for the commits.

ksuess · 2025-03-04T15:59:14Z

Actually a stunning good starting point, but its not correct in all places.

Thank you, @jensens, for the review. I wait with applying your suggestions, cause the spontaneous idea of this PR is more a starting point to collect some insights, experiences and opinions.

ksuess · 2025-03-04T16:10:31Z

One Shot prompts are not the best approach suitable for our needs

As a one shot prompt the approach is a nice try, but there is a lot possible to improve the result by setting up an AI toolchain based on current serious experiences with this very fast evolving technology.

Topics are:

* System Prompts (Meta-Prompting)
  
  * Adjusting temperature to zero etc.

* Existing Content as RAG

* Feature Store and Feedback Management

* Multistep Workflows (aka AI Agents)

* Quality Assurance
  
  * This should be split into a pipe of actions (not properly ordered:
    
    * Add a Link to the meta documentation of the Plone Documentation as requirements for new Docs for Plone 6 as a reference for the AI Meta-Prompt prerequisites
    * List all the code parts involved in the Plone 5 documentation mentioned
    * Compare the relevant related parts with the code in Plone 6
    * Was anything changed in the underlying code except the estimated renaming from `Event Handlers` to `Subscribers`
    * Is something available from the Roadmap of Plone6/7
    * is something relevant in the forum available (examples for subscribers are available)
      I stop here for now
  * using a different AI to qa AI output

You have obviously deeper insights and experience with AI tools. How is your estimation for the time commitment needed for let's say one chapter, that is by now only a placeholder: Setting up a toolchain with constraints and requirements for Plone 6 documentation, focus (developer, not marketing), copyright, example code, etc., generating also meta data, keywords and description for SEO?

acsr · 2025-03-04T17:15:43Z

You have obviously deeper insights and experience with AI tools. How is you estimation for the time commitment needed for let's say one chapter, that is by now only a placeholder: Setting up a toolchain with constraints and requirements for Plone 6 documentation, focus (developer, not marketing), copyright, example code, etc., generating also meta data, keywords and description for SEO?

I was working on a toolchain based on PrivateGPT with RAG and custom system prompts over one year ago as a test balloon. Open source models from Mistral and later LLama 3.1 8B and 70B were used on a local MacBookPro M1 MaxPro with 64GB RAM. The 70B was at that time was at the limit of the hardware. For us the use of the public APIs was not an option.

Currently my next steps are waiting for customer requirements to update to the latest toolchains. There was very much progress in the AI world during the last months. I now would take a look at something like AnythingLLM as a frontend framework. But to create a reliable toolchain usable by a wider audience the complexity is high, when you want to constantly rely on that. This is why I am focusing to replace the workflow and result management solutions out there with actually a Plone System used similar to the eea.daviz concepts.

I am sorry to have skip the estimate question. We are also in R&D for that. I took a touch on the topic in my talk during the Plone Tagung 2024 in Gießen.

stevepiercy · 2025-03-04T23:59:19Z

The first rule is to read the response of the AI tool. The second is to check the sources. Before these checks it is lazy to post something. And it's lazy not to even mention that an AI tool has been used for the commits.

I included these points in my recommendation to the Board about an AI policy. It touches on risks of violating copyright law and ethical concerns of being a member of the Plone Foundation or GitHub organization. I'll follow up with the Board to get the status of this item.

ksuess requested review from stevepiercy, jensens, thet, pbauer and MrTango March 4, 2025 09:24

ksuess marked this pull request as ready for review March 4, 2025 09:32

prompt: for better review experience: split text in one line per sentece

39c8435

ksuess requested a review from petschki March 4, 2025 10:30

jensens requested changes Mar 4, 2025

View reviewed changes

stevepiercy mentioned this pull request Mar 4, 2025

Purge empty pages and todos #1882

Open

40 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test if content creation with AI could stimulate the creation of reviewed content #1887

Test if content creation with AI could stimulate the creation of reviewed content #1887

ksuess commented Mar 4, 2025 •

edited

Loading

acsr commented Mar 4, 2025

acsr commented Mar 4, 2025 •

edited

Loading

acsr commented Mar 4, 2025

jensens left a comment

jensens Mar 4, 2025

jensens Mar 4, 2025

jensens Mar 4, 2025

jensens Mar 4, 2025 •

edited

Loading

stevepiercy commented Mar 4, 2025

ksuess commented Mar 4, 2025 •

edited

Loading

ksuess commented Mar 4, 2025 •

edited

Loading

ksuess commented Mar 4, 2025

ksuess commented Mar 4, 2025 •

edited

Loading

One Shot prompts are not the best approach suitable for our needs

acsr commented Mar 4, 2025 •

edited

Loading

stevepiercy commented Mar 4, 2025

	In Plone 6, the event system provides a robust mechanism for developers to execute custom actions in response to various occurrences within the system.
	In Plone 6, the event system provides a mechanism for developers to execute custom actions in response to various occurrences within the system.

	Events are typically defined by interfaces and may carry pertinent information about the event.
	Events are matching interfaces and may carry relevant information about the event.


		## Registering Event Handlers

		Event handlers can be registered in two primary ways: using ZCML (Zope Configuration Markup Language) or Python decorators.

Test if content creation with AI could stimulate the creation of reviewed content #1887

Are you sure you want to change the base?

Test if content creation with AI could stimulate the creation of reviewed content #1887

Conversation

ksuess commented Mar 4, 2025 • edited Loading

acsr commented Mar 4, 2025

acsr commented Mar 4, 2025 • edited Loading

Misleading interpretation of focus

One Shot prompts are not the best approach suitable for our needs

Make this not a blocker

Long term discussion on AI generated docs, based on existing docs

acsr commented Mar 4, 2025

jensens left a comment

Choose a reason for hiding this comment

jensens Mar 4, 2025

Choose a reason for hiding this comment

jensens Mar 4, 2025

Choose a reason for hiding this comment

jensens Mar 4, 2025

Choose a reason for hiding this comment

jensens Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

stevepiercy commented Mar 4, 2025

ksuess commented Mar 4, 2025 • edited Loading

ksuess commented Mar 4, 2025 • edited Loading

ksuess commented Mar 4, 2025

ksuess commented Mar 4, 2025 • edited Loading

One Shot prompts are not the best approach suitable for our needs

acsr commented Mar 4, 2025 • edited Loading

stevepiercy commented Mar 4, 2025

ksuess commented Mar 4, 2025 •

edited

Loading

acsr commented Mar 4, 2025 •

edited

Loading

jensens Mar 4, 2025 •

edited

Loading

ksuess commented Mar 4, 2025 •

edited

Loading

ksuess commented Mar 4, 2025 •

edited

Loading

ksuess commented Mar 4, 2025 •

edited

Loading

acsr commented Mar 4, 2025 •

edited

Loading