Bug Report - AIOps 90 minute workshop

[amg301]

Problem Summary

Part 1 Section 2 - Remediation Workflow
When running the "Remediation Workflow" the survey provides the option to either use the default prompt or the AI generated prompt by copying the AI example into the prompt to use box.

With the current iteration the AI provided prompt "Restart the httpd service after correcting the invalid directive in the Apache configuration file" will not result in a playbook being generated that will actually remediate the issue. It creates a playbook that restarts httpd but does not remove the InvalidDirectiveHere line within the configuration file first.

Copying in the error information from the mattermost output also does not provide a workable solution.

Re-running the "Break Apache" playbook forced a re-generation of the "Remediation Workflow" and re-generation of the gpt_generated_prompt for the survey definition. The 2nd attempt was better. This is instability and unpredictability in the workflow.

Extra vars file

None

Ansible Playbook Output

Following first prompt resulted in this playbook.

• name: Restarting Apache
  hosts: node1
  tasks:
  • name: Restart Apache Services
    ansible.builtin.service:
    name: httpd
    state: restarted

The updated prompt from re-running the Break Apache resulted in this playbook.

• name: correct the Apache configuration file
  hosts: all
  tasks:
  • name: remove the incorrect configuration
    ansible.builtin.lineinfile:
    path: /etc/httpd/conf/httpd.conf
    regexp: ^InvalidDirectiveHere
    state: absent
  • name: restart the httpd service
    ansible.builtin.service:
    name: httpd
    state: restarted

[IPvSean]

Thank you for the detailed feedback and reproduction steps.

What you are observing is an inherent characteristic of generative AI systems rather than a deterministic workflow failure. The remediation prompt generation currently relies on LLM-based inference, which means responses may vary between executions even when the underlying issue is similar.

In the first run, the generated remediation focused only on restarting the httpd service:

state: restarted

while the second generation correctly inferred the need to first remove the invalid Apache directive before restarting the service:

regexp: ^InvalidDirectiveHere
state: absent

This variability highlights an important limitation of using generative AI to create executable automation directly from inferred intent. The model does not truly “understand” the environment or validate remediation correctness prior to playbook generation, so different runs may produce different levels of accuracy or completeness.

As a result, we are actively evolving the workshop and demo experience toward a more human-in-the-loop approach. Rather than having AI generate executable remediation playbooks entirely “out of thin air,” future iterations will emphasize:

• AI-assisted recommendations and draft remediation guidance
• Curated and validated automation patterns
• Human review/approval before execution
• Greater use of trusted remediation templates and workflows

This approach better reflects how enterprise customers are operationalizing AI-assisted automation today: leveraging AI for acceleration and guidance while maintaining deterministic, governed execution through validated automation content.

The current workshop scenario was intentionally designed to explore agentic remediation concepts, but your feedback reinforces why guardrails and human oversight are critical components of production-ready AI automation systems.