RHAI-ENG-1551-Ragas-docs-changes - initial draft of Llama-stack speci…

[chtyler]

…fic docs changes

Description

Documentation related to Llama-Stack-centric ragas evaluation.

How Has This Been Tested?

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

Summary by CodeRabbit

• Documentation
  • Added comprehensive guides for evaluating RAG systems with multiple provider options
  • Introduced documentation on available evaluation providers: RAGAS, BEIR, and TrustyAI
  • New guide for Ragas integration covering inline and remote deployment modes
  • Documented key evaluation metrics: faithfulness, answer relevancy, and context precision

[coderabbitai]

Walkthrough

New documentation introduced for evaluating RAG systems with Llama Stack, covering evaluation providers, Ragas integration, and related concepts. The main assembly document is updated to reference the new evaluation documentation modules.

Changes

Cohort / File(s)	Summary
New RAG Evaluation Assembly assemblies/evaluating-rag-systems-with-llama-stack.adoc	New AsciiDoc module defining evaluation scope, abstract, and references to three supporting documentation modules with conditional context scaffolding.
New RAG Evaluation Documentation Modules modules/understanding-rag-evaluation-providers.adoc, modules/using-ragas-with-llama-stack.adoc	Two new concept pages explaining evaluation providers (RAGAS, BEIR, TrustyAI) and Ragas integration with deployment modes (inline and remote).
Assembly Document Update working-with-llama-stack.adoc	Modified to replace a single RAG stack deployment include with two separate includes: one for deployment and one for evaluation guidance.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

• All changes are documentation additions and straightforward assembly references with no logic or code modifications.

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title references a Jira ticket (RHAI-ENG-1551) and mentions 'Ragas-docs-changes' and 'Llama-stack', which align with the PR's documentation updates for RAG evaluation with Llama Stack. However, it's truncated mid-word ('speci…'), uses a ticket prefix that adds noise, and is somewhat generic about 'initial draft'.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c7dc427 and db97db3.

📒 Files selected for processing (4)

• assemblies/evaluating-rag-systems-with-llama-stack.adoc (1 hunks)
• modules/understanding-rag-evaluation-providers.adoc (1 hunks)
• modules/using-ragas-with-llama-stack.adoc (1 hunks)
• working-with-llama-stack.adoc (1 hunks)

🧰 Additional context used

🧠 Learnings (1)

📚 Learning: 2025-09-09T14:54:18.536Z

Learnt from: kelbrown20
Repo: opendatahub-io/opendatahub-documentation PR: 929
File: assemblies/deploying-a-rag-stack-in-a-data-science-project.adoc:20-20
Timestamp: 2025-09-09T14:54:18.536Z
Learning: In the opendatahub-documentation repository, the assembly "Deploying a RAG stack in a data science project" (assemblies/deploying-a-rag-stack-in-a-data-science-project.adoc) includes modules/overview-of-rag.adoc as a submodule to provide RAG-specific context within the RAG deployment workflow, even when the parent page focuses on Llama Stack.

Applied to files:

• working-with-llama-stack.adoc

🔇 Additional comments (5)

working-with-llama-stack.adoc (1)

24-25: Clean addition of evaluation assembly include.

The new include follows the established pattern and file structure. Placement after the deployment assembly makes logical sense within the documentation workflow. Based on learnings from similar assembly patterns in this repository, this change aligns with best practices for modular documentation organization.

modules/understanding-rag-evaluation-providers.adoc (1)

1-14: Well-structured concept module with consistent variable usage.

The module follows proper AsciiDoc conventions for concept pages, with appropriate metadata, context-aware ID, and consistent use of product variables ({productname-short}, {org-name}). The provider descriptions are concise and set appropriate context for downstream modules.

modules/using-ragas-with-llama-stack.adoc (1)

1-19: Clear, practical guidance on Ragas deployment modes.

The module provides actionable guidance for users to choose between inline and remote deployment modes based on their workflow needs. The abstract clearly sets expectations, and the deployment mode descriptions are practical. Consistent use of product variables supports maintainability.

assemblies/evaluating-rag-systems-with-llama-stack.adoc (2)

1-16: PR readiness check: merge criteria not yet satisfied.

Per the PR objectives, the following merge checklist items remain unchecked:

• Commits not yet squashed with meaningful messages
• Testing instructions not added to PR body
• Developer has not manually tested and verified the changes

These items should be addressed before merging. Specifically, consider adding a testing procedure that demonstrates the new evaluation documentation renders correctly in the documentation build and that all module includes resolve without errors.

---

1-16: All referenced module files exist; original concern is invalid.

The verification confirms that all three module files referenced in the assembly (understanding-rag-evaluation-providers.adoc, using-ragas-with-llama-stack.adoc, and benchmarking-embedding-models-with-beir-datasets-and-llama-stack.adoc) are present in the repository. The assembly will render without file-not-found errors.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}