Skip to content

[OPIK-5036] [DOCS] docs: add evaluation suites documentation page#5828

Draft
alexkuzmik wants to merge 3 commits intomainfrom
aliaksandrk/OPIK-5036-evaluation-suite-docs
Draft

[OPIK-5036] [DOCS] docs: add evaluation suites documentation page#5828
alexkuzmik wants to merge 3 commits intomainfrom
aliaksandrk/OPIK-5036-evaluation-suite-docs

Conversation

@alexkuzmik
Copy link
Copy Markdown
Collaborator

Details

Add a new documentation page for Evaluation Suites covering the full SDK workflow: creating suites with assertions and execution policies, adding test items (batch and single with item-level assertions/policies), defining task functions, running evaluations, and comparing prompt versions. Includes a runnable quickstart script and three screenshots (terminal output, experiment items, assertion detail).

  • New MDX page added to Fern docs under Evaluation section
  • Standalone evaluation_suite_quickstart.py example script
  • Three screenshots: terminal output, passed suite, failed run detail
  • Registered in docs.yml navigation

Change checklist

  • User facing
  • Documentation update

Issues

  • OPIK-5036

AI-WATERMARK

AI-WATERMARK: yes

  • If yes:
    • Tools: Claude Code
    • Model(s): Claude Opus 4.6
    • Scope: Full implementation
    • Human verification: Manual review of rendered docs page

Testing

Ran Fern dev server locally and verified the page renders correctly with all code blocks, screenshots, and MDX components (Tip, Note, Frame).

Documentation

This PR is the documentation change itself.

alexkuzmik and others added 3 commits March 20, 2026 17:58
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions bot added documentation Improvements or additions to documentation python Pull requests that update Python code labels Mar 24, 2026
@github-actions
Copy link
Copy Markdown
Contributor

🌿 Preview your docs: https://opik-preview-838c7d17-7398-4584-8f8e-5b6a561e2043.docs.buildwithfern.com/docs/opik

No broken links found


📌 Results for commit 816cd5e

@alexkuzmik alexkuzmik added test-environment Deploy Opik adhoc environment and removed test-environment Deploy Opik adhoc environment labels Mar 24, 2026
@github-actions
Copy link
Copy Markdown
Contributor

🔄 Test environment deployment process has started

Phase 1: Deploying base version 1.10.47-4623 (from main branch) if environment doesn't exist
Phase 2: Building new images from PR branch aliaksandrk/OPIK-5036-evaluation-suite-docs
Phase 3: Will deploy newly built version after build completes

You can monitor the progress here.

Comment on lines +1 to +5
---
headline: Evaluation Suites | Opik Documentation
og:description: Create reusable test suites with natural-language assertions to evaluate
AI agents and LLM-powered applications across iterations.
og:site_name: Opik Documentation
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

evaluation_suites.mdx uses underscores instead of the kebab-case required by apps/opik-documentation/AGENTS.md, should we rename it to evaluation-suites.mdx?
evaluation_suites.mdx => evaluation-suites.mdx

Finding type: AI Coding Guidelines | Severity: 🟢 Low


Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents:

Before applying, verify this suggestion against the current code. In
apps/opik-documentation/documentation/fern/docs/evaluation/evaluation_suites.mdx around
lines 1-5, the filename uses underscores instead of the repo's required kebab-case.
Rename the file to
apps/opik-documentation/documentation/fern/docs/evaluation/evaluation-suites.mdx. After
renaming, search the repository for references to evaluation_suites.mdx and update them
to evaluation-suites.mdx (links, imports, or documentation indexes) so nothing breaks.

@comet-ml comet-ml deleted a comment from CometActions Mar 24, 2026
@CometActions
Copy link
Copy Markdown
Collaborator

🌙 Nightly cleanup: The test environment for this PR (pr-5828) has been cleaned up to free cluster resources. PVCs are preserved — re-deploy to restore the environment.

4 similar comments
@CometActions
Copy link
Copy Markdown
Collaborator

🌙 Nightly cleanup: The test environment for this PR (pr-5828) has been cleaned up to free cluster resources. PVCs are preserved — re-deploy to restore the environment.

@CometActions
Copy link
Copy Markdown
Collaborator

🌙 Nightly cleanup: The test environment for this PR (pr-5828) has been cleaned up to free cluster resources. PVCs are preserved — re-deploy to restore the environment.

@CometActions
Copy link
Copy Markdown
Collaborator

🌙 Nightly cleanup: The test environment for this PR (pr-5828) has been cleaned up to free cluster resources. PVCs are preserved — re-deploy to restore the environment.

@CometActions
Copy link
Copy Markdown
Collaborator

🌙 Nightly cleanup: The test environment for this PR (pr-5828) has been cleaned up to free cluster resources. PVCs are preserved — re-deploy to restore the environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation python Pull requests that update Python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants