Skip to content

Users/singankit/update evaluator prompts #40469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Users/singankit/update evaluator prompts #40469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c87107e
Update coherence.prompty
singankit Apr 10, 2025
90a5aba
Update fluency.prompty
singankit Apr 10, 2025
c6a14e8
Update groundedness_with_query.prompty
singankit Apr 10, 2025
e33a04b
Update groundedness_without_query.prompty
singankit Apr 10, 2025
09a78dd
Update relevance.prompty
singankit Apr 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion ...aluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_coherence/coherence.prompty
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ user:
**Query:** Discuss the significance of the Monroe Doctrine in shaping U.S. foreign policy.
**Response:** The Monroe Doctrine was a pivotal policy declared in 1823 that asserted U.S. opposition to European colonization in the Americas. By stating that any intervention by external powers in the Western Hemisphere would be viewed as a hostile act, it established the U.S. as a protector of the region. This doctrine shaped U.S. foreign policy by promoting isolation from European conflicts while justifying American influence and expansion in the hemisphere. Its long-term significance lies in its enduring influence on international relations and its role in defining the U.S. position in global affairs.

Note that the QUERY can either be a string with a user request or an entire conversation history including previous requests and responses from the assistant.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and RESPONSE

In this case, the assistant's response should be evaluated in the context of the entire conversation but the focus should be on the last query and response.

# Data
QUERY: {{query}}
Expand All @@ -96,4 +98,4 @@ RESPONSE: {{response}}


## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
# Output
# Output
4 changes: 3 additions & 1 deletion sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/fluency.prompty
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ user:

**Response:** Technology revolutionizes modern education by providing interactive learning platforms, enabling personalized learning experiences, and connecting students worldwide, thereby transforming how knowledge is acquired and shared.

Note that the QUERY can either be a string with a user request or an entire conversation history including previous requests and responses from the assistant.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RESPONSE instead of QUERY

In this case, the assistant's response should be evaluated in the context of the entire conversation but the focus should be on the last query and response.

# Data
RESPONSE: {{response}}
Expand All @@ -83,4 +85,4 @@ RESPONSE: {{response}}


## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
# Output
# Output
4 changes: 3 additions & 1 deletion ...-evaluation/azure/ai/evaluation/_evaluators/_groundedness/groundedness_with_query.prompty
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@ user:
**Query:** By what date must participants register to receive early bird pricing?
**Response:** Participants must register by May 31st to receive early bird pricing.

Note that the QUERY can either be a string with a user request or an entire conversation history including previous requests and responses from the assistant.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Note that the QUERY can either be a string with a user request or an entire conversation history including previous requests and responses from the assistant.
Note that the QUERY can either be a string with a user request or an entire conversation history including previous requests and responses from the assistant. And CONTEXT can either be a string with retrieved context or an entire conversation history including previous requests and responses from the assistant.

In this case, the assistant's response should be evaluated in the context of the entire conversation but the focus should be on the last query and response.

# Data
CONTEXT: {{context}}
Expand All @@ -110,4 +112,4 @@ RESPONSE: {{response}}


## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
# Output
# Output
4 changes: 3 additions & 1 deletion ...aluation/azure/ai/evaluation/_evaluators/_groundedness/groundedness_without_query.prompty
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ user:
**Context:** The new smartphone model features a larger display, improved battery life, and an upgraded camera system.
**Response:** The new smartphone model features a larger display, improved battery life, and an upgraded camera system.

Note that the QUERY can either be a string with a user request or an entire conversation history including previous requests and responses from the assistant.
In this case, the assistant's response should be evaluated in the context of the entire conversation but the focus should be on the last query and response.

# Data
CONTEXT: {{context}}
Expand All @@ -96,4 +98,4 @@ RESPONSE: {{response}}


## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
# Output
# Output
5 changes: 3 additions & 2 deletions ...aluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/relevance.prompty
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,8 @@ user:
**Query:** What topics will the conference cover?
**Response:** The conference will cover renewable energy, climate change, and sustainability practices, bringing together global experts to discuss these critical issues.


Note that the QUERY can either be a string with a user request or an entire conversation history including previous requests and responses from the assistant.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And RESPONSE

In this case, the assistant's response should be evaluated in the context of the entire conversation but the focus should be on the last query and response.

# Data
QUERY: {{query}}
Expand All @@ -97,4 +98,4 @@ RESPONSE: {{response}}


## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
# Output
# Output
Loading