Failed to export annotatations with the spans

[hristogg]

Hi,

I am instrumenting a llamaindex workflow and am updating the CHAIN with user feedback like this:

client = httpx.Client()
    annotation_payload = {
        "data": [
            {
                "span_id": span_id,
                "name": "user feedback",
                "annotator_kind": "HUMAN",
                "result": {"label": "one-to-five", "score": score},
                "metadata": {"user comment": usr_comment},
            }
        ]
    }

    headers = {'api_key': api_key}

    client.post(
        "https://app.phoenix.arize.com/v1/span_annotations?sync=false",
        json=annotation_payload,
        headers=headers
    )

In the UI the annotation is properly reflected:

However I am trying to query all the spans together with the feedback annotation and see no way of doing this...

tried through px_client.query_spans however there if I understood correctly annotations are not supported, tried from px_client.get_evaluations however this returns an empty list.

From the UI I can actaully filter on the feedback through the spans:
annotations['user feedback'].score >= 0
And I can see them in the 'user feedback' column however did not manage to actually query and select this column programatically. any help would be highly appreciated :)

[dosubot[bot]]

To query spans along with their feedback annotations programmatically in Arize Phoenix, you can use the SpanQuery class with the px_client.query_spans method. Although annotations are not directly supported in the query_spans method, you can filter spans based on evaluation results or metadata using the domain-specific language (DSL) provided by SpanQuery.

Here's an example of how you might construct a query to filter spans based on the feedback annotation:

import phoenix as px
from phoenix.trace.dsl import SpanQuery

# Create a query to filter spans based on the feedback annotation
query = SpanQuery().where(
    "annotations['user feedback'].score >= 0"
)

# Use the Phoenix Client to execute the query
px_client = px.Client()
result_df = px_client.query_spans(query)

# Display the result
print(result_df)

This query filters spans where the user feedback annotation's score is greater than or equal to 0. The result will be a DataFrame containing the spans that match this condition.

If you need to access specific annotation details, you might need to handle annotations separately by querying them through the API and then merging the results with the spans DataFrame programmatically. Unfortunately, direct support for annotations in query_spans is not available, so this workaround is necessary to achieve your goal.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[dosubot[bot]]

To access and visualize annotations like "user feedback" in the returned pandas DataFrame, you can try extracting the spanAnnotations field from each span and then process it to include the desired annotation details in the DataFrame. Here's a step-by-step approach:

1. Query the Spans: Use the query_spans method to get the spans.

2. Extract Annotations: Iterate over the spanAnnotations field in each span to extract the relevant annotation details.

3. Add to DataFrame: Add the extracted annotation details as new columns in the DataFrame.

Here's an example of how you might achieve this:

import phoenix as px
from phoenix.trace.dsl import SpanQuery
import pandas as pd

# Create a query to get all spans
query = SpanQuery()

# Use the Phoenix Client to execute the query
px_client = px.Client()
result_df = px_client.query_spans(query)

# Function to extract 'user feedback' annotation score
def extract_user_feedback(span_annotations):
    for annotation in span_annotations:
        if annotation['name'] == 'user feedback':
            return annotation['score']
    return None

# Add a new column for 'user feedback' score
result_df['user_feedback_score'] = result_df['spanAnnotations'].apply(extract_user_feedback)

# Display the result
print(result_df[['user_feedback_score']])

This code snippet adds a new column user_feedback_score to the DataFrame, which contains the score of the "user feedback" annotation for each span. Adjust the extraction logic if you need to include more details or handle different annotation structures.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[hristogg]

Thanks @dosu, but the resulting dataframe has no column 'spanAnnotations' here are the columns being return from query_spans - 'name', 'span_kind', 'parent_id', 'start_time', 'end_time',
'status_code', 'status_message', 'events', 'context.span_id',
'context.trace_id', 'attributes.input.value',
'attributes.output.mime_type', 'attributes.output.value',
'attributes.input.mime_type', 'attributes.openinference.span.kind',
'attributes.embedding.model_name', 'attributes.embedding.embeddings',
'attributes.retrieval.documents',
'attributes.llm.invocation_parameters', 'attributes.llm.model_name',
'attributes.llm.output_messages', 'attributes.llm.input_messages',
'attributes.llm.prompt_template.template',
'attributes.llm.prompt_template.variables', 'attributes.llm.prompts'

[dosubot[bot]]

It seems that the query_spans method is not returning the spanAnnotations field directly in the DataFrame. To work around this, you can retrieve annotations separately and then merge them with the spans DataFrame. Here's how you can do it:

1. Query Spans: Use the query_spans method to get the spans.

2. Retrieve Annotations: Use the API to fetch annotations separately.

3. Merge Data: Combine the annotations with the spans DataFrame based on the span_id.

Here's an example approach:

import phoenix as px
import pandas as pd
import httpx

# Step 1: Query the spans
query = px.SpanQuery()
px_client = px.Client()
spans_df = px_client.query_spans(query)

# Step 2: Retrieve annotations using the API
def fetch_annotations(api_key):
    client = httpx.Client()
    headers = {'api_key': api_key}
    response = client.get("https://app.phoenix.arize.com/v1/span_annotations")
    annotations = response.json().get('data', [])
    return pd.DataFrame(annotations)

# Replace '<your_api_key>' with your actual API key
annotations_df = fetch_annotations('<your_api_key>')

# Step 3: Merge spans with annotations
# Assuming 'span_id' is the key to join on
merged_df = pd.merge(spans_df, annotations_df, left_on='context.span_id', right_on='span_id', how='left')

# Display the merged DataFrame
print(merged_df)

This code snippet demonstrates how to fetch annotations separately and merge them with the spans DataFrame. Adjust the API endpoint and key as needed. This approach allows you to include annotation details in your analysis.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[hristogg]

@dosu I dont think the span_annotations endpoint has a GET method if I am looking at the documentation correctly.

[dosubot[bot]]

If the span_annotations endpoint does not support a GET method, you can retrieve annotations programmatically using the SpanAnnotationsDataLoader class. This class allows you to query SpanAnnotation objects based on a list of keys (span row IDs). Here's an example of how you might use this class:

from your_module import SpanAnnotationsDataLoader, DbSessionFactory

# Initialize the database session factory
db_session_factory = DbSessionFactory()

# Create an instance of the data loader
data_loader = SpanAnnotationsDataLoader(db=db_session_factory)

# Define the keys (span row IDs) you want to retrieve annotations for
keys = [1, 2, 3]  # Example keys

# Use the data loader to fetch the annotations
annotations = await data_loader.load_many(keys)

# Process the annotations as needed
for annotation_list in annotations:
    for annotation in annotation_list:
        print(annotation)

This code snippet demonstrates how to use the SpanAnnotationsDataLoader to asynchronously load annotations for specific span row IDs ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[hristogg]

If someone can pitch in with an idea, so far I have reached the conclusion that it is not possible to export the user feedback, which is a bit weird in my opinion. I can export it through the UI if I create an experiment, and I do have the inpout and output which is great but I dont have the timestamp then... The feedback from dosu is appreciate but I don think it is correct...

[hristogg]

Hi @axiomofjoy and thank you for the suggestion! I was under the same impression, but what I am doing is the following:
span_evals = px_client.get_evaluations(project_name='my-llm-app')
and here is a screenshot from app.phoenix.arize.com:

I tried doing a request manually and I get 404, which made me question whether I am doing it correctly but if I look into the get_evaluations I saw the following:

if response.status_code == 404:
            logger.info("No evaluations found.")
            return []

So I guessed I am getting the same result from both...

[axiomofjoy]

Hey @hristogg, this is the response I would expect to receive if my project has no span annotations. Can you confirm that you have annotated spans in this particular project?

[hristogg]

Hi @axiomofjoy in the UI I do see the annotation as user feedback, here is a screenshot to a particular span:

Here is also a screenshot from when opening up the span itself:

Could it be that human annotations are something different from annotations? :)

[axiomofjoy]

@hristogg You're absolutely right. It looks like get_evaluations currently only fetches LLM evaluations, not human annotations.

[axiomofjoy]

We are tracking this issue here. Thanks for raising!