Attaching user feedback to traces

[shinji-Yama77]

Hi,

I’m currently using the M365 Agents SDK together with the Microsoft Agent Framework to call Azure AI Foundry agents.

I’m attaching user feedback (e.g., thumbs-up/down) to the traces, but it’s currently being recorded as a custom metric. Instead, I’d prefer for this feedback to appear in the right-side “User Feedback” panel in Azure AI Foundry — the same place where manual evaluations and ratings are normally displayed.

Could you clarify how to properly log or tag user feedback so it’s surfaced in that panel and linked to the corresponding gen_ai.response.id rather than as a separate custom metric?

async def handle_reaction(self, context: TurnContext, state: TurnState):
        """Handles incoming reaction"""

        user_id = context.activity.from_property.id
        activity = context.activity


        # restore trace context
        parent_ctx = self.telemetry_service.restore_or_create_context(state, user_id)
        tracer = self.telemetry_service.get_tracer()

        # reactions don't need a persistent thread
        try:

            with tracer.start_as_current_span(
                f"Reaction_{user_id}",
                kind=trace.SpanKind.CLIENT,
                context=parent_ctx
            ) as span:
                for reaction in (activity.reactions_added or []):

                    reaction_type = reaction.type.lower()

                    # Map reaction type -> evaluation score
                    score = 1 if reaction_type == "like" else -1

                    span.add_event(
                        name="gen_ai.evaluation.user_feedback",
                        attributes={
                            "gen_ai_evalution.score": score,
                            "feedback.type": reaction.type.lower(),
                            "feedback.comment": f"User added {reaction.type.lower()} reaction"
                        },
                    )
                    logger.info(f"User added reaction: {reaction.type}")

                # Handle removed reactions
                for reaction in (activity.reactions_removed or []):
                    reaction_type = reaction.type.lower()
                    span.add_event(
                        name="gen_ai.evaluation.user_feedback",
                        attributes={
                            "gen_ai_evalution.score": 0,
                            "feedback.type": reaction_type.lower(),
                            "feedback.comment": f"User removed {reaction_type} reaction",
                        },
                    )
                    logger.info(f"User removed reaction: {reaction_type}")

        except Exception as e:
            logger.exception(f"Error handling reaction: {e}")

I would prefer is the metric is instead in the rightmost down panel where it is titled User Feedback. The query below is used by Azure AI Foundry to display results in the User Feedback Panel.

set query_take_max_records=10001;
set truncationmaxsize=67108864;
let application=dynamic(['*']);
let model=dynamic(['*']);
let response_ids=(
    dependencies
    | where customDimensions["gen_ai.operation.name"] == "chat" and ("*" in (model) or array_length(model) == 0 or customDimensions["gen_ai.response.model"] in (model))
    | where ('*' in (application) or array_length(application) == 0 or cloud_RoleName in (application)) and timestamp between(datetime("2025-11-06T01:26:43.114Z") .. datetime("2025-11-07T01:26:43.114Z"))
    | project customDimensions.["gen_ai.response.id"]
    ); 
traces
| where customDimensions["event.name"] == "gen_ai.evaluation.user_feedback"
| where ('*' in (application) or array_length(application) == 0 or cloud_RoleName in (application)) and timestamp between(datetime("2025-11-06T01:26:43.114Z") .. datetime("2025-11-07T01:26:43.114Z")) 
| where customDimensions.["gen_ai.response.id"] in (response_ids)
| extend FeedbackType = case(toint(customDimensions["gen_ai.evaluation.score"]) == -1, "Positive", toint(customDimensions["gen_ai.evaluation.score"]) == 1, "Negative", "Unknown")
| summarize Count = count() by FeedbackType