[Feature Request] Have option to return trace as part of out put of a PromptFlow when deployed as endpoint

[adampolak-vertex]

Is your feature request related to a problem? Please describe.
I want instant feedback after running a deployed prompt so I can put it in my database.

I currently need to keep polling for the azure insights to find the data I want so I need a background job to update the table.

Describe the solution you'd like
Make it possible to output the trace as part of the endpoint response. This way when I get the results I also have the telemetry so I can apply to tables along with results.

Describe alternatives you've considered
I have created a background worker to keep polling the azure insights for the prompt event which adds unnecessary complexity. It would be much easier if I can get the data immediately.

Additional context
This is a deployed endpoint.

[zhengfeiwang]

Hi @adampolak-vertex , thanks for reaching out!

Based on my understanding of trace and open telemetry, when the endpoint returns response, the corresponding trace/span is not close yet, so it shall not be carried to the endpoint response. Besides, as deployed endpoint does not expose any other API to read, so it can be difficult to make you access the trace data.

For your scenario, is it possible for you to host a separate server somewhere else, and it is accessible for you? If so, I think you can host a prompt flow service there, and pointing your deployed endpoint to export traces to that compute; then you are able to access the trace data freely - feel free to share your thought on this solution, and I can provide more information if it works for you; otherwise, I can sync with my team and brainstorm some other possible solution(s).

[wxpjimmy]

Hi @adampolak-vertex, instead of sending the trace back as part of the response, is it possible that you implement your own trace exporter which directly send the trace to your database? This is the suggested solution for exporting OTel traces.

[adampolak-vertex]

Hi @adampolak-vertex , thanks for reaching out!

Based on my understanding of trace and open telemetry, when the endpoint returns response, the corresponding trace/span is not close yet, so it shall not be carried to the endpoint response. Besides, as deployed endpoint does not expose any other API to read, so it can be difficult to make you access the trace data.

For your scenario, is it possible for you to host a separate server somewhere else, and it is accessible for you? If so, I think you can host a prompt flow service there, and pointing your deployed endpoint to export traces to that compute; then you are able to access the trace data freely - feel free to share your thought on this solution, and I can provide more information if it works for you; otherwise, I can sync with my team and brainstorm some other possible solution(s).

Thank you for your quick response.

I am currently using the Azure UI to deploy as that is a direction our team wants to go for ease of use to other stakeholders.

How can I point a trace to a different server from my endpoint?

In returning it with the api I did not mean a new route, but rather another key it could return.

So it would be: output.answers and output.trace, so a single response would return both as different keys.

[adampolak-vertex]

Hi @adampolak-vertex, instead of sending the trace back as part of the response, is it possible that you implement your own trace exporter which directly send the trace to your database? This is the suggested solution for exporting OTel traces.

How would I do this using the Azure UI? Is that possible or does it need to not be deployed through azure portal or does that not make a difference?

I would be happy to add a trace to the beginning of my promptflow and then export it out at the end along with the output, but how would that work?

[zhengfeiwang]

To point traces to different server, you can use OTel environment variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to do so, PF will honor this setting and export traces to that endpoint - this requires you to host a trace server on that place.

For returning in API, I think the answer is still no, as I mentioned, when API returns the response, the trace is still unavailable at that time.

BTW, could you please describe more on the database/table where you store the traces? I agree with Xiaopeng's suggestion that you can write your own exporter to export traces to your database, as it's the recommended way from OTel - this is something written in your code.

[adampolak-vertex]

To point traces to different server, you can use OTel environment variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to do so, PF will honor this setting and export traces to that endpoint - this requires you to host a trace server on that place.

For returning in API, I think the answer is still no, as I mentioned, when API returns the response, the trace is still unavailable at that time.

BTW, could you please describe more on the database/table where you store the traces? I agree with Xiaopeng's suggestion that you can write your own exporter to export traces to your database, as it's the recommended way from OTel - this is something written in your code.

How would I set the environment variable for PromptFlow if I deployed as an endpoint using the Azure Portal UI?

I know OpenAI responds with tokens used per call, so at the least could the tokens be returned as part of the API response?

[zhengfeiwang]

I don't see place to set environment variable in portal UI, not sure it is supported from UI; however, is it possible to set it in your flow? I understand those code are controlable from your side?

[zhengfeiwang]

Meanwhile, token should be available, but it's different from tracing. Providing token usage in endpoint response should be a separate topic from prompt flow; tagging Xiaopeng @wxpjimmy to add some comments on this.

[adampolak-vertex]

Meanwhile, token should be available, but it's different from tracing. Providing token usage in endpoint response should be a separate topic from prompt flow; tagging Xiaopeng @wxpjimmy to add some comments on this.

That would be a good first step. Because

• time spent (should be instant)
• prompt tokens (should be instant)
• inputs and outputs of every module (should be instant)

None of these are post-processing values so it would be great to have an option "include logs in response.logs and the result is in response.output.

[wxpjimmy]

If you want to get token, you can write your own llm tool, in that tool you can parse the response and return both output message and token info as output, PF doesn't have any plan to provide these info in the response. cc @zhengfeiwang

[wxpjimmy]

@adampolak-vertex what you need is still the trace data, please follow the suggested practice to define your own exporter to collect those info, PF doesn't have plan to provide that info in the response from infra perspective. cc @zhengfeiwang

[github-actions]

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!