Retrieving scores updated in the UI via the API

[chriswmann]

I'm having difficulty retrieving a score (annotation) which was updated via the Langfuse web app using the python SDK.

(I created the score, and score config, for test purposes. It's the only score for the given trace, of the given config and in the given queue.)

The score was updated less than a day after it was created. It's a categorical score. Both the score and the comment were altered.

However, when I query the API (using the python SDK's score_v_2 method without any filters) the returned data hasn't changed - a single score is returned with the original value, timestamp, updated at, etc. are unchanged.

I've found quite a lot of information about the behaviour and mutability of the score data when updated via the API but nothing about this specific situation.

At the moment, I have one score showing on the UI but a different one being retrieved via the API.

Any advice would be much appreciated.

[inkeep-ai-assistant[bot]]

From November 20, 2025, I can help you understand the issue you're experiencing with scores updated via the Langfuse UI not reflecting in API queries.

When you update a score through the Langfuse web UI, you need to understand how score updates work with idempotency keys. The documentation explains that by default, Langfuse allows multiple scores of the same name on the same trace(1). To prevent duplicate scores or update an existing score, you need to create an idempotency key on the score and add this as an id (JS/TS) / score_id (Python) when creating the score, for example <trace_id>-<score_name>(1).

When you created the score initially (either via SDK or UI), if no explicit score_id was provided, Langfuse would have auto-generated one. When you then updated the score through the UI, the UI likely updated that specific score instance using its internal ID. However, when you query via the SDK's score_v_2 method without filters, you may be retrieving a different score instance if:

1. Multiple score instances exist for the same trace and score name
2. The query is returning a different score instance than the one you updated in the UI

To resolve this, you should:

1. Check if multiple scores exist for that trace with the same name by examining all scores for that trace
2. When querying, use more specific filters to target the exact score you updated (such as filtering by the score's unique ID if you know it)
3. Consider using explicit score_id values when creating scores to ensure you can reliably update and retrieve the same score instance(1)

The documentation also notes that there can be race conditions with score ingestion, particularly in self-hosted environments(2), though this typically affects score creation rather than updates.

📚 Sources:

1. Custom Scores
2. bug: Scoring is not working with langgraph integration and infinitely hanging flush call in latest python SDK (v3.2.7)

---

Have another question? Just tag @inkeep.

[chriswmann]

I have already tried setting score_ids with the score's ID and it returns the same, single score that I get when I don't specify the ID.

I assume searching without specifying the score ID and seeing only the one score returned eliminates the possibility that there are multiple scores.

I don't think I can use score_id when creating a score since it's being done via the Langfuse app.

When I change the score on the app, the "Score data saved" animates, suggesting that the new value is saved (and indeed the new value is what is displayed when I go to view the score again).

[jannikmaierhoefer]

Hi @chriswmann, thanks. Are you self-hosting Langfuse? If yes, which version are you running?

[chriswmann]

Hi @jannikmaierhoefer ! No, we're using the hosted service.

[jannikmaierhoefer]

@chriswmann, thanks. Could you send me a link to the score? Happy to have a look at what might cause this.

[chriswmann]

Thanks for looking into this @jannikmaierhoefer . Here's the link: https://cloud.langfuse.com/project/cmhj1h2w901jqad07y3voipxp/annotation-queues/cmhn7lb4c009jad079089c910/items/cmi653k3z007bad081cl1jrpe?singleItem=true

[chriswmann]

Hi @jannikmaierhoefer , just to let you know, the updated score is now showing via the API. Not sure if this is something you've done or just a really long delay.

[Steffen911]

@chriswmann Thank you for the report! Updates to scores are processed on our backend like a "new" insert and then eventually deduplicated in the database and also deduplicated on reads. It seems that there was a bug where we took a random version of the score for the public API result. This will be fixed in #10634 with a cloud release happening later today and an OSS release following soon after.

[chriswmann]

Hi @jannikmaierhoefer. Apologies but can you check something else for me? I can see the updated score now has the new value and comment but the updated_at timestamp still matches created_at. This makes it cumbersome to determine if a score has been updated or to retrieve any new or updated scores after from_timestamp.

[chriswmann]

I forgot from_timestamp filters on created_at. I think that means the only way we can programmatically detect if a score has changed via the python SDK is to retrieve them all and compare the values and comments (or if updated_at is updated, then checking if updated_at <> created_at). Is my understanding here correct?

[Steffen911]

@chriswmann What's the usual time between creation and update in your case? In some cases, we process both together which produces a single row in ClickHouse. If you need to reliably detect whether a score was updated, I would recommend to set a corresponding metadata property, as we don't provide guarantees that updates are processed independently (although that mostly affects <1min updates).

[chriswmann]

Hi @Steffen911 . Thanks very much for the clarification, that's really useful. The updates could be anywhere from seconds to days but would typically be in the minutes - hours range. We'll use the metadata property approach. I appreciate the help from you and @jannikmaierhoefer.

[Steffen911]

@chriswmann Happy to hear it! Let us know if you need anything else.