Selective deletion of traces with associated observations older than 7 days by trace name and project

[Haoping-Xiao]

Describe your question

We are currently storing a large number of traces in Langfuse for logging purposes. The growing volume of data has made queries on our self-hosted ClickHouse cluster extremely slow. I would like to selectively delete traces, along with their associated observations, that are older than 7 days, filtered by trace name and project. At the same time, I need to retain certain traces for collecting training data. What is the recommended approach to achieve this?

Langfuse Cloud or Self-Hosted?

Self-Hosted

If self-hosted, what version are you running?

3.132.0

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

For selective deletion of traces older than 7 days filtered by trace name and project in your self-hosted Langfuse v3.132.0 deployment, you have several options:

Deletion Methods

Delete by Query (UI)(1)
You can configure your desired filter in the traces list (including trace name and time range), select all items matching the filter, and choose Delete in the Actions dropdown(1). This will delete traces along with related entities like scores and observations across all data storages(1).

API-Based Deletion(1)
For programmatic deletion, you can use:

• Single trace deletion: DELETE /api/public/traces/{traceId}(1)
• Batch deletion: DELETE /api/public/traces(1)

Important Limitations

Deletion is Not Instantaneous(1)
Trace deletions do not happen instantly(1). Removing records from the data warehouse is resource-intensive, so Langfuse rate limits deletions(1). Trace data is typically deleted within 15 minutes of the delete call, and there is no confirmation(1). You'll need to query again to verify deletion(1).

Data Retention Feature(2)
For automated cleanup, Langfuse offers a Data Retention feature that controls how long event data is stored(2). However, this feature is only available in Pro, Enterprise, and Self-Hosted Enterprise Edition plans—not in the open-source self-hosted version(2). The OSS version stores event data indefinitely by default(2).

Alternative Approaches for OSS Users

According to community discussions, for self-hosted OSS users without access to automated data retention, you would need to implement your own solution(3). This could involve writing SQL scripts or cron jobs to remove records older than 7 days directly from your database, ensuring you handle related data (like observations and scores) to maintain referential integrity(3).

Performance Considerations

If slow queries are your primary concern, the documentation also recommends reviewing your ClickHouse instance size and ensuring time filters are applied, as tracing data is indexed by projectId and time(4). Adding filter conditions on these should significantly improve performance(4). For larger deployments, at least 16 GiB of memory is recommended for ClickHouse(4).

Note: There is no built-in feature in the OSS version to selectively retain certain traces while automatically deleting others based on custom criteria like trace name. You would need to implement custom logic using the deletion APIs or database scripts.

📚 Sources:

1. Data Deletion
2. Data Retention
3. Deleting traces and events older than 30 days in Langfuse self-hosted OSS
4. Scaling

---

Have another question? Just tag @inkeep.

[Lotte-Verheyden]

Hi @Haoping-Xiao, was Inkeep's answer helpful for you?