Controlling the cleanup of DBStream

[zhongmiaoretell]

Hi folks, wondering if you guys can provide some functions that can allow controlling when cleanup is triggered for DBStream.
I can kinda achieve it by changing cleanup_internal and monitor the timestamp, but it feels very hacky and can easily break.

[MaxHalford]

Hey! Honestly we're open to proposals here. Do you have an idea in mind? Have you DBStream in production for instance? If so, what happened?

[zhongmiaoretell]

Hi Max, Actually if you could take a look and see if DBStream is suitable for our use case it would be great: 1. We have questions from customers arriving and we want to be able to quickly find out the top questions within a time period. 2. So whenever a question arrives, we add it to the data set and find its clusters using DBStream, and record the cluster for that question in ClickHouse (an analytical DB). 3. To find top questions within a time period, we find the cluster with the highest count within that time. We are making the following adaptations for DBStream to fit our scenario: 1. We do not have a lot of data arriving. So instead of running DBStream online, after adding a data point, we will persist the model in S3 and then load it when another data point arrives. 2. After cleanup, the cluster assignment of some data points will change. To ensure the top question count still accurate, we need to update the cluster assignments in ClickHouse. The second problem is where controllable cleanup would be very helpful, so that we can trigger the cleanup whenever needed and update the cluster assignments. Also, we want the cleanup to be triggered more frequently in the beginning, but much slower later when the model already has lots of data. Lemme know if the above makes sense, thank you!

…

On Wed, Sep 17, 2025 at 8:41 AM Max Halford ***@***.***> wrote: *MaxHalford* left a comment (online-ml/river#1709) <#1709 (comment)> Hey! Honestly we're open to proposals here. Do you have an idea in mind? Have you DBStream in production for instance? If so, what happened? — Reply to this email directly, view it on GitHub <#1709 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BQ7O7JL4XN4VS3RLND6YFT33TF6MHAVCNFSM6AAAAACGTQU6VGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMBTGU4DOMRZGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[zhongmiaoretell]

Hi Max, what do you think?

[MaxHalford]

I'm sorry I haven't dedicated time to answering questions in last months. Do you still need an answer from me?