[Bug]: StreamingNode Memory keeps increasing

[cxyz1]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.6.8
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):2.6.5
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

There's a 10-second scheduled task upserting data to Mulvus. The total number of upserts within those 10 seconds is approximately 10,000, in batches of 2,000. Data is being written normally, but the streaming memory usage keeps increasing, and the segments show no data. I've tried manually flushing, but it didn't work.

Expected Behavior

No response

Steps To Reproduce

Milvus Log

No response

Anything else?

No response

[cxyz1]

It has been upgraded to 2.6.8, but the problem doesn't seem to be resolved.

[chyezh]

/assign @chyezh

[yanliang567]

@cxyz1 it does not reproduce to me. I upsert 50 batches with 2000 entities for each batch, the memory usage of streaming node increased for 1 few mins, and went down after upsert completed.

Please share more details about how to reprodce this issue: the code snippet, milvus metrics and milvus logs for all pods.

/assign @cxyz1
/unassign

[cxyz1]

@yanliang567 I think I know where the problem lies. It's probably because I used partial_update, which caused a large accumulation of Delta Log during upsert. Switching to a full update has fixed it. However, I originally intended for a partial update, which is also a problem.