Skip to content

Scale-in nodes failover compact the StreamSetObjects #706

Open
@superhx

Description

@superhx

Who is this for and what problem do they have today?

Why is solving this problem impactful?

Compact the StreamSetObjects of the scale-in nodes with splitThreshold=1MiB :

  • Most of the data: Most of the data is split into StreamObjects to achieve independent TTL and better data cohesiveness.
  • Remaining 5000 partitions * 1MB: The remaining data will be periodically scanned and eventually expire through StreamControlManager#cleanupScaleInNodes.

Additional notes

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions