Skip to content

Add prerequisite documentation to deploy a dedicated Iceberg compactor when using internal iceberg (engine=iceberg) #1010

@rmbeddor14

Description

@rmbeddor14

When using engine=iceberg, is deploying a dedicated Iceberg compactor node still required?

Easy to Miss Infra Requirement in Doc
If so, this requirement is currently not documented in the Iceberg docs. The only place I see a note on it is in this document - https://docs.risingwave.com/iceberg/maintenance, but it is easy to miss, and instructs the user to contact the RW team on slack. It is also filed under table maintenance, which doesn't make sense for me if deploying a compactor is a prerequisite. Also, infrastructure deployment isn't really "table maintenance", though I understand the infrastructure component serves the purpose of maintaining tables through compaction.

Consider Adding or Adjusting Documentation to Give Instructions
We can adjust the documentation to have clean, useful instructions for on-prem deployment by adding another page or editing a section in quickstart or https://docs.risingwave.com/iceberg/ov-internal or both.

can we add a section that explains the following for on-prem users:

  • Explains the need for a dedicated Iceberg compactor
  • Provides guidance on configuring Kubernetes (Helm and Operator) to deploy it. Ideally including a code sample for yaml syntax that is used to do the deployment.
  • Includes recommended sizing guidelines - teach the user how to determine how many iceberg compactors they will need and at what size

This will prevent misconfiguration and clarify deployment requirements for on-prem users.

Let me know if we still require a separate iceberg compactor and what you need for us to kick off this documentation improvement!

Thanks!

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions