-
Notifications
You must be signed in to change notification settings - Fork 15
Description
When using engine=iceberg, is deploying a dedicated Iceberg compactor node still required?
Easy to Miss Infra Requirement in Doc
If so, this requirement is currently not documented in the Iceberg docs. The only place I see a note on it is in this document - https://docs.risingwave.com/iceberg/maintenance, but it is easy to miss, and instructs the user to contact the RW team on slack. It is also filed under table maintenance, which doesn't make sense for me if deploying a compactor is a prerequisite. Also, infrastructure deployment isn't really "table maintenance", though I understand the infrastructure component serves the purpose of maintaining tables through compaction.
Consider Adding or Adjusting Documentation to Give Instructions
We can adjust the documentation to have clean, useful instructions for on-prem deployment by adding another page or editing a section in quickstart or https://docs.risingwave.com/iceberg/ov-internal or both.
can we add a section that explains the following for on-prem users:
- Explains the need for a dedicated Iceberg compactor
- Provides guidance on configuring Kubernetes (Helm and Operator) to deploy it. Ideally including a code sample for yaml syntax that is used to do the deployment.
- Includes recommended sizing guidelines - teach the user how to determine how many iceberg compactors they will need and at what size
This will prevent misconfiguration and clarify deployment requirements for on-prem users.
Let me know if we still require a separate iceberg compactor and what you need for us to kick off this documentation improvement!
Thanks!