Skip to content

Latest commit

 

History

History
1614 lines (1185 loc) · 47.5 KB

cluster.md

File metadata and controls

1614 lines (1185 loc) · 47.5 KB
page_title subcategory description
redpanda_cluster Resource - terraform-provider-redpanda

redpanda_cluster (Resource)

Enables the provisioning and management of Redpanda clusters on AWS and GCP. A cluster must always have a network and resource group.

Schema

Required

  • cluster_type (String) Cluster type. Type is immutable and can only be set on cluster creation.
  • connection_type (String) Cluster connection type. Private clusters are not exposed to the internet. For BYOC clusters, Private is best-practice.
  • name (String) Unique name of the cluster.
  • network_id (String) Network ID where cluster is placed.
  • resource_group_id (String) Resource group ID of the cluster.
  • throughput_tier (String) Throughput tier of the cluster.

Optional

  • allow_deletion (Boolean) Allows deletion of the cluster. Defaults to true. Should probably be set to false for production use.
  • aws_private_link (Attributes) AWS PrivateLink configuration. (see below for nested schema)
  • azure_private_link (Attributes) Azure Private Link configuration. (see below for nested schema)
  • cloud_provider (String) Cloud provider where resources are created.
  • connectivity (Attributes) Cloud provider-specific connectivity configuration. (see below for nested schema)
  • customer_managed_resources (Attributes) Customer managed resources configuration for the cluster. (see below for nested schema)
  • gcp_private_service_connect (Attributes) GCP Private Service Connect configuration. (see below for nested schema)
  • http_proxy (Attributes) HTTP Proxy properties. (see below for nested schema)
  • kafka_api (Attributes) Cluster's Kafka API properties. (see below for nested schema)
  • kafka_connect (Attributes) Kafka Connect configuration. (see below for nested schema)
  • maintenance_window_config (Attributes) Maintenance window configuration for the cluster. (see below for nested schema)
  • read_replica_cluster_ids (List of String) IDs of clusters that can create read-only topics from this cluster.
  • redpanda_version (String) Current Redpanda version of the cluster.
  • region (String) Cloud provider region. Region represents the name of the region where the cluster will be provisioned.
  • schema_registry (Attributes) Schema Registry properties. (see below for nested schema)
  • tags (Map of String) Tags placed on cloud resources. If the cloud provider is GCP and the name of a tag has the prefix "gcp.network-tag.", the tag is a network tag that will be added to the Redpanda cluster GKE nodes. Otherwise, the tag is a normal tag. For example, if the name of a tag is "gcp.network-tag.network-tag-foo", the network tag named "network-tag-foo" will be added to the Redpanda cluster GKE nodes. Note: The value of a network tag will be ignored. See the details on network tags at https://cloud.google.com/vpc/docs/add-remove-network-tags.
  • zones (List of String) Zones of the cluster. Must be valid zones within the selected region. If multiple zones are used, the cluster is a multi-AZ cluster.

Read-Only

  • cluster_api_url (String) The URL of the cluster API.
  • created_at (String) Timestamp when the cluster was created.
  • id (String) ID of the cluster. ID is an output from the Create Cluster endpoint and cannot be set by the caller.
  • prometheus (Attributes) Prometheus metrics endpoint properties. (see below for nested schema)
  • redpanda_console (Attributes) Redpanda Console properties. (see below for nested schema)
  • state (String) Current state of the cluster.
  • state_description (Attributes) Detailed state description when cluster is in a non-ready state. (see below for nested schema)

Nested Schema for aws_private_link

Required:

  • allowed_principals (List of String) The ARN of the principals that can access the Redpanda AWS PrivateLink Endpoint Service. To grant permissions to all principals, use an asterisk (*).
  • connect_console (Boolean) Whether Console is connected via PrivateLink.
  • enabled (Boolean) Whether AWS PrivateLink is enabled.

Read-Only:

Nested Schema for aws_private_link.status

Read-Only:

  • console_port (Number) Port for Redpanda Console.
  • created_at (String) When the PrivateLink service was created.
  • deleted_at (String) When the PrivateLink service was deleted.
  • kafka_api_node_base_port (Number) Base port for Kafka API nodes.
  • kafka_api_seed_port (Number) Port for Kafka API seed brokers.
  • redpanda_proxy_node_base_port (Number) Base port for HTTP proxy nodes.
  • redpanda_proxy_seed_port (Number) Port for HTTP proxy.
  • schema_registry_seed_port (Number) Port for Schema Registry.
  • service_id (String) The PrivateLink service ID.
  • service_name (String) The PrivateLink service name.
  • service_state (String) Current state of the PrivateLink service.
  • vpc_endpoint_connections (Attributes List) List of VPC endpoint connections. (see below for nested schema)

Nested Schema for aws_private_link.status.vpc_endpoint_connections

Read-Only:

  • connection_id (String) The connection ID.
  • created_at (String) When the endpoint connection was created.
  • dns_entries (Attributes List) DNS entries for the endpoint. (see below for nested schema)
  • id (String) The endpoint connection ID.
  • load_balancer_arns (List of String) ARNs of associated load balancers.
  • owner (String) Owner of the endpoint connection.
  • state (String) State of the endpoint connection.

Nested Schema for aws_private_link.status.vpc_endpoint_connections.dns_entries

Read-Only:

  • dns_name (String) The DNS name.
  • hosted_zone_id (String) The hosted zone ID.

Nested Schema for azure_private_link

Required:

  • allowed_subscriptions (List of String) The subscriptions that can access the Redpanda Azure PrivateLink Endpoint Service. To grant permissions to all principals, use an asterisk (*).
  • connect_console (Boolean) Whether Console is connected in Redpanda Azure Private Link Service.
  • enabled (Boolean) Whether Redpanda Azure Private Link Endpoint Service is enabled.

Read-Only:

Nested Schema for azure_private_link.status

Read-Only:

  • approved_subscriptions (List of String) List of approved Azure subscription IDs.
  • console_port (Number) Port for Redpanda Console.
  • created_at (String) When the Private Link service was created.
  • deleted_at (String) When the Private Link service was deleted.
  • dns_a_record (String) DNS A record for the service.
  • kafka_api_node_base_port (Number) Base port for Kafka API nodes.
  • kafka_api_seed_port (Number) Port for Kafka API seed brokers.
  • private_endpoint_connections (Attributes List) List of private endpoint connections. (see below for nested schema)
  • redpanda_proxy_node_base_port (Number) Base port for HTTP proxy nodes.
  • redpanda_proxy_seed_port (Number) Port for HTTP proxy.
  • schema_registry_seed_port (Number) Port for Schema Registry.
  • service_id (String) The Private Link service ID.
  • service_name (String) The Private Link service name.

Nested Schema for azure_private_link.status.private_endpoint_connections

Read-Only:

  • connection_id (String) ID of the connection.
  • connection_name (String) Name of the connection.
  • created_at (String) When the endpoint connection was created.
  • private_endpoint_id (String) ID of the private endpoint.
  • private_endpoint_name (String) Name of the private endpoint.
  • status (String) Status of the endpoint connection.

Nested Schema for connectivity

Optional:

Nested Schema for connectivity.gcp

Required:

  • enable_global_access (Boolean) Whether global access is enabled.

Nested Schema for customer_managed_resources

Optional:

Nested Schema for customer_managed_resources.aws

Required:

Nested Schema for customer_managed_resources.aws.agent_instance_profile

Required:

  • arn (String) ARN for the agent instance profile

Nested Schema for customer_managed_resources.aws.cloud_storage_bucket

Required:

  • arn (String) ARN for the cloud storage bucket

Nested Schema for customer_managed_resources.aws.cluster_security_group

Required:

  • arn (String) ARN for the cluster security group

Nested Schema for customer_managed_resources.aws.connectors_node_group_instance_profile

Required:

  • arn (String) ARN for the connectors node group instance profile

Nested Schema for customer_managed_resources.aws.connectors_security_group

Required:

  • arn (String) ARN for the connectors security group

Nested Schema for customer_managed_resources.aws.k8s_cluster_role

Required:

  • arn (String) ARN for the Kubernetes cluster role

Nested Schema for customer_managed_resources.aws.node_security_group

Required:

  • arn (String) ARN for the node security group

Nested Schema for customer_managed_resources.aws.permissions_boundary_policy

Required:

  • arn (String) ARN for the permissions boundary policy

Nested Schema for customer_managed_resources.aws.redpanda_agent_security_group

Required:

  • arn (String) ARN for the redpanda agent security group

Nested Schema for customer_managed_resources.aws.redpanda_node_group_instance_profile

Required:

  • arn (String) ARN for the redpanda node group instance profile

Nested Schema for customer_managed_resources.aws.redpanda_node_group_security_group

Required:

  • arn (String) ARN for the redpanda node group security group

Nested Schema for customer_managed_resources.aws.utility_node_group_instance_profile

Required:

  • arn (String) ARN for the utility node group instance profile

Nested Schema for customer_managed_resources.aws.utility_security_group

Required:

  • arn (String) ARN for the utility security group

Nested Schema for customer_managed_resources.gcp

Required:

Optional:

  • psc_nat_subnet_name (String) NAT subnet name if GCP Private Service Connect is enabled.

Nested Schema for customer_managed_resources.gcp.agent_service_account

Required:

  • email (String) GCP service account email.

Nested Schema for customer_managed_resources.gcp.connector_service_account

Required:

  • email (String) GCP service account email.

Nested Schema for customer_managed_resources.gcp.console_service_account

Required:

  • email (String) GCP service account email.

Nested Schema for customer_managed_resources.gcp.gke_service_account

Required:

  • email (String) GCP service account email.

Nested Schema for customer_managed_resources.gcp.redpanda_cluster_service_account

Required:

  • email (String) GCP service account email.

Nested Schema for customer_managed_resources.gcp.subnet

Required:

  • k8s_master_ipv4_range (String) Kubernetes Master IPv4 range, e.g. 10.0.0.0/24.
  • name (String) Subnet name.
  • secondary_ipv4_range_pods (Attributes) Secondary IPv4 range for pods. (see below for nested schema)
  • secondary_ipv4_range_services (Attributes) Secondary IPv4 range for services. (see below for nested schema)

Nested Schema for customer_managed_resources.gcp.subnet.secondary_ipv4_range_pods

Required:

  • name (String) Secondary IPv4 range name for pods.

Nested Schema for customer_managed_resources.gcp.subnet.secondary_ipv4_range_services

Required:

  • name (String) Secondary IPv4 range name for services.

Nested Schema for customer_managed_resources.gcp.tiered_storage_bucket

Required:

  • name (String) GCP storage bucket name.

Nested Schema for gcp_private_service_connect

Required:

  • consumer_accept_list (Attributes List) List of consumers that are allowed to connect to Redpanda GCP PSC (Private Service Connect) service attachment. (see below for nested schema)
  • enabled (Boolean) Whether Redpanda GCP Private Service Connect is enabled.
  • global_access_enabled (Boolean) Whether global access is enabled.

Read-Only:

Nested Schema for gcp_private_service_connect.consumer_accept_list

Required:

  • source (String) Either the GCP project number or its alphanumeric ID.

Nested Schema for gcp_private_service_connect.status

Read-Only:

  • connected_endpoints (Attributes List) List of connected endpoints. (see below for nested schema)
  • created_at (String) When the Private Service Connect service was created.
  • deleted_at (String) When the Private Service Connect service was deleted.
  • dns_a_records (List of String) DNS A records for the service.
  • kafka_api_node_base_port (Number) Base port for Kafka API nodes.
  • kafka_api_seed_port (Number) Port for Kafka API seed brokers.
  • redpanda_proxy_node_base_port (Number) Base port for HTTP proxy nodes.
  • redpanda_proxy_seed_port (Number) Port for HTTP proxy.
  • schema_registry_seed_port (Number) Port for Schema Registry.
  • seed_hostname (String) Hostname for the seed brokers.
  • service_attachment (String) The service attachment identifier.

Nested Schema for gcp_private_service_connect.status.connected_endpoints

Read-Only:

  • connection_id (String) The connection ID.
  • consumer_network (String) The consumer network.
  • endpoint (String) The endpoint address.
  • status (String) Status of the endpoint connection.

Nested Schema for http_proxy

Optional:

Read-Only:

  • url (String) The HTTP Proxy URL.

Nested Schema for http_proxy.mtls

Optional:

  • ca_certificates_pem (List of String) CA certificate in PEM format.
  • enabled (Boolean) Whether mTLS is enabled.
  • principal_mapping_rules (List of String) Principal mapping rules for mTLS authentication. See the Redpanda documentation on configuring authentication.

Nested Schema for kafka_api

Optional:

Read-Only:

  • seed_brokers (List of String) List of Kafka broker addresses.

Nested Schema for kafka_api.mtls

Optional:

  • ca_certificates_pem (List of String) CA certificate in PEM format.
  • enabled (Boolean) Whether mTLS is enabled.
  • principal_mapping_rules (List of String) Principal mapping rules for mTLS authentication. See the Redpanda documentation on configuring authentication.

Nested Schema for kafka_connect

Optional:

  • enabled (Boolean) Whether Kafka Connect is enabled.

Nested Schema for maintenance_window_config

Optional:

  • anytime (Boolean) If true, maintenance can occur at any time.
  • day_hour (Attributes) (see below for nested schema)

Read-Only:

  • unspecified (Boolean) If true, maintenance window is unspecified.

Nested Schema for maintenance_window_config.day_hour

Optional:

  • day_of_week (String) Day of week.
  • hour_of_day (Number) Hour of day.

Nested Schema for schema_registry

Optional:

Read-Only:

  • url (String) The Schema Registry URL.

Nested Schema for schema_registry.mtls

Optional:

  • ca_certificates_pem (List of String) CA certificate in PEM format.
  • enabled (Boolean) Whether mTLS is enabled.
  • principal_mapping_rules (List of String) Principal mapping rules for mTLS authentication. See the Redpanda documentation on configuring authentication.

Nested Schema for prometheus

Read-Only:

  • url (String) The Prometheus metrics endpoint URL.

Nested Schema for redpanda_console

Read-Only:

  • url (String) The Redpanda Console URL.

Nested Schema for state_description

Read-Only:

  • code (Number) Error code if cluster is in error state.
  • message (String) Detailed error message if cluster is in error state.

Usage

On AWS

provider "redpanda" {}

resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"
  cidr_block        = "10.0.0.0/20"
}

resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  tags = {
    "key" = "value"
  }
  # aws_private_link = {
  #   enabled         = true
  #   connect_console = true
  #   allowed_principals = ["arn:aws:iam::123456789024:root"]
  # }
}

variable "resource_group_name" {
  default = "testname"
}

variable "network_name" {
  default = "testname"
}

variable "cluster_name" {
  default = "testname"
}

variable "region" {
  default = "us-east-2"
}

variable "zones" {
  default = ["use2-az1", "use2-az2", "use2-az3"]
}

variable "cloud_provider" {
  default = "aws"
}

variable "throughput_tier" {
  default = "tier-1-aws-v2-arm"
}

resource "redpanda_user" "test" {
  name            = var.user_name
  password        = var.user_pw
  mechanism       = var.mechanism
  cluster_api_url = redpanda_cluster.test.cluster_api_url
}

resource "redpanda_topic" "test" {
  name               = var.topic_name
  partition_count    = var.partition_count
  replication_factor = var.replication_factor
  cluster_api_url    = redpanda_cluster.test.cluster_api_url
  allow_deletion     = true
}


resource "redpanda_acl" "test" {
  resource_type         = "TOPIC"
  resource_name         = redpanda_topic.test.name
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.test.name}"
  host                  = "*"
  operation             = "READ"
  permission_type       = "ALLOW"
  cluster_api_url       = redpanda_cluster.test.cluster_api_url
}


variable "user_name" {
  default = "test-username"
}

variable "user_pw" {
  default = "password"
}

variable "mechanism" {
  default = "scram-sha-256"
}

variable "topic_name" {
  default = "test-topic"
}

variable "partition_count" {
  default = 3
}

variable "replication_factor" {
  default = 3
}

On GCP

provider "redpanda" {}

resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"
  cidr_block        = "10.0.0.0/20"
}

resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  ## This is a reference for GCP tags
  #   tags = {
  #     "key" = "value"
  #   }
  ## This is a reference for GCP Private Service Connect
  #   gcp_private_service_connect = {
  #     enabled               = true
  #     global_access_enabled = true
  #     consumer_accept_list = [
  #       {
  #         source = "projects/123456789012"
  #       }
  #     ]
  #   }
}

variable "cluster_name" {
  default = ""
}

variable "resource_group_name" {
  default = ""
}

variable "network_name" {
  default = ""
}

variable "region" {
  default = "us-central1"
}

variable "zones" {
  default = ["us-central1-a", "us-central1-b", "us-central1-c"]
}

variable "cloud_provider" {
  default = "gcp"
}

variable "throughput_tier" {
  default = "tier-1-gcp-um4g"
}


resource "redpanda_user" "test" {
  name            = var.user_name
  password        = var.user_pw
  mechanism       = var.mechanism
  cluster_api_url = redpanda_cluster.test.cluster_api_url
}

resource "redpanda_topic" "test" {
  name               = var.topic_name
  partition_count    = var.partition_count
  replication_factor = var.replication_factor
  cluster_api_url    = redpanda_cluster.test.cluster_api_url
  allow_deletion     = true
}


resource "redpanda_acl" "test" {
  resource_type         = "TOPIC"
  resource_name         = redpanda_topic.test.name
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.test.name}"
  host                  = "*"
  operation             = "READ"
  permission_type       = "ALLOW"
  cluster_api_url       = redpanda_cluster.test.cluster_api_url
}


variable "user_name" {
  default = "test-username"
}

variable "user_pw" {
  default = "password"
}

variable "mechanism" {
  default = "scram-sha-256"
}

variable "topic_name" {
  default = "test-topic"
}

variable "partition_count" {
  default = 3
}

variable "replication_factor" {
  default = 3
}

On Azure

provider "redpanda" {}

resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"
  cidr_block        = "10.0.0.0/20"
}

resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  tags = {
    "key" = "value"
  }
#   azure_private_link = {
#     enabled         = true
#     connect_console = true
#     allowed_subscriptions = ["12345678-1234-1234-1234-123456789012"]
#   }
}

variable "resource_group_name" {
  default = "testname"
}

variable "network_name" {
  default = "testname"
}

variable "cluster_name" {
  default = "testname"
}

variable "cloud_provider" {
  default = "azure"
}

variable "region" {
  default = "eastus"
}

variable "zones" {
  default = ["eastus-az1", "eastus-az2", "eastus-az3"]
}

variable "throughput_tier" {
  default = "tier-1-azure-v3-x86"
}


resource "redpanda_user" "test" {
  name            = var.user_name
  password        = var.user_pw
  mechanism       = var.mechanism
  cluster_api_url = redpanda_cluster.test.cluster_api_url
}

resource "redpanda_topic" "test" {
  name               = var.topic_name
  partition_count    = var.partition_count
  replication_factor = var.replication_factor
  cluster_api_url    = redpanda_cluster.test.cluster_api_url
  allow_deletion     = true
}


resource "redpanda_acl" "test" {
  resource_type         = "TOPIC"
  resource_name         = redpanda_topic.test.name
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.test.name}"
  host                  = "*"
  operation             = "READ"
  permission_type       = "ALLOW"
  cluster_api_url       = redpanda_cluster.test.cluster_api_url
}


variable "user_name" {
  default = "test-username"
}

variable "user_pw" {
  default = "password"
}

variable "mechanism" {
  default = "scram-sha-256"
}

variable "topic_name" {
  default = "test-topic"
}

variable "partition_count" {
  default = 3
}

variable "replication_factor" {
  default = 3
}

BYOC

This configuration of cluster allows the end user to provide access to their cloud account to the provider so that it can create the necessary infrastructure in their account rather than in Redpanda's Cloud.

Additional Requirements

To build a BYOC cluster you must provide credentials that enable the provider to authenticate to the relevant cloud provider. How this works will depend on which cloud provider you are using.

AWS BYOC

To create a BYOC AWS cluster you must provide an AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The account must have fairly wide ranging permissions to create the necessary infrastructure.

provider "redpanda" {}

resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "byoc"
  cidr_block        = "10.0.0.0/20"
}

resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = redpanda_network.test.cloud_provider
  region            = redpanda_network.test.region
  cluster_type      = redpanda_network.test.cluster_type
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  tags = {
    "key" = "value"
  }
  # aws_private_link = {
  #   enabled         = true
  #   connect_console = true
  #   allowed_principals = ["arn:aws:iam::123456789024:root"]
  # }
}

variable "resource_group_name" {
  default = "testname"
}

variable "network_name" {
  default = "testname"
}

variable "cluster_name" {
  default = "testname"
}

variable "region" {
  default = "us-east-2"
}

variable "zones" {
  default = ["use2-az1", "use2-az2", "use2-az3"]
}

variable "cloud_provider" {
  default = "aws"
}

variable "throughput_tier" {
  default = "tier-1-aws-v2-x86"
}

resource "redpanda_user" "test" {
  name            = var.user_name
  password        = var.user_pw
  mechanism       = var.mechanism
  cluster_api_url = redpanda_cluster.test.cluster_api_url
}

resource "redpanda_topic" "test" {
  name               = var.topic_name
  partition_count    = var.partition_count
  replication_factor = var.replication_factor
  cluster_api_url    = redpanda_cluster.test.cluster_api_url
  allow_deletion     = true
}


resource "redpanda_acl" "test" {
  resource_type         = "TOPIC"
  resource_name         = redpanda_topic.test.name
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.test.name}"
  host                  = "*"
  operation             = "READ"
  permission_type       = "ALLOW"
  cluster_api_url       = redpanda_cluster.test.cluster_api_url
}


variable "user_name" {
  default = "test-username"
}

variable "user_pw" {
  default = "password"
}

variable "mechanism" {
  default = "scram-sha-256"
}

variable "topic_name" {
  default = "test-topic"
}

variable "partition_count" {
  default = 3
}

variable "replication_factor" {
  default = 3
}

GCP BYOC

To create a GCP BYOC cluster you must provide a GCP_PROJECT_ID and GOOGLE_CREDENTIALS. We also accept the credentials encoded in base64 format if you use GOOGLE_CREDENTIALS_BASE64. The account must have fairly wide ranging permissions to create the necessary infrastructure.

provider "redpanda" {}

resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "byoc"
  cidr_block        = "10.0.0.0/20"
}

resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = redpanda_network.test.cloud_provider
  region            = redpanda_network.test.region
  cluster_type      = redpanda_network.test.cluster_type
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  ## This is a reference for GCP tags
  #   tags = {
  #     "key" = "value"
  #   }
  ## This is a reference for GCP Private Service Connect
  #   gcp_private_service_connect = {
  #     enabled               = true
  #     global_access_enabled = true
  #     consumer_accept_list = [
  #       {
  #         source = "projects/123456789012"
  #       }
  #     ]
  #   }
}

variable "cluster_name" {
  default = ""
}

variable "resource_group_name" {
  default = ""
}

variable "network_name" {
  default = ""
}

variable "region" {
  default = "us-central1"
}

variable "zones" {
  default = ["us-central1-a", "us-central1-b", "us-central1-c"]
}

variable "cloud_provider" {
  default = "gcp"
}

variable "throughput_tier" {
  default = "tier-1-gcp-um4g"
}


resource "redpanda_user" "test" {
  name            = var.user_name
  password        = var.user_pw
  mechanism       = var.mechanism
  cluster_api_url = redpanda_cluster.test.cluster_api_url
}

resource "redpanda_topic" "test" {
  name               = var.topic_name
  partition_count    = var.partition_count
  replication_factor = var.replication_factor
  cluster_api_url    = redpanda_cluster.test.cluster_api_url
  allow_deletion     = true
}


resource "redpanda_acl" "test" {
  resource_type         = "TOPIC"
  resource_name         = redpanda_topic.test.name
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.test.name}"
  host                  = "*"
  operation             = "READ"
  permission_type       = "ALLOW"
  cluster_api_url       = redpanda_cluster.test.cluster_api_url
}


variable "user_name" {
  default = "test-username"
}

variable "user_pw" {
  default = "password"
}

variable "mechanism" {
  default = "scram-sha-256"
}

variable "topic_name" {
  default = "test-topic"
}

variable "partition_count" {
  default = 3
}

variable "replication_factor" {
  default = 3
}

Azure BYOC

To create a BYOC Azure cluster you must provide Azure credentials, be logged in to the Azure CLI, or specify an Azure authentication method. This provider supports the same authentication methods and environment variables as the official AzureRM provider. For example, to use a service principal and client certificate, you can pass the environment variables ARM_CLIENT_ID, ARM_CLIENT_SECRET, ARM_TENANT_ID, and ARM_SUBSCRIPTION_ID.

The account must have fairly wide ranging permissions to create the necessary infrastructure.

provider "redpanda" {}

resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "byoc"
  cidr_block        = "10.0.0.0/20"
}

resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = redpanda_network.test.cloud_provider
  region            = redpanda_network.test.region
  cluster_type      = redpanda_network.test.cluster_type
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  tags = {
    "key" = "value"
  }
  # azure_private_link = {
  #   enabled         = true
  #   connect_console = true
  #   allowed_subscriptions = ["12345678-1234-1234-1234-123456789012"]
  # }
}

variable "resource_group_name" {
  default = "testname"
}

variable "network_name" {
  default = "testname"
}

variable "cluster_name" {
  default = "testname"
}

variable "cloud_provider" {
  default = "azure"
}

variable "region" {
  default = "eastus"
}

variable "zones" {
  default = ["eastus-az1", "eastus-az2", "eastus-az3"]
}

variable "throughput_tier" {
  default = "tier-1-azure-v3-x86"
}


resource "redpanda_user" "test" {
  name            = var.user_name
  password        = var.user_pw
  mechanism       = var.mechanism
  cluster_api_url = redpanda_cluster.test.cluster_api_url
}

resource "redpanda_topic" "test" {
  name               = var.topic_name
  partition_count    = var.partition_count
  replication_factor = var.replication_factor
  cluster_api_url    = redpanda_cluster.test.cluster_api_url
  allow_deletion     = true
}


resource "redpanda_acl" "test" {
  resource_type         = "TOPIC"
  resource_name         = redpanda_topic.test.name
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.test.name}"
  host                  = "*"
  operation             = "READ"
  permission_type       = "ALLOW"
  cluster_api_url       = redpanda_cluster.test.cluster_api_url
}


variable "user_name" {
  default = "test-username"
}

variable "user_pw" {
  default = "password"
}

variable "mechanism" {
  default = "scram-sha-256"
}

variable "topic_name" {
  default = "test-topic"
}

variable "partition_count" {
  default = 3
}

variable "replication_factor" {
  default = 3
}

BYOVPC

This accepts a network and other elements created by the end user inside their cloud provider account (currently limited to AWS) and builds a Redpanda Cluster inside it.

There is a module provided for convenience of the end user here that handles the necessary setup. It contains outputs for the inputs the provider requires.

AWS BYOVPC

Has the same requirements as the AWS BYOC Cluster in addition to ARNs for numerous resources that the end user must create.

provider "redpanda" {}

provider aws {
    region = var.region
}

module "redpanda_byovpc" {
  source = "redpanda-data/redpanda-byovpc/aws"

  region = var.region
  public_subnet_cidrs = [
    "10.0.16.0/20", "10.0.32.0/20", "10.0.48.0/20"
  ]
}


resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "byoc"
  customer_managed_resources = {
    aws = {
      management_bucket = {
        arn = redpanda_byovpc.management_bucket_arn
      }
      dynamodb_table = {
        arn = red
      }
      vpc = {
        arn = redpanda_byovpc.vpc_arn
      }
      private_subnets = {
        arns = redpanda_byovpc.private_subnet_arns
      }
    }
  }
}

resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = redpanda_network.test.cloud_provider
  region            = redpanda_network.test.region
  cluster_type      = redpanda_network.test.cluster_type
  connection_type   = "private"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  tags = {
    "key" = "value"
  }
  customer_managed_resources = {
    aws = {
      aws_permissions_boundary_policy_arn = {
        arn = redpanda_byovpc.permissions_boundary_policy_arn
      }
      agent_instance_profile = {
        arn = redpanda_byovpc.agent_instance_profile_arn
      }
      connectors_node_group_instance_profile = {
        arn = redpanda_byovpc.connectors_node_group_instance_profile_arn
      }
      utility_node_group_instance_profile = {
        arn = redpanda_byovpc.utility_node_group_instance_profile_arn
      }
      redpanda_node_group_instance_profile = {
        arn = redpanda_byovpc.redpanda_node_group_instance_profile_arn
      }
      k8s_cluster_role = {
        arn = redpanda_byovpc.k8s_cluster_role_arn
      }
      redpanda_agent_security_group = {
        arn = redpanda_byovpc.redpanda_agent_security_group_arn
      }
      connectors_security_group = {
        arn = redpanda_byovpc.connectors_security_group_arn
      }
      redpanda_node_group_security_group = {
        arn = redpanda_byovpc.redpanda_node_group_security_group_arn
      }
      utility_security_group = {
        arn = redpanda_byovpc.utility_security_group_arn
      }
      cluster_security_group = {
        arn = redpanda_byovpc.cluster_security_group_arn
      }
      node_security_group = {
        arn = redpanda_byovpc.node_security_group_arn
      }
      cloud_storage_bucket = {
        arn = redpanda_byovpc.cloud_storage_bucket_arn
      }
      permissions_boundary_policy = {
        arn = redpanda_byovpc.permissions_boundary_policy_arn
      }
    }
  }
}

resource "redpanda_user" "test" {
  name            = var.user_name
  password        = var.user_pw
  mechanism       = var.mechanism
  cluster_api_url = redpanda_cluster.test.cluster_api_url
}

resource "redpanda_topic" "test" {
  name               = var.topic_name
  partition_count    = var.partition_count
  replication_factor = var.replication_factor
  cluster_api_url    = redpanda_cluster.test.cluster_api_url
  allow_deletion     = true
}

resource "redpanda_acl" "test" {
  resource_type         = "TOPIC"
  resource_name         = redpanda_topic.test.name
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.test.name}"
  host                  = "*"
  operation            = "READ"
  permission_type      = "ALLOW"
  cluster_api_url      = redpanda_cluster.test.cluster_api_url
}


# Existing variables from original configuration
variable "resource_group_name" {
  default = "testname"
}

variable "network_name" {
  default = "testname"
}

variable "cluster_name" {
  default = "testname"
}

variable "region" {
  default = "us-east-2"
}

variable "zones" {
  default = ["use2-az1", "use2-az2", "use2-az3"]
}

variable "cloud_provider" {
  default = "aws"
}

variable "throughput_tier" {
  default = "tier-1-aws-v2-x86"
}

variable "user_name" {
  default = "test-username"
}

variable "user_pw" {
  default = "password"
}

variable "mechanism" {
  default = "scram-sha-256"
}

variable "topic_name" {
  default = "test-topic"
}

variable "partition_count" {
  default = 3
}

variable "replication_factor" {
  default = 3
}

variable "aws_access_key" {
  type = string
}

variable "aws_secret_key" {
  type = string
}

Limitations

We are not currently able to support GCP or Azure BYOVPC clusters.

Example Usage of a data source BYOC to manage users and ACLs

provider "redpanda" {}

variable "cluster_id" {
  default = ""
}

data "redpanda_cluster" "test" {
  id = var.cluster_id
}

resource "redpanda_topic" "test" {
  name               = var.topic_name
  partition_count    = var.partition_count
  replication_factor = var.replication_factor
  cluster_api_url    = data.redpanda_cluster.test.cluster_api_url
  allow_deletion     = true
  configuration      = var.topic_config
}

resource "redpanda_user" "test" {
  name            = var.user_name
  password        = var.user_pw
  mechanism       = var.mechanism
  cluster_api_url = data.redpanda_cluster.test.cluster_api_url
}

resource "redpanda_acl" "test" {
  resource_type         = "CLUSTER"
  resource_name         = "kafka-cluster"
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.test.name}"
  host                  = "*"
  operation             = "ALTER"
  permission_type       = "ALLOW"
  cluster_api_url       = data.redpanda_cluster.test.cluster_api_url
}

variable "topic_config" {
  default = {
    "cleanup.policy"   = "compact"
    "flush.ms"         = 100
    "compression.type" = "snappy"
  }
}

variable "user_name" {
  default = "data-test-username"
}

variable "user_pw" {
  default = "password"
}

variable "mechanism" {
  default = "scram-sha-256"
}

variable "topic_name" {
  default = "data-test-topic"
}

variable "partition_count" {
  default = 3
}

variable "replication_factor" {
  default = 3
}

Import

terraform import resource.redpanda_cluster.example clusterId