Skip to content

Latest commit

 

History

History
192 lines (135 loc) · 8.35 KB

destination_aws_datalake.md

File metadata and controls

192 lines (135 loc) · 8.35 KB
page_title subcategory description
airbyte_destination_aws_datalake Resource - terraform-provider-airbyte
DestinationAwsDatalake Resource

airbyte_destination_aws_datalake (Resource)

DestinationAwsDatalake Resource

Example Usage

resource "airbyte_destination_aws_datalake" "my_destination_awsdatalake" {
  configuration = {
    aws_account_id = "111111111111"
    bucket_name    = "...my_bucket_name..."
    bucket_prefix  = "...my_bucket_prefix..."
    credentials = {
      iam_role = {
        role_arn = "...my_role_arn..."
      }
    }
    format = {
      json_lines_newline_delimited_json = {
        compression_codec = "UNCOMPRESSED"
        format_type       = "JSONL"
      }
      parquet_columnar_storage = {
        compression_codec = "GZIP"
        format_type       = "Parquet"
      }
    }
    glue_catalog_float_as_decimal             = true
    lakeformation_database_default_tag_key    = "pii_level"
    lakeformation_database_default_tag_values = "private,public"
    lakeformation_database_name               = "...my_lakeformation_database_name..."
    lakeformation_governed_tables             = true
    partitioning                              = "DAY"
    region                                    = "ap-southeast-4"
  }
  definition_id = "aa9c2d01-84b7-4474-ba6f-e45dbbc28cdd"
  name          = "...my_name..."
  workspace_id  = "3df68150-9956-454d-8144-1645f409cdd1"
}

Schema

Required

  • configuration (Attributes) (see below for nested schema)
  • name (String) Name of the destination e.g. dev-mysql-instance.
  • workspace_id (String)

Optional

  • definition_id (String) The UUID of the connector definition. One of configuration.destinationType or definitionId must be provided. Requires replacement if changed.

Read-Only

  • created_at (Number)
  • destination_id (String)
  • destination_type (String)
  • resource_allocation (Attributes) actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level. (see below for nested schema)

Nested Schema for configuration

Required:

  • bucket_name (String) The name of the S3 bucket. Read more here.
  • credentials (Attributes) Choose How to Authenticate to AWS. (see below for nested schema)
  • lakeformation_database_name (String) The default database this destination will use to create tables in per stream. Can be changed per connection by customizing the namespace.

Optional:

  • aws_account_id (String) target aws account id
  • bucket_prefix (String) S3 prefix
  • format (Attributes) Format of the data output. (see below for nested schema)
  • glue_catalog_float_as_decimal (Boolean) Cast float/double as decimal(38,18). This can help achieve higher accuracy and represent numbers correctly as received from the source. Default: false
  • lakeformation_database_default_tag_key (String) Add a default tag key to databases created by this destination
  • lakeformation_database_default_tag_values (String) Add default values for the Tag Key to databases created by this destination. Comma separate for multiple values.
  • lakeformation_governed_tables (Boolean) Whether to create tables as LF governed tables. Default: false
  • partitioning (String) Partition data by cursor fields when a cursor field is a date. Default: "NO PARTITIONING"; must be one of ["NO PARTITIONING", "DATE", "YEAR", "MONTH", "DAY", "YEAR/MONTH", "YEAR/MONTH/DAY"]
  • region (String) The region of the S3 bucket. See here for all region codes. Default: ""; must be one of ["", "af-south-1", "ap-east-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-south-1", "ap-south-2", "ap-southeast-1", "ap-southeast-2", "ap-southeast-3", "ap-southeast-4", "ca-central-1", "ca-west-1", "cn-north-1", "cn-northwest-1", "eu-central-1", "eu-central-2", "eu-north-1", "eu-south-1", "eu-south-2", "eu-west-1", "eu-west-2", "eu-west-3", "il-central-1", "me-central-1", "me-south-1", "sa-east-1", "us-east-1", "us-east-2", "us-gov-east-1", "us-gov-west-1", "us-west-1", "us-west-2"]

Nested Schema for configuration.credentials

Optional:

Nested Schema for configuration.credentials.iam_role

Required:

  • role_arn (String, Sensitive) Will assume this role to write data to s3

Nested Schema for configuration.credentials.iam_user

Required:

  • aws_access_key_id (String, Sensitive) AWS User Access Key Id
  • aws_secret_access_key (String, Sensitive) Secret Access Key

Nested Schema for configuration.format

Optional:

Nested Schema for configuration.format.json_lines_newline_delimited_json

Optional:

  • compression_codec (String) The compression algorithm used to compress data. Default: "UNCOMPRESSED"; must be one of ["UNCOMPRESSED", "GZIP"]
  • format_type (String) Default: "JSONL"; must be "JSONL"

Nested Schema for configuration.format.parquet_columnar_storage

Optional:

  • compression_codec (String) The compression algorithm used to compress data. Default: "SNAPPY"; must be one of ["UNCOMPRESSED", "SNAPPY", "GZIP", "ZSTD"]
  • format_type (String) Default: "Parquet"; must be "Parquet"

Nested Schema for resource_allocation

Read-Only:

Nested Schema for resource_allocation.default

Read-Only:

  • cpu_limit (String)
  • cpu_request (String)
  • ephemeral_storage_limit (String)
  • ephemeral_storage_request (String)
  • memory_limit (String)
  • memory_request (String)

Nested Schema for resource_allocation.job_specific

Read-Only:

  • job_type (String) enum that describes the different types of jobs that the platform runs. must be one of ["get_spec", "check_connection", "discover_schema", "sync", "reset_connection", "connection_updater", "replicate"]
  • resource_requirements (Attributes) optional resource requirements to run workers (blank for unbounded allocations) (see below for nested schema)

Nested Schema for resource_allocation.job_specific.resource_requirements

Read-Only:

  • cpu_limit (String)
  • cpu_request (String)
  • ephemeral_storage_limit (String)
  • ephemeral_storage_request (String)
  • memory_limit (String)
  • memory_request (String)

Import

Import is supported using the following syntax:

terraform import airbyte_destination_aws_datalake.my_airbyte_destination_aws_datalake ""