Skip to content

Latest commit

 

History

History
175 lines (122 loc) · 8.93 KB

destination_s3_glue.md

File metadata and controls

175 lines (122 loc) · 8.93 KB
page_title subcategory description
airbyte_destination_s3_glue Resource - terraform-provider-airbyte
DestinationS3Glue Resource

airbyte_destination_s3_glue (Resource)

DestinationS3Glue Resource

Example Usage

resource "airbyte_destination_s3_glue" "my_destination_s3glue" {
  configuration = {
    access_key_id     = "A012345678910EXAMPLE"
    file_name_pattern = "{date}"
    format = {
      json_lines_newline_delimited_json = {
        compression = {
          # ...
        }
        flattening  = "No flattening"
        format_type = "JSONL"
      }
    }
    glue_database              = "airbyte_database"
    glue_serialization_library = "org.openx.data.jsonserde.JsonSerDe"
    s3_bucket_name             = "airbyte_sync"
    s3_bucket_path             = "data_sync/test"
    s3_bucket_region           = "us-gov-west-1"
    s3_endpoint                = "http://localhost:9000"
    s3_path_format             = "${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_"
    secret_access_key          = "a012345678910ABCDEFGH/AbCdEfGhEXAMPLEKEY"
  }
  definition_id = "7a9c505b-1990-4b38-b243-1c254ff78c36"
  name          = "...my_name..."
  workspace_id  = "e7df4138-2176-42e6-aa11-ed72fe60b365"
}

Schema

Required

  • configuration (Attributes) (see below for nested schema)
  • name (String) Name of the destination e.g. dev-mysql-instance.
  • workspace_id (String)

Optional

  • definition_id (String) The UUID of the connector definition. One of configuration.destinationType or definitionId must be provided. Requires replacement if changed.

Read-Only

  • created_at (Number)
  • destination_id (String)
  • destination_type (String)
  • resource_allocation (Attributes) actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level. (see below for nested schema)

Nested Schema for configuration

Required:

  • format (Attributes) Format of the data output. See here for more details (see below for nested schema)
  • glue_database (String) Name of the glue database for creating the tables, leave blank if no integration
  • s3_bucket_name (String) The name of the S3 bucket. Read more here.
  • s3_bucket_path (String) Directory under the S3 bucket where data will be written. Read more here

Optional:

  • access_key_id (String, Sensitive) The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here.
  • file_name_pattern (String) The pattern allows you to set the file-name format for the S3 staging file(s)
  • glue_serialization_library (String) The library that your query engine will use for reading and writing data in your lake. Default: "org.openx.data.jsonserde.JsonSerDe"; must be one of ["org.openx.data.jsonserde.JsonSerDe", "org.apache.hive.hcatalog.data.JsonSerDe"]
  • s3_bucket_region (String) The region of the S3 bucket. See here for all region codes. Default: ""; must be one of ["", "af-south-1", "ap-east-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-south-1", "ap-south-2", "ap-southeast-1", "ap-southeast-2", "ap-southeast-3", "ap-southeast-4", "ca-central-1", "ca-west-1", "cn-north-1", "cn-northwest-1", "eu-central-1", "eu-central-2", "eu-north-1", "eu-south-1", "eu-south-2", "eu-west-1", "eu-west-2", "eu-west-3", "il-central-1", "me-central-1", "me-south-1", "sa-east-1", "us-east-1", "us-east-2", "us-gov-east-1", "us-gov-west-1", "us-west-1", "us-west-2"]
  • s3_endpoint (String) Your S3 endpoint url. Read more here. Default: ""
  • s3_path_format (String) Format string on how data will be organized inside the S3 bucket directory. Read more here
  • secret_access_key (String, Sensitive) The corresponding secret to the access key ID. Read more here

Nested Schema for configuration.format

Optional:

Nested Schema for configuration.format.json_lines_newline_delimited_json

Optional:

  • compression (Attributes) Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). (see below for nested schema)
  • flattening (String) Whether the input json data should be normalized (flattened) in the output JSON Lines. Please refer to docs for details. Default: "Root level flattening"; must be one of ["No flattening", "Root level flattening"]
  • format_type (String) Default: "JSONL"; must be "JSONL"

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression

Optional:

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression.gzip

Optional:

  • compression_type (String) Default: "GZIP"; must be "GZIP"

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression.no_compression

Optional:

  • compression_type (String) Default: "No Compression"; must be "No Compression"

Nested Schema for resource_allocation

Read-Only:

Nested Schema for resource_allocation.default

Read-Only:

  • cpu_limit (String)
  • cpu_request (String)
  • ephemeral_storage_limit (String)
  • ephemeral_storage_request (String)
  • memory_limit (String)
  • memory_request (String)

Nested Schema for resource_allocation.job_specific

Read-Only:

  • job_type (String) enum that describes the different types of jobs that the platform runs. must be one of ["get_spec", "check_connection", "discover_schema", "sync", "reset_connection", "connection_updater", "replicate"]
  • resource_requirements (Attributes) optional resource requirements to run workers (blank for unbounded allocations) (see below for nested schema)

Nested Schema for resource_allocation.job_specific.resource_requirements

Read-Only:

  • cpu_limit (String)
  • cpu_request (String)
  • ephemeral_storage_limit (String)
  • ephemeral_storage_request (String)
  • memory_limit (String)
  • memory_request (String)

Import

Import is supported using the following syntax:

terraform import airbyte_destination_s3_glue.my_airbyte_destination_s3_glue ""