Skip to content

Latest commit

 

History

History
317 lines (200 loc) · 15.9 KB

destination_gcs.md

File metadata and controls

317 lines (200 loc) · 15.9 KB
page_title subcategory description
airbyte_destination_gcs Resource - terraform-provider-airbyte
DestinationGcs Resource

airbyte_destination_gcs (Resource)

DestinationGcs Resource

Example Usage

resource "airbyte_destination_gcs" "my_destination_gcs" {
  configuration = {
    credential = {
      hmac_key = {
        credential_type    = "HMAC_KEY"
        hmac_key_access_id = "1234567890abcdefghij1234"
        hmac_key_secret    = "1234567890abcdefghij1234567890ABCDEFGHIJ"
      }
    }
    format = {
      json_lines_newline_delimited_json = {
        compression = {
          # ...
        }
        format_type = "JSONL"
      }
    }
    gcs_bucket_name   = "airbyte_sync"
    gcs_bucket_path   = "data_sync/test"
    gcs_bucket_region = "australia-southeast2"
  }
  definition_id = "fbdeaa3f-58d4-41ca-a298-54ecee1615a0"
  name          = "...my_name..."
  workspace_id  = "0ffee906-5986-410b-95a7-80c6f12aeac3"
}

Schema

Required

  • configuration (Attributes) (see below for nested schema)
  • name (String) Name of the destination e.g. dev-mysql-instance.
  • workspace_id (String)

Optional

  • definition_id (String) The UUID of the connector definition. One of configuration.destinationType or definitionId must be provided. Requires replacement if changed.

Read-Only

  • created_at (Number)
  • destination_id (String)
  • destination_type (String)
  • resource_allocation (Attributes) actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level. (see below for nested schema)

Nested Schema for configuration

Required:

  • credential (Attributes) An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here. (see below for nested schema)
  • format (Attributes) Output data format. One of the following formats must be selected - AVRO format, PARQUET format, CSV format, or JSONL format. (see below for nested schema)
  • gcs_bucket_name (String) You can find the bucket name in the App Engine Admin console Application Settings page, under the label Google Cloud Storage Bucket. Read more here.
  • gcs_bucket_path (String) GCS Bucket Path string Subdirectory under the above bucket to sync the data into.

Optional:

  • gcs_bucket_region (String) Select a Region of the GCS Bucket. Read more here. Default: "us"; must be one of ["northamerica-northeast1", "northamerica-northeast2", "us-central1", "us-east1", "us-east4", "us-west1", "us-west2", "us-west3", "us-west4", "southamerica-east1", "southamerica-west1", "europe-central2", "europe-north1", "europe-west1", "europe-west2", "europe-west3", "europe-west4", "europe-west6", "asia-east1", "asia-east2", "asia-northeast1", "asia-northeast2", "asia-northeast3", "asia-south1", "asia-south2", "asia-southeast1", "asia-southeast2", "australia-southeast1", "australia-southeast2", "asia", "eu", "us", "asia1", "eur4", "nam4"]

Nested Schema for configuration.credential

Optional:

Nested Schema for configuration.credential.hmac_key

Required:

  • hmac_key_access_id (String, Sensitive) When linked to a service account, this ID is 61 characters long; when linked to a user account, it is 24 characters long. Read more here.
  • hmac_key_secret (String, Sensitive) The corresponding secret for the access ID. It is a 40-character base-64 encoded string. Read more here.

Optional:

  • credential_type (String) Default: "HMAC_KEY"; must be "HMAC_KEY"

Nested Schema for configuration.format

Optional:

Nested Schema for configuration.format.avro_apache_avro

Required:

  • compression_codec (Attributes) The compression algorithm used to compress data. Default to no compression. (see below for nested schema)

Optional:

  • format_type (String) Default: "Avro"; must be "Avro"

Nested Schema for configuration.format.avro_apache_avro.compression_codec

Optional:

Nested Schema for configuration.format.avro_apache_avro.compression_codec.bzip2

Optional:

  • codec (String) Default: "bzip2"; must be "bzip2"

Nested Schema for configuration.format.avro_apache_avro.compression_codec.deflate

Optional:

  • codec (String) Default: "Deflate"; must be "Deflate"
  • compression_level (Number) 0: no compression & fastest, 9: best compression & slowest. Default: 0

Nested Schema for configuration.format.avro_apache_avro.compression_codec.no_compression

Optional:

  • codec (String) Default: "no compression"; must be "no compression"

Nested Schema for configuration.format.avro_apache_avro.compression_codec.snappy

Optional:

  • codec (String) Default: "snappy"; must be "snappy"

Nested Schema for configuration.format.avro_apache_avro.compression_codec.xz

Optional:

  • codec (String) Default: "xz"; must be "xz"
  • compression_level (Number) The presets 0-3 are fast presets with medium compression. The presets 4-6 are fairly slow presets with high compression. The default preset is 6. The presets 7-9 are like the preset 6 but use bigger dictionaries and have higher compressor and decompressor memory requirements. Unless the uncompressed size of the file exceeds 8 MiB, 16 MiB, or 32 MiB, it is waste of memory to use the presets 7, 8, or 9, respectively. Read more here for details. Default: 6

Nested Schema for configuration.format.avro_apache_avro.compression_codec.zstandard

Optional:

  • codec (String) Default: "zstandard"; must be "zstandard"
  • compression_level (Number) Negative levels are 'fast' modes akin to lz4 or snappy, levels above 9 are generally for archival purposes, and levels above 18 use a lot of memory. Default: 3
  • include_checksum (Boolean) If true, include a checksum with each data block. Default: false

Nested Schema for configuration.format.csv_comma_separated_values

Optional:

  • compression (Attributes) Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".csv.gz"). (see below for nested schema)
  • flattening (String) Whether the input JSON data should be normalized (flattened) in the output CSV. Please refer to docs for details. Default: "No flattening"; must be one of ["No flattening", "Root level flattening"]
  • format_type (String) Default: "CSV"; must be "CSV"

Nested Schema for configuration.format.csv_comma_separated_values.compression

Optional:

Nested Schema for configuration.format.csv_comma_separated_values.compression.gzip

Optional:

  • compression_type (String) Default: "GZIP"; must be "GZIP"

Nested Schema for configuration.format.csv_comma_separated_values.compression.no_compression

Optional:

  • compression_type (String) Default: "No Compression"; must be "No Compression"

Nested Schema for configuration.format.json_lines_newline_delimited_json

Optional:

  • compression (Attributes) Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). (see below for nested schema)
  • format_type (String) Default: "JSONL"; must be "JSONL"

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression

Optional:

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression.gzip

Optional:

  • compression_type (String) Default: "GZIP"; must be "GZIP"

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression.no_compression

Optional:

  • compression_type (String) Default: "No Compression"; must be "No Compression"

Nested Schema for configuration.format.parquet_columnar_storage

Optional:

  • block_size_mb (Number) This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. Default: 128 MB. Default: 128
  • compression_codec (String) The compression algorithm used to compress data pages. Default: "UNCOMPRESSED"; must be one of ["UNCOMPRESSED", "SNAPPY", "GZIP", "LZO", "BROTLI", "LZ4", "ZSTD"]
  • dictionary_encoding (Boolean) Default: true. Default: true
  • dictionary_page_size_kb (Number) There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. Default: 1024 KB. Default: 1024
  • format_type (String) Default: "Parquet"; must be "Parquet"
  • max_padding_size_mb (Number) Maximum size allowed as padding to align row groups. This is also the minimum size of a row group. Default: 8 MB. Default: 8
  • page_size_kb (Number) The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. Default: 1024 KB. Default: 1024

Nested Schema for resource_allocation

Read-Only:

Nested Schema for resource_allocation.default

Read-Only:

  • cpu_limit (String)
  • cpu_request (String)
  • ephemeral_storage_limit (String)
  • ephemeral_storage_request (String)
  • memory_limit (String)
  • memory_request (String)

Nested Schema for resource_allocation.job_specific

Read-Only:

  • job_type (String) enum that describes the different types of jobs that the platform runs. must be one of ["get_spec", "check_connection", "discover_schema", "sync", "reset_connection", "connection_updater", "replicate"]
  • resource_requirements (Attributes) optional resource requirements to run workers (blank for unbounded allocations) (see below for nested schema)

Nested Schema for resource_allocation.job_specific.resource_requirements

Read-Only:

  • cpu_limit (String)
  • cpu_request (String)
  • ephemeral_storage_limit (String)
  • ephemeral_storage_request (String)
  • memory_limit (String)
  • memory_request (String)

Import

Import is supported using the following syntax:

terraform import airbyte_destination_gcs.my_airbyte_destination_gcs ""