This module creates a log sink, pub/sub topic, and pub/sub subscription needed to facilitate the collection of asset inventory records, metrics and logs from GCP for a given project.
This module also creates a Cloud Function to fetch some data through the GCP REST API.
Here is an example manifest for collecting data from a Google Cloud organization.
After running terraform apply, data should start flowing into Pub/Sub. In the Observe
UI, one would set up the GCP app. The info from the terraform output and terraform output -raw service_account_private_key are needed to set up the GCP App pollers.
provider "google" {
project = "YOUR_PROJECT_ID"
region = "YOUR_DEFAULT_REGION"
}
module "observe_gcp_collection" {
source = "observeinc/collection/google"
name = "observe"
resource = "projects/YOUR_PROJECT_ID"
}
output "project" {
description = "The Pub/Sub project of the subcription (to be passed to the Pub/Sub poller)"
value = module.observe_gcp_collection.project
}
# To extract correct value - terraform output -json | jq -r '.subscription.value.name'
output "subscription" {
description = "The Pub/Sub subscription created by this module (to be passed to the Pub/Sub poller)"
value = module.observe_gcp_collection.subscription
}
# To extract properly formatted string - terraform output -json | jq -r '.service_account_private_key.value'
output "service_account_private_key" {
description = "A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring"
value = base64decode(module.observe_gcp_collection.service_account_key.private_key)
sensitive = true
}
| Name | Version |
|---|---|
| terraform | >= 0.12.21 |
| >= 4.15 | |
| random | ~> 3.0 |
| Name | Version |
|---|---|
| >= 4.15 | |
| random | ~> 3.0 |
No modules.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| bucket_lifecycle_abort_upload_days | The number of days to wait before deleting AbortIncompleteMultipartUpload. | number |
7 |
no |
| bucket_lifecycle_delete_days | The number of days to wait before Delete of temporary bucket files. | number |
14 |
no |
| cloud_function_debug_level | The debug level for the GCP cloud functions | string |
"WARNING" |
no |
| enable_asset_tracking | Whether to enable the Cloud function that tracks GCP assets. | bool |
true |
no |
| enable_function | DEPRECATED: This variable has been renamed to 'enable_asset_tracking'. Please update your configuration to use 'enable_asset_tracking' instead. | bool |
null |
no |
| folder_include_children | Whether to include all children Projects of a Folder when collecting logs | bool |
true |
no |
| function_available_memory_mb | Memory (in MB), available to the function. Default value is 512. Possible values include 128, 256, 512, 1024, etc. | number |
4096 |
no |
| function_bucket | GCS bucket containing the Cloud Function source code | string |
"observeinc" |
no |
| function_disable_logging | Whether to disable function logging. | bool |
false |
no |
| function_max_instances | The limit on the maximum number of function instances that may coexist at a given time. | number |
100 |
no |
| function_object | GCS object key of the Cloud Function source code zip file. Will use the latest release unless modified. | string |
"google-cloud-functions-latest.zip" |
no |
| function_roles | A list of IAM roles to give the Cloud Function. | set(string) |
[ |
no |
| function_schedule_frequency | Cron schedule for the job | string |
"0 * * * *" |
no |
| function_schedule_frequency_rest_of_assets | Cron schedule for the job | string |
"*/5 * * * *" |
no |
| function_timeout | Timeout (in seconds) for the function. Default value is 300 seconds. Cannot be more than 540 seconds. | number |
300 |
no |
| gcp_region | The location where the Task Queue will be created. | string |
"us-central1" |
no |
| labels | A map of labels to add to resources (https://cloud.google.com/resource-manager/docs/creating-managing-labels)" Note: Many, but not all, Google Cloud SDK resources support labels. |
map(string) |
{} |
no |
| logging_exclusions | Log entries that match any of these exclusion filters will not be exported. If a log entry is matched by both logging_filter and one of logging_exclusions it will not be exported. Relevant docs: https://cloud.google.com/logging/docs/reference/v2/rest/v2/billingAccounts.exclusions#LogExclusion |
list(object({ |
[] |
no |
| logging_filter | An advanced logs filter. The only exported log entries are those that are in the resource owning the sink and that match the filter. Relevant docs: https://cloud.google.com/logging/docs/view/building-queries |
string |
"" |
no |
| max_attempts | The maximum number of retry attempts for a task in case of failure. | number |
-1 |
no |
| max_concurrent_dispatches | The maximum number of tasks that can be dispatched concurrently. | number |
2 |
no |
| max_dispatches_per_second | The maximum rate at which tasks can be dispatched per second. | number |
2 |
no |
| max_retry_duration | The time limit for retrying a task in seconds | string |
"7200s" |
no |
| min_backoff | The minimum amount of time to wait between retries in seconds | string |
"30s" |
no |
| name | Module name. Used as a name prefix. | string |
"obs" |
no |
| poller_roles | A list of IAM roles to give the Observe poller (through the service account key output). | set(string) |
[ |
no |
| project_id | Billing Project ID needed for asset feed. | string |
null |
no |
| pubsub_ack_deadline_seconds | Ack deadline for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | number |
60 |
no |
| pubsub_maximum_backoff | Retry policy maximum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | string |
"600s" |
no |
| pubsub_message_retention_duration | Message retention for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | string |
"86400s" |
no |
| pubsub_minimum_backoff | Retry policy minimum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | string |
"10s" |
no |
| resource | The identifier of the GCP Resource to monitor. The resource can be a project, folder, or organization. Examples: "projects/my_project-123", "folders/1234567899", "organizations/34739118321" |
string |
n/a | yes |
| Name | Description |
|---|---|
| project | The ID of the Project in which resources were created |
| service_account_key | A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring |
| subscription | The Pub/Sub subscription created by this module. |
| topic | The Pub/Sub topic created by this module. |