Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions terraform/modules/scheduled-job/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,7 @@ module "data_processor" {
- `main_file` - Python file name (e.g., "main.py"), relative to `source_dir`.
- `schedule` - Cron expression (e.g., "0 9 * * 1-5")
- `description` - Function/job description
- `owner` - The owner/team responsible for this scheduled job
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Do we want to use some consistent way of naming the team? Like should it be a github tag (@owner or #team), or the list of teams in ownership_data.json, or something else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I should probably add a predefined list of teams here.


### Optional (with defaults)
- `execution_type` - "function" or "job" ("function")
Expand All @@ -230,6 +231,7 @@ module "data_processor" {
- `timeout_seconds` - Timeout for functions (60)
- `environment_variables` - Environment vars ({})
- `secrets` - Secret Manager secrets ([])
- `tags` - A map of tags to assign to all resources ({})

### Cloud Run Job specific (when `execution_type = "job"`)
- `job_cpu` - CPU allocation (e.g., "1000m", "2") ("1000m")
Expand Down Expand Up @@ -389,6 +391,42 @@ Or use Cloud Build directly:
gcloud builds submit --tag gcr.io/YOUR_PROJECT_ID/YOUR_JOB_NAME:latest ./jobs/your-job
```

## Resource Tagging

All resources created by this module are automatically tagged with common metadata:

### Automatic Tags
- `terraform_module` - Set to "scheduled-job"
- `scheduled_job_name` - The name of your function/job
- `owner` - The owner/team responsible for this scheduled job

### Custom Tags
You can add custom tags using the `tags` variable:

```hcl
module "my_function" {
source = "git::https://github.com/Khan/terraform-modules.git//terraform/modules/scheduled-job?ref=v1.0.0"

job_name = "my-function"
owner = "data-team"
# ... other configuration

tags = {
"environment" = "production"
"team" = "data-engineering"
"cost-center" = "infrastructure"
}
}
```

### Supported Resources
The following resources support tagging/labeling:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if you try to add tags somewhere else? It seems like it's just silently ignored?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you add a field not supported in a terraform resource (in this case label or tag), it fails well before an apply, I think it's during the check stage.

- **Storage Buckets** - Labels applied
- **Storage Objects** - Metadata applied
- **PubSub Topics** - Labels applied
- **Cloud Functions** - Labels applied
- **Cloud Run Jobs** - Labels applied

## Common Cron Patterns

| Schedule | Description |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,14 @@ module "daily_health_check" {
version = "latest"
}
]

owner = "platform-team"

tags = {
environment = "example"
team = "platform"
cost-center = "infrastructure"
}
}

# Output the function details
Expand Down
15 changes: 15 additions & 0 deletions terraform/modules/scheduled-job/examples/simple-job/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ module "daily_data_processor_image" {
context_path = "./job-code"
project_id = var.project_id
image_tag_suffix = "latest"

tags = {
environment = "example"
team = "data-engineering"
cost-center = "infrastructure"
owner = "data-team"
}
}

# Simple daily job example
Expand Down Expand Up @@ -69,6 +76,14 @@ module "daily_data_processor" {
version = "latest"
}
]

owner = "data-team"

tags = {
environment = "example"
team = "data-engineering"
cost-center = "infrastructure"
}
}

# Output the job details
Expand Down
18 changes: 18 additions & 0 deletions terraform/modules/scheduled-job/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,15 @@ terraform {
}
}

# Common tags for all resources
locals {
common_tags = merge(var.tags, {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems misnamed -- I'd expect common_tags to just be the 3 common tags you list below, not that merged with var.tags. Should it be all_tags maybe?

Also, who wins in case of a merge conflict? I'm guessing that the ones below take precedence over the ones in var.tags. Is that the desired behavior?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this means "common for all resources generated by this usage of this module". Does "all_tags" sound better to you?

Good call on the merge conflicts, I'll make sure your guess is correct and document it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all_tags sounds good to me!

"terraform_module" = "scheduled-job"
"scheduled_job_name" = var.job_name
"owner" = var.owner
})
}

# Service account for the Cloud Function/Job
resource "google_service_account" "function_sa" {
project = var.project_id
Expand All @@ -34,6 +43,8 @@ resource "google_storage_bucket" "function_bucket" {
location = var.region
uniform_bucket_level_access = true
force_destroy = true

labels = local.common_tags
}

# Create function source archive (only for Cloud Functions)
Expand All @@ -57,6 +68,8 @@ resource "google_storage_bucket_object" "function_archive" {
name = "${var.job_name}-function-${data.archive_file.function_archive[0].output_sha}.zip"
bucket = google_storage_bucket.function_bucket[0].name
source = data.archive_file.function_archive[0].output_path

metadata = local.common_tags
}

# PubSub topic for triggering the Cloud Function (only created when execution_type is "function")
Expand All @@ -65,6 +78,8 @@ resource "google_pubsub_topic" "function_topic" {

project = var.project_id
name = "${var.job_name}-topic"

labels = local.common_tags
}

# Cloud Scheduler job for Cloud Function (only created when execution_type is "function")
Expand Down Expand Up @@ -104,6 +119,8 @@ resource "google_cloudfunctions2_function" "function" {
description = var.description
location = var.region

labels = local.common_tags

build_config {
runtime = var.runtime
entry_point = var.entry_point
Expand Down Expand Up @@ -159,6 +176,7 @@ resource "google_cloud_run_v2_job" "job" {
project = var.project_id
name = var.job_name
location = var.region
labels = local.common_tags

lifecycle {
precondition {
Expand Down
11 changes: 11 additions & 0 deletions terraform/modules/scheduled-job/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -194,3 +194,14 @@ variable "job_image" {
type = string
default = null
}

variable "tags" {
description = "A map of tags to assign to all resources created by this module"
type = map(string)
default = {}
}

variable "owner" {
description = "The owner/team responsible for this scheduled job"
type = string
}