google_app_engine_standard_app_version seems to be ignoring the provided `project`

[llarco]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.7.0
on linux_amd64

• provider registry.terraform.io/hashicorp/google v6.27.0

Affected Resource(s)

google_app_engine_standard_app_version

Terraform Configuration

resource "google_app_engine_standard_app_version" "default" {
  project = <project 1 name>

  service    = "service-name"
  version_id = "v1"
  runtime    = "static" # Use the 'static' runtime
  entrypoint {
    shell = "echo 'Serving static content'" # Required, but doesn't do anything for static runtime.
  }

  deployment {
    zip {
      source_url = "gs://bucket/file.zip"
    }
  }

  handlers {
    url_regex = ".*"
    static_files {
      application_readable = true
      upload_path_regex    = ".*" # Serve all files from the zip
    }
  }

  service_account = google_service_account.gae_sa.email
}

Debug Output

  Error: Error creating StandardAppVersion: googleapi: Error 403: App Engine Admin API has not been used in project <project 2 ID> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/appengine.googleapis.com/overview?project=<project 2 ID> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Expected Behavior

google_app_engine_standard_app_version is created successfully in project 1.

Actual Behavior

Error is thrown indicating that appengine.googleapis.com must be enabled in project 2

Steps to reproduce

No response

Important Factoids

We use guardian to apply our TF changes. When we don't set the project field in any TF resource, it uses the GCP project of guardian by default (in the error logs above referred as <project 2>).

Although we are setting project to <project 1> in google_app_engine_standard_app_version, it seems to be ignored.

References

Looking at the implementation code, it seems that even if you provide a project, it might get overriden if a billing_project is found:

terraform-provider-google/google/services/appengine/resource_app_engine_standard_app_version.go

Lines 617 to 620 in 6719d6d

	// err == nil indicates that the billing_project value was found
	if bp, err := tpgresource.GetBillingProject(d, config); err == nil {
	billingProject = bp
	}

[SirGitsalot]

I've confirmed that it's having the billing project set that results in the error - it appears that the AppEngine API needs to be enabled in both the target project and the billing project. Do you (or any of the folks who have thumbs-up'd) happen to know if this is new behavior? And if so, about when did it start?

For reference, here's the config I used:

provider "google" {
  user_project_override = true
  billing_project     = "<project 2>"
  region      = "us-central1"
  zone        = "us-central1-c"
}

resource "google_app_engine_standard_app_version" "default" {
  project = "<project 1>"

  service    = "service-name"
  version_id = "v1"
  runtime    = "static" # Use the 'static' runtime
  entrypoint {
    shell = "echo 'Serving static content'" # Required, but doesn't do anything for static runtime.
  }

  deployment {
    zip {
      source_url = "gs://bucket/file.zip"
    }
  }

  handlers {
    url_regex = ".*"
    static_files {
      application_readable = true
      upload_path_regex    = ".*" # Serve all files from the zip
    }
  }
}

And the API request it generated:

POST /v1/apps/<project 1>/services/service-name/versions?alt=json HTTP/1.1
Host: appengine.googleapis.com
User-Agent: Terraform/1.11.3 (+https://www.terraform.io) Terraform-Plugin-SDK/2.36.0 terraform-provider-google/6.27.0
Content-Length: 241
Content-Type: application/json
X-Goog-User-Project: <project 2>
Accept-Encoding: gzip
2025-03-28T16:00:41.530-0700 [DEBUG] provider.terraform-provider-google_v6.27.0_x5
{
 "deployment": {
  "zip": {
   "sourceUrl": "gs://bucket/file.zip"
  }
 },
 "entrypoint": {
  "shell": "echo 'Serving static content'"
 },
 "handlers": [
  {
   "staticFiles": {
    "applicationReadable": true,
    "uploadPathRegex": ".*"
   },
   "urlRegex": ".*"
  }
 ],
 "id": "v1",
 "runtime": "static"
}

HTTP/2.0 403 Forbidden
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: application/json; charset=UTF-8
Date: Fri, 28 Mar 2025 23:00:41 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 403,
    "message": "App Engine Admin API has not been used in project <project 2> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/appengine.googleapis.com/overview?project=<project 2> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "SERVICE_DISABLED",
        "domain": "googleapis.com",
        "metadata": {
          "activationUrl": "https://console.developers.google.com/apis/api/appengine.googleapis.com/overview?project=<project 2>",
          "containerInfo": "<project 2>",
          "serviceTitle": "App Engine Admin API",
          "consumer": "projects/<project 2>",
          "service": "appengine.googleapis.com"
        }
      },
      {
        "@type": "type.googleapis.com/google.rpc.LocalizedMessage",
        "locale": "en-US",
        "message": "App Engine Admin API has not been used in project <project 2> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/appengine.googleapis.com/overview?project=<project 2> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry."
      },
      {
        "@type": "type.googleapis.com/google.rpc.Help",
        "links": [
          {
            "description": "Google developers console API activation",
            "url": "https://console.developers.google.com/apis/api/appengine.googleapis.com/overview?project=<project 2>"
          }
        ]
      }
    ]
  }
}

[llarco]

@SirGitsalot this is my first time attempting to use this specific resource, so I wouldn't be able to tell whether this is a new behavior or not

[ggtisc]

Hi @llarco

Since you are using a static runtime instead of a programming language, could you please share an example without sensitive information of what you are uploading in the zip?

source_url = "gs://bucket/file.zip"

For sensitive data you could use examples like:

1. project = "project-22095"
2. member = "user:user-22095@domain-22095.com"