feat: Start using an OGC version compatible with OIDC

[fr4nc1sc0-r4m0n]

Describe the enhancement:

Start using an OGC version which uses OIDC to authenticate to GCP

The current version of OGC needs a specific credentials json file which contains the Private Key of a Service Account Key and this is a security concern because it's using the legacy secret kv/ci-shared/observability-ingest/cloud/gcp.

We want to use a OGC versions which uses credentials generated by OIDC to log in the Google Cloud and stop using this legacy secret which implies a security concern.

Describe a specific use case for the enhancement or feature:

The testing process uses an OGC container and therefore that legacy secret.

What is the definition of done?

OGC is using OIDC credentials and not the legacy secret anymore.

Depends on the new OGC release

[pchila]

In elastic-agent pipeline we already moved away from ogc for the majority of our CI test steps, I wonder if it would be easier/faster to remove ogc for tests that use serverless projects as it's been already done for the rest of the pipeline.

Latest terraform ec provider supports creating serverless projects anyways and the serverless tests have been moved to their own package and can be launched using go test like the other integration/e2e tests we have.

/cc @ebeahan @cmacknz

[cmacknz]

Agree, I'd rather replace OGC with Terraform everywhere that orchestration is still needed. This has been a long standing idea we have never been able to get to.

[ebeahan]

Agree, I'd rather replace OGC with Terraform everywhere that orchestration is still needed

++