Client metrics for cross-project GCS traffic

[dcrodman]

Hi, I apologize if this duplicate or already documented somewhere I have not encountered but I've not found a record of this problem.

We are using the Go gRPC GCS client running in a self-hosted GCP K8s cluster to write to a mix of GCS buckets, some of which are in another project. After sorting out some other issues with the client (notably the prior GCP resource detection bug), we're now seeing logs like this from the metric exporter:

rpc error: code = InvalidArgument desc = Field timeSeries[0].resource.labels.project_id had an invalid value of "(redacted)": if present, must be the project number or ID in the request name

(where "(redacted)" is a correctly form project ID, seemingly pulled from the GCE instance metadata).

That is the entirety of the error message so it's difficult to be completely certain where the issue is, but I believe this is coming from the Cloud Monitoring CreateServiceTimeSeries API route. Digging around in there led me to this TODO:

opentelemetry-operations-go/exporter/metric/metric.go

Line 249 in 9faf11b

// TODO: When this exporter is rewritten, support writing to multiple

Seeing as there's a TODO, I'm guessing this is a currently known/expected shortcoming? If so, is anyone able to share whether this is currently being addressed or planned in the near-mid future? Either way, if it's a limitation that is likely to exist for a while then from the perspective of an end user I'd lightly suggest that this would be worth mentioning on the gRPC client-side metrics documentation as it would've saved me (and possibly others) quite a bit of time.

[dashpole]

That TODO is unlikely to be implemented, and I don't think would solve this issue. It is to support writing to multiple projects in a single request, which I don't think is the issue here.

What options are being passed to the exporter? I'm not familiar with gRPC client-side metrics at all

[dcrodman]

With the disclaimer that I'm unfamiliar with either of these codebases, best I can tell this is where the exporter from this repository is configured when setting up the gRPC client: https://github.com/googleapis/google-cloud-go/blob/535f52e4a698b80b8c25a5f43dc8d6ffb3c913a5/storage/grpc_metrics.go#L54. The only thing that stands out to me there is that a project ID is specified, which had me wondering the if the error message (again, presumably from Cloud Monitoring) is due to the exporter submitting metrics for requests to a bucket under a different project ID and that ID not matching the project ID set by the linked code.

I'm not sure which library/component is "responsible" for this error and figured I'd start here since it's what is logging the error, but if you think I should take this to google-cloud-go or somewhere else then let me know.

[dashpole]

cc @frankyn

[dashpole]

Interesting... Reading that, it looks like project_id and the WithProjectID option are always set to smr.project. It might be an issue with the WithProjectID option...

[quentinmit]

You can't write metrics for multiple projects in a single CreateTimeSeries request, so it seems like this is WAI to me. If you want to write to multiple projects I think today you need one exporter per project. The exporter could theoretically be taught how to batch requests by project, but that seems like a lot of complexity.

[dashpole]

@quentinmit it looks like the exporter hard-codes the same string in the project_id resource attribute and passed via WithProjectID, so I don't think this should be trying to batch multiple resources to different projects. I suspect this is either an issue where we aren't properly respecting WithProjectID, or an issue with running multiple instances of the exporter with different project IDs set.