Skip to content

[bot] Bedrock Runtime CountTokens API is not instrumented #212

Description

@braintrust-bot

<!-- provider-gap-audit: bedrock-counttokens-not-instrumented -->

What instrumentation is missing

Amazon Bedrock Runtime's CountTokens operation (POST /model/{modelId}/count-tokens) returns the token count for a given InvokeModel- or Converse-shaped input, at no cost, so callers can estimate usage/cost before running inference. It's a real, current Bedrock Runtime execution-adjacent API — not a CRUD/admin/config endpoint — but this repo's Bedrock middleware has no tracer for it.

Where it breaks

trace/contrib/bedrockruntime/tracebedrockruntime.go, pickTracer (lines 169-181):

func pickTracer(cfg *middlewareConfig, opName string) opTracer {
	switch opName {
	case "Converse":
		return &converseTracer{cfg: cfg}
	case "ConverseStream":
		return &converseStreamTracer{cfg: cfg}
	case "InvokeModel":
		return &invokeModelTracer{cfg: cfg}
	case "InvokeModelWithResponseStream":
		return &invokeModelStreamTracer{cfg: cfg}
	}
	return nil
}

There is no case "CountTokens", so a CountTokens call falls through to the nil default and produces no span. The package doc comment (lines 31-36) enumerates the intentionally-unsupported surfaces ("bidirectional streams, management APIs") but doesn't mention CountTokens, and none of the existing operation tracers (converse.go, invokemodel.go, stream.go) cover it either — it's simply missing, not a deliberate exclusion.

Braintrust docs status: not_found

https://www.braintrust.dev/docs/integrations/ai-providers/bedrock enumerates exactly four traced operations for every language SDK it documents (Converse, ConverseStream, InvokeModel, InvokeModelWithResponseStream for Go), and explicitly calls out for Java that "InvokeModel and ApplyGuardrail are not traced" — but CountTokens isn't mentioned as traced or explicitly excluded for any language.

Upstream sources

Local repo files inspected

  • trace/contrib/bedrockruntime/tracebedrockruntime.gopickTracer (lines 169-181), package doc (lines 1-37), NewMiddleware/initializeHandler/deserializeHandler (lines 98-241)
  • trace/contrib/bedrockruntime/converse.go, invokemodel.go, stream.go — confirmed none implement a CountTokens-shaped tracer

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions