feat(epp): add OpenTelemetry spans for the scheduler scoring path

[mvanhorn]

What type of PR is this?

/kind feature

Summary

Adds fine-grained OpenTelemetry spans to the EPP scheduler scoring path so a single request trace shows which scorers ran, how long each took, and the aggregate score signals each produced.

SchedulerProfile.runScorerPlugins now opens a parent llm_d.epp.scoring span over the scorer chain, and each scorer invocation runs inside an llm_d.epp.scorer.<type> child span. The child spans nest under the parent, and the existing plugin-internal span (e.g. score_prefix_cache) nests under its scorer span. This mirrors the span-emission pattern already used by the precise-prefix-cache scorer.

Attributes are request- and chain-level only:

• parent: llm_d.epp.scorer.count, llm_d.epp.scoring.candidate_endpoints, plus gen_ai.request.model / gen_ai.request.id when present.
• per-scorer: llm_d.epp.scorer.type, llm_d.epp.scorer.name, llm_d.epp.scorer.weight, llm_d.epp.scorer.candidate_endpoints, and aggregate llm_d.epp.scorer.score.max / .avg / endpoints_scored derived from the returned score map.

No per-pod / per-endpoint attribute keys are emitted, keeping span cardinality bounded.

Why this matters

Issue #1692 ("Add span for scheduler score") is the scoped sub-task of umbrella #1483, which asks for spans across the scheduling/scoring pipeline so operators can see scorer-level behavior inside a request trace instead of only the top-level gateway.request / gateway.request_orchestration spans. Today the scorer chain is invisible in traces: there is no way to tell which scorer dominated a routing decision or which one was slow.

This package is allocation-sensitive (the surrounding code documents per-request allocation work in runScorerPlugins and runPickerPlugin), so the scoring path stays allocation-free when tracing is disabled: the parent span's IsRecording() is checked once and all attribute and child-span construction is skipped on the default no-op path, with the tracer and span-kind option resolved once rather than per scorer. BenchmarkSchedule confirms the disabled path holds at the prior baseline.

Testing

• Parent llm_d.epp.scoring span recorded with one llm_d.epp.scorer.<type> child per scorer, children nested under the parent, with correct weight, candidate-count, and aggregate max/avg attributes.
• No per-pod / per-endpoint attribute keys present on any emitted span.
• Scoring results and scorer call counts are unchanged with a no-op tracer; no spans recorded on the disabled path.
• A scorer returning an empty score map still emits its span and omits aggregates without dividing by zero.

Which issue(s) this PR fixes:

Fixes #1692

Release note:

Add OpenTelemetry spans for the EPP scheduler scoring path: a parent `llm_d.epp.scoring` span over the scorer chain and per-scorer `llm_d.epp.scorer.<type>` child spans carrying scorer identity, weight, candidate count, and aggregate score signals. No per-endpoint attributes are emitted, and spans are skipped entirely when tracing is disabled.

[elevran]

Is there a need to have a span per plugin or we can live with a single span for scheduling?
Any reason to suspect that a scorer would take noticeable time to warrant creating such fine grained spans?
It would be good to have some rationale for the production of detailed OTel traces at the granularity of internal function calls.

[github-actions]

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

[mvanhorn]

Fair question — the rationale, and I'm happy to drop to a single span if it doesn't hold up.

Why per-scorer rather than one scheduling span: the scorer chain is the part of the EPP path where the plugin set is operator-configured, so the interesting number is which scorer is slow on a given deployment, not that scoring took N ms. With one span you get the aggregate and then have to bisect by disabling scorers; with a child span per scorer the trace answers it directly. It also gives the existing plugin-internal spans (e.g. score_prefix_cache) a natural parent instead of hanging them off the request span.

On cost: I don't assume scorers are individually slow — the child spans are there for the tail, not the median. The hot path is guarded: tracer.Start on the parent returns a non-recording span when tracing is disabled, and span.IsRecording() gates every attribute and child-span construction after it, so with tracing off the added work is one Start call and a boolean check. Attributes are request- and chain-level only, no per-endpoint keys, so span cardinality stays bounded by the configured scorer count.

If you'd rather keep the trace coarse, I'll collapse to the single llm_d.epp.scoring span and keep the per-scorer latency metric, which covers the "which scorer is slow" question at lower trace cost. Say which you prefer and I'll push that — the branch needs a rebase anyway.