feat(trace,sls): differentiate serviceNamePrefix for user vs managed backends (#167)

Managed (inner) trace and log backends can now report under a distinct
service name from user backends, via a top-level `serviceNamePrefix` in
configs/inner/data_config.json. Falls back to the user prefix when unset,
so the default single-service-name behavior is unchanged.

- OTLP trace: convert once per distinct service.name and fan out each group
  to its own backends (reuses the existing convertKey multi-state path).
- SLS: per-endpoint service name injected into the __service_name__ tag at
  flush time.
- Dedup keys (OTLP + SLS) include serviceName so a shared URL under two
  service names is kept as two distinct backends.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Author: fangxiu-wf

docs/trace-output.md

@@ -82,10 +82,10 @@ Trace export sends the **same** converted spans to **every** configured backend
 
 Backends are **deduplicated** by normalized endpoint URL + full request headers, so listing the same backend twice (e.g. once as a user backend and once as a managed backend) results in a single export. Two backends that share a URL but differ in auth headers / workspace / license are kept as distinct.
 
-Shared vs. per-backend settings (because spans are converted once):
+Shared vs. per-backend settings (spans are converted once per distinct `service.name`):
 
-- **Shared across all backends:** `serviceName`, `resourceAttributes`, `captureMessageContent`, `resourceAttributeKeys`, `maxExportBatchBytes`, `turnIdleTimeoutMs`.
-- **Per-backend:** endpoint URL, headers, compression.
+- **Shared across all backends:** `resourceAttributes`, `captureMessageContent`, `resourceAttributeKeys`, `maxExportBatchBytes`, `turnIdleTimeoutMs`.
+- **Per-backend:** endpoint URL, headers, compression, and `service.name` (see below — user vs. managed backends can differ).
 
 A failing backend is isolated — it does not block the healthy backends, and its failed spans are persisted separately under `~/.loongsuite-pilot/logs/otlp-failed/<service>-<agent>__<backend-name>.jsonl`.
 
@@ -96,6 +96,7 @@ Managed/hosted deployments can push extra trace backends via `~/.loongsuite-pilo
 ```json
 {
   "sls": [ /* managed SLS endpoints (unchanged) */ ],
+  "serviceNamePrefix": "managed-service",
   "otlp": [
     {
       "name": "team-collector",
@@ -118,6 +119,7 @@ Managed/hosted deployments can push extra trace backends via `~/.loongsuite-pilo
 
 | Field | Applies to | Description |
 |-------|-----------|-------------|
+| `serviceNamePrefix` | top-level | Service-name prefix for **all** managed backends — trace (`otlp[]`/`cms[]`, as `service.name`) and log (`sls[]`, as the `__service_name__` tag) — keeping them distinct from user backends. Optional; falls back to the user `serviceNamePrefix` when omitted (no differentiation). |
 | `otlp[].name` / `cms[].name` | both | Label used in logs and in the per-backend failed-log filename. Optional (defaults to `inner-otlp-<i>` / `inner-cms-<i>`). |
 | `otlp[].endpoint` | otlp | OTLP HTTP base URL (`/v1/traces` auto-appended). |
 | `otlp[].headers` | otlp | Request headers (e.g. auth token). |
@@ -127,7 +129,7 @@ Managed/hosted deployments can push extra trace backends via `~/.loongsuite-pilo
 | `cms[].project` | cms | Sent as `x-arms-project`. Extracted from the endpoint hostname if omitted. |
 | `cms[].workspace` | cms | Sent as `x-cms-workspace`. |
 
-Each `cms[]` entry is expanded into an OTLP endpoint with the corresponding `x-arms-*` / `x-cms-*` headers, and any CMS backend adds the `acs.arms.service.feature=genai_app` resource attribute (shared, as noted above). Malformed managed config (e.g. `otlp`/`cms` written as a non-array) is ignored rather than failing collection.
+Each `cms[]` entry is expanded into an OTLP endpoint with the corresponding `x-arms-*` / `x-cms-*` headers, and any CMS backend adds the `acs.arms.service.feature=genai_app` resource attribute (shared, as noted above). When `serviceNamePrefix` is set here, managed backends report under `<serviceNamePrefix>-<agent>` while user backends keep the user prefix; spans are then converted once per distinct `service.name` (typically twice — user and managed). Malformed managed config (e.g. `otlp`/`cms` written as a non-array) is ignored rather than failing collection.
 
 ## Backend Examples
 

docs/zh-CN/trace-output.md

@@ -82,10 +82,10 @@ Trace 导出会把**同一批**转换后的 span **同时**发往**所有**已
 
 后端会按"规范化 URL + 完整请求 headers"**去重**,因此把同一个后端列两遍(例如既作为用户后端又作为托管后端)只会导出一次;URL 相同但鉴权 header / workspace / license 不同的两个后端会被视为不同后端各自保留。
 
-共享 vs 单后端设置(因为 span 只转换一次):
+共享 vs 单后端设置(span 按不同 `service.name` 各转换一次):
 
-- **所有后端共享:** `serviceName`、`resourceAttributes`、`captureMessageContent`、`resourceAttributeKeys`、`maxExportBatchBytes`、`turnIdleTimeoutMs`。
-- **每后端独立:** endpoint URL、headers、compression。
+- **所有后端共享:** `resourceAttributes`、`captureMessageContent`、`resourceAttributeKeys`、`maxExportBatchBytes`、`turnIdleTimeoutMs`。
+- **每后端独立:** endpoint URL、headers、compression,以及 `service.name`(见下文——用户后端与托管后端可不同)。
 
 某个后端失败会被隔离——不会阻塞健康后端,其失败 span 会单独落盘到 `~/.loongsuite-pilot/logs/otlp-failed/<service>-<agent>__<后端名>.jsonl`。
 
@@ -96,6 +96,7 @@ Trace 导出会把**同一批**转换后的 span **同时**发往**所有**已
 ```json
 {
   "sls": [ /* 托管 SLS endpoint(不变) */ ],
+  "serviceNamePrefix": "managed-service",
   "otlp": [
     {
       "name": "team-collector",
@@ -118,6 +119,7 @@ Trace 导出会把**同一批**转换后的 span **同时**发往**所有**已
 
 | 字段 | 适用 | 说明 |
 |------|------|------|
+| `serviceNamePrefix` | 顶层 | **所有**托管后端的 service name 前缀——trace(`otlp[]`/`cms[]`,作为 `service.name`)与 log(`sls[]`,作为 `__service_name__` tag)——用于与用户后端区分。可选;省略时回退到用户的 `serviceNamePrefix`(即不做区分)。 |
 | `otlp[].name` / `cms[].name` | 两者 | 用于日志与失败日志文件名的标签。可选(默认 `inner-otlp-<i>` / `inner-cms-<i>`)。 |
 | `otlp[].endpoint` | otlp | OTLP HTTP 基础 URL(自动补 `/v1/traces`)。 |
 | `otlp[].headers` | otlp | 请求 header(如鉴权 token)。 |
@@ -127,7 +129,7 @@ Trace 导出会把**同一批**转换后的 span **同时**发往**所有**已
 | `cms[].project` | cms | 作为 `x-arms-project` 发送。省略时从 endpoint 域名提取。 |
 | `cms[].workspace` | cms | 作为 `x-cms-workspace` 发送。 |
 
-每个 `cms[]` 条目会展开为一个带 `x-arms-*` / `x-cms-*` header 的 OTLP 后端;只要存在 CMS 后端,就会附加 `acs.arms.service.feature=genai_app` 资源属性(如上所述为共享)。托管配置格式错误(例如 `otlp`/`cms` 写成非数组)会被忽略,而不会导致采集失败。
+每个 `cms[]` 条目会展开为一个带 `x-arms-*` / `x-cms-*` header 的 OTLP 后端;只要存在 CMS 后端,就会附加 `acs.arms.service.feature=genai_app` 资源属性(如上所述为共享)。此处设置 `serviceNamePrefix` 后,托管后端会以 `<serviceNamePrefix>-<agent>` 上报,用户后端仍用用户前缀;此时 span 会按不同 `service.name` 各转换一次(通常两次——用户与托管)。托管配置格式错误(例如 `otlp`/`cms` 写成非数组)会被忽略,而不会导致采集失败。
 
 ## 后端接入示例
 

src/core/config-loader.ts

@@ -56,6 +56,7 @@ export interface InnerDataConfig {
   sls?: SlsEndpointEntry[];
   otlp?: OtlpEndpointEntry[];
   cms?: CmsEndpointEntry[];
+  serviceNamePrefix?: string;
 }
 
 /**
@@ -224,7 +225,11 @@ export async function loadConfig(): Promise<AnalyticsConfig> {
     cms: buildCmsConfig(file),
     otlpTrace: buildOtlpTraceRawConfig(file),
     innerTrace: innerDataConfig
-      ? { otlp: innerDataConfig.otlp, cms: innerDataConfig.cms }
+      ? {
+          otlp: innerDataConfig.otlp,
+          cms: innerDataConfig.cms,
+          serviceNamePrefix: innerDataConfig.serviceNamePrefix,
+        }
       : undefined,
     autoUpdate: buildAutoUpdateConfig(file),
 
@@ -487,6 +492,13 @@ export function buildOtlpTraceConfig(config: AnalyticsConfig): OtlpTraceFlusherC
   const endpoints: OtlpEndpoint[] = [];
   // Collected from any ARMS/CMS backend; merged into the shared resource.
   const armsResourceAttributes: Record<string, string> = {};
+  // User backends use the top-level (shared) serviceName. Managed backends may
+  // override it via inner serviceNamePrefix; only tag them when it actually
+  // differs, so the default case keeps user/inner backends dedup-identical and
+  // leaves the flusher on its single-conversion path.
+  const userServiceName = config.otlpTrace?.serviceName ?? (config.serviceNamePrefix || 'loongsuite-pilot');
+  const innerPrefix = config.innerTrace?.serviceNamePrefix;
+  const innerServiceName = innerPrefix && innerPrefix !== userServiceName ? innerPrefix : undefined;
 
   // 1. User generic OTLP (endpoint via env or config).
   const userOtlpEndpoint = env('LOONGSUITE_PILOT_OTLP_ENDPOINT') ?? config.otlpTrace?.endpoint;
@@ -528,22 +540,25 @@ export function buildOtlpTraceConfig(config: AnalyticsConfig): OtlpTraceFlusherC
       endpoint: ep.endpoint,
       headers: ep.headers,
       compression: ep.compression,
+      serviceName: innerServiceName,
     });
   });
 
   // 4. Inner managed CMS/ARMS shorthand backends.
   const innerCms = Array.isArray(config.innerTrace?.cms) ? config.innerTrace!.cms : [];
   innerCms.forEach((ep, i) => {
     if (!ep.endpoint) return;
-    endpoints.push(cmsEntryToOtlpEndpoint(ep.name ?? `inner-cms-${i}`, ep, armsResourceAttributes));
+    endpoints.push(
+      cmsEntryToOtlpEndpoint(ep.name ?? `inner-cms-${i}`, ep, armsResourceAttributes, innerServiceName),
+    );
   });
 
   const deduped = dedupOtlpEndpoints(endpoints);
   if (deduped.length === 0) return undefined;
 
   const otlp = config.otlpTrace;
   const captureMessageContent = otlp?.captureMessageContent ?? resolveCaptureMessageContent(config.agents);
-  const serviceName = otlp?.serviceName ?? (config.serviceNamePrefix || 'loongsuite-pilot');
+  const serviceName = userServiceName;
   const resourceAttributes = { ...(otlp?.resourceAttributes ?? {}), ...armsResourceAttributes };
 
   return {
@@ -565,14 +580,15 @@ function cmsEntryToOtlpEndpoint(
   name: string,
   cms: { endpoint: string; licenseKey?: string; workspace?: string; project?: string },
   armsResourceAttributes: Record<string, string>,
+  serviceName?: string,
 ): OtlpEndpoint {
   const headers: Record<string, string> = {};
   const armsProject = cms.project || extractArmsProject(cms.endpoint);
   if (cms.licenseKey) headers['x-arms-license-key'] = cms.licenseKey;
   if (armsProject) headers['x-arms-project'] = armsProject;
   if (cms.workspace) headers['x-cms-workspace'] = cms.workspace;
   armsResourceAttributes['acs.arms.service.feature'] = 'genai_app';
-  return { name, endpoint: cms.endpoint, headers };
+  return { name, endpoint: cms.endpoint, headers, serviceName };
 }
 
 /** Stable serialization of headers (sorted keys) for dedup keying. */
@@ -585,17 +601,18 @@ function stableHeaderKey(headers?: Record<string, string>): string {
 }
 
 /**
- * Dedup by normalized URL + full headers. Because the CMS shorthand encodes
- * license-key / project / workspace as headers, this subsumes those fields and
- * also distinguishes generic OTLP backends that share a URL but differ in auth
+ * Dedup by normalized URL + full headers + serviceName. Because the CMS shorthand
+ * encodes license-key / project / workspace as headers, this subsumes those fields
+ * and also distinguishes generic OTLP backends that share a URL but differ in auth
  * headers — so a managed backend is never silently folded into a user endpoint.
- * First occurrence wins.
+ * serviceName is included so the same URL under two service names is kept as two
+ * distinct backends. First occurrence wins.
  */
 function dedupOtlpEndpoints(endpoints: OtlpEndpoint[]): OtlpEndpoint[] {
   const seen = new Set<string>();
   const result: OtlpEndpoint[] = [];
   for (const ep of endpoints) {
-    const key = `${normalizeEndpointUrl(ep.endpoint)}|${stableHeaderKey(ep.headers)}`;
+    const key = `${normalizeEndpointUrl(ep.endpoint)}|${stableHeaderKey(ep.headers)}|${ep.serviceName ?? ''}`;
     if (seen.has(key)) continue;
     seen.add(key);
     result.push(ep);
@@ -696,9 +713,16 @@ function buildSlsConfig(file: ConfigFile | null, serviceNamePrefix: string, inne
   }
 
   if (innerDataConfig?.sls && Array.isArray(innerDataConfig.sls)) {
+    // Managed endpoints get their own __service_name__; only tag them when the
+    // inner prefix differs, so the default case stays byte-identical to before.
+    const innerPrefix = innerDataConfig.serviceNamePrefix;
+    const innerServiceName = innerPrefix && innerPrefix !== serviceNamePrefix ? innerPrefix : undefined;
     const innerEndpoints = innerDataConfig.sls
       .filter(ep => ep.endpoint && ep.logstore)
-      .map((ep, i) => parseSlsEndpointEntry(ep, i));
+      .map((ep, i) => {
+        const parsed = parseSlsEndpointEntry(ep, i);
+        return innerServiceName ? { ...parsed, serviceName: innerServiceName } : parsed;
+      });
     endpoints = [...endpoints, ...innerEndpoints];
   }
 
@@ -788,7 +812,7 @@ function dedupSlsEndpoints(endpoints: SlsEndpoint[]): SlsEndpoint[] {
   const seen = new Set<string>();
   const result: SlsEndpoint[] = [];
   for (const ep of endpoints) {
-    const key = `${normalizeEndpointUrl(ep.endpoint)}|${ep.project}|${ep.logstore}`;
+    const key = `${normalizeEndpointUrl(ep.endpoint)}|${ep.project}|${ep.logstore}|${ep.serviceName ?? ''}`;
     if (seen.has(key)) continue;
     seen.add(key);
     result.push(ep);

src/flushers/otlp-trace-flusher.ts

@@ -70,6 +70,7 @@ interface ResolvedOtlpEndpoint {
   url: string;
   headers: Record<string, string>;
   compression: CompressionAlgorithm;
+  serviceName: string;
 }
 
 interface AgentExportState {
@@ -167,6 +168,7 @@ export class OtlpTraceFlusher extends BaseFlusher {
       url: resolveEndpointUrl(ep.endpoint),
       headers: ep.headers ?? {},
       compression: ep.compression === 'none' ? CompressionAlgorithm.NONE : CompressionAlgorithm.GZIP,
+      serviceName: ep.serviceName || cfg.serviceName,
     }));
     const dataDir = cfg.dataDir ?? os.homedir() + '/.loongsuite-pilot';
     this.pilotVersion = readInstalledVersion(dataDir);
@@ -296,11 +298,17 @@ export class OtlpTraceFlusher extends BaseFlusher {
   // --- Test seam ---
 
   async exportSpansForAgent(agentType: string, spans: ReadableSpan[]): Promise<void> {
-    const exportState = this.getOrCreateExportState(agentType);
     if (this.cfg.debug) {
       await this.writeDebugLog(agentType, spans);
     }
-    await this.exportInBatches(exportState, agentType, spans);
+    // Fan out to every backend; each endpoint belongs to exactly one serviceName
+    // group, so the spans reach each backend once.
+    const serviceNames = [...new Set(this.endpoints.map((e) => e.serviceName))];
+    await Promise.all(
+      serviceNames.map((serviceName) =>
+        this.exportInBatches(this.getOrCreateExportState(agentType, serviceName), agentType, spans),
+      ),
+    );
   }
 
   // --- Internal ---
@@ -374,25 +382,35 @@ export class OtlpTraceFlusher extends BaseFlusher {
   ): Promise<void> {
     if (records.length === 0) return;
     const projectedResourceAttributes = this.collectResourceAttributes(records);
-    const convertKey = this.buildConvertStateKey(agentType, projectedResourceAttributes);
-    const prev = this.convertLocks.get(convertKey) ?? Promise.resolve();
-    const current = prev.then(() => this.doConvertAndExport(
-      agentType,
-      records,
-      projectedResourceAttributes,
-      convertKey,
-    ));
-    this.convertLocks.set(convertKey, current.catch(() => {}));
-    await current;
+    // Convert once per distinct service.name (backends may split into user/inner
+    // service names). Each service name owns an independent convert state, so the
+    // common single-name case still converts exactly once.
+    const serviceNames = [...new Set(this.endpoints.map((e) => e.serviceName))];
+    await Promise.all(
+      serviceNames.map((serviceName) => {
+        const convertKey = this.buildConvertStateKey(agentType, serviceName, projectedResourceAttributes);
+        const prev = this.convertLocks.get(convertKey) ?? Promise.resolve();
+        const current = prev.then(() => this.doConvertAndExport(
+          agentType,
+          serviceName,
+          records,
+          projectedResourceAttributes,
+          convertKey,
+        ));
+        this.convertLocks.set(convertKey, current.catch(() => {}));
+        return current;
+      }),
+    );
   }
 
   private async doConvertAndExport(
     agentType: string,
+    serviceName: string,
     records: AgentActivityEntry[],
     projectedResourceAttributes: Record<string, ResourceProjectionValue>,
     convertKey: string,
   ): Promise<void> {
-    const convertState = this.getOrCreateConvertState(agentType, projectedResourceAttributes, convertKey);
+    const convertState = this.getOrCreateConvertState(agentType, serviceName, projectedResourceAttributes, convertKey);
     const { handler, provider, inMem } = convertState;
     convertState.active += 1;
 
@@ -434,7 +452,7 @@ export class OtlpTraceFlusher extends BaseFlusher {
 
       if (spans.length === 0) return;
 
-      const exportState = this.getOrCreateExportState(agentType);
+      const exportState = this.getOrCreateExportState(agentType, serviceName);
 
       if (this.cfg.debug) {
         await this.writeDebugLog(agentType, spans);
@@ -520,8 +538,9 @@ export class OtlpTraceFlusher extends BaseFlusher {
 
   private getOrCreateConvertState(
     agentType: string,
+    serviceName: string,
     projectedResourceAttributes: Record<string, ResourceProjectionValue> = {},
-    key = this.buildConvertStateKey(agentType, projectedResourceAttributes),
+    key = this.buildConvertStateKey(agentType, serviceName, projectedResourceAttributes),
   ): AgentConvertState {
     let state = this.agentConvertStates.get(key);
     if (state) {
@@ -530,7 +549,7 @@ export class OtlpTraceFlusher extends BaseFlusher {
       return state;
     }
 
-    const resource = this.buildResource(agentType, projectedResourceAttributes);
+    const resource = this.buildResource(agentType, serviceName, projectedResourceAttributes);
     const inMem = new InMemorySpanExporter();
     const provider = new BasicTracerProvider({
       resource,
@@ -564,9 +583,10 @@ export class OtlpTraceFlusher extends BaseFlusher {
 
   private buildConvertStateKey(
     agentType: string,
+    serviceName: string,
     projectedResourceAttributes: Record<string, ResourceProjectionValue>,
   ): string {
-    return `${agentType}|${this.stableJson(projectedResourceAttributes)}`;
+    return `${agentType}|${serviceName}|${this.stableJson(projectedResourceAttributes)}`;
   }
 
   private stableJson(value: Record<string, ResourceProjectionValue>): string {
@@ -635,27 +655,31 @@ export class OtlpTraceFlusher extends BaseFlusher {
     return undefined;
   }
 
-  private getOrCreateExportState(agentType: string): AgentExportState {
-    let state = this.agentExportStates.get(agentType);
+  private getOrCreateExportState(agentType: string, serviceName: string): AgentExportState {
+    const key = `${agentType}|${serviceName}`;
+    let state = this.agentExportStates.get(key);
     if (state) return state;
 
-    const exporters = this.endpoints.map((ep) => ({
-      name: ep.name,
-      exporter: this.exporterFactory({
-        url: ep.url,
-        headers: ep.headers,
-        compression: ep.compression,
+    const exporters = this.endpoints
+      .filter((ep) => ep.serviceName === serviceName)
+      .map((ep) => ({
         name: ep.name,
-      }),
-    }));
+        exporter: this.exporterFactory({
+          url: ep.url,
+          headers: ep.headers,
+          compression: ep.compression,
+          name: ep.name,
+        }),
+      }));
 
     state = { exporters };
-    this.agentExportStates.set(agentType, state);
+    this.agentExportStates.set(key, state);
     return state;
   }
 
   private buildResource(
     agentType: string,
+    serviceName: string,
     projectedResourceAttributes: Record<string, ResourceProjectionValue> = {},
   ): Resource {
     const userAttrs: Record<string, string> = {};
@@ -686,7 +710,7 @@ export class OtlpTraceFlusher extends BaseFlusher {
     }
 
     return new Resource({
-      'service.name': `${this.cfg.serviceName}-${agentType}`,
+      'service.name': `${serviceName}-${agentType}`,
       'service.version': this.pilotVersion,
       'service.instance.id': this.instanceId,
       'service.namespace': 'loongsuite-pilot',