test(plugin): version-by-version plugin test framework + CAP scenario (closes #565)

.github/workflows/plugin-test.yml

@@ -0,0 +1,50 @@
+name: Plugin Test
+
+# Per-plugin tests: each plugin scenario runs against a dedicated mock collector and asserts the REAL
+# emitted segments, version-by-version (see test/plugin/). One matrix job PER PLUGIN so results show
+# per plugin in the checks UI (like apache/skywalking-java's plugin-test). Heavier than unit tests
+# (Docker + a Java collector build), so it runs only when the agent, a plugin, or the framework changes.
+on:
+  push:
+    branches: [main]
+    paths: ['src/**', 'test/plugin/**', '.github/workflows/plugin-test.yml']
+  pull_request:
+    paths: ['src/**', 'test/plugin/**', '.github/workflows/plugin-test.yml']
+
+concurrency:
+  group: plugin-test-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  # Discover the scenarios so each becomes its own matrix entry (no need to edit this file per plugin).
+  discover:
+    runs-on: ubuntu-latest
+    outputs:
+      scenarios: ${{ steps.list.outputs.scenarios }}
+    steps:
+      - uses: actions/checkout@v4
+      - id: list
+        run: |
+          scenarios=$(ls -d test/plugin/scenarios/*/ | xargs -n1 basename | jq -R . | jq -cs .)
+          echo "scenarios=$scenarios" >> "$GITHUB_OUTPUT"
+          echo "scenarios: $scenarios"
+
+  plugin-test:
+    needs: discover
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    strategy:
+      fail-fast: false
+      matrix:
+        scenario: ${{ fromJSON(needs.discover.outputs.scenarios) }}
+    name: ${{ matrix.scenario }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive   # the protocol-v3 codegen source
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '11'
+      - name: Run ${{ matrix.scenario }} (build collector + per-version build/run/validate)
+        run: bash test/plugin/run.sh ${{ matrix.scenario }}

CLAUDE.md

@@ -43,6 +43,11 @@ themes/hugo-... , hugo.toml, i18n/   docs site theme + config (.github/workflows
 - Config: `skyapm.json` (or env `SKYWALKING__...`, double-underscore). Agent no-ops if `ServiceName`
   empty or `SkyWalking:Enable=false`. Full reference: `content/docs/guides/skyapm_config.md`.
 - Plugins: a fixed default set is auto-registered; others are opt-in via the `AddSkyAPM(ext => …)` lambda.
+- Plugin tests (`test/plugin/`): each scenario runs the **real** instrumented library against a dedicated
+  mock collector and asserts the **real emitted segments**, version-by-version (`run.sh`). Tests must reflect
+  an **actual run** — never fabricate or assume a trace: capture via the collector's `/receiveData`, validate
+  via `/dataValidate`, and base `config/expectedData.yaml` on that. Only list a `<TFM> <version>` in
+  `support-version.list` once it has actually been run and passed (don't declare versions you haven't validated).
 - The repo also pins per-TFM package versions via `Condition="'$(TargetFramework)' == 'netX'"` blocks.
 - Project is **independent** — reports to Apache SkyWalking but is not an ASF/SkyWalking sub-project.
 - Commits: do **not** add a `Co-Authored-By: Claude …` trailer (or any AI co-author trailer) to commit messages.

content/docs/guides/plugin-development.md

@@ -0,0 +1,177 @@
+---
+title: "Plugin development"
+weight: 40
+---
+
+SkyAPM-dotnet instruments libraries through **diagnostic plugins**. A plugin is a thin adapter
+that subscribes to a library's [`DiagnosticSource`](https://learn.microsoft.com/dotnet/api/system.diagnostics.diagnosticsource)
+events and turns them into SkyWalking **trace segments**. This guide shows how to write one,
+using the existing plugins (`SkyApm.Diagnostics.HttpClient`, `SkyApm.Diagnostics.SqlClient`,
+`SkyApm.Diagnostics.CAP`, …) as the reference shape.
+
+## How a plugin works
+
+Most .NET libraries already emit `DiagnosticSource` events (ASP.NET Core, HttpClient,
+EF Core, gRPC, CAP, MongoDB, …). A plugin listens to the named listener and maps each event to a
+span. The agent owns the wiring:
+
+```
+library DiagnosticSource  ──►  TracingDiagnosticProcessorObserver  (matches ListenerName)
+                                        │
+                                        ▼
+                          your ITracingDiagnosticProcessor          (one [DiagnosticName] method per event)
+                                        │  CreateEntry/Local/ExitSegmentContext + AddTag/AddLog
+                                        ▼
+                                  ITracingContext.Release  ──►  emitted SegmentRequest (the real trace)
+```
+
+You implement only the middle box.
+
+## 1. Implement `ITracingDiagnosticProcessor`
+
+Create a class in a `SkyApm.Diagnostics.<Library>` project that implements
+[`ITracingDiagnosticProcessor`](https://github.com/SkyAPM/SkyAPM-dotnet/blob/main/src/SkyApm.Abstractions/Diagnostics/ITracingDiagnosticProcessor.cs).
+Its `ListenerName` must equal the library's `DiagnosticSource`/`DiagnosticListener` name, and each
+event handler is annotated with `[DiagnosticName("<the.event.name>")]`:
+
+```csharp
+public class MyLibTracingDiagnosticProcessor : ITracingDiagnosticProcessor
+{
+    public string ListenerName => "MyLib.DiagnosticListener";   // the source's name
+
+    private readonly ITracingContext _tracingContext;
+    private readonly IExitSegmentContextAccessor _exitSegmentContextAccessor;
+    private readonly TracingConfig _tracingConfig;
+
+    public MyLibTracingDiagnosticProcessor(ITracingContext tracingContext,
+        IExitSegmentContextAccessor exitSegmentContextAccessor,
+        IConfigAccessor configAccessor)
+    {
+        _tracingContext = tracingContext;
+        _exitSegmentContextAccessor = exitSegmentContextAccessor;
+        _tracingConfig = configAccessor.Get<TracingConfig>();
+    }
+
+    [DiagnosticName("MyLib.Command.Before")]
+    public void BeforeCommand([Object] CommandEventData data)   // [Object] binds the raw event payload
+    {
+        var context = _tracingContext.CreateExitSegmentContext("MyLib/" + data.Operation, data.Host);
+        context.Span.SpanLayer = SpanLayer.DB;
+        context.Span.Component = Components.MYLIB;          // see "Components" below
+        context.Span.AddTag(Tags.DB_STATEMENT, data.Sql);
+    }
+
+    [DiagnosticName("MyLib.Command.After")]
+    public void AfterCommand([Object] CommandEventData data)
+    {
+        var context = _exitSegmentContextAccessor.Context;
+        if (context == null) return;
+        _tracingContext.Release(context);                  // Release emits the segment
+    }
+
+    [DiagnosticName("MyLib.Command.Error")]
+    public void ErrorCommand([Object] CommandErrorEventData data)
+    {
+        var context = _exitSegmentContextAccessor.Context;
+        if (context == null) return;
+        context.Span.ErrorOccurred(data.Exception, _tracingConfig);
+        _tracingContext.Release(context);
+    }
+}
+```
+
+Key points:
+
+- The `[Object]` parameter attribute tells the dispatcher to pass the event's raw payload object;
+  its type is the library's event class, which the plugin references at compile time.
+- A handler that throws does not crash the app — `TracingDiagnosticObserver` catches it. A
+  **structural binding error** (`MissingMethodException`/`TypeLoadException`, i.e. the library
+  version is incompatible with what the plugin compiled against) is logged **once** and that handler
+  is then disabled, rather than spamming on every event (see *Version coupling* below).
+
+## 2. Create segments with `ITracingContext`
+
+[`ITracingContext`](https://github.com/SkyAPM/SkyAPM-dotnet/blob/main/src/SkyApm.Abstractions/Tracing/ITracingContext.cs)
+is the only API you need to produce spans. Pick the span kind by where the work happens:
+
+| Method | Use for | Accessor to read it back |
+|---|---|---|
+| `CreateEntrySegmentContext(operationName, carrierHeader)` | inbound work (a received request/message) | `IEntrySegmentContextAccessor` |
+| `CreateLocalSegmentContext(operationName)` | in-process work (persistence, serialization) | `ILocalSegmentContextAccessor` |
+| `CreateExitSegmentContext(operationName, networkAddress[, carrierHeader])` | outbound work (a DB/HTTP/MQ call) | `IExitSegmentContextAccessor` |
+
+Always pair a `Create…` with a `Release(context)` (typically on the *After*/*Error* event). On the
+matching *After* you usually read the context back from the accessor rather than threading it through.
+On every span set:
+
+- `Span.SpanLayer` — `Http` / `Database` (DB) / `RPCFramework` / `MQ` / `Cache` / `Unknown`.
+- `Span.Component` — the component id (below).
+- `Span.Peer` — the remote address for exit spans (a [peer formatter](#peers) helps here).
+- `Span.AddTag(Tags.X, value)` and `Span.AddLog(LogEvent.…)` for detail; `Span.ErrorOccurred(ex, config)` on failure.
+
+## 3. Cross-process context propagation
+
+For spans that cross a process boundary (an MQ message, an outbound RPC), inject/extract the
+SkyWalking `sw8` headers so the trace stays connected:
+
+- **Producer / client (exit):** build an `ICarrierHeaderCollection` over the outbound message
+  headers and call `ICarrierPropagator.Inject(context, header)` so the downstream side continues the trace.
+- **Consumer / server (entry):** wrap the inbound headers in an `ICarrierHeaderCollection` and pass it
+  to `CreateEntrySegmentContext(operationName, carrierHeader)`; the agent extracts the parent ref.
+
+`SkyApm.Diagnostics.CAP`'s `CapCarrierHeaderCollection` is a concrete example of adapting a library's
+message headers to `ICarrierHeaderCollection`.
+
+## Components
+
+Span components are numeric ids from the shared SkyWalking registry
+([`component-libraries.yml`](https://github.com/apache/skywalking/blob/master/oap-server/server-starter/src/main/resources/component-libraries.yml)),
+surfaced as `SkyApm.Common.Components`. Reuse an existing id where one fits (e.g. `Components.HTTPCLIENT`,
+`Components.SQLCLIENT`, `Components.CAP`); a brand-new library needs an id registered upstream in
+SkyWalking first, otherwise the UI shows it as "Unknown".
+
+## Peers
+
+Exit spans should carry the remote address in `Span.Peer`. Where the address must be parsed from a
+connection/config object, add a `SkyApm.PeerFormatters.<Library>` implementing `IDbPeerFormatter`
+(see `SkyApm.PeerFormatters.MySqlConnector`) and resolve it through the injected `IPeerFormatter`.
+
+## 4. Register the plugin
+
+Expose an opt-in extension on `SkyApmExtensions` that registers your processor as an
+`ITracingDiagnosticProcessor` singleton — the agent discovers all registered processors and
+subscribes each to its `ListenerName`:
+
+```csharp
+public static class SkyWalkingBuilderExtensions
+{
+    public static SkyApmExtensions AddMyLib(this SkyApmExtensions extensions)
+    {
+        extensions.Services.AddSingleton<ITracingDiagnosticProcessor, MyLibTracingDiagnosticProcessor>();
+        return extensions;
+    }
+}
+```
+
+Users opt in via the setup lambda:
+
+```csharp
+services.AddSkyAPM(ext => ext.AddMyLib());
+```
+
+A small, always-on default set is registered automatically; everything else (CAP, MassTransit,
+MongoDB, …) is opt-in like the above.
+
+## Version coupling (important)
+
+A plugin is **compiled against a specific version** of the library it instruments, but at runtime
+NuGet unifies that library to the **application's** installed version. If the app's version changed
+the diagnostic event types in a binary-incompatible way, the handler throws `MissingMethodException`
+at runtime (see [#565](https://github.com/SkyAPM/SkyAPM-dotnet/issues/565)). The agent now degrades
+gracefully (logs once, disables the handler), but the plugin still won't trace that library until the
+versions line up. Two consequences for plugin authors:
+
+- Pin the plugin's `DotNetCore.X` reference per target framework (see the existing per-TFM
+  `Condition="'$(TargetFramework)' == 'netX'"` `PackageReference` blocks) and document the supported range.
+- **Test the plugin against the real library, version by version** — see
+  [Plugin testing](plugin-testing.md).

content/docs/guides/plugin-testing.md

@@ -0,0 +1,95 @@
+---
+title: "Plugin testing"
+weight: 41
+---
+
+A diagnostic plugin is only correct if the agent **emits the right trace** — and the emitted trace is
+the only thing that is real (intermediate state isn't). So plugin tests run the **real** instrumented
+library, let the agent report to a **dedicated mock collector**, and assert on the **captured
+segments**. Because a plugin is compiled against one version of a library but runs against whatever
+version the application installs (the cause of [#565](https://github.com/SkyAPM/SkyAPM-dotnet/issues/565)),
+each scenario runs **version-by-version**.
+
+This mirrors the [Apache SkyWalking Java plugin-test framework](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/plugin-test/).
+The harness lives in [`test/plugin/`](https://github.com/SkyAPM/SkyAPM-dotnet/tree/main/test/plugin);
+this guide explains how it works and how to add a scenario. (For a full OAP + storage backend test
+instead, see the heavier [`test/e2e/`](https://github.com/SkyAPM/SkyAPM-dotnet/tree/main/test/e2e).)
+
+## How it works
+
+```
+scenario app  ──gRPC :19876──►  mock collector  ──HTTP :12800──►  /receiveData  /dataValidate
+(agent from source,             (a SkyWalking receiver,           (captured       (validate vs
+ real library)                   not real OAP+DB)                  segments)        expectedData.yaml)
+```
+
+For each scenario and each `<TFM> <version>` line in its `support-version.list`, `run.sh`:
+
+1. builds the scenario app image — the **agent from source** plus the instrumented library at that
+   version (NuGet unifies the version up, so the plugin runs against it just like a real app);
+2. boots the app together with the mock collector;
+3. drives the scenario's `/case/<name>` entry endpoint;
+4. POSTs the scenario's `config/expectedData.yaml` to the collector's `/dataValidate`; `200`/`success`
+   passes, otherwise the validator prints which span/field differed.
+
+### The dedicated collector
+
+The collector is `apache/skywalking-agent-test-tool`'s `mock-collector`, built from a **pinned commit**
+(the same one `apache/skywalking-java` pins today). The published `skyapm/mock-collector:latest` is from
+2019 and predates the V8 management protocol the current agent needs, so do **not** use it — `run.sh`
+builds the pinned version for you (override with `COLLECTOR_IMAGE=…` to reuse a pre-built one).
+
+## Authoring `expectedData.yaml`
+
+Run the scenario once and read what the collector actually captured:
+
+```bash
+curl -s http://localhost:12800/receiveData
+```
+
+Then keep the segments/spans you want to assert and replace volatile fields with matchers. The validator
+([SkyWalking validator DSL](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/plugin-test/#expecteddatayaml-format))
+matches **expected segments as a subset** of the captured ones (so you needn't describe every segment),
+but within a matched segment it checks **span count, tags, and span logs exactly** — so describe a span's
+full tag/log set, using matchers for anything variable.
+
+Matchers: `not null`, `not blank`, `nq <v>` (not-equal), `gt`/`ge <v>`, `eq <v>` (default), `start with`,
+`end with`. Example from the CAP scenario (the publisher span):
+
+```yaml
+segmentItems:
+  - serviceName: cap-scenario
+    segmentSize: nq 0
+    segments:
+      - segmentId: not null
+        spans:
+          - operationName: CAP/skyapm.plugin.test.cap/Publisher
+            spanLayer: MQ
+            componentId: 3004        # Components.CAP
+            spanType: Entry
+            startTime: nq 0          # volatile -> matcher
+            endTime: nq 0
+            isError: false
+            peer: localhost
+            tags:
+              - {key: mq.topic, value: skyapm.plugin.test.cap}
+              - {key: mq.broker, value: localhost}
+            logs:
+              - logEvent: [{key: event, value: Event Publishing Start}]
+              - logEvent: [{key: message, value: CAP message publishing start...}]
+              - logEvent: [{key: event, value: Event Publishing End}]
+              - logEvent: [{key: message, value: not blank}]   # variable content
+```
+
+## Adding a scenario
+
+1. Create `test/plugin/scenarios/<lib>-scenario/` with a minimal app referencing the agent and the
+   plugin **from source** and the instrumented library at a build-arg version, exposing `/case/<name>`
+   (which performs the instrumented action) and `/case/healthCheck`.
+2. Add `skyapm.json` (report to `mock-collector:19876`), a `Dockerfile` (agent built from source;
+   library version as a build arg), and a `docker-compose.yml` (app + collector).
+3. Add `support-version.list` — the **latest patch of each supported minor** of the library.
+4. Author `config/expectedData.yaml` from a real `/receiveData` capture, as above.
+5. Run `bash test/plugin/run.sh <lib>-scenario` until it passes.
+
+See [Plugin development](plugin-development.md) for writing the plugin itself.

src/SkyApm.Core/Diagnostics/TracingDiagnosticObserver.cs

@@ -17,6 +17,7 @@
  */
 
 using System;
+using System.Collections.Concurrent;
 using System.Collections.Generic;
 using System.Linq;
 using SkyApm.Logging;
@@ -26,6 +27,7 @@ namespace SkyApm.Diagnostics
     internal class TracingDiagnosticObserver : IObserver<KeyValuePair<string, object>>
     {
         private readonly Dictionary<string, TracingDiagnosticMethod> _methodCollection;
+        private readonly ConcurrentDictionary<string, byte> _disabledDiagnostics = new ConcurrentDictionary<string, byte>();
         private readonly ILogger _logger;
 
         public TracingDiagnosticObserver(ITracingDiagnosticProcessor tracingDiagnosticProcessor,
@@ -54,10 +56,26 @@ public void OnNext(KeyValuePair<string, object> value)
             if (!_methodCollection.TryGetValue(value.Key, out var method))
                 return;
 
+            if (_disabledDiagnostics.ContainsKey(value.Key))
+                return;
+
             try
             {
                 method.Invoke(value.Key, value.Value);
             }
+            catch (Exception exception) when (exception is MissingMemberException || exception is TypeLoadException)
+            {
+                // A binding/structural error means the instrumented library's version is incompatible
+                // with this plugin (e.g. a diagnostic event type or property changed across versions).
+                // It fails deterministically for every event, so disable this handler after logging once
+                // instead of spamming the same error on every call (issue #565).
+                if (_disabledDiagnostics.TryAdd(value.Key, 0))
+                {
+                    _logger.Warning($"Disabled diagnostic handler '{value.Key}' due to an incompatible " +
+                                    "instrumented library version; ensure the installed package version matches " +
+                                    $"this SkyAPM plugin. {exception.GetType().Name}: {exception.Message}");
+                }
+            }
             catch (Exception exception)
             {
                 _logger.Error("Invoke diagnostic method exception.", exception);

src/SkyApm.Diagnostics.CAP/CapCarrierHeaderCollection.cs

@@ -44,7 +44,10 @@ public IEnumerator<KeyValuePair<string, string>> GetEnumerator()
 
         public void Add(string key, string value)
         {
-            _messageHeaders.Add(key, value);
+            // Overwrite rather than Dictionary.Add: a message can already carry an sw8 header when
+            // the transport re-publishes it (e.g. CAP's in-memory queue), and Add throws on a
+            // duplicate key. Injecting the carrier must be idempotent.
+            _messageHeaders[key] = value;
         }
 
         IEnumerator IEnumerable.GetEnumerator()