Skip to content

Commit 1b12cf7

Browse files
janhavisjanhavis
authored
Merge branch 'master' into apm-ai-toolkit/new_integration/mqtt/20260310-011726
2 parents f9158bd + dbbc711 commit 1b12cf7

File tree

492 files changed

+26364
-5726
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

492 files changed

+26364
-5726
lines changed

‎.agents/skills/apm-integrations/SKILL.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
---
2+
name: apm-integrations
3+
description: |
4+
This skill should be used when the user asks to "add a new integration",
5+
"instrument a library", "add instrumentation for",
6+
"create instrumentation", "new dd-trace integration",
7+
"add tracing for", "TracingPlugin", "DatabasePlugin", "CachePlugin",
8+
"ClientPlugin", "ServerPlugin", "CompositePlugin", "ConsumerPlugin",
9+
"ProducerPlugin", "addHook", "shimmer.wrap", "orchestrion",
10+
"bindStart", "bindFinish", "startSpan", "diagnostic channel",
11+
"runStores", "reference plugin", "example plugin", "similar integration",
12+
or needs to build, modify, or debug the instrumentation and plugin layers
13+
for a third-party library in dd-trace-js.
14+
---
15+
16+
# APM Integrations
17+
18+
dd-trace-js provides automatic tracing for 100+ third-party libraries. Each integration consists of two decoupled layers communicating via Node.js diagnostic channels.
19+
20+
## Architecture
21+
22+
```
23+
┌──────────────────────────┐ diagnostic channels ┌─────────────────────────┐
24+
│ Instrumentation │ ──────────────────────────▶ │ Plugin │
25+
│ datadog-instrumentations │ apm:<name>:<op>:start │ datadog-plugin-<name> │
26+
│ │ apm:<name>:<op>:finish │ │
27+
│ Hooks into library │ apm:<name>:<op>:error │ Creates spans, sets │
28+
│ methods, emits events │ │ tags, handles errors │
29+
└──────────────────────────┘ └─────────────────────────┘
30+
```
31+
32+
**Instrumentation** (`packages/datadog-instrumentations/src/`):
33+
Hooks into a library's internals and publishes events with context data to named diagnostic channels. Has zero knowledge of tracing — only emits events.
34+
35+
**Plugin** (`packages/datadog-plugin-<name>/src/`):
36+
Subscribes to diagnostic channel events and creates APM spans with service name, resource, tags, and error metadata. Extends a base class providing lifecycle management.
37+
38+
Both layers are always needed for a new integration.
39+
40+
## Instrumentation: Orchestrion First
41+
42+
**Orchestrion is the required default for all new instrumentations.** It is an AST rewriter that automatically wraps methods via JSON configuration, with correct CJS and ESM handling built in. Orchestrion handles ESM code far more reliably than traditional shimmer-based wrapping, which struggles with ESM's static module structure.
43+
44+
Config lives in `packages/datadog-instrumentations/src/helpers/rewriter/instrumentations/<name>.js`. See [Orchestrion Reference](references/orchestrion.md) for the full config format and examples.
45+
46+
### When Shimmer Is Necessary Instead
47+
48+
Shimmer (`addHook` + `shimmer.wrap`) should **only** be used when orchestrion cannot handle the pattern. When using shimmer, **always include a code comment explaining why orchestrion is not viable.** Valid reasons:
49+
50+
- **Dynamic method interception** — methods created at runtime or on prototype chains that orchestrion's static analysis cannot reach
51+
- **Factory patterns** — wrapping return values of factory functions
52+
- **Argument modification** — instrumentations that need to mutate arguments before the original call
53+
54+
If none of these apply, use orchestrion. For shimmer patterns, refer to existing shimmer-based instrumentations in the codebase (e.g., `packages/datadog-instrumentations/src/pg.js`). Always try to use Orchestrion when beginning a new integration!
55+
56+
## Plugin Base Classes
57+
58+
Plugins extend a base class matching the library type. The base class provides automatic channel subscriptions, span lifecycle, and type-specific tags.
59+
60+
```
61+
Plugin
62+
├── CompositePlugin — Multiple sub-plugins (produce + consume)
63+
├── LogPlugin — Log correlation injection (no spans)
64+
├── WebPlugin — Base web plugin
65+
│ └── RouterPlugin — Web frameworks with middleware
66+
└── TracingPlugin — Base for all span-creating plugins
67+
├── InboundPlugin — Inbound calls
68+
│ ├── ServerPlugin — HTTP servers
69+
│ └── ConsumerPlugin — Message consumers (DSM)
70+
└── OutboundPlugin — Outbound calls
71+
├── ProducerPlugin — Message producers (DSM)
72+
└── ClientPlugin — HTTP/RPC clients
73+
└── StoragePlugin — Storage systems
74+
├── DatabasePlugin — Database clients (DBM, db.* tags)
75+
└── CachePlugin — Key-value caches
76+
```
77+
78+
**Wrong base class = complex workarounds.** Always match the library type to the base class.
79+
80+
## Key Concepts
81+
82+
### The `ctx` Object
83+
Context flows from instrumentation to plugin:
84+
85+
- **Orchestrion**: automatically provides `ctx.arguments` (method args) and `ctx.self` (instance)
86+
- **Shimmer**: instrumentation sets named properties (`ctx.sql`, `ctx.client`, etc.)
87+
- **Plugin sets**: `ctx.currentStore` (span), `ctx.parentStore` (parent span)
88+
- **On completion**: `ctx.result` or `ctx.error`
89+
90+
### Channel Event Lifecycle
91+
- `runStores()` for **start** events — establishes async context (always)
92+
- `publish()` for **finish/error** events — notification only
93+
- `hasSubscribers` guard — skip instrumentation when no plugin listens (performance fast path)
94+
- When shimmer is necessary, prefer `tracingChannel` (from `dc-polyfill`) over manual channels — it provides `start/end/asyncStart/asyncEnd/error` events automatically
95+
96+
### Channel Prefix Patterns
97+
- **Orchestrion**: `tracing:orchestrion:<npm-package>:<channelName>` (set via `static prefix`)
98+
- **Shimmer + `tracingChannel`** (preferred): `tracing:apm:<name>:<operation>` (set via `static prefix`)
99+
- **Shimmer + manual channels** (legacy): `apm:{id}:{operation}` (default, no `static prefix` needed)
100+
101+
### `bindStart` / `bindFinish`
102+
Primary plugin methods. Base classes handle most lifecycle; often only `bindStart` is needed to create the span and set tags.
103+
104+
## Reference Integrations
105+
106+
**Always read 1-2 references of the same type before writing or modifying code.**
107+
108+
| Library Type | Plugin | Instrumentation | Base Class |
109+
|---|---|---|---|
110+
| Database | `datadog-plugin-pg` | `src/pg.js` | `DatabasePlugin` |
111+
| Cache | `datadog-plugin-redis` | `src/redis.js` | `CachePlugin` |
112+
| HTTP client | `datadog-plugin-fetch` | `src/fetch.js` | `HttpClientPlugin` (extends `ClientPlugin`) |
113+
| Web framework | `datadog-plugin-express` | `src/express.js` | `RouterPlugin` |
114+
| Message queue | `datadog-plugin-kafkajs` | `src/kafkajs.js` | `Producer`/`ConsumerPlugin` |
115+
| Orchestrion | `datadog-plugin-langchain` | `rewriter/instrumentations/langchain.js` | `TracingPlugin` |
116+
117+
For the complete list by base class, see [Reference Plugins](references/reference-plugins.md).
118+
119+
## Debugging
120+
121+
- `DD_TRACE_DEBUG=true` to see channel activity
122+
- Log `Object.keys(ctx)` in `bindStart` to inspect available context
123+
- Spans missing → verify `hasSubscribers` guard; check channel names match between layers
124+
- Context lost → ensure `runStores()` (not `publish()`) for start events
125+
- ESM fails but CJS works → check `esmFirst: true` in hooks.js (or switch to orchestrion)
126+
127+
## Implementation Workflow
128+
129+
Follow these steps when creating or modifying an integration:
130+
131+
1. **Investigate** — Read 1-2 reference integrations of the same type (see table above). Understand the instrumentation and plugin patterns before writing code.
132+
2. **Implement instrumentation** — Create the instrumentation in `packages/datadog-instrumentations/src/`. Use orchestrion for instrumentation.
133+
3. **Implement plugin** — Create the plugin in `packages/datadog-plugin-<name>/src/`. Extend the correct base class.
134+
4. **Register** — Add entries in `packages/dd-trace/src/plugins/index.js`, `index.d.ts`, `docs/test.ts`, `docs/API.md`, and `.github/workflows/apm-integrations.yml`.
135+
5. **Write tests** — Add unit tests and ESM integration tests. See [Testing](references/testing.md) for templates.
136+
6. **Run tests** — Validate with:
137+
```bash
138+
# Run plugin tests (preferred CI command — handles yarn services automatically)
139+
PLUGINS="<name>" npm run test:plugins:ci
140+
141+
# If the plugin needs external services (databases, message brokers, etc.),
142+
# check docker-compose.yml for available service names, then:
143+
docker compose up -d <service>
144+
PLUGINS="<name>" npm run test:plugins:ci
145+
```
146+
7. **Verify** — Confirm all tests pass before marking work as complete.
147+
148+
## Reference Files
149+
150+
- **[New Integration Guide](references/new-integration-guide.md)** — Step-by-step guide and checklist for creating a new integration end-to-end
151+
- **[Orchestrion Reference](references/orchestrion.md)** — JSON config format, channel naming, function kinds, plugin subscription
152+
- **[Plugin Patterns](references/plugin-patterns.md)** — `startSpan()` API, `ctx` object details, `CompositePlugin`, channel subscriptions, code style
153+
- **[Testing](references/testing.md)** — Unit test and ESM integration test templates
154+
- **[Reference Plugins](references/reference-plugins.md)** — All plugins organized by base class

‎.agents/skills/apm-integrations/references/async-iterator-pattern.md‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# AsyncIterator Orchestrion Transform
2+
3+
**CRITICAL:** If you are working with async iterators or async generators (methods like `stream()`, `*generate()`, or anything returning `Promise<AsyncIterable>`), you **MUST** read and follow this entire document. The AsyncIterator pattern requires TWO plugins and has specific implementation requirements.
4+
5+
## When to Use AsyncIterator
6+
7+
Use `kind: 'AsyncIterator'` in your Orchestrion config when the target method:
8+
9+
- Returns `Promise<AsyncIterable<T>>`
10+
- Returns `Promise<AsyncIterableIterator<T>>`
11+
- Returns `Promise<IterableReadableStream<T>>`
12+
- Is an async generator function: `async *methodName()`
13+
- Returns any promise that resolves to an async iterable
14+
15+
**Examples:**
16+
```javascript
17+
// These ALL need kind: 'AsyncIterator'
18+
async stream(input) { /* returns Promise<AsyncIterable> */ }
19+
async *generate() { /* async generator */ }
20+
async getStream() { /* returns Promise<ReadableStream> */ }
21+
```
22+
23+
## Two-Channel Pattern
24+
25+
**When `kind: 'AsyncIterator'` is used, Orchestrion automatically creates TWO channels:**
26+
27+
1. **Base channel**: `tracing:orchestrion:{package}:{channelName}:*`
28+
- Fires when the method is called (before iteration starts)
29+
- Used to create the span
30+
31+
2. **Next channel**: `tracing:orchestrion:{package}:{channelName}_next:*`
32+
- Fires on EACH iteration (`next()` call)
33+
- Used to finish the span when `result.done === true`
34+
35+
## Critical Implementation Requirements
36+
37+
You **MUST** create TWO plugins to handle both channels. See the complete LangGraph example below for the full implementation pattern.
38+
39+
### 1. Channel Naming
40+
- Base channel: Uses `channelName` from config exactly as-is
41+
- Next channel: Automatically appends `_next` to `channelName`
42+
- Plugin prefix MUST match the full channel name including `_next`
43+
44+
### 2. Plugin Class Relationship
45+
- Next plugin typically extends the main plugin for consistency
46+
- Both plugins MUST use the same `static id`
47+
- Both plugins handle the same integration
48+
49+
### 3. Span Lifecycle
50+
- **Main plugin `bindStart()`**: Creates span via `this.startSpan()`
51+
- **Next plugin `bindStart()`**: Returns inherited store (NO new span)
52+
- **Next plugin `asyncEnd()`**: Finishes span ONLY when `ctx.result.done === true`
53+
- **Either plugin `error()`**: Finishes span immediately on error
54+
55+
### 4. Plugin Export and Registration
56+
Both plugins MUST be:
57+
- Exported from the plugin file: `module.exports = [StreamPlugin, NextStreamPlugin]`
58+
- Registered in the plugin system (see LangGraph example below)
59+
60+
## Common Mistakes
61+
62+
### ❌ Only creating one plugin
63+
64+
### ❌ Creating new span in Next plugin
65+
66+
### ❌ Finishing span on every iteration
67+
68+
### ❌ Wrong channel suffix
69+
70+
## Complete Example: LangGraph Stream
71+
72+
### Orchestrion Config
73+
```javascript
74+
// packages/datadog-instrumentations/src/helpers/rewriter/instrumentations/langgraph.js
75+
module.exports = [
76+
{
77+
module: {
78+
name: '@langchain/langgraph',
79+
versionRange: '>=1.2.0',
80+
filePath: 'dist/pregel/index.js'
81+
},
82+
functionQuery: {
83+
methodName: 'stream',
84+
className: 'Pregel',
85+
kind: 'AsyncIterator' // ← Critical
86+
},
87+
channelName: 'Pregel_stream'
88+
}
89+
]
90+
```
91+
92+
### Plugin Implementation
93+
```javascript
94+
// packages/datadog-plugin-langchain-langgraph/src/tracing.js
95+
const { TracingPlugin } = require('../../dd-trace/src/plugins/tracing')
96+
97+
class StreamPlugin extends TracingPlugin {
98+
static id = 'langgraph'
99+
static prefix = 'tracing:orchestrion:@langchain/langgraph:Pregel_stream'
100+
101+
bindStart (ctx) {
102+
const input = ctx.arguments?.[0]
103+
104+
this.startSpan('langgraph.stream', {
105+
service: this.config.service,
106+
kind: 'internal',
107+
component: 'langgraph',
108+
meta: {
109+
'langgraph.input': JSON.stringify(input)
110+
}
111+
}, ctx)
112+
113+
return ctx.currentStore
114+
}
115+
}
116+
117+
class NextStreamPlugin extends StreamPlugin {
118+
static id = 'langgraph'
119+
static prefix = 'tracing:orchestrion:@langchain/langgraph:Pregel_stream_next'
120+
121+
bindStart (ctx) {
122+
return ctx.currentStore // Inherit span from StreamPlugin
123+
}
124+
125+
asyncEnd (ctx) {
126+
const span = ctx.currentStore?.span
127+
if (!span) return
128+
129+
if (ctx.result.done === true) {
130+
span.setTag('langgraph.chunks', ctx.result.value?.length || 0)
131+
span.finish()
132+
}
133+
}
134+
135+
error (ctx) {
136+
const span = ctx.currentStore?.span
137+
if (span) {
138+
this.addError(ctx?.error, span)
139+
span.finish()
140+
}
141+
}
142+
}
143+
144+
module.exports = [StreamPlugin, NextStreamPlugin]
145+
```
146+
147+
## Testing AsyncIterator Integrations
148+
149+
When testing AsyncIterator instrumentation:
150+
151+
1. **Test span creation**: Verify span starts when method is called
152+
2. **Test iteration**: Verify span stays open during iteration
153+
3. **Test completion**: Verify span finishes when iterator is exhausted
154+
4. **Test early termination**: Verify span finishes if iteration stops early
155+
5. **Test error handling**: Verify span finishes and captures error
156+
157+
```javascript
158+
it('should trace stream() method with AsyncIterator', async () => {
159+
const result = await myLib.stream(input)
160+
161+
// Iterate through results
162+
const chunks = []
163+
for await (const chunk of result) {
164+
chunks.push(chunk)
165+
}
166+
167+
// Verify span exists and finished
168+
await agent.assertSomeTraces(traces => {
169+
const span = traces[0][0]
170+
expect(span.name).to.equal('mylib.stream')
171+
expect(span.meta.component).to.equal('mylib')
172+
// Span should be complete after iteration finishes
173+
})
174+
})
175+
```
176+
177+
## Summary Checklist
178+
179+
When implementing AsyncIterator instrumentation:
180+
181+
- [ ] Orchestrion config uses `kind: 'AsyncIterator'`
182+
- [ ] Created TWO plugin classes (Main + Next)
183+
- [ ] Next plugin prefix has `_next` suffix
184+
- [ ] Both plugins use same `static id`
185+
- [ ] Main plugin creates span in `bindStart()`
186+
- [ ] Next plugin returns inherited store in `bindStart()`
187+
- [ ] Next plugin checks `result.done === true` before finishing span
188+
- [ ] Both plugins handle errors and finish span
189+
- [ ] Both plugins exported in module.exports array
190+
- [ ] Tests verify span lifecycle (start, iteration, completion)

0 commit comments

Comments
 (0)