Skip to content

docs(langsmith): add Engine webhook events reference#4008

Draft
Andy Young (ayoung19) wants to merge 4 commits into
mainfrom
andyy/engine-webhook-events
Draft

docs(langsmith): add Engine webhook events reference#4008
Andy Young (ayoung19) wants to merge 4 commits into
mainfrom
andyy/engine-webhook-events

Conversation

@ayoung19
Copy link
Copy Markdown
Contributor

@ayoung19 Andy Young (ayoung19) commented May 14, 2026

Summary

Adds a Stripe-style reference page documenting the webhook events the LangSmith Engine emits today. The Engine page already mentions webhook configuration but does not describe the wire format consumers need to handle the events, and the Issues UI is expected to link to this page from the webhook settings panel.

The page covers:

  • The two event types currently emitted (issue.created, issue.trace.added) with a full example payload for each.
  • The shared envelope (id, type, created, request_id, data) and the data.object / data.trace schemas.
  • Delivery semantics: POST with JSON body, 20s per-attempt timeout, up to 4 total attempts (1 initial + 3 retries with exponential backoff), byte-identical retried payloads (dedupe on id), and request_id-based batch coalescing.
  • Severity threshold filtering (0=urgent … 3=low).
  • Security guidance: SSRF policy validates URLs at write- and connect-time; no HMAC signing today, so authenticate via a custom header (e.g. Authorization: Bearer …); https:// is strongly recommended even though the SSRF default policy accepts both http and https.

New file lives next to the existing langsmith/engine page in the Observability tab and is registered in src/docs.json.

Known gaps — acknowledged but punted

Cross-checking the page against the in-repo Slack consumer (smith-issues-agent/webhook_consumers/slack/handler.ts, PR langchain-ai/langchainplus#24520) surfaced two trace-linking details a consumer can't fully reconstruct from this page alone:

  1. trace://<run-id> Markdown link convention inside data.object.description and data.trace.comment. The agent is prompted to embed trace references as [label](trace://<run-id>) rather than raw IDs (smith-issues-agent/smith_issues_agent/prompt.py:1234). The Slack consumer rewrites these to clickable LangSmith trace URLs.
  2. No trace URL is shipped in the payload. Consumers who want a clickable trace link have to hand-construct it from tenant_id + session_id + trace_id, the way the Slack consumer does. Compare data.object.url, which we ship precisely so consumers don't reverse-engineer the issue URL.

Both are intentionally out of scope here. Trace linking has known issues across the product and the behavior is expected to change; we want to nail that down before committing to it as a webhook wire contract. Once that lands, fold (1) into the description/comment field docs and add (2) as either a shipped trace.url field or a documented URL template.

Provenance

Verified against origin/main (langchain-ai/langchainplus). Implementation PRs:

  • langchain-ai/langchainplus#23509 — initial webhooks (CRUD + delivery)
  • langchain-ai/langchainplus#24342 — switched to the current Stripe-style envelope and singular issue.created naming
  • langchain-ai/langchainplus#24236 — added issue.trace.added
  • langchain-ai/langchainplus#24520 — in-repo val.town Slack consumer (used here as a cross-check)

Release Note

none

Document the two webhook events the LangSmith Engine emits today
(issue.created, issue.trace.added), the shared envelope, payload shapes,
delivery semantics, severity filtering, and security guidance.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added internal langsmith For docs changes to LangSmith labels May 14, 2026
Andy Young (ayoung19) and others added 3 commits May 14, 2026 13:23
- SSRF DefaultPolicy allows both http and https; correct the page from
  "https only" to "both accepted, https recommended".
- asynq MaxRetry is retries, not total attempts. webhookMaxRetry=3 yields
  up to 4 total attempts (1 initial + 3 retries).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

Mintlify preview branch generated: preview-andyye-1778820730-7214c62

Site preview: https://langchain-5e9cc07a-preview-andyye-1778820730-7214c62.mintlify.app

Preview links may take a few minutes to start working while the deployment finishes.

Changed documentation pages (preview deep links):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Andy Young (@ayoung19) please review the changes I committed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal langsmith For docs changes to LangSmith

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants