Add issue-detection-triage skill reference - #64
Open
annabellscha wants to merge 2 commits into
Open
Conversation
Adds references/issue-detection-triage.md: a workflow for sweeping recent production traffic across every dimension where issues hide, ranking findings P0-P3 with a reproducible decision rule, and reporting them with evidence and ready-to-paste fix prompts. Also fixes the pagination tip in cli.md (verified against the CLI: traces/scores are page-based, only observations is cursor-based, metrics has no pagination flags) and bumps both plugin manifests to 1.3.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lotte-Verheyden
requested changes
Jul 15, 2026
Lotte-Verheyden
left a comment
Lotte-Verheyden
left a comment
Member
There was a problem hiding this comment.
I left some smaller comments on changes to existing files, and read through the new file too. Some thoughts on the new file, but I didn't test it myself so I can't really say whether they are valid or not:
- the file is very long, I often saw duplicate content or content that looks redundant. If there's a reason for adding this many lines/stating things multiple times after testing it's okay, but if it's only because the agent added it I'd cut the content significantly. Thinking about (1) token cost when each LLM request has this skill file in context and (2) how well it would work with a smaller model like sonnet, I can imagine it starts missing things
- instead of showing cli commands explicitly, best to describe what the agent should do with the CLI, and instruct it to look up the CLI schema if necessary (although I think the agent would do that automatically). Most reliable way to avoid giving outdated instructions via our skill
- one thing I always test is whether the new reference file gets invoked when I expect it to, and gets ignored when it should be ignored. You maybe already did this, but if not, best to spin up some agents with different input prompts
| - Prefer `scores` over `legacy-score-v1s` for list/get operations | ||
| - For broad trace queries, `traces list` can time out on Langfuse Cloud — use `observations list` (with `--trace-id` if you're traversing from a known trace) instead. See the [Observations API docs](https://langfuse.com/docs/api-and-data-platform/features/observations-api) for the v1 → v2 mapping. | ||
| - Pagination: legacy v1 endpoints use `--limit` and `--page`; modern endpoints (`observations`, `metrics`, `scores`) use cursor-based pagination — pass `--limit`, then thread `meta.cursor` from the response into the next request's `--cursor` | ||
| - Pagination: `traces` and `scores` use `--limit` + `--page` (starts at 1; `meta.totalPages` tells you how many pages). `observations` uses cursor-based pagination — pass `--limit`, then thread `meta.cursor` (NOT `meta.nextCursor`) from the response into the next request's `--cursor`. `metrics` takes a query object and has no pagination flags. |
Member
There was a problem hiding this comment.
Was this change intentional or just the agent changing something because it did something wrong? Not sure if we should be so specific (e.g. "meta.cursor (NOT meta.nextCursor)")
| ### Detailed CLI Reference | ||
|
|
||
| For common workflows, tips, and full usage patterns, see [references/cli.md](references/cli.md). | ||
| For common workflows, tips, pagination, and endpoint preferences, see [references/cli.md](references/cli.md). Use-case-specific CLI recipes live in the relevant reference file. |
Member
There was a problem hiding this comment.
If not necessary I'd discard this change.
| @@ -1 +1,3 @@ | |||
| .DS_Store | |||
| skills/langfuse/references/.env | |||
| skills/langfuse/references/.env2 | |||
|
|
||
| ### 0. Connect | ||
|
|
||
| Credentials and CLI basics: follow the **`langfuse` skill** CLI section and [cli.md](cli.md). Verify the connection with a single one-row `traces list` before proceeding. |
Member
There was a problem hiding this comment.
Try not to include specific code/CLI examples (these get outdated) and instead instruct the agent to look up the schema for a specific command
| - **Window:** default to the user's ask ("last hour", "today"). Get current UTC, compute `--from-timestamp`. If unsure of data recency, list the latest traces first and anchor on the most recent timestamp. | ||
| - **Separate real app traffic from machinery.** This is critical and easy to get wrong. Langfuse projects mix: | ||
| - Real application traces (what you want). | ||
| - **Evaluator executions** — LLM-as-a-judge runs, usually `environment = "langfuse-llm-as-a-judge"` and named `"Execute evaluator: <name>"`. **Exclude these** unless the user is asking about the evaluators themselves. |
Member
There was a problem hiding this comment.
I'd expect things like "environment = "langfuse-llm-as-a-judge"" to cause hallucinations when users use a different name for that environment
Member
There was a problem hiding this comment.
If there was a reason for adding this line you can keep it of course. But otherwise best to remove these
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new use-case reference,
skills/langfuse/references/issue-detection-triage.md: a systematic triage workflow over a Langfuse project's recent production traffic — sweep every dimension where issues can hide (reliability, latency, cost, tool/agent behavior, scores/evaluators, user friction, output quality, inputs/scope, segmentation, coverage), rank findings P0–P3 with a reproducible decision rule, and report them with evidence, example trace links, and ready-to-paste fix prompts.Also included
SKILL.md: one-line routing entry for the new reference (peragents.md, "when to use" lives only in the list entry + the reference frontmatter), plus a small update to the CLI-reference pointer and a typo fix.cli.mdpagination fix: verified against the CLI's own--help—traces listandscores listuse--limit+--page(withmeta.totalPages), onlyobservations listis cursor-based (next cursor ismeta.cursor, notmeta.nextCursor), andmetrics listtakes a query object with no pagination flags. The previous tip incorrectly claimedscoresandmetricsare cursor-based.agents.md).The companion
improvement-loop.mdreference is intentionally not part of this PR and will follow separately.🤖 Generated with Claude Code