Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
{
"name": "grafana-k6",
"version": "0.1.0",
"version": "0.2.0",
"description": "Claude Skills for k6 performance testing with plan-builder-validate lifecycle: planning strategy, runnable artifact generation, and script/config quality validation",
"author": {
"name": "charlyautomatiza",
"url": "https://github.com/charlyautomatiza/grafana-k6-skill"
"url": "https://github.com/charlyautomatiza/grafana-k6-plugin"
},
"repository": {
"type": "git",
"url": "https://github.com/charlyautomatiza/grafana-k6-plugin.git"
},
"repository": "charlyautomatiza/grafana-k6-skill",
"license": "MIT"
}
9 changes: 8 additions & 1 deletion skills/k6-builder/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,12 @@ Do not emit final artifact content after this fallback.
- Use `__ENV.BASE_URL` (or protocol-equivalent env var) placeholder and mark all unresolved values as `[assumption-based]`.
- Append a `pending_questions` block with one direct question for the unresolved target.

**Upstream exploratory handoff policy:**
- If input from `k6-plan` includes `plan_mode: exploratory`, `ready_for_builder: false`, `inferred_targets`, `inferred_flows`, or `pending_clarifications`, do not treat that input as executable-ready.
- Preserve every `inferred` or `[assumption-based]` field in the builder output and carry unresolved items into a `pending_questions` block.
- Generate a partial template only when the user explicitly wants to continue from the exploratory plan; otherwise ask for the highest-priority runnable blocker.
- Never upgrade inferred routes, auth requirements, or SLA values into confirmed build inputs without explicit confirmation.

Additional questions must be integrated into the same question system, not handled as a separate side flow:
- Add an HTTP method question when `protocol=http` and the method cannot be inferred safely.
- Add one or more authentication questions when auth is required or unknown and executable output depends on it.
Expand Down Expand Up @@ -113,7 +119,8 @@ Apply this policy in every builder response:
1. Report only data that is verifiably known from user input or direct tool/runtime evidence in the current task.
2. Do not present inferred values as facts in the final output.
3. If a value is required but not verified, mark it as `unknown` and ask a single unblocker question.
4. Do not fabricate machine/runtime numbers in summaries.
4. If upstream planning metadata is exploratory, preserve its uncertainty markers and keep `ready_for_builder: false` semantics intact.
5. Do not fabricate machine/runtime numbers in summaries.

## Terminology Contract

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,6 @@ Use this policy for all user-facing summaries.
1. If value is not verified, mark as `unknown`.
2. Ask one unblocker question for critical unknowns.
3. Do not mix inferred and verified data in the same fact line.
4. Keep final summary concise and practical.
4. Preserve `[assumption-based]`, `inferred_*`, `pending_clarifications`, and `ready_for_builder: false` markers received from `k6-plan`.
5. Do not promote exploratory planning metadata to confirmed runnable facts during generation.
6. Keep final summary concise and practical.
100 changes: 68 additions & 32 deletions skills/k6-plan/SKILL.md

Large diffs are not rendered by default.

56 changes: 56 additions & 0 deletions skills/k6-plan/evals/evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,62 @@
"name": "journey_steps",
"text": "Plan must preserve the full requested journey sequence"
},
},
{
"id": 13,
"prompt": "Quiero diseñar una prueba de performance para validar si podré soportar 2 mil usuarios concurrentes durante 1h constante para los flujos principales de mi actual backend, pero no sé qué SLA debería exigir.",
"expected_output": "exploratory-backend-concurrency-plan",
"files": [],
"assertions": [
{
"name": "exploratory_mode_enabled",
"text": "Must activate exploratory mode and label the plan as provisional or exploratory instead of stopping at missing SLA"
},
{
"name": "suggested_sla_candidates",
"text": "Must include one or more suggested SLA candidates sourced from documented defaults and keep them explicitly unconfirmed"
},
{
"name": "pending_clarifications_present",
"text": "Must include pending clarifications for unresolved target, auth, or flow confirmation details"
}
]
},
{
"id": 14,
"prompt": "Plan a load test for my current application codebase to validate the main user flows. I have not mapped endpoints or auth yet, but please suggest the likely flows from the app context and keep going.",
"expected_output": "application-context-exploratory-plan",
"files": [],
"assertions": [
{
"name": "codebase_flow_exploration",
"text": "Must prefer application-context exploration over immediately asking for endpoints or auth when the workspace likely contains the app under test"
},
{
"name": "inferred_targets_and_flows",
"text": "Must include inferred targets or inferred flows with brief evidence and prioritization"
},
{
"name": "ready_for_builder_false",
"text": "Must keep the exploratory handoff marked as not ready for runnable artifact generation"
}
]
},
{
"id": 15,
"prompt": "I need a performance plan for my service but I'm not sure whether this should be load or stress, and I also don't know the protocol yet.",
"expected_output": "scenario-ambiguity-clarification",
"files": [],
"assertions": [
{
"name": "clarification_over_exploration",
"text": "Must ask for clarification instead of activating exploratory mode when scenario type remains ambiguous"
},
{
"name": "canonical_missing_fields",
"text": "Must emit the canonical missing/why/next_question clarification contract"
}
]
{
"name": "frontend_sla_threshold",
"text": "Plan must include threshold/KPI aligned to p95<5s"
Expand Down
21 changes: 20 additions & 1 deletion skills/k6-plan/references/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Complete reference materials for comprehensive k6 performance test planning.
- **[protocol-guide.md](protocol-guide.md)** — HTTP, gRPC, and browser protocol patterns and examples
- **[data-integration.md](data-integration.md)** — CSV/JSON/environment variable data source patterns
- **[certainty-and-compact-output.md](certainty-and-compact-output.md)** — Verified-facts-only and compact response guidance
- **[exploratory-planning.md](exploratory-planning.md)** — Auto-detected exploratory mode, inferred flows, and provisional handoff rules

## Core Topics

Expand All @@ -18,6 +19,7 @@ Complete reference materials for comprehensive k6 performance test planning.
- Protocol-specific planning (HTTP, gRPC, browser VU requirements)
- Data source selection and integration strategies
- HTTP method confirmation and auth discovery for executable plans
- Exploratory planning from estimated load goals or application codebase context
- Deterministic next-step selection from unresolved dependencies
- Cloud runtime-family gating for v1.x/v2.x compatibility
- Cloud distribution validation (`options.cloud.distribution` percent sum must equal 100)
Expand All @@ -42,17 +44,19 @@ Verified cloud routing/auth env keys:
- **Scenario type**: objective shape (`load`, `stress`, `spike`, `soak`, `smoke`) used to select executor strategy.
- **Profile**: intensity preset (`minimal`, `standard`, `aggressive`) used to derive default `vus` and `duration` when missing.
- **Round**: one consolidated question block in the same adaptive question system; baseline questions are Round 1 and optional tie-break is Round 2.
- **Exploratory mode**: suggested plan generation from partial requirements or codebase evidence, with explicit pending clarifications.
- **Explicit** means user-provided; **derived** means deterministic defaults declared in assumptions.

## Implementation Rules

1. **Plans must include explicit thresholds** — provided by user or derived from defaults
1. **Plans must include explicit thresholds** — provided by user, derived from defaults, or emitted as `suggested_sla` candidates in exploratory mode
2. **Plans must include explicit load profile** — minimum: `vus` and `duration`, or staged equivalents
3. **Missing critical parameters trigger the question protocol** before plan generation
4. **All assumptions must be listed** in output when defaults are applied
5. **Each plan includes executor recommendation** with explicit rationale
6. **Runnable URL hard-coding is forbidden** in executable output; require `__ENV.BASE_URL` style variables
7. **Auth discovery is mandatory** before final handoff to k6-builder when auth requirements are uncertain
8. **Exploratory plans must remain non-runnable** until `ready_for_builder` blockers are cleared

## Clarification Contract (Minimal)

Expand All @@ -73,6 +77,7 @@ Rules:

- If core test shape is clear but `target` or `sla` is missing, generate a provisional plan with `[assumption-based]` labels.
- Always include `pending_questions` listing each assumption and why it is needed.
- If application-context exploration yields candidate routes or flows, include them under `inferred_targets` or `inferred_flows` and keep `ready_for_builder: false`.

## HTTP Planning Requirements

Expand Down Expand Up @@ -110,6 +115,7 @@ When building a test plan, the skill gathers these decisions:
3. **SLA requirements** — latency (p95/p99), error rate, success rate?
4. **Protocol context** — HTTP, gRPC, browser, or mixed?
5. **Load profile** — explicit VUs/duration, or derive from scenario?
6. **Exploratory evidence** — is there enough app or request context to suggest targets, flows, and SLA candidates?

## Example Workflow

Expand All @@ -120,3 +126,16 @@ User asks: "plan a k6 load test for my API"
3. Executor asks: What are your SLA requirements? (default: standard profile)
4. Executor derives: HTTP protocol, ramping-vus executor, standard load profile
5. Executor generates: Complete plan with thresholds, stages, and assumptions listed

Exploratory variant:

1. User asks: "Quiero validar si soportaré 2 mil usuarios concurrentes durante 1h para los flujos principales de mi backend"
2. Executor derives: `scenario=load` from the sustained validation goal and inspects the available app context if the workspace contains the backend
3. Executor suggests: candidate routes or business flows, one or more `suggested_sla` values from profile defaults, and explicit `pending_clarifications`
4. Executor marks: `plan_mode: exploratory` and `ready_for_builder: false`

1. Executor asks: What's the target URL?
2. Executor asks: What scenario type? (default: load)
3. Executor asks: What are your SLA requirements? (default: standard profile)
4. Executor derives: HTTP protocol, ramping-vus executor, standard load profile
5. Executor generates: Complete plan with thresholds, stages, and assumptions listed
5 changes: 4 additions & 1 deletion skills/k6-plan/references/certainty-and-compact-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,13 @@ Use this guide to keep planning responses strict and concise.

1. Do not present inferred values as confirmed facts.
2. If a required planning field is missing, set it to `unknown`.
3. Ask one unblocker question for unresolved critical fields.
3. Ask one unblocker question for unresolved critical fields unless exploratory mode is active.
4. In exploratory mode, isolate unconfirmed data under explicit labels such as `inferred_targets`, `inferred_flows`, `suggested_sla`, or `[assumption-based]`.
5. Never copy exploratory assumptions into finalized summaries without retaining their uncertainty markers.

## Compact Rules

1. Keep output focused on executor, profile, thresholds, and one next step.
2. Avoid narrative repetition when a short bullet is sufficient.
3. Include only details needed for handoff to k6-builder.
4. In exploratory mode, keep candidate flows and pending clarifications short and prioritized.
62 changes: 62 additions & 0 deletions skills/k6-plan/references/exploratory-planning.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Exploratory Planning

Use this guide when the user provides an estimated performance goal but lacks full endpoint, auth, or SLA detail, or when the workspace appears to contain the application under test.

## Activation Rules

Activate exploratory mode only when all of these are true:

1. The request contains a concrete test intent such as concurrency, duration, throughput, sustained validation, or named business flows.
2. `scenario` is explicit or directly inferable from the user's wording.
3. `target`, `sla`, auth, or exact journey details are incomplete.

Do not activate exploratory mode when `scenario` is still ambiguous after one interpretation pass.

## Evidence Order

Use evidence in this order:

1. Explicit user requirements
2. Direct workspace evidence from the application code
3. Deterministic profile defaults already documented in the skill
4. `[assumption-based]` placeholders for unresolved execution details

Never skip over stronger evidence to use weaker assumptions.

## Application-Context Exploration

When the workspace likely contains the system under test:

1. Inspect API routes, controllers, service entry points, frontend navigation, HTTP clients, auth middleware, queues, streaming paths, and dashboards.
2. Prioritize flows that are business-critical, write-heavy, auth-gated, or frequently repeated.
3. For each inferred candidate, record:
- `name`
- `type`: endpoint, journey, rpc, websocket, or mixed
- `evidence`
- `why_this_flow_matters`
4. Keep the list short and ordered by likely business impact.

## Output Requirements

Exploratory plans must include:

1. `plan_mode: exploratory`
2. `assumption_basis`
3. `inferred_targets`
4. `inferred_flows`
5. `pending_clarifications`
6. `ready_for_builder: false`

Use `pending_clarifications` to mark what still blocks runnable artifacts or validation-complete output.

## Suggested SLA Rules

1. Suggested SLAs must come only from `sla-defaults.md` or from explicit user placeholder values.
2. Label every non-confirmed threshold as `suggested_sla`.
3. If multiple candidates are offered, explain the comparison briefly and keep the number of candidates low.

## Handoff Limits

1. Do not present exploratory plans as executable-ready.
2. Do not set `ready_for_builder: true` while target, auth, or SLA blockers remain unresolved.
3. Preserve all `inferred` or `[assumption-based]` markers in downstream handoff metadata.
22 changes: 22 additions & 0 deletions skills/k6-plan/references/sla-defaults.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,28 @@
| standard | <500ms | <900ms | <1% | >95% |
| aggressive | <300ms | <700ms | <0.5% | >99% |

## Exploratory Suggested SLA Policy

When the user does not know the SLA but gives a concrete goal such as concurrency, throughput, duration, or named business flows:

1. Reuse only the documented profile defaults from this file.
2. Emit the result as `suggested_sla`, never as a confirmed SLA.
3. Prefer a single suggested SLA when the request clearly maps to one profile.
4. Prefer multiple suggested SLA candidates only when the request is exploratory enough to justify a comparison, for example `standard` vs `aggressive` for a sustained backend load goal.
5. If the user provides a placeholder threshold such as "around p95<700ms", keep that as user input and supplement only the missing dimensions from the matching profile.

Example exploratory output shape:

```yaml
suggested_sla:
- profile: standard
thresholds: p95<500ms,p99<900ms,error<1%,checks>95%
reason: sustained validation goal with no explicit SLA
- profile: aggressive
thresholds: p95<300ms,p99<700ms,error<0.5%,checks>99%
reason: stricter candidate for core backend flows under elevated concurrency
```

## Parsing Examples

### Input: `p95<400ms,p99<900ms,error<1%`
Expand Down
10 changes: 10 additions & 0 deletions skills/k6-validate/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,16 @@ Incomplete-data rules:
- **Anonymous default export function** — `export default function() {}` without a name is a quality violation. Flag as `WARNING`: "Default export function should be named for traceability and debuggability. Example: `export default function runLoad() {}`". The naming convention is `run<ScenarioType>` or `run<Protocol><ScenarioType>`.
</validation-rules>

## Provisional Artifact Handling

When the artifact or upstream handoff is explicitly exploratory or provisional (`plan_mode: exploratory`, `ready_for_builder: false`, `pending_clarifications`, or pervasive `[assumption-based]` markers):

1. Keep security, lifecycle hygiene, cloud compatibility, and cloud paid execution guardrails as hard gates.
2. Treat missing confirmed thresholds, incomplete load-profile confirmation, or missing current-host capacity evidence as `WARNING` instead of `ERROR` only when the provisional state is explicit and the unresolved evidence is called out clearly.
3. Do not issue `PASS` for provisional artifacts; the highest allowed status is `WARN` until runnable blockers are resolved.
4. In `Scope and Assumptions`, list the provisional markers exactly as they appear in the artifact or handoff.
5. In `Next Step`, prioritize the first clarification required to convert the artifact into a final runnable candidate.

## Required k6 Invariants

Always enforce these validations as mandatory checks:
Expand Down
2 changes: 2 additions & 0 deletions skills/k6-validate/references/certainty-and-compact-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@ Use this guide for deterministic and cost-efficient validation reports.
1. Every finding must be backed by direct evidence.
2. Unknown or unverified context must be labeled `unknown`.
3. Do not convert assumptions into summary facts.
4. When validating exploratory or provisional artifacts, keep `[assumption-based]`, `pending_clarifications`, and `ready_for_builder: false` visible in the report instead of normalizing them away.

## Compact Rules

1. Prioritize ERROR and WARNING findings.
2. Keep findings table one-line per row where possible.
3. Move long explanations to `Suggested Fixes` only when required.
4. For provisional artifacts, spend summary budget on unresolved runnable blockers before optional INFO items.