From 3acad0e6524cc8b47b1cee23197cb7a138fac2e1 Mon Sep 17 00:00:00 2001 From: Charly Automatiza Date: Fri, 31 Jul 2026 12:37:52 -0300 Subject: [PATCH 1/2] feat: update skills content --- skills/k6-builder/SKILL.md | 9 +- .../certainty-and-evidence-policy.md | 4 +- skills/k6-plan/SKILL.md | 100 ++++++++++++------ skills/k6-plan/evals/evals.json | 56 ++++++++++ skills/k6-plan/references/README.md | 21 +++- .../certainty-and-compact-output.md | 5 +- .../references/exploratory-planning.md | 62 +++++++++++ skills/k6-plan/references/sla-defaults.md | 22 ++++ skills/k6-validate/SKILL.md | 10 ++ .../certainty-and-compact-output.md | 2 + 10 files changed, 255 insertions(+), 36 deletions(-) create mode 100644 skills/k6-plan/references/exploratory-planning.md diff --git a/skills/k6-builder/SKILL.md b/skills/k6-builder/SKILL.md index d64fc93..67f9c7f 100644 --- a/skills/k6-builder/SKILL.md +++ b/skills/k6-builder/SKILL.md @@ -77,6 +77,12 @@ Do not emit final artifact content after this fallback. - Use `__ENV.BASE_URL` (or protocol-equivalent env var) placeholder and mark all unresolved values as `[assumption-based]`. - Append a `pending_questions` block with one direct question for the unresolved target. + **Upstream exploratory handoff policy:** + - If input from `k6-plan` includes `plan_mode: exploratory`, `ready_for_builder: false`, `inferred_targets`, `inferred_flows`, or `pending_clarifications`, do not treat that input as executable-ready. + - Preserve every `inferred` or `[assumption-based]` field in the builder output and carry unresolved items into a `pending_questions` block. + - Generate a partial template only when the user explicitly wants to continue from the exploratory plan; otherwise ask for the highest-priority runnable blocker. + - Never upgrade inferred routes, auth requirements, or SLA values into confirmed build inputs without explicit confirmation. + Additional questions must be integrated into the same question system, not handled as a separate side flow: - Add an HTTP method question when `protocol=http` and the method cannot be inferred safely. - Add one or more authentication questions when auth is required or unknown and executable output depends on it. @@ -113,7 +119,8 @@ Apply this policy in every builder response: 1. Report only data that is verifiably known from user input or direct tool/runtime evidence in the current task. 2. Do not present inferred values as facts in the final output. 3. If a value is required but not verified, mark it as `unknown` and ask a single unblocker question. -4. Do not fabricate machine/runtime numbers in summaries. +4. If upstream planning metadata is exploratory, preserve its uncertainty markers and keep `ready_for_builder: false` semantics intact. +5. Do not fabricate machine/runtime numbers in summaries. ## Terminology Contract diff --git a/skills/k6-builder/references/certainty-and-evidence-policy.md b/skills/k6-builder/references/certainty-and-evidence-policy.md index e2a5966..fa714d3 100644 --- a/skills/k6-builder/references/certainty-and-evidence-policy.md +++ b/skills/k6-builder/references/certainty-and-evidence-policy.md @@ -19,4 +19,6 @@ Use this policy for all user-facing summaries. 1. If value is not verified, mark as `unknown`. 2. Ask one unblocker question for critical unknowns. 3. Do not mix inferred and verified data in the same fact line. -4. Keep final summary concise and practical. +4. Preserve `[assumption-based]`, `inferred_*`, `pending_clarifications`, and `ready_for_builder: false` markers received from `k6-plan`. +5. Do not promote exploratory planning metadata to confirmed runnable facts during generation. +6. Keep final summary concise and practical. diff --git a/skills/k6-plan/SKILL.md b/skills/k6-plan/SKILL.md index 4507319..0ebd60b 100644 --- a/skills/k6-plan/SKILL.md +++ b/skills/k6-plan/SKILL.md @@ -1,6 +1,6 @@ --- name: k6-plan -description: Plan deterministic k6 performance tests from goals, SLA, and protocol context. Use when users ask to plan a load test, set up a stress/spike/soak strategy, or request a full k6 test blueprint. Route implementation output requests to k6-builder. +description: Plan deterministic k6 performance tests from goals, SLA, protocol context, or exploratory application context. Use when users ask to plan a load test, set up a stress/spike/soak strategy, request a full k6 test blueprint, or need a suggested plan from estimated load goals and app flows. Route implementation output requests to k6-builder. user-invocable: true disable-model-invocation: false license: MIT @@ -21,10 +21,10 @@ At the beginning of the workflow, detect and use interaction tools in this order 4. Else emit the exact fallback and end the turn: ```md -> [?] MISSING REQUIREMENT: Missing target, scenario, SLA, or protocol detail -missing: target, scenario type, SLA requirements, protocol -why: deterministic planning cannot proceed without baseline planning inputs -next_question: What target URL or endpoint should this plan use? +> [?] MISSING REQUIREMENT: Missing scenario or protocol detail for planning +missing: scenario type, protocol +why: planning cannot continue safely until the test objective shape and protocol are known or inferable +next_question: What scenario type and protocol should this test use? ``` Do not continue plan generation after fallback. @@ -51,6 +51,18 @@ Do not emit final plan content after this fallback. - What are your SLA requirements? Example: p95<500ms,error<1% (if `sla` missing) - What protocol should this test use? Options: http, grpc, browser (if `protocol` missing) + **Exploratory planning mode:** + - Auto-detect exploratory mode when the user provides a concrete load objective, concurrency goal, duration, throughput target, or named business flow but omits `target`, `sla`, or both. + - Auto-detect exploratory mode when the active workspace appears to contain the application under test and the user has not provided concrete endpoints, routes, or user journeys. + - Exploratory mode is allowed only when `scenario` is explicit or can be determined directly from the user's wording. If `scenario` remains ambiguous, stop and ask for clarification. + - In exploratory mode, prefer generating a suggested plan with explicit assumptions over stopping at the first missing `target` or `sla`. + + **Application-context exploration:** + - Before asking for endpoints, auth, or exact routes, inspect the available application context when the workspace likely contains the system under test. + - Look for likely performance-critical surfaces such as API routes, frontend journeys, HTTP clients, auth entry points, dashboards, search flows, write-heavy transactional flows, and polling or streaming paths. + - Produce a short prioritized candidate list with evidence-backed labels such as `inferred target`, `inferred flow`, and `why this flow matters`. + - If exploration yields viable candidates, use them to build a provisional plan directly and move unresolved details into `pending_clarifications` instead of blocking on questions. + **Edge case hardening**: When user input is ambiguous or conflicts with best practices: - If SLA thresholds conflict (e.g., "p95<100ms" for a high-latency service), ask clarification. - If scenario and SLA are mismatched (e.g., "smoke test with SLA p99<50ms"), flag and ask confirmation. @@ -62,11 +74,18 @@ Do not emit final plan content after this fallback. - If required inputs are still unresolved after Round 2, emit the interoperability fallback and end the turn. **Provisional plan policy:** - - Default behavior is strict certainty: do not generate provisional assumptions in final plan output. - - If critical data is missing, return `unknown` for unresolved fields and ask one unblocker question. - - Only generate provisional plans when the user explicitly asks for assumption-based output. + - Default behavior remains strict certainty for finalized plans. + - If `scenario` and `protocol` are clear but `target`, `sla`, auth, or exact flow details are missing, exploratory mode may generate a provisional plan with explicit `inferred` or `[assumption-based]` markers. + - If critical data is missing outside exploratory mode, return `unknown` for unresolved fields and ask one unblocker question. + - Suggested SLA values in exploratory mode must come only from documented profile defaults or explicit user placeholders; do not invent new threshold heuristics. - If scenario type is ambiguous, do not generate a provisional plan; ask clarification using the Clarification Output Contract. + **Exploratory output contract:** + - Provisional plans must include `plan_mode: exploratory`, `ready_for_builder: false`, `assumption_basis`, `inferred_targets`, `inferred_flows`, and `pending_clarifications`. + - `pending_clarifications` must list each unresolved field, why it matters, and whether it blocks runnable artifact generation. + - Exploratory plans may recommend one or more candidate SLAs, but each candidate must be labeled `suggested_sla` and tied to a named profile or user-provided placeholder. + - Exploratory plans must not present inferred routes, auth requirements, or SLA values as confirmed facts. + Clarification-mode hard stop: - When clarification mode is triggered, emit only the canonical clarification block and stop. - Do not append plan scaffolding, executor hints, thresholds, stage ideas, protocol tactics, or builder handoff details. @@ -76,7 +95,7 @@ Do not emit final plan content after this fallback. - Add an HTTP method question when `protocol=http` and the method cannot be inferred safely. - Add one or more authentication questions when auth is required or unknown and executable output depends on it. - Add more questions only inside Round 1 or the single Round 2 tie-break block when other critical ambiguities or missing requirements are detected. - - Do not finalize the plan or builder handoff until all required questions from this same system are resolved. + - Do not finalize a runnable builder handoff until all required questions from this same system are resolved or explicitly moved into exploratory handoff metadata. 2. **Load Profile Defaults** (when `profile` is not specified): - `minimal`: 5 VUs, 1m duration, smoke testing @@ -84,12 +103,14 @@ Do not emit final plan content after this fallback. - `aggressive`: 120 VUs, 14m duration, stress testing 3. **Output Format**: Primary output is a textual execution plan with: + - `plan_mode`: `finalized` or `exploratory` - Recommended executor type - VU count and stages - Duration estimate - - SLA-derived thresholds + - SLA-derived thresholds or `suggested_sla` candidates - Protocol-specific recommendations - Data integration suggestions (CSV/JSON) + - Exploratory metadata when applicable: `assumption_basis`, `inferred_targets`, `inferred_flows`, `pending_clarifications`, `ready_for_builder` - Exactly one deterministic `Next recommended step` 4. **Determinism**: Same inputs produce identical outputs every time. @@ -104,14 +125,16 @@ Do not emit final plan content after this fallback. 1. Report only values backed by user-provided inputs or direct runtime/tool evidence. 2. Do not present inferred values as final facts. -3. If unresolved, mark as `unknown` and ask one unblocker question. -4. Keep assumptions out of final summaries unless user explicitly requests assumption-based mode. +3. If unresolved, mark as `unknown` and ask one unblocker question unless exploratory mode is active. +4. In exploratory mode, inferred values must be isolated under explicit labels such as `inferred`, `suggested_sla`, or `[assumption-based]`. +5. Keep exploratory assumptions out of finalized summaries and runnable handoff fields unless they remain explicitly marked as provisional. ## Terminology Contract - **Scenario type** means the test objective shape (`load`, `stress`, `spike`, `soak`, `smoke`). - **Profile** means default intensity presets (`minimal`, `standard`, `aggressive`) used when explicit `vus`/`duration` are missing. - **Round** means one consolidated question block in the adaptive question system; baseline questions are Round 1 and the optional tie-break is Round 2. +- **Exploratory mode** means the skill is generating a suggested plan from partial requirements or codebase evidence while keeping unresolved fields explicit. - Scenario type selects the executor strategy; profile sets default intensity values. ## Language Policy @@ -353,6 +376,7 @@ Rules: - Keep `why` strictly unblocker-focused. Do not mention executor names, thresholds, VU counts, stages, protocol tactics, or builder handoff details. - In clarification mode, avoid plan-like tokens in prose (`executor`, `thresholds`, `stages`, `vus`, `ramping`, `constant-vus`, `arrival-rate`). - If multiple fields are missing, list them all in `missing` but ask only the single most-blocking question in `next_question`. +- Do not use clarification mode when exploratory mode is allowed and sufficient evidence exists to suggest a provisional plan. - After emitting the clarification block, end the response. Do not add caveats or partial analysis below it. ## Required k6 Invariants @@ -361,7 +385,8 @@ Always enforce these validations before returning the plan: 1. **Thresholds are required** - Parse thresholds from SLA if provided. - - If SLA is not provided, derive profile-based defaults and show them explicitly. + - If SLA is not provided in finalized mode, derive profile-based defaults and show them explicitly. + - If SLA is not provided in exploratory mode, emit one or more `suggested_sla` candidates derived only from documented profile defaults and keep them marked as unconfirmed. 2. **Load profile is required** - Plan must include explicit VUs and duration (or explicit stage set with equivalent duration and target VUs). - If `vus`/`duration` are missing, derive from profile defaults and state assumptions. @@ -369,7 +394,8 @@ Always enforce these validations before returning the plan: 3. **Runnable URL hard-coding is forbidden** - Do not generate runnable scripts with fixed live target URLs. - Require `__ENV.BASE_URL` (or equivalent) for executable output. - - If target is missing, ask for it instead of using a default live URL. + - If target is missing in finalized mode, ask for it instead of using a default live URL. + - If target is missing in exploratory mode, list `inferred_targets` or `unknown` and keep `ready_for_builder: false`. 4. **Parameter coherence is required** - Derived or explicit profile values must map to explicit `vus` and `duration`, or explicit staged equivalents. - If write methods are planned (`POST`/`PUT`/`PATCH`), payload assumptions and expected status must be explicit. @@ -399,6 +425,10 @@ Always enforce these validations before returning the plan: - Output must match runtime family cloud allowlist. - Distribution math must pass strict 100% validation. - Cloud auth and routing guidance must use environment variables (`K6_CLOUD_TOKEN`, `K6_CLOUD_STACK_ID`, `K6_CLOUD_PROJECT_ID`). +9. **Exploratory handoff safety is required** + - Exploratory plans must include `ready_for_builder: false` until target, protocol, auth, and SLA blockers for runnable generation are confirmed or intentionally deferred. + - Every inferred route, auth assumption, or business flow used in the plan must appear in `assumption_basis` or `pending_clarifications`. + - Do not present exploratory plans as runnable or validation-complete artifacts. ## Output Contract @@ -414,9 +444,12 @@ Every response must include these sections in order: Dynamic capacity reporting requirements: -- `Planning Inputs Summary` must include `execution_context`. +- `Planning Inputs Summary` must include `plan_mode` and `execution_context`. +- `Planning Inputs Summary` must include `assumption_basis` when exploratory mode is active. - `Load Profile` must include `projected_load` and `machine_profile`. +- `Thresholds` must switch from confirmed SLA to `suggested_sla` candidates when exploratory mode is active. - `Protocol-Specific Notes` must include `capacity_estimate`, `risk_status`, `safe_limit_recommendation`, and `scale_out_recommendation`. +- `Protocol-Specific Notes` must include `inferred_targets`, `inferred_flows`, and `ready_for_builder` when exploratory mode is active. - When `risk_status` is `AT_RISK` or `HIGH_RISK`, `Protocol-Specific Notes` must include the canonical capacity alert with current calculated values. Guardrail Validation checklist (minimum): @@ -424,6 +457,7 @@ Guardrail Validation checklist (minimum): - [ ] Final summary includes only verified facts or explicit `unknown` - [ ] Missing critical data is surfaced with one unblocker question - [ ] No assumption-based values are presented as confirmed +- [ ] Exploratory outputs keep `ready_for_builder: false` until runnable blockers are resolved ## Scenario to Executor Mapping @@ -512,25 +546,27 @@ Keep this file focused on core planning workflow. Place deep guidance in: When user invokes this skill: 1. Parse provided parameters (`target`, `scenario`, `sla`, `profile`, `protocol`, `duration`, `vus`, `output`). -2. Run Tool Discovery Protocol when critical inputs are missing. -3. Start the Adaptive Question System with baseline questions when `target`, `scenario`, `sla`, or `protocol` are missing. +2. Run Tool Discovery Protocol when `scenario` or `protocol` is missing and cannot be inferred safely. +3. Detect whether exploratory mode should activate from partial requirements or application-context evidence. +4. If the workspace likely contains the application under test and `target` is missing, inspect the available code and derive prioritized `inferred_targets` and `inferred_flows` before asking endpoint/auth questions. +5. Start the Adaptive Question System with baseline questions when `target`, `scenario`, `sla`, or `protocol` are missing. - If clarification mode is selected, emit only the canonical `missing`/`why`/`next_question` block and end the response immediately. - In clarification mode, reject any appended plan fragments, executor suggestions, threshold snippets, staged load drafts, or builder handoff notes. -4. Apply load profile defaults based on `profile`. -5. Add an HTTP method question to the same question system when protocol is HTTP and the method is still ambiguous. -6. Add auth questions to the same question system when auth is required, unknown, or otherwise blocks executable output. -7. Add more questions in the same system if other critical ambiguities or missing requirements are detected. -8. Select executor based on scenario type. -9. Parse SLA thresholds or apply deterministic defaults. -10. Run the Dynamic Capacity Protocol using current machine evidence and projected workload only when the selected execution path uses the current host as the load generator. -11. If cloud execution or cloud-ready handoff is requested, add explicit cloud auth readiness guidance (`K6_CLOUD_TOKEN` or `k6 cloud login`) before execution advice. -12. For journey-style plans, preserve the full requested sequence and add session/data-state handling strategy. -13. Validate explicit or derived VUs and duration. -14. Generate textual plan with recommendations. -15. Validate output structure using the Output Contract section order. -16. Add exactly one deterministic `Next recommended step` based on first unresolved dependency. -17. If `output=script` or user explicitly requests runnable code, route to k6-builder with accumulated plan parameters (`target`, `scenario`, `sla`, `protocol`, `profile`, `method`, `auth`, `duration`, `vus`) plus the current capacity assessment fields. -18. Return the plan and assumptions summary. +6. Apply load profile defaults based on `profile`. +7. Add an HTTP method question to the same question system when protocol is HTTP and the method is still ambiguous. +8. Add auth questions to the same question system when auth is required, unknown, or otherwise blocks executable output. +9. Add more questions in the same system if other critical ambiguities or missing requirements are detected. +10. Select executor based on scenario type. +11. Parse SLA thresholds, apply deterministic defaults, or emit `suggested_sla` candidates for exploratory mode. +12. Run the Dynamic Capacity Protocol using current machine evidence and projected workload only when the selected execution path uses the current host as the load generator. +13. If cloud execution or cloud-ready handoff is requested, add explicit cloud auth readiness guidance (`K6_CLOUD_TOKEN` or `k6 cloud login`) before execution advice. +14. For journey-style plans, preserve the full requested sequence and add session/data-state handling strategy. +15. Validate explicit or derived VUs and duration. +16. Generate textual plan with recommendations and exploratory metadata when applicable. +17. Validate output structure using the Output Contract section order. +18. Add exactly one deterministic `Next recommended step` based on first unresolved dependency. +19. If `output=script` or user explicitly requests runnable code, route to k6-builder with accumulated plan parameters (`target`, `scenario`, `sla`, `protocol`, `profile`, `method`, `auth`, `duration`, `vus`) plus the current capacity assessment fields and exploratory metadata. +20. Return the plan and assumptions summary. ## Local Evaluation Workspace Policy diff --git a/skills/k6-plan/evals/evals.json b/skills/k6-plan/evals/evals.json index cc61db9..65fc66c 100644 --- a/skills/k6-plan/evals/evals.json +++ b/skills/k6-plan/evals/evals.json @@ -15,6 +15,62 @@ "name": "journey_steps", "text": "Plan must preserve the full requested journey sequence" }, + }, + { + "id": 13, + "prompt": "Quiero diseñar una prueba de performance para validar si podré soportar 2 mil usuarios concurrentes durante 1h constante para los flujos principales de mi actual backend, pero no sé qué SLA debería exigir.", + "expected_output": "exploratory-backend-concurrency-plan", + "files": [], + "assertions": [ + { + "name": "exploratory_mode_enabled", + "text": "Must activate exploratory mode and label the plan as provisional or exploratory instead of stopping at missing SLA" + }, + { + "name": "suggested_sla_candidates", + "text": "Must include one or more suggested SLA candidates sourced from documented defaults and keep them explicitly unconfirmed" + }, + { + "name": "pending_clarifications_present", + "text": "Must include pending clarifications for unresolved target, auth, or flow confirmation details" + } + ] + }, + { + "id": 14, + "prompt": "Plan a load test for my current application codebase to validate the main user flows. I have not mapped endpoints or auth yet, but please suggest the likely flows from the app context and keep going.", + "expected_output": "application-context-exploratory-plan", + "files": [], + "assertions": [ + { + "name": "codebase_flow_exploration", + "text": "Must prefer application-context exploration over immediately asking for endpoints or auth when the workspace likely contains the app under test" + }, + { + "name": "inferred_targets_and_flows", + "text": "Must include inferred targets or inferred flows with brief evidence and prioritization" + }, + { + "name": "ready_for_builder_false", + "text": "Must keep the exploratory handoff marked as not ready for runnable artifact generation" + } + ] + }, + { + "id": 15, + "prompt": "I need a performance plan for my service but I'm not sure whether this should be load or stress, and I also don't know the protocol yet.", + "expected_output": "scenario-ambiguity-clarification", + "files": [], + "assertions": [ + { + "name": "clarification_over_exploration", + "text": "Must ask for clarification instead of activating exploratory mode when scenario type remains ambiguous" + }, + { + "name": "canonical_missing_fields", + "text": "Must emit the canonical missing/why/next_question clarification contract" + } + ] { "name": "frontend_sla_threshold", "text": "Plan must include threshold/KPI aligned to p95<5s" diff --git a/skills/k6-plan/references/README.md b/skills/k6-plan/references/README.md index 3e60a9c..783316e 100644 --- a/skills/k6-plan/references/README.md +++ b/skills/k6-plan/references/README.md @@ -9,6 +9,7 @@ Complete reference materials for comprehensive k6 performance test planning. - **[protocol-guide.md](protocol-guide.md)** — HTTP, gRPC, and browser protocol patterns and examples - **[data-integration.md](data-integration.md)** — CSV/JSON/environment variable data source patterns - **[certainty-and-compact-output.md](certainty-and-compact-output.md)** — Verified-facts-only and compact response guidance +- **[exploratory-planning.md](exploratory-planning.md)** — Auto-detected exploratory mode, inferred flows, and provisional handoff rules ## Core Topics @@ -18,6 +19,7 @@ Complete reference materials for comprehensive k6 performance test planning. - Protocol-specific planning (HTTP, gRPC, browser VU requirements) - Data source selection and integration strategies - HTTP method confirmation and auth discovery for executable plans +- Exploratory planning from estimated load goals or application codebase context - Deterministic next-step selection from unresolved dependencies - Cloud runtime-family gating for v1.x/v2.x compatibility - Cloud distribution validation (`options.cloud.distribution` percent sum must equal 100) @@ -42,17 +44,19 @@ Verified cloud routing/auth env keys: - **Scenario type**: objective shape (`load`, `stress`, `spike`, `soak`, `smoke`) used to select executor strategy. - **Profile**: intensity preset (`minimal`, `standard`, `aggressive`) used to derive default `vus` and `duration` when missing. - **Round**: one consolidated question block in the same adaptive question system; baseline questions are Round 1 and optional tie-break is Round 2. +- **Exploratory mode**: suggested plan generation from partial requirements or codebase evidence, with explicit pending clarifications. - **Explicit** means user-provided; **derived** means deterministic defaults declared in assumptions. ## Implementation Rules -1. **Plans must include explicit thresholds** — provided by user or derived from defaults +1. **Plans must include explicit thresholds** — provided by user, derived from defaults, or emitted as `suggested_sla` candidates in exploratory mode 2. **Plans must include explicit load profile** — minimum: `vus` and `duration`, or staged equivalents 3. **Missing critical parameters trigger the question protocol** before plan generation 4. **All assumptions must be listed** in output when defaults are applied 5. **Each plan includes executor recommendation** with explicit rationale 6. **Runnable URL hard-coding is forbidden** in executable output; require `__ENV.BASE_URL` style variables 7. **Auth discovery is mandatory** before final handoff to k6-builder when auth requirements are uncertain +8. **Exploratory plans must remain non-runnable** until `ready_for_builder` blockers are cleared ## Clarification Contract (Minimal) @@ -73,6 +77,7 @@ Rules: - If core test shape is clear but `target` or `sla` is missing, generate a provisional plan with `[assumption-based]` labels. - Always include `pending_questions` listing each assumption and why it is needed. +- If application-context exploration yields candidate routes or flows, include them under `inferred_targets` or `inferred_flows` and keep `ready_for_builder: false`. ## HTTP Planning Requirements @@ -110,6 +115,7 @@ When building a test plan, the skill gathers these decisions: 3. **SLA requirements** — latency (p95/p99), error rate, success rate? 4. **Protocol context** — HTTP, gRPC, browser, or mixed? 5. **Load profile** — explicit VUs/duration, or derive from scenario? +6. **Exploratory evidence** — is there enough app or request context to suggest targets, flows, and SLA candidates? ## Example Workflow @@ -120,3 +126,16 @@ User asks: "plan a k6 load test for my API" 3. Executor asks: What are your SLA requirements? (default: standard profile) 4. Executor derives: HTTP protocol, ramping-vus executor, standard load profile 5. Executor generates: Complete plan with thresholds, stages, and assumptions listed + +Exploratory variant: + +1. User asks: "Quiero validar si soportaré 2 mil usuarios concurrentes durante 1h para los flujos principales de mi backend" +2. Executor derives: `scenario=load` from the sustained validation goal and inspects the available app context if the workspace contains the backend +3. Executor suggests: candidate routes or business flows, one or more `suggested_sla` values from profile defaults, and explicit `pending_clarifications` +4. Executor marks: `plan_mode: exploratory` and `ready_for_builder: false` + +1. Executor asks: What's the target URL? +2. Executor asks: What scenario type? (default: load) +3. Executor asks: What are your SLA requirements? (default: standard profile) +4. Executor derives: HTTP protocol, ramping-vus executor, standard load profile +5. Executor generates: Complete plan with thresholds, stages, and assumptions listed diff --git a/skills/k6-plan/references/certainty-and-compact-output.md b/skills/k6-plan/references/certainty-and-compact-output.md index d4c9c64..d6b5e0b 100644 --- a/skills/k6-plan/references/certainty-and-compact-output.md +++ b/skills/k6-plan/references/certainty-and-compact-output.md @@ -6,10 +6,13 @@ Use this guide to keep planning responses strict and concise. 1. Do not present inferred values as confirmed facts. 2. If a required planning field is missing, set it to `unknown`. -3. Ask one unblocker question for unresolved critical fields. +3. Ask one unblocker question for unresolved critical fields unless exploratory mode is active. +4. In exploratory mode, isolate unconfirmed data under explicit labels such as `inferred_targets`, `inferred_flows`, `suggested_sla`, or `[assumption-based]`. +5. Never copy exploratory assumptions into finalized summaries without retaining their uncertainty markers. ## Compact Rules 1. Keep output focused on executor, profile, thresholds, and one next step. 2. Avoid narrative repetition when a short bullet is sufficient. 3. Include only details needed for handoff to k6-builder. +4. In exploratory mode, keep candidate flows and pending clarifications short and prioritized. diff --git a/skills/k6-plan/references/exploratory-planning.md b/skills/k6-plan/references/exploratory-planning.md new file mode 100644 index 0000000..9ec4327 --- /dev/null +++ b/skills/k6-plan/references/exploratory-planning.md @@ -0,0 +1,62 @@ +# Exploratory Planning + +Use this guide when the user provides an estimated performance goal but lacks full endpoint, auth, or SLA detail, or when the workspace appears to contain the application under test. + +## Activation Rules + +Activate exploratory mode only when all of these are true: + +1. The request contains a concrete test intent such as concurrency, duration, throughput, sustained validation, or named business flows. +2. `scenario` is explicit or directly inferable from the user's wording. +3. `target`, `sla`, auth, or exact journey details are incomplete. + +Do not activate exploratory mode when `scenario` is still ambiguous after one interpretation pass. + +## Evidence Order + +Use evidence in this order: + +1. Explicit user requirements +2. Direct workspace evidence from the application code +3. Deterministic profile defaults already documented in the skill +4. `[assumption-based]` placeholders for unresolved execution details + +Never skip over stronger evidence to use weaker assumptions. + +## Application-Context Exploration + +When the workspace likely contains the system under test: + +1. Inspect API routes, controllers, service entry points, frontend navigation, HTTP clients, auth middleware, queues, streaming paths, and dashboards. +2. Prioritize flows that are business-critical, write-heavy, auth-gated, or frequently repeated. +3. For each inferred candidate, record: + - `name` + - `type`: endpoint, journey, rpc, websocket, or mixed + - `evidence` + - `why_this_flow_matters` +4. Keep the list short and ordered by likely business impact. + +## Output Requirements + +Exploratory plans must include: + +1. `plan_mode: exploratory` +2. `assumption_basis` +3. `inferred_targets` +4. `inferred_flows` +5. `pending_clarifications` +6. `ready_for_builder: false` + +Use `pending_clarifications` to mark what still blocks runnable artifacts or validation-complete output. + +## Suggested SLA Rules + +1. Suggested SLAs must come only from `sla-defaults.md` or from explicit user placeholder values. +2. Label every non-confirmed threshold as `suggested_sla`. +3. If multiple candidates are offered, explain the comparison briefly and keep the number of candidates low. + +## Handoff Limits + +1. Do not present exploratory plans as executable-ready. +2. Do not set `ready_for_builder: true` while target, auth, or SLA blockers remain unresolved. +3. Preserve all `inferred` or `[assumption-based]` markers in downstream handoff metadata. diff --git a/skills/k6-plan/references/sla-defaults.md b/skills/k6-plan/references/sla-defaults.md index 3916d33..b57f0f1 100644 --- a/skills/k6-plan/references/sla-defaults.md +++ b/skills/k6-plan/references/sla-defaults.md @@ -8,6 +8,28 @@ | standard | <500ms | <900ms | <1% | >95% | | aggressive | <300ms | <700ms | <0.5% | >99% | +## Exploratory Suggested SLA Policy + +When the user does not know the SLA but gives a concrete goal such as concurrency, throughput, duration, or named business flows: + +1. Reuse only the documented profile defaults from this file. +2. Emit the result as `suggested_sla`, never as a confirmed SLA. +3. Prefer a single suggested SLA when the request clearly maps to one profile. +4. Prefer multiple suggested SLA candidates only when the request is exploratory enough to justify a comparison, for example `standard` vs `aggressive` for a sustained backend load goal. +5. If the user provides a placeholder threshold such as "around p95<700ms", keep that as user input and supplement only the missing dimensions from the matching profile. + +Example exploratory output shape: + +```yaml +suggested_sla: + - profile: standard + thresholds: p95<500ms,p99<900ms,error<1%,checks>95% + reason: sustained validation goal with no explicit SLA + - profile: aggressive + thresholds: p95<300ms,p99<700ms,error<0.5%,checks>99% + reason: stricter candidate for core backend flows under elevated concurrency +``` + ## Parsing Examples ### Input: `p95<400ms,p99<900ms,error<1%` diff --git a/skills/k6-validate/SKILL.md b/skills/k6-validate/SKILL.md index 375b709..4b12d0f 100644 --- a/skills/k6-validate/SKILL.md +++ b/skills/k6-validate/SKILL.md @@ -237,6 +237,16 @@ Incomplete-data rules: - **Anonymous default export function** — `export default function() {}` without a name is a quality violation. Flag as `WARNING`: "Default export function should be named for traceability and debuggability. Example: `export default function runLoad() {}`". The naming convention is `run` or `run`. +## Provisional Artifact Handling + +When the artifact or upstream handoff is explicitly exploratory or provisional (`plan_mode: exploratory`, `ready_for_builder: false`, `pending_clarifications`, or pervasive `[assumption-based]` markers): + +1. Keep security, lifecycle hygiene, cloud compatibility, and cloud paid execution guardrails as hard gates. +2. Treat missing confirmed thresholds, incomplete load-profile confirmation, or missing current-host capacity evidence as `WARNING` instead of `ERROR` only when the provisional state is explicit and the unresolved evidence is called out clearly. +3. Do not issue `PASS` for provisional artifacts; the highest allowed status is `WARN` until runnable blockers are resolved. +4. In `Scope and Assumptions`, list the provisional markers exactly as they appear in the artifact or handoff. +5. In `Next Step`, prioritize the first clarification required to convert the artifact into a final runnable candidate. + ## Required k6 Invariants Always enforce these validations as mandatory checks: diff --git a/skills/k6-validate/references/certainty-and-compact-output.md b/skills/k6-validate/references/certainty-and-compact-output.md index 9d8bd94..877c65d 100644 --- a/skills/k6-validate/references/certainty-and-compact-output.md +++ b/skills/k6-validate/references/certainty-and-compact-output.md @@ -7,9 +7,11 @@ Use this guide for deterministic and cost-efficient validation reports. 1. Every finding must be backed by direct evidence. 2. Unknown or unverified context must be labeled `unknown`. 3. Do not convert assumptions into summary facts. +4. When validating exploratory or provisional artifacts, keep `[assumption-based]`, `pending_clarifications`, and `ready_for_builder: false` visible in the report instead of normalizing them away. ## Compact Rules 1. Prioritize ERROR and WARNING findings. 2. Keep findings table one-line per row where possible. 3. Move long explanations to `Suggested Fixes` only when required. +4. For provisional artifacts, spend summary budget on unresolved runnable blockers before optional INFO items. From a8cd7d50373f5a42204ef22e7b9a99a03e8373a5 Mon Sep 17 00:00:00 2001 From: Charly Automatiza Date: Fri, 31 Jul 2026 12:41:31 -0300 Subject: [PATCH 2/2] Update plugin metadata --- .claude-plugin/plugin.json | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index be99e36..5d2cacb 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,11 +1,14 @@ { "name": "grafana-k6", - "version": "0.1.0", + "version": "0.2.0", "description": "Claude Skills for k6 performance testing with plan-builder-validate lifecycle: planning strategy, runnable artifact generation, and script/config quality validation", "author": { "name": "charlyautomatiza", - "url": "https://github.com/charlyautomatiza/grafana-k6-skill" + "url": "https://github.com/charlyautomatiza/grafana-k6-plugin" + }, + "repository": { + "type": "git", + "url": "https://github.com/charlyautomatiza/grafana-k6-plugin.git" }, - "repository": "charlyautomatiza/grafana-k6-skill", "license": "MIT" }