chore(skills): add eval manifests and host audits

Author: charlyautomatiza

.gitignore

@@ -30,6 +30,16 @@ summary.json
 *.har
 *.html
 k6-results/
+skills/k6-plan/k6-plan-workspace/
+skills/k6-builder/k6-builder-workspace/
+skills/k6-validate/k6-validate-workspace/
+**/iteration-*/
+**/benchmark.json
+**/benchmark.md
+**/grading.json
+**/timing.json
+**/response.txt
+**/review.html
 
 # Temporary files
 tmp/

skills/k6-builder/SKILL.md

@@ -141,6 +141,82 @@ Then:
 2. If language is not explicit, default to English.
 3. Keep command names, k6 metric keys, and code identifiers in English.
 
+## Dynamic Capacity Protocol
+
+Run this protocol before emitting any runnable artifact.
+
+Cross-skill required fields:
+
+- `execution_context`
+- `machine_profile`
+- `projected_load`
+- `capacity_estimate`
+- `risk_status`
+- `safe_limit_recommendation`
+- `scale_out_recommendation`
+
+High-risk gate rule (mandatory):
+
+- Before generating or orchestrating any script that projects more than `500` VUs, run host audit silently with `node scripts/audit-host.js`.
+- If Node.js is unavailable, use the Python fallback: `python3 scripts/audit-host.py`.
+- Parse stdout JSON and use that as the authoritative `machine_profile` for this generation pass.
+
+Contextual deterministic formulas:
+
+- `vus_max = (logical_cpu * cpu_factor) + (free_ram_gb * ram_factor)`
+- `arrival_rate_max = floor(vus_max / max(expected_iteration_duration_seconds, 1))`
+
+Default context factors:
+
+- `local`: `cpu_factor=50`, `ram_factor=10`
+- `ci`: `cpu_factor=30`, `ram_factor=6`
+- `container`: `cpu_factor=25`, `ram_factor=5`
+- `cloud`: `cpu_factor=60`, `ram_factor=12`
+- `distributed`: compute per node, then aggregate
+
+Evaluation order:
+
+1. Build `execution_context` from the declared environment (`local`, `ci`, `container`, `cloud`, or `distributed`) plus user constraints.
+2. Derive `machine_profile` from current evidence: `logical_cpu`, `free_ram_gb`, `fd_limit`, `ephemeral_port_budget`, environment type, and declared user restrictions.
+3. If projected load is expected to exceed `500` VUs, run host audit (`node scripts/audit-host.js`, fallback `python3 scripts/audit-host.py`) and overwrite missing or stale machine evidence with audit output.
+4. Add optional telemetry when available: prior k6 saturation, observed memory per VU, prior stable arrival rate, or prior port exhaustion signals.
+5. Derive `projected_load` from the runnable executor shape: target VUs, arrival rate, duration, and stages.
+6. Calculate `capacity_estimate.vus_max` and `capacity_estimate.arrival_rate_max` from the current `machine_profile` using contextual deterministic formulas. Never use a universal fixed VU ceiling.
+7. Classify `risk_status` as `SAFE`, `AT_RISK`, or `HIGH_RISK`.
+8. Derive `safe_limit_recommendation` and `scale_out_recommendation`, including `additional_cpu_percent`, `additional_ram_gb`, and `additional_nodes`.
+
+Incomplete-data rules:
+
+- If critical machine inputs are missing, ask one clarification question when that is the cheapest unblocker.
+- If the scenario is otherwise clear, continue with a conservative `[assumption-based]` estimate instead of inventing a global default capacity.
+- Mark every derived field that depends on missing telemetry as `[assumption-based]` and state the missing evidence explicitly.
+
+Builder-stage gate behavior:
+
+- `SAFE`: emit runnable artifacts normally.
+- `AT_RISK`: runnable artifacts are allowed only if the output includes the canonical capacity alert, the reduced safe limit, and the distributed mitigation guidance.
+- `HIGH_RISK`: block runnable single-node artifact emission. Replace the runnable artifact section with a blocked-output explanation plus a reduced-load or distributed execution path.
+
+When `HIGH_RISK`, the blocked-output explanation must use this exact format:
+
+`Tu máquina solo tiene [X]GB de RAM disponibles/libres. Intentar correr [Y] VUs localmente colapsará el generador. Te recomiendo limitar la prueba local a [Z] VUs o exportar este diseño para k6-operator (Configuración distribuida).`
+
+Canonical alert format:
+
+```md
+LOAD GENERATOR CAPACITY ALERT
+Risk Detected: The requested scenario ([X] VUs) exceeds the load generator operating limit ([Y] VUs).
+Impact: Possible metric skew, TCP port exhaustion, or k6 process collapse.
+Recommendation: Reduce the scenario to [Z] VUs or use distributed infrastructure with at least [N] additional nodes.
+```
+
+Determinism rules:
+
+- `Y` must come from the current `capacity_estimate`.
+- `Z` must come from the current `safe_limit_recommendation`.
+- `N` must come from the current `scale_out_recommendation.additional_nodes`.
+- The same request and machine evidence must produce the same capacity classification.
+
 ## Required k6 Invariants
 
 Always enforce these validations before returning output:
@@ -394,7 +470,26 @@ Every response must include these sections in order:
    /k6-validate <your-plan-or-test-config>
      ```
    - **For other scenarios** (load, stress, soak, spike): Include validation recommendation that references k6-validate.
-10. Next recommended step
+10. Execution Hints (required)
+   - Include at least one concrete run command that can be copied as-is.
+   - Include required environment variables used by that command.
+   - Use deterministic command shape (`k6 run <script>` plus `-e` flags when needed).
+   - Keep hints runnable and aligned with the emitted artifact names.
+   - Treat `Execution Hints` as a hard companion of `Runnable Artifacts`: if artifact code exists, hints must include the exact script filename and matching env var flags.
+   - Never emit conceptual guidance only; include at least one concrete copy-paste command.
+11. Next recommended step
+
+Dynamic capacity reporting requirements:
+
+- `Build Inputs Summary` must include `execution_context`.
+- `Internal Minimal Plan` must include `machine_profile` and `projected_load`.
+- `Guardrail Validation` must also include these checklist items:
+   - [ ] Dynamic capacity gate executed with current `machine_profile`
+   - [ ] `capacity_estimate`, `risk_status`, and `safe_limit_recommendation` are present
+   - [ ] `scale_out_recommendation` includes `additional_cpu_percent`, `additional_ram_gb`, and `additional_nodes`
+   - [ ] Canonical capacity alert emitted when `risk_status` is `AT_RISK` or `HIGH_RISK`
+
+- When `risk_status` is `HIGH_RISK`, `Runnable Artifacts` must not include a single-node runnable artifact.
 
 ## Runnable Artifact Rules
 
@@ -445,8 +540,17 @@ Keep this file focused on generation workflow. Place deep guidance in:
 8. Select executor and derive coherent scenario config.
 9. Enforce executor coherence: if scenario type is `load` and the user did not explicitly request rate-based control, recommend and emit `ramping-vus`.
 10. Parse SLA thresholds or apply deterministic defaults.
-11. For multi-environment requests, choose architecture using Multi-Environment Architecture Options (default single-script unless explicitly overridden).
-12. Generate runnable script/options/config outputs.
-13. Apply dashboard and secrets safety policies.
-14. Validate all required invariants, including: (a) the generated default export function is named — reject anonymous `export default function () {}`; (b) SLA thresholds are identical across all environments — reject any per-environment threshold relaxation. Both are hard invariant violations that block output emission.
-15. Return output in Output Contract order.
+11. Run the Dynamic Capacity Protocol using current machine evidence and projected workload (mandatory host audit first when projected load exceeds 500 VUs).
+12. For multi-environment requests, choose architecture using Multi-Environment Architecture Options (default single-script unless explicitly overridden).
+13. Generate runnable script/options/config outputs only when the capacity gate allows them.
+14. Apply dashboard and secrets safety policies.
+15. Validate all required invariants, including: (a) the generated default export function is named — reject anonymous `export default function () {}`; (b) SLA thresholds are identical across all environments — reject any per-environment threshold relaxation. Both are hard invariant violations that block output emission.
+16. Return output in Output Contract order.
+
+## Local Evaluation Workspace Policy
+
+For official skill evaluation runs in this repository:
+
+- Store artifacts under `skills/k6-builder/k6-builder-workspace/iteration-N/`.
+- Keep each run isolated inside its own `iteration-N` directory.
+- Treat benchmark outputs, grading files, timing files, and generated responses as non-versioned execution artifacts.

skills/k6-builder/evals/evals.json

@@ -21,7 +21,7 @@
         },
         {
           "name": "runnable_artifact",
-          "text": "Must produce runnable script artifact and execution hints"
+          "text": "Must produce a runnable script artifact and execution hints that include at least one concrete k6 run command and required env vars"
         }
       ]
     },
@@ -200,6 +200,90 @@
       ],
       "files": [],
       "expected_output": "browser-smoke-runnable"
+    },
+    {
+      "id": 8,
+      "prompt": "Build a runnable k6 HTTP load script for a local workstation with 8 logical CPUs, 6 GB free RAM, fd limit 10240, and ephemeral port budget 16000. Target load is 1200 VUs for 15m against https://perf.example.com/checkout. SLA: p95<600ms and error<1%.",
+      "expected_output": "builder-local-at-risk-capacity-gate",
+      "files": [],
+      "assertions": [
+        {
+          "name": "capacity_gate_fields",
+          "text": "Must include execution_context, machine_profile, projected_load, capacity_estimate, risk_status, and safe_limit_recommendation"
+        },
+        {
+          "name": "canonical_capacity_alert",
+          "text": "Must emit the canonical LOAD GENERATOR CAPACITY ALERT when projected load exceeds the operating limit"
+        },
+        {
+          "name": "mitigation_guidance",
+          "text": "Must include scale_out_recommendation with additional_cpu_percent, additional_ram_gb, and additional_nodes"
+        },
+        {
+          "name": "at_risk_warning_required",
+          "text": "Must not emit an at-risk runnable artifact silently; the warning and mitigation must be explicit"
+        }
+      ]
+    },
+    {
+      "id": 9,
+      "prompt": "Build a runnable k6 stress script for 6000 VUs on a local Docker Desktop environment with 4 logical CPUs, 4 GB free RAM, fd limit 4096, and ephemeral port budget 8000. Target is https://stress.example.com/api. SLA: p95<900ms.",
+      "expected_output": "builder-high-risk-capacity-block",
+      "files": [],
+      "assertions": [
+        {
+          "name": "high_risk_block",
+          "text": "Must block single-node runnable artifact emission when the capacity gate classifies the scenario as HIGH_RISK"
+        },
+        {
+          "name": "canonical_capacity_alert",
+          "text": "Must emit the canonical LOAD GENERATOR CAPACITY ALERT for the blocked scenario"
+        },
+        {
+          "name": "reduced_or_distributed_path",
+          "text": "Must replace the runnable artifact with a reduced-load or distributed execution path"
+        }
+      ]
+    },
+    {
+      "id": 10,
+      "prompt": "Build a distributed k6 configuration for 9000 VUs across cloud load generators. Each node has 8 logical CPUs, 16 GB free RAM, fd limit 65535, and environment type cloud. Use https://api.example.net/orders and SLA p95<750ms.",
+      "expected_output": "builder-distributed-cloud-config",
+      "files": [],
+      "assertions": [
+        {
+          "name": "distributed_capacity_model",
+          "text": "Must model per-node capacity instead of applying a single fixed global VU limit"
+        },
+        {
+          "name": "node_count_guidance",
+          "text": "Must include scale_out_recommendation.additional_nodes derived from the current capacity estimate"
+        },
+        {
+          "name": "distributed_runnable_output",
+          "text": "Must produce a runnable distributed configuration when the cloud/distributed capacity gate allows it"
+        }
+      ]
+    },
+    {
+      "id": 11,
+      "prompt": "Build a rate-driven k6 config for 2600 RPS over 20m on a cloud generator with 16 logical CPUs, 32 GB free RAM, fd limit 65535, and ephemeral port budget 50000. Target is https://rate.example.net and SLA p95<450ms, error<0.5%.",
+      "expected_output": "builder-rate-driven-cloud-capacity",
+      "files": [],
+      "assertions": [
+        {
+          "name": "arrival_rate_capacity_fields",
+          "text": "Must include projected arrival rate and capacity_estimate.arrival_rate_max based on the provided machine profile"
+        },
+        {
+          "name": "rate_driven_scale_out_math",
+          "text": "Must include scale_out_recommendation.additional_nodes derived from arrival-rate capacity rather than VU-only math"
+        },
+        {
+          "name": "capacity_gate_coherent_with_output",
+          "text": "Must keep the emitted configuration coherent with the resulting SAFE or AT_RISK capacity assessment"
+        }
+      ]
     }
   ]
 }

skills/k6-builder/references/executor-decision-matrix.md

@@ -142,6 +142,13 @@
 
 ## Capacity Guardrails
 
+### Mandatory pre-generation audit for high load
+
+- If projected load exceeds `500` VUs, execute host audit before script generation:
+  - Primary: `node scripts/audit-host.js`
+  - Fallback: `python3 scripts/audit-host.py`
+- Use audit JSON as the canonical machine evidence (`logical_cpu`, `free_ram_gb`, `fd_limit`, `ephemeral_port_budget`).
+
 ### For arrival-rate executors:
 
 **Dimensioning Formula:**
@@ -168,6 +175,23 @@ required_vus = 100 × 0.5 = 50 VUs
 - Monitor CPU/memory on load generator
 - **Rule:** 1 GB RAM ≈ 50–100 VUs sustained
 
+### Contextual deterministic capacity model
+
+Use context factors instead of a universal ceiling:
+
+```
+vus_max = (logical_cpu * cpu_factor) + (free_ram_gb * ram_factor)
+arrival_rate_max = floor(vus_max / max(expected_iteration_duration_seconds, 1))
+```
+
+Default factors:
+
+- local: `cpu_factor=50`, `ram_factor=10`
+- ci: `cpu_factor=30`, `ram_factor=6`
+- container: `cpu_factor=25`, `ram_factor=5`
+- cloud: `cpu_factor=60`, `ram_factor=12`
+- distributed: compute per node, then aggregate
+
 ## Threshold Mapping by Executor
 
 | Executor Type | Typical Threshold | Rationale |

skills/k6-builder/references/sla-defaults.md

@@ -20,26 +20,42 @@ When no explicit profile is specified, the following environment-to-profile mapp
 
 This mapping ensures consistent profile selection across different deployment contexts. Users can override by explicitly specifying a profile (e.g., `--profile=standard`).
 
-## VU Capacity Boundaries
+## Dynamic Capacity Assessment Baseline
 
-Load generator capacity varies by execution context. Recommended VU limits:
+Load generator capacity must be calculated from the current execution context rather than from a universal VU table.
 
-| Context | Recommended Max VUs | Rationale |
-|---------|-------------------|-----------|
-| Local (laptop/desktop) | 50-100 | Limited CPU/memory; avoid overwhelming local machine |
-| CI/CD (standard runner) | 100-200 | Shared resources; keep tests fast and reliable |
-| Cloud (dedicated instance) | 500-1000+ | Higher capacity; can scale with larger instance types |
+Required inputs:
 
-**Behavior on Out-of-Range Values:**
-- If user requests VUs exceeding context limits, the skill should:
-  1. Issue a `WARNING` noting the potential resource constraint
-  2. Suggest alternative: scale horizontally (multiple load generators) or use cloud execution
-  3. Proceed with user-specified value (explicit override) unless it's clearly invalid (e.g., negative VUs)
+- `logical_cpu`
+- `free_ram_gb`
+- `fd_limit`
+- `ephemeral_port_budget`
+- environment type (`local`, `ci`, `container`, `cloud`, or `distributed`)
+- user-declared restrictions
 
-**Example Warning:**
-```
-⚠️  WARNING: Requested 500 VUs exceeds recommended limit for local execution (50-100 VUs).
-Consider running this test on a cloud load generator or reducing VU count to avoid resource exhaustion.
+Optional telemetry:
+
+- prior stable VU count
+- prior stable arrival rate
+- observed CPU saturation
+- observed memory per VU
+- prior port exhaustion or socket-pressure signals
+
+Required behavior:
+
+1. Derive `capacity_estimate.vus_max` and `capacity_estimate.arrival_rate_max` from the current machine evidence.
+2. Derive `safe_limit_recommendation` from the same estimate.
+3. Derive `scale_out_recommendation.additional_nodes`, `additional_cpu_percent`, and `additional_ram_gb` whenever projected load exceeds the safe operating range.
+4. If critical inputs are missing, use a conservative `[assumption-based]` estimate and state what evidence is missing.
+5. Do not rely on a global fixed maximum such as a single laptop or CI runner VU ceiling.
+
+Canonical alert when risk exists:
+
+```md
+LOAD GENERATOR CAPACITY ALERT
+Risk Detected: The requested scenario ([X] VUs) exceeds the load generator operating limit ([Y] VUs).
+Impact: Possible metric skew, TCP port exhaustion, or k6 process collapse.
+Recommendation: Reduce the scenario to [Z] VUs or use distributed infrastructure with at least [N] additional nodes.
 ```
 
 ## Parsing Examples