[Spike][Security Solution] Detection Emulation Skill (Epic #15974) — substrate + orchestration + production-opt-in readiness pack

[patrykkopycinski]

Summary

Closes security-team#15974.

Lands the Detection Emulation Skill end-to-end on a single branch. An Agent Builder skill takes a candidate detection rule + target host(s), runs an emulated attack (real EDR action OR ECS log injection), polls the Detection Engine for the alerts the rule produces, and returns a ValidationReport with a confidence score.

Every action is gated by:

• two off-by-default experimental flags (detectionEmulationRealExecution, detectionEmulationLogInjection)
• a runtime kill switch (xpack.securitySolution.detectionEmulation.realExecution.enabled) for fast disable without redeploy
• a default-deny endpoint allowlist (operator must explicitly permit hosts)
• per-command RBAC re-checked at every tool boundary
• a Zod-enforced ≤5 endpoint fanout cap per call
• per-space (100/h) and per-host (3/h) rate limits with atomic acquire-with-rollback
• ≤1 in-flight real_execution scenario per Kibana space
• mandatory HITL confirmation on every destructive command path
• a regex allowlist for free-form execute payloads (allowedExecuteCommandPatterns)
• full actor attribution (user vs agent-builder, with conversationId / runId / toolCallId / SHA-256 prompt hash) in the audit trail
• idempotency cache on tool dispatch gates (prevents double-fire on retried LLM tool calls)

93 files / +17,020 against upstream/main. Server-side orchestration only; existing UI surfaces unchanged. Allowlist + rate-limiter knobs are exposed as Stack Advanced Settings.

Architecture

What's new

Component	Path	Notes
Multi-EDR runner + route	lib/detection_emulation/execution/runner.ts, api/dispatch/route.ts	Vendor-agnostic dispatcher (execute, runscript, kill-process, isolate, release).
Payload library	lib/detection_emulation/payloads/{payloads.json,index.ts}	12 Wave-1 entries, hard-capped at 15. Data-driven.
Scenario generator	lib/detection_emulation/scenario_generator.ts	Pure function. Deterministic scenarioId = sha256(...). Typed errors no_mitre_tags, no_supported_techniques.
Log injection mode	lib/detection_emulation/log_injection/{generator,index_template,executor}.ts	Dedicated .kibana-security-emulation-logs-<spaceId>-* index template w/ 7-day ILM. Synthetic ECS docs stamped with event.dataset, event.module: 'emulation', and tagged with emulation: { mode, emulationId, scenarioId }.
Telemetry collector	lib/detection_emulation/telemetry_collector.ts	poll + one_shot modes. AbortController. Wall-budget enforcement. Queries alerts by kibana.alert.original_event.module: emulation OR emulation tag via bool.should.
Confidence scorer	lib/detection_emulation/confidence_scorer.ts	confidence = round(coverage * 0.6 + precision * 0.4, 2) clamped [0, 1]. Pure.
Validation gate	lib/detection_emulation/execution/validation_gate.ts	Pre-dispatch gate that combines RBAC, allowlist, fanout, regex allowlist for free-form execute, and rate-limit acquire.
Gate primitives	agent_builder/skills/detection_emulation/gate_checks.ts	Reusable withCommandGates decorator extracted from per-family tool boilerplate. Composes feature-flag → auth → RBAC → allowlist → per-host rate → per-space rate → fanout cap → validation gate → audit in a single pipeline.
Emulation history SO	lib/detection_emulation/emulation_history/{create,get,find,index}.ts + emulation_report_type.ts	Hidden SO, write-once via scenarioFingerprint dedup, namespace-scoped. Model version 2 adds actor field with data_backfill: { kind: 'user' } for old rows.
validateRule route	lib/detection_emulation/api/validate_rule/route.ts	POST /internal/detection_engine/emulation/validate_rule. Eight-step pipeline. Typed 4xx errors. All strings i18n-translated.
Skill tools (six)	agent_builder/skills/detection_emulation/{validate_rule_tool,get_emulation_history_tool,create_run_family_command_tool}.ts	Factory-based per-family tools (process / file / network / execution) built via createRunFamilyCommandTool. Each runs through withCommandGates. validateRule and getEmulationHistory are standalone tools.
HITL primitives	agent_builder/skills/detection_emulation/build_emulation_confirmation.ts + per-family tools	Each per-family tool declares confirmation: { askUser: 'once', getConfirmation }; validateRule does an on-demand HITL prompt when mode === 'real_execution'. Skipped only in executionMode === 'standalone' (eval / A2A).
EmulationToolError	agent_builder/skills/detection_emulation/emulation_tool_errors.ts	Typed error builder for structured error codes (feature_flag_disabled, endpoint_not_allowed, etc.) with consistent shape across all tools.
Skill content	agent_builder/skills/detection_emulation/detection_emulation_skill.ts	Six tools registered. referencedContent array with MITRE ATT&CK overview. Section order: When-to-Use → Process → Examples → Guardrails → Response Format.
Advanced Settings	common/experimental_features.ts, server/config.ts, runtime config resolver	Allowlist + rate-limiter knobs surface in Stack Management → Advanced Settings. Operators can override per-space without restart.
kbn-evals suite	packages/kbn-evals-suite-detection-emulation/evals/{validate_rule_dataset,validate_rule.spec}.ts	Dataset name security: detection-emulation-validate-rule. Eight examples (success / failure / log-injection / distractor / userDeclines HITL rejection / realExecBlocked allowlist 403). Evaluators: toolSelection, schemaCompliance, criteria, plus a trajectory evaluator that scores tool_sequence from execution traces.
Shared utilities	resolveCurrentUsername moved to shared lib, savedObjectsClient threaded through gates	Aligns with patterns from andrew-goldstein's Workflows stack.

Pipeline (route execution order)

 1. Feature-flag gate     detectionEmulationRealExecution      (always required)
                          detectionEmulationLogInjection       (required if mode='log_injection')
 2. Runtime kill switch   xpack.securitySolution.detectionEmulation.realExecution.enabled
 3. Authentication        emulation actions are attributable
 4. RBAC                  per-command authz over the selected payload
 5. Schema gates          MAX_ENDPOINT_FANOUT (5) at Zod boundary
 6. Allowlist             default-deny; operator config drives allowedHosts
 7. Per-space rate        100 commands / space / hour
 8. Per-host rate         3 commands / host / hour (atomic acquire-with-rollback)
 9. Concurrency gate      ≤1 real_execution in flight per space
10. Validation gate       free-form `execute` matched against allowedExecuteCommandPatterns
11. HITL                  framework prompts user; skipped only in standalone executionMode
12. Scenario generator    rule MITRE tags → payload set → deterministic scenarioId
13. Per-payload dispatch  real_execution → execution/runner.ts (multi-EDR)
                          log_injection  → log_injection/executor.ts (synthetic ECS docs)
14. Audit                 actor.kind discriminator + via= suffix on response-action comment
15. Telemetry collector   poll Detection Engine alerts filtered by original_event.module + scenarioId
16. Confidence scorer     coverage * 0.6 + precision * 0.4
17. History write         persist detection-emulation-report SO (model version 2 with actor)
   ────────────────────   ValidationReport (HTTP 200) or typed error (4xx/5xx)

Failure-mode coverage

HTTP	errorCode	When
403	feature_flag_disabled	Mode requires a flag that's off, or runtime kill switch is set.
403	endpoint_not_allowed	endpointId not on the operator allowlist.
403	command_not_allowed	Free-form execute payload didn't match any allowedExecuteCommandPatterns.
403	user_declined	HITL prompt rejected by the user.
422	endpoint_fanout_exceeded	More than MAX_ENDPOINT_FANOUT (5) endpoints in one call.
422	no_mitre_tags	Rule has no MITRE techniques.
422	no_supported_techniques	Rule MITRE tags don't intersect Wave-1.
404	rule_not_found	ruleId doesn't resolve.
429	rate_limit_exceeded	Per-space (100/h) or per-host (3/h) bucket exhausted. Response includes blockedEndpoints + Retry-After.
429	concurrency_exceeded	Another real_execution scenario is in flight in this space. Response includes inflight_scenario_fingerprint + Retry-After.
200 + caveats	wall_budget_exceeded	Partial result; score over partial observations.
500	es_bulk_error	log_injection ES bulk write failure.

Key refactors (latest iteration)

• Factory-based per-family tools: createRunFamilyCommandTool replaces copy-pasted per-family tool files. One factory, parameterized by family name and command schema.
• Reusable gate primitives: withCommandGates extracted from per-family tools — the REST route now reuses the same gate pipeline.
• Typed error builder: EmulationToolError.from(code, message) replaces ad-hoc error construction.
• Dead code removal: removed unused imports, legacy helpers, and orphaned test fixtures.
• resolveCurrentUsername shared: moved to shared lib for reuse across tools and routes.
• savedObjectsClient threading: threaded through gate checks instead of resolving ad-hoc per tool.
• Telemetry collector query fix: uses bool.should with minimum_should_match: 1 to match alerts by kibana.alert.original_event.module: emulation OR emulation tag — ensures confidence > 0 for log_injection mode.
• Skill referencedContent schema compliance: corrected from object to array format per referencedContentSchema in type_definition.ts.
• Idempotency cache: tool dispatch gates cache recent calls to prevent double-fire on retried LLM tool invocations.

Demo plan

1. Enable both experimental flags in config/kibana.dev.yml (full snippet in DEMO.md).
2. Configure a restrictive allowlist: xpack.securitySolution.detectionEmulation.allowlist.endpointIds: [<your-pilot-host>].
3. Pick a rule whose MITRE technique intersects Wave-1 (e.g. T1059.001 Windows PowerShell).
4. From Agent Builder UI: "Validate detection rule <ruleId> against endpoint <endpointId> using log injection."
5. The framework prompts for HITL confirmation. Approve.
6. Inspect the ValidationReport: confidence, coverage, precision, per-phase breakdown, history SO id.
7. Inspect the response-action comment for via=agent-builder/conv:<id>/run:<id> actor attribution.
8. Try fanning out to 6+ endpoints — schema rejects with endpoint_fanout_exceeded.
9. Try a second real_execution while the first is in flight — second is rejected with concurrency_exceeded + Retry-After.
10. Follow up: "Show recent emulation runs for rule <ruleId>." → getEmulationHistory returns paginated history with the actor field populated.

Full walkthrough: x-pack/solutions/security/plugins/security_solution/server/lib/detection_emulation/DEMO.md.

Test plan

• eslint clean on changed files
• type_check clean on security_solution
• jest green: node scripts/jest --testPathPattern='detection_emulation|emulation_report_type|validation_gate'
• detectionEmulationLogInjection defaults to false
• Allowlist defaults to deny-all; operator config required to permit any host
• Audit trail attributes every action to actor.kind (user vs agent-builder) with conversation/run/tool ids and prompt hash
• Endpoint fanout capped at MAX_ENDPOINT_FANOUT = 5 at the Zod boundary
• Per-host rate limit enforced (3/host/hour, atomic acquire-with-rollback)
• Concurrency gate caps in-flight real_execution per space (1)
• Free-form execute payloads gated by allowedExecuteCommandPatterns regex allowlist
• Runtime kill switch (detectionEmulation.realExecution.enabled) returns 403 when disabled
• HITL prompt on every destructive command path; skipped only in standalone executionMode
• Skill content follows mandated section order; mentions every gate in the Guardrails table
• CODEOWNERS updated for new directories
• E2E log_injection pipeline verified: emulation log → detection rule fires → alert with original_event.module: emulation → telemetry collector finds alert → confidence > 0
• Agent Builder skill loads without ZodError (referencedContent schema compliance)
• kbn-evals dataset run against a live cluster (needs ES + a connector — to be run manually before flipping the flag in any beta env)
• Smoke spec (detection_emulation.integration.test.ts) with EMULATION_SMOKE_ES_URL against a live ES (manual; documented in README)

Out of scope

• Beyond Wave-1 payloads. Hard-capped at 15 entries by design. Adding more is a follow-up after operator feedback on the Wave-1 set.
• Cross-space concurrency. The gate is per-space. If real_execution should be globally serialized, that's a follow-up keyed on operator demand.
• UI for emulation history. The getEmulationHistory tool exposes the SO via the Agent Builder; a dedicated stack-management UI is intentionally out of scope.
• Live-cluster eval baseline. The kbn-evals suite ships with deterministic mocks; a live-cluster baseline run + result publication is a separate follow-up because it requires connector + cluster setup outside of CI.

[cla-checker-service]

❌ Author of the following commits did not sign a Contributor Agreement:
5638057, de325db, d119178, 6f86b85, cfc20c1

Please, read and sign the above mentioned agreement if you want to contribute to this project

[infra-vault-gh-plugin-prod]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!
• Click to trigger kibana-entity-store-performance-from-pr for this PR!
• Click to trigger kibana-storybooks-from-pr for this PR!

[patrykkopycinski]

/ci

[kibanamachine]

🤖 Prompt Changes Detected

Changes have been detected to one or more prompt files in the Elastic Assistant plugin.

Please remember to update the integrations repository with your prompt changes to ensure consistency across all deployments.

Next Steps:

1. Follow the documentation in x-pack/solutions/security/packages/security-ai-prompts/README.md to update the corresponding prompt files
2. Make the changes in the integrations repository
3. Test your changes in the integrations environment
4. Ensure prompt consistency across all deployments

This is an automated reminder to help maintain prompt consistency across repositories.

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[kibanamachine]

💔 Build Failed

• Buildkite Build
• Commit: 23aee62

Failed CI Steps

• Checks
• Check Types

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	9408	9417	+9

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	12.1MB	12.1MB	+8.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	153.5KB	159.3KB	+5.8KB

Unknown metric groups

async chunk count

id	before	after	diff
securitySolution	113	112	-1

ESLint disabled in files

id	before	after	diff
securitySolution	106	107	+1

ESLint disabled line counts

id	before	after	diff
securitySolution	739	740	+1

Total ESLint disabled count

id	before	after	diff
securitySolution	845	847	+2

History

• 💔 Build #442393 failed 9e892fb
• 💔 Build #442329 failed 6981eeb
• 💔 Build #442316 failed 8aabd0f
• 💔 Build #442298 failed e477cd1

[patrykkopycinski]

Observation: EmulationRunner + withCommandGates duplicate ResponseActionsClient capabilities

While reviewing the shared infrastructure for the upcoming Endpoint Response Actions Skill (#17508), I noticed the emulation skill rebuilds several capabilities that BaseResponseActionsClient already provides on main. Consolidating would reduce ~300 lines and prevent the two paths from diverging.

What BaseResponseActionsClient already handles

Capability	Location	What it does
RBAC (per-command)	validateRequest() → isActionSupportedByAgentType()	Checks command is supported for agent type + action mode (manual/automated)
Enterprise license gate	validateRequest() → getLicenseService().isEnterprise()	Blocks automated actions without Enterprise license
Space-scoped agent validation	validateRequest() → fetchAgentPolicyInfo()	Validates agents are in the active space
Audit trail	writeActionRequestToEndpointIndex()	Writes to .logs-endpoint.actions-* with full attribution
Cases attachment	updateCases()	Attaches action to Security cases
Telemetry	sendActionSentTelemetry() / sendActionResponseTelemetry()	Reports ENDPOINT_RESPONSE_ACTION_SENT_EVENT
Action expiration	getActionRequestExpiration()	Sets TTL on action requests
Dispatch (all 12 commands)	EndpointActionsClient.isolate(), .execute(), etc.	Typed dispatch via Fleet actions API

What the emulation skill re-implements

Gate in withCommandGates / gate_checks.ts	Overlap with client
checkRbac() — maps command → RESPONSE_CONSOLE_ACTION_COMMANDS_TO_REQUIRED_AUTHZ → endpointAuthz[key]	validateRequest() already checks isActionSupportedByAgentType(agentType, command, actionType). The emulation RBAC gate adds a finer-grained check (per-console-command privilege), which the base client doesn't do — this is a genuine addition
checkAuth() — resolves username via _security/_authenticate	The client constructor takes username as a required param — the caller already resolved it. The emulation skill re-resolves because of the Task Manager fakeRequest issue, but for interactive Agent Builder calls the request already carries the user
EmulationRunner.dispatch() — exhaustive switch over all 12 command types	EndpointActionsClient has the identical switch — .isolate(), .execute(), .killProcess(), etc. The runner is a pass-through wrapper
EmulationRunner.createResponseActionsClient() — getResponseActionsClient(agentType, opts)	One-liner factory call, same as what any consumer does

What's genuinely emulation-specific (should stay)

These are NOT in the base client and rightfully belong in the emulation layer:

• EmulationAllowlist — operator-controlled host allowlist (Advanced Settings)
• EmulationRateLimiter — per-space (100/h) + per-host (3/h) sliding windows
• EmulationIdempotencyCache — dedup retried LLM tool calls
• checkRealExecutionFeatureFlags() — emulation-specific feature flag + runtime kill switch
• checkValidation() — curated-only mode, allowedExecuteCommandPatterns regex allowlist, allowedScriptIds
• buildEmulationComment() — audit attribution with conversationId/runId/toolCallId/SHA-256 prompt hash
• EmulationRunner.resolveRuleBinding() — rule context lookup for emulation actions

Suggested simplification

The EmulationRunner class could be reduced to a thin wrapper that:

1. Resolves the rule binding (emulation-specific)
2. Builds the emulation comment with actor attribution (emulation-specific)
3. Calls client.isolate() / client.execute() / etc. directly — no dispatch switch needed

The exhaustive dispatch() switch duplicates the typed interface that ResponseActionsClient already enforces. If a new command is added, it needs to be added in both places today — single point of truth would be better.

// Before: EmulationRunner.dispatch() has 12-case switch
// After: direct client call
const client = getResponseActionsClient('endpoint', constructorOptions);
const actionDetails = await client[commandMethodMap[input.command]](request, options);

The withCommandGates pipeline is valuable — the emulation-specific gates (allowlist, rate limiter, idempotency, feature flags, validation) are genuine additions. But the RBAC check + auth check + dispatch could lean on the client directly rather than re-implementing them.

This isn't blocking — the current implementation works and is well-tested. But when the Response Actions Skill (#17508) ships, it will use ResponseActionsClient directly (no runner, no custom dispatch switch). Having two patterns for the same underlying dispatch in the same codebase will cause confusion about which one to use for future response-action surfaces.

---

Context: this came up while planning the shared infrastructure between the detection emulation skill and the upcoming endpoint response actions skill. The response actions skill will be ~50 lines of tool handler code because it delegates everything to ResponseActionsClient + getActionDetailsById() directly.