[streams][features] schema changes (elastic#251196)

## Summary

- Rename name into subtype as the former is not aligned with the value
we currently populate (eg logging_agent, operating_system, ...)
- Delete name
- Remove the deterministic _id generation. Uses an uuid instead
- Create an LLM-generated id field that we'll use to deduplicate
features. We currently deduplicate in storage (overwrite if it already
exists). We can remove that step if/when we need history
- Rename value into properties


<img width="588" height="803" alt="features-schema-change"
src="https://github.com/user-attachments/assets/bac641e8-c851-4323-bdf3-cfb0c2bbda7e"
/>

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Author: klacabane

x-pack/platform/packages/shared/kbn-streams-ai/src/features/prompt.ts

@@ -20,21 +20,25 @@ const featuresSchema = {
       items: {
         type: 'object',
         properties: {
+          id: {
+            type: 'string',
+            description: 'Unique identifier for the feature.',
+          },
           type: {
             type: 'string',
           },
-          description: {
+          subtype: {
             type: 'string',
-            description: 'A summary of the feature.',
           },
-          name: {
+          description: {
             type: 'string',
+            description: 'A summary of the feature.',
           },
           title: {
             type: 'string',
             description: 'Very short human-readable title for UI (e.g. table, flyout header).',
           },
-          value: {
+          properties: {
             type: 'object',
             properties: {},
           },
@@ -65,11 +69,12 @@ const featuresSchema = {
           },
         },
         required: [
+          'id',
           'type',
+          'subtype',
           'description',
-          'name',
           'title',
-          'value',
+          'properties',
           'confidence',
           'evidence',
           'tags',

x-pack/platform/packages/shared/kbn-streams-ai/src/features/system_prompt.text

@@ -2,68 +2,75 @@ You are extracting **features** from log data. Features are stable facts about r
 
 Every feature you output MUST include ALL required fields:
 - `type` (string): one of `infrastructure`, `technology`, `dependency`
-- `name` (string): generic snake_case name
+- `subtype` (string): categorization within the type (e.g. `operating_system`, `programming_language`, `service_dependency`)
+- `id` (string): unique concise identifier for deduplication across runs (e.g. "aws-deployment", "log4j-2.14.1", "api-user-http")
 - `title` (string): very short human-readable title for UI display (e.g. "Ubuntu 20.04", "Log4j 2.14.1", "api-service → user-service"). Keep it to a few words; it can summarize type + key value (e.g. technology + version or source → target for dependencies).
 - `description` (string): a short summary of what the feature represents
-- `value` (object): stable, low-cardinality properties for deduplication
+- `properties` (object): stable, low-cardinality key facts for deduplication
 - `confidence` (number): 0–100
 - `evidence` (array of strings): supporting evidence from logs (2–5 items)
 - `tags` (array of strings): descriptive tags
-- `meta` (object): high-cardinality or variable data (can be `{}`)
+- `meta` (object): supplementary information that doesn't fit in `properties` - use for high-cardinality data, contextual notes, or interesting observations (can be `{}`)
 
 ## Feature Types
 
 Extract features in three categories:
 - **infrastructure**: cloud provider/deployment, container orchestration, operating systems, networking, hardware
-  - Example names: `cloud_deployment`, `container_orchestration`, `operating_system`
+  - Example subtypes: `cloud_deployment`, `container_orchestration`, `operating_system`
 - **technology**: programming languages, web servers, databases, libraries, frameworks
-  - Example names: `programming_language`, `web_server`, `database`, `logging_library`
+  - Example subtypes: `programming_language`, `web_server`, `database`, `logging_library`
 - **dependency**: explicit relationships between systems (service-to-service calls, DB connections, API integrations)
-  - Example names: `service_dependency`, `database_connection`, `api_integration`
+  - Example subtypes: `service_dependency`, `database_connection`, `api_integration`
 
 ## Consolidation Rules
 
 **Consolidate** when properties belong to the same entity and appear together in logs:
-- Good: A single cloud deployment feature with provider in `value` and regions/zones in `meta`
-- Good: A single container orchestration feature for Kubernetes with stable platform/version in `value`
+- Good: A single cloud deployment feature with provider in `properties` and regions/zones in `meta`
+- Good: A single container orchestration feature for Kubernetes with stable platform/version in `properties`
 
 **Separate** distinct technologies even if related:
 - Good: Separate features for `web_server` (nginx), `database` (postgresql), `cache` (redis)
 - Bad: Do not combine multiple distinct technologies into one feature
 
-Also: **do not emit multiple features with the same (`type`, `name`, `value`) tuple**. Merge evidence/tags/meta instead.
+Also: **do not emit multiple features with the same (`type`, `subtype`, `properties`) tuple**. Merge evidence/tags/meta instead.
 
 ## Naming Conventions
 
-Use **generic names** with specific values in the `value` object:
-- Good: `{ "name": "programming_language", "value": { "language": "java", "version": "11" } }`
-- Bad: `{ "name": "java_runtime", "value": { "version": "11" } }`
+Use **generic subtypes** with specific values in the `properties` object:
+- Good: `{ "subtype": "programming_language", "properties": { "language": "java", "version": "11" } }`
+- Bad: `{ "subtype": "java_runtime", "properties": { "version": "11" } }`
 
-Rules:
-- Use `snake_case` for `name`
-- Keep names descriptive but concise
-- Put specificity in `value`, not in `name`
+Subtype rules:
+- Use `snake_case` for `subtype`
+- Keep subtypes descriptive but concise
+- Put specificity in `properties`, not in `subtype`
 
-## Value vs Meta Fields
+ID rules:
+- Generate a short, stable identifier based on key properties
+- Use hyphens for readability (e.g. "aws-deployment", "log4j-2.14.1", "api-user-http")
+- Include distinguishing characteristics (provider, name, version, or source/target for dependencies)
+- Keep it concise (typically 2-5 tokens)
 
-The `value` field MUST contain **stable, low-cardinality properties** that enable deduplication across many log lines and deployments.
+## Properties vs Meta Fields
 
-- Use `value` for: cloud provider (`aws`, `gcp`, `azure`), technology/library name, protocol, major/normalized version, stable service names
-- Use `meta` for: regions/availability zones, hostnames, instance IDs, pod/container names, IPs, URLs/paths, request/trace IDs, endpoint lists, build hashes/edition labels
+The `properties` field MUST contain **stable, low-cardinality key facts** that enable deduplication across many log lines and deployments.
+
+- Use `properties` for: cloud provider (`aws`, `gcp`, `azure`), technology/library name, protocol, major/normalized version, stable service names
+- Use `meta` for: supplementary details like regions/availability zones, hostnames, instance IDs, pod/container names, IPs, URLs/paths, request/trace IDs, endpoint lists, build hashes/edition labels, contextual notes, security observations, or any interesting information that doesn't fit as a stable property
 
 Example (cloud deployment):
-- Good: `value: { "provider": "aws" }` (stable)
-- Good: `meta: { "regions": ["eu-west-1"], "availability_zones": ["eu-west-1a"] }` (variable/high-cardinality)
+- Good: `properties: { "provider": "aws" }` (stable)
+- Good: `meta: { "regions": ["eu-west-1"], "availability_zones": ["eu-west-1a"], "note": "Multi-region deployment pattern observed" }` (variable/high-cardinality + contextual info)
 
 **Conflict & cardinality rules**:
-- If multiple values are observed for the same property, prefer the most frequently supported value in `value` and record alternates in `meta.observed_*` with evidence.
+- If multiple values are observed for the same property, prefer the most frequently supported value in `properties` and record alternates in `meta.observed_*` with evidence.
 - Avoid emitting separate features that differ only by high-cardinality metadata. Merge into one feature and store varying details in `meta`.
 
 ## Inference & Confidence
 
 **One-level inference is allowed** when strong patterns exist, but use lower confidence and clearly label it:
 - Tag inferred features with `"inferred"` in `tags`
-- Explain the inference briefly in `meta.notes`
+- Explain the inference briefly in `meta.note`
 
 Confidence bands:
 - **90–100**: explicit, unambiguous evidence
@@ -81,21 +88,22 @@ Evidence requirements:
 - Evidence must directly support the feature claim
 
 Version formatting rules (important for CVE/vulnerability analysis):
-- Prefer normalized numeric versions in `value.version` (e.g., `"11"`, `"11.0"`, `"11.0.2"`)
+- Prefer normalized numeric versions in `properties.version` (e.g., `"11"`, `"11.0"`, `"11.0.2"`)
 - Strip leading `v` and surrounding text; keep only the numeric portion when possible
-- If the original version contains labels (LTS/Enterprise/codename/build metadata), store the normalized numeric part in `value.version` and store the original in `meta.raw_version`
-- Remove codenames/release names/edition labels from `value.version`
+- If the original version contains labels (LTS/Enterprise/codename/build metadata), store the normalized numeric part in `properties.version` and store the original in `meta.raw_version`
+- Remove codenames/release names/edition labels from `properties.version`
 
 ## Examples
 
 **Example 1 - Infrastructure with clean version**
 ```
 {
   "type": "infrastructure",
-  "name": "operating_system",
+  "subtype": "operating_system",
+  "id": "ubuntu-20.04.6",
   "title": "Ubuntu 20.04.6",
   "description": "Ubuntu Linux operating system version 20.04.6",
-  "value": {
+  "properties": {
     "os": "ubuntu",
     "version": "20.04.6"
   },
@@ -111,14 +119,15 @@ Version formatting rules (important for CVE/vulnerability analysis):
 }
 ```
 
-**Example 2 - Infrastructure showing value vs meta**
+**Example 2 - Infrastructure showing properties vs meta**
 ```
 {
   "type": "infrastructure",
-  "name": "cloud_deployment",
+  "subtype": "cloud_deployment",
+  "id": "aws-deployment",
   "title": "AWS",
   "description": "AWS cloud deployment observed across one or more regions/availability zones",
-  "value": {
+  "properties": {
     "provider": "aws"
   },
   "confidence": 92,
@@ -130,7 +139,8 @@ Version formatting rules (important for CVE/vulnerability analysis):
   "tags": ["infrastructure", "cloud"],
   "meta": {
     "regions": ["eu-west-1"],
-    "availability_zones": ["eu-west-1a", "eu-west-1b"]
+    "availability_zones": ["eu-west-1a", "eu-west-1b"],
+    "note": "Multi-AZ deployment pattern observed"
   }
 }
 ```
@@ -139,10 +149,11 @@ Version formatting rules (important for CVE/vulnerability analysis):
 ```
 {
   "type": "technology",
-  "name": "logging_library",
+  "subtype": "logging_library",
+  "id": "log4j-2.14.1",
   "title": "Log4j 2.14.1",
   "description": "Apache Log4j logging library version 2.14.1",
-  "value": {
+  "properties": {
     "library": "log4j",
     "version": "2.14.1"
   },
@@ -153,7 +164,7 @@ Version formatting rules (important for CVE/vulnerability analysis):
   ],
   "tags": ["technology", "library", "logging"],
   "meta": {
-    "security_note": "Library versions can be used for CVE/vulnerability queries."
+    "note": "Version 2.14.1 may have known CVEs; suitable for vulnerability queries"
   }
 }
 ```
@@ -162,10 +173,11 @@ Version formatting rules (important for CVE/vulnerability analysis):
 ```
 {
   "type": "technology",
-  "name": "programming_language",
+  "subtype": "programming_language",
+  "id": "java",
   "title": "Java",
   "description": "Java programming language (inferred from exception patterns)",
-  "value": {
+  "properties": {
     "language": "java"
   },
   "confidence": 45,
@@ -175,19 +187,20 @@ Version formatting rules (important for CVE/vulnerability analysis):
   ],
   "tags": ["technology", "inferred"],
   "meta": {
-    "notes": "Inferred from Java exception class names and .java stack trace references."
+    "note": "Inferred from Java exception class names and .java stack trace references"
   }
 }
 ```
 
-**Example 5 - Dependency feature showing value vs meta + aggregation/capping**
+**Example 5 - Dependency feature showing properties vs meta + aggregation/capping**
 ```
 {
   "type": "dependency",
-  "name": "service_dependency",
+  "subtype": "service_dependency",
+  "id": "api-user-http",
   "title": "api-service → user-service",
   "description": "Service-to-service HTTP dependency from api-service to user-service",
-  "value": {
+  "properties": {
     "source": "api-service",
     "target": "user-service",
     "protocol": "http"
@@ -201,7 +214,7 @@ Version formatting rules (important for CVE/vulnerability analysis):
   "meta": {
     "endpoints": ["/users", "/users/:id", "/users/:id/profile", "/users/search"],
     "methods": ["GET", "POST", "PUT"],
-    "notes": "Aggregate endpoints under one dependency; cap the list (e.g., max 10) and summarize additional entries."
+    "note": "Aggregate endpoints under one dependency; cap the list (e.g., max 10) and summarize additional entries"
   }
 }
 ```
@@ -210,7 +223,7 @@ Version formatting rules (important for CVE/vulnerability analysis):
 
 - Extract all features that meet the confidence threshold and have supporting evidence.
 - Prefer fewer, higher-confidence features over many speculative ones.
-- Sort features by descending `confidence` (and within ties, stable alphabetical order by `type`, then `name`).
+- Sort features by descending `confidence` (and within ties, stable alphabetical order by `type`, then `subtype`).
 - Dependency anti-spam: only emit a dependency feature when logs contain explicit evidence of a relationship; aggregate endpoints in `meta.endpoints` and cap the list.
 
 Extract all features that meet the confidence threshold and have supporting evidence. Use the finalize_features tool to return the results.

x-pack/platform/packages/shared/kbn-streams-schema/src/feature.ts

@@ -13,30 +13,32 @@ export type FeatureStatus = (typeof featureStatus)[number];
 export const featureStatusSchema = z.enum(featureStatus);
 
 export interface BaseFeature {
+  id: string;
   type: string;
-  name: string;
+  subtype?: string;
   title?: string;
   description: string;
-  value: Record<string, any>;
+  properties: Record<string, string>;
   confidence: number;
   evidence: string[];
   tags: string[];
   meta: Record<string, any>;
 }
 
 export interface Feature extends BaseFeature {
-  id: string;
+  uuid: string;
   status: FeatureStatus;
   last_seen: string;
   expires_at?: string;
 }
 
 export const baseFeatureSchema: z.Schema<BaseFeature> = z.object({
+  id: z.string(),
   type: z.string(),
-  name: z.string(),
+  subtype: z.string().optional(),
   title: z.string().optional(),
   description: z.string(),
-  value: z.record(z.string(), z.any()),
+  properties: z.record(z.string(), z.string()),
   confidence: z.number().min(0).max(100),
   evidence: z.array(z.string()),
   tags: z.array(z.string()),
@@ -45,7 +47,7 @@ export const baseFeatureSchema: z.Schema<BaseFeature> = z.object({
 
 export const featureSchema: z.Schema<Feature> = baseFeatureSchema.and(
   z.object({
-    id: z.string(),
+    uuid: z.string(),
     status: featureStatusSchema,
     last_seen: z.string(),
     expires_at: z.string().optional(),