find_structure API unable to process nested fields from ndjson

[rseldner]

Elasticsearch Version

8.16.5

Installed Plugins

No response

Java Version

bundled

OS Version

Elastic Cloud

Problem Description

The find_structure API in elasticsearch (doc) is unable to find fields nested under an object.

This results in limiting the Kibana Data Visualizer file upload functionality

• Kibana analyzeFile function: https://github.com/elastic/kibana/blob/162d3abc53cc161e485b570ea6f8587d5cea9f2a/x-pack/platform/plugins/private/file_upload/server/analyze_file.tsx#L16-L29
  • calls _text_structure/find_structure https://github.com/elastic/kibana/blob/162d3abc53cc161e485b570ea6f8587d5cea9f2a/src/platform/plugins/shared/console/server/lib/spec_definitions/json/generated/text_structure.find_structure.json#L34-L36

It would be ideal to document as a known issue or limitation in both products for now

{
    "overrides": {
        "explain": "true"
    },
    "results": {
        "num_lines_analyzed": 3,
        "num_messages_analyzed": 3,
-        "sample_start": "{\"host\": {\"id\": \"1\", \"category\": \"NETWORKING DEVICE\"}}\n{\"host\": {\"id\": \"2\", \"category\": \"NETWORKING DEVICE\"}}\n",
        "charset": "UTF-8",
        "has_byte_order_marker": false,
        "format": "ndjson",
        "ecs_compatibility": "disabled",
        "need_client_timezone": false,
-        "mappings": {
-            "properties": {
-                "host": {
-                    "type": "object"
-                }
-            }
-        },
        "explanation": [
            "Using character encoding [UTF-8], which matched the input with [15%] confidence - first [8kB] of input was pure ASCII",
            "Deciding sample is newline delimited NDJSON"
        ]
    }
}

Steps to Reproduce

Submit an ndjson with nested fields

POST _text_structure/find_structure?filter_path=mappings
{"host": {"id": "1", "category": "NETWORKING DEVICE"}}
{"host": {"id": "2", "category": "NETWORKING DEVICE"}}

Only the parent level object is detected

{
  "properties": {
    "host": {
      "type": "object"
    }
  }
}

We have to flatten the structure to work around this:

POST _text_structure/find_structure?filter_path=mappings
{"host.id": 1, "host.category": "NETWORKING DEVICE"}
{"host.id": 2, "host.category": "NETWORKING DEVICE"}

{
  "mappings": {
    "properties": {
      "host.category": {
        "type": "keyword"
      },
      "host.id": {
        "type": "long"
      }
    }
  }
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)