Watches configured with mapping types break Watcher UI / Watcher APIs in version 8.x

[romain-chanu]

Elasticsearch Version

8.5.3

Installed Plugins

Elasticsearch Service (ESS)

Java Version

bundled

OS Version

Elasticsearch Service (ESS)

Problem Description

1. After upgrading to version 8.x, watches cannot be accessed / edited via the Watcher UI in Kibana. Running Watcher APIs also fails.

Similar error messages can be observed:

could not parse [search] input for watch [my_watch]. failed to parse [my_request]

or,

could not parse [index] action [my_watch/my_action]. unexpected string field [doc_type]

2. Before version 8.x, it was possible to configure watches to use mapping types in the following:

a) Search input:

request.types

b) Search transform:

request.types

c) Index action:

doc_type

3. As of Elasticsearch version 8.0.0, mapping types are no longer supported. (c.f Removal of mapping types). When checking the deprecation logs in version 7.x (c.f Prepare to upgrade from 7.x), users should address any critical deprecations about the use of types in the watches. For example:

[deprecation.elasticsearch][CRITICAL] [types removal] Specifying types in a watcher search request is deprecated.

4. If users do not address these critical deprecations, this will result in the reported behaviour in this issue.

5. As discussed internally with @rjernst , this should be handled in the system indices migration feature as the .watches index is a system index. Moreover, while the above deprecation message is visible in the Elasticsearch deprecation logs - it is not listed in the critical deprecation in the Kibana Upgrade Assistant.

Steps to Reproduce

1. Create a cluster version 6.8.23

2. Create an index with mapping type and add a document:

PUT logs
{
  "mappings": {
    "event": {
      "properties": {
        "id": {
          "type": "integer"
        }
      }
    }
  }
}

POST logs/event
{
  "id": 1
}

3. Create a watch with a search input targeting the above index:

PUT _xpack/watcher/watch/my_watch
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          "logs"
        ],
        "types": [
          "event"
        ],
        "body": {
          "query": {
            "match_all": {}
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 0
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 0."
      }
    }
  }
}

4. Upgrade the cluster to version 7.17.8

5. Go to the Kibana Upgrade Assistant:

a) Migrate system indices. The migration of system indices will fail (i.e An error ocurred while migrating system indices for watcher: illegal_state_exception) because of the legacy index templates. Delete these templates:

• DELETE _template/.watches
• DELETE _template/.triggered_watches

b) Migrate system indices and the migration should be successful.

c) Resolve the critical Elasticsearch deprecation issues listed in the UI by deleting or reindexing the indices created in version 6.8.23 (i.e indices logs and .watcher-history-9-YYYY.MM.DD). Note: the critical Elasticsearch deprecation issue about the types removal is not listed in the UI. At this point, users may not be aware of this particular critical deprecation and may decide to proceed with the upgrade.

6. Upgrade the cluster to version 8.5.3

7. Go to the Watcher UI and observe that the existing watch cannot be edited. The following error message will be displayed when editing the watch:

Error loading watch
could not parse [search] input for watch [my_watch]. failed to parse [request]

8. Accessing the watch details using the Watcher APIs also fails:

GET _watcher/watch/my_watch

returns

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "could not read search request. unsupported non-empty array field [types]"
      }
    ],
    "type": "parse_exception",
    "reason": "could not parse [search] input for watch [my_watch]. failed to parse [request]",
    "caused_by": {
      "type": "parse_exception",
      "reason": "could not read search request. unsupported non-empty array field [types]"
    }
  },
  "status": 400
}

Logs (if relevant)

c.f above.

Workaround

1. Login to Kibana with the elastic superuser

2. Go to Kibana > Dev Tools > Console and run GET .watches/_count

3. Run GET .watches/_search?size=X where X is the count value from the count API

4. In the list of watches documents, identify which input/transform/action contains doc_type or types

a) Example 1: in the below example, input.search.my_request.types is the field that results in a parsing failure in version 8.x:

{
    "_index": ".watches",
    "_id": "my_watch",
    "_score": 1,
    "_source": {
        ...
        ...
        "input": {
            "search": {
                "my_request": {
                    "search_type": "query_then_fetch",
                    "indices": [
                        "tests-*"
                    ],
                    "types": [
                        "my_custom_doc_type"
                    ]
                }
            }
        }
    }
}

b) Example 2: in the below example, actions.my_action.index.doc_type is the field that results in a parsing failure in version 8.x:

{
    "_index": ".watches",
    "_id": "my_watch",
    "_score": 1,
    "_source": {
        ...
        ...
        "actions": {
            "my_action": {
                "index": {
                    "index": "my_test_index",
                    "doc_type": "_doc",
                    "execution_time_field": "timestamp"
                }
            }
        }
    }
}

c) Important: if any of the watches refer to the below watches where an index action contains doc_type, these can safely be deleted:

• X-Pack Monitoring: Nodes Changed
• X-Pack Monitoring: Cluster Status
• X-Pack Monitoring: Logstash Version Mismatch
• X-Pack Monitoring: Elasticsearch Version Mismatch
• X-Pack Monitoring: License Expiration
• X-Pack Monitoring: Kibana Version Mismatch

These built-in watches are obsolete and have been replaced by Kibana alerts (c.f Cluster alerting).

5. Run the below APIs to create a temp_user who can access the .watches index:

POST _security/role/watcher_superuser
{
  "indices": [
    {
      "names": [
        ".watch*"
      ],
      "privileges": [
        "all"
      ],
      "allow_restricted_indices": true
    }
  ]
}

POST _security/user/temp_user
{
  "password": "temp_password",
  "roles": [
    "superuser",
    "watcher_superuser"
  ]
}

6. Log-out from Kibana and log-in again with temp_user (password is temp_password as per the above API)

7. Run the below APIs to create an ingest pipeline that removes the problematic field previously identified. For example, the below ingest pipeline removes the actions.my_action.index.doc_type field from all watches that contain this field:

PUT _ingest/pipeline/watcher-pipeline
{
 "processors": [
  {
   "remove": {
   "field": "actions.my_action.index.doc_type",
   "if": "ctx?.actions?.my_action?.index?.doc_type != null"
   }
  }
 ]
}

Important: modify the ingest pipeline based on the field that needs to be removed.

8. Run the below API which will update the .watches index using the above ingest pipeline to remove the desired field:

POST .watches/_update_by_query?pipeline=watcher-pipeline&refresh=true

9. Log-out from Kibana and log-in again with the elastic superuser

10. Go to Stack Management > Alerts and Insights > Watcher and verify that watches can now be accessed / edited.

11. If the Watcher UI and Watcher APIs are operational, delete the user/role/pipeline created earlier:

DELETE _security/user/temp_user
DELETE _security/role/watcher_superuser
DELETE _ingest/pipeline/watcher-pipeline

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)