Watcher Endpoint Unable To Actually Sort Or Filter On Any Fields

[lukewhiting]

Elasticsearch Version

7.x, 8.x, 9.x

Installed Plugins

No response

Java Version

bundled

OS Version

All

Problem Description

Kibana have requested the ability to sort, filter and paginate watches server side instead of having to do it locally in the browser.

Currently the GET /_watcher/_query/watches endpoint does support accepting a sort and query object however it's real world usefulness for this scenario is limited due to a number of factors:

• The default sort method for the Kibana watch management page is by Watch name however the watch name is only stored as the ID of the .watches document that defines a watch. Since 8.0 we prevented users from sorting and filtering in queries by the document ID due to performance concerns meaning server side sorting isn't possible
  • We do have a dedicated metadata.name field on the documents but this is only set / enforced by Kibana so any Watches added directly via the ES API's can't be relied upon to have this field, meaning client sorting is still necessary
• We don't have any mappings on the .watches index so performant sorting and filtering by other fields such as creator or creating date is currently not possible either

This has been filed as a bug as the presence of sort and query params on the endpoint imply this functionality should be possible already.

Proposed solution as discussed at 2025 Engineering All Hands:

• Create a one time migration task to copy the watch ID field into the metadata.name field for all existing watches
• Modify the Watcher create / update APIs to store the watch ID in the name field if no metadata.name field is explicitly set
• Update the mappings on the .watches system index to ensure all fields currently sortable in the Kibana watch management UI are mapped

Steps to Reproduce

• Create Watches
• Attempt to sort by ID using GET /_watcher/_query/watches

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)