Reduce usage of `TransportMasterNodeReadAction`

[DaveCTurner]

TransportMasterNodeReadAction is intended for cases where we need to collect some state held only on the elected master, for instance related to shard allocation or data stream errors. However, many TransportMasterNodeReadAction implementations work as a pure function of the cluster state, which is held on every node, so there is no need to route these requests to the master for processing. Moreover, some of these requests may be quite expensive to process in large clusters, so routing them all to the master represents a scalability bottleneck. We should reconsider each usage of TransportMasterNodeReadAction and decide whether it needs to run on the master or not. If not, we should convert them to regular local-only transport actions (e.g. using TransportLocalClusterStateAction).

Additionally, many of these actions are not currently cancellable, but they (or at least the expensive ones) should be. Experience shows that we're not great at spotting the expensive ones ahead of time, so IMO we should err on the side of caution and make each one cancellable unless we have a good reason for not doing so.

Note that attempting to route these requests to the current master does not give them any stronger consistency guarantees, because the node that does the work does not validate that it is the master before responding. It's possible that a new master has been elected, and the cluster state updated, without the responding node knowing about it.

• TransportGetDataStreamsAction (org.elasticsearch.datastreams.action) Run TransportGetDataStreamsAction on local node #122852
• GetPipelineTransportAction (org.elasticsearch.action.ingest) Run GetPipelineTransportAction on local node #120445
• TransportClusterGetSettingsAction (org.elasticsearch.action.admin.cluster.settings) Run TransportClusterGetSettingsAction on local node #119831
• TransportClusterHealthAction (org.elasticsearch.action.admin.cluster.health)
• TransportClusterInfoAction (org.elasticsearch.action.support.master.info)
• TransportClusterSearchShardsAction (org.elasticsearch.action.admin.cluster.shards)
• TransportClusterStateAction (org.elasticsearch.action.admin.cluster.state)
• TransportDeprecationInfoAction (org.elasticsearch.xpack.deprecation)
• TransportEnrichStatsAction (org.elasticsearch.xpack.enrich.action) Run TransportEnrichStatsAction on local node #121256
• TransportExplainDataStreamLifecycleAction (org.elasticsearch.datastreams.lifecycle.action), see Run TransportExplainDataStreamLifecycleAction on local node #120567 (comment)
• TransportExplainLifecycleAction (org.elasticsearch.xpack.ilm.action) Run TransportExplainLifecycleAction on local node #122885
• TransportFollowInfoAction (org.elasticsearch.xpack.ccr.action)
• TransportGetAliasesAction (org.elasticsearch.action.admin.indices.alias.get) Run TransportGetAliasesAction on local node #101815
• TransportGetAnalyticsCollectionAction (org.elasticsearch.xpack.application.analytics.action)
• TransportGetAutoFollowPatternAction (org.elasticsearch.xpack.ccr.action)
• TransportGetBasicStatusAction (org.elasticsearch.license)
• TransportGetComponentTemplateAction (org.elasticsearch.action.admin.indices.template.get) Run TransportGetComponentTemplateAction on local node #116868
• TransportGetComposableIndexTemplateAction (org.elasticsearch.action.admin.indices.template.get) Run TransportGetComposableIndexTemplate on local node #119830
• TransportGetDataStreamLifecycleAction (org.elasticsearch.datastreams.lifecycle.action) Run TransportGetDataStreamLifecycleAction on local node #125214
• TransportGetDataStreamLifecycleStatsAction (org.elasticsearch.datastreams.lifecycle.action), depends on DataStreamLifecycleErrorStore which only runs on the master node.
• TransportGetDataStreamOptionsAction (org.elasticsearch.datastreams.options.action) Run TransportGetDataStreamOptionsAction on local node #125213
• TransportGetDatafeedsAction (org.elasticsearch.xpack.ml.action)
• TransportGetDesiredBalanceAction (org.elasticsearch.action.admin.cluster.allocation), see Reduce usage of TransportMasterNodeReadAction #101805 (comment)
• TransportGetDesiredNodesAction (org.elasticsearch.action.admin.cluster.desirednodes)
• TransportGetEnrichPolicyAction (org.elasticsearch.xpack.enrich.action) Run TransportGetEnrichPolicyAction on local node #121124
• TransportGetIndexAction (org.elasticsearch.action.admin.indices.get)
• TransportGetIndexTemplatesAction (org.elasticsearch.action.admin.indices.template.get) Run TransportGetIndexTemplatesAction on local node #119837
• TransportGetJobModelSnapshotsUpgradeStatsAction (org.elasticsearch.xpack.ml.action)
• TransportGetJobsAction (org.elasticsearch.xpack.ml.action)
• TransportGetLicenseAction (org.elasticsearch.license)
• TransportGetLifecycleAction (org.elasticsearch.xpack.ilm.action) Run TransportGetLifecycleAction on local node #126002
• TransportGetMappingsAction (org.elasticsearch.action.admin.indices.mapping.get) Run TransportGetMappingsAction on local node #122921
• TransportGetRepositoriesAction (org.elasticsearch.action.admin.cluster.repositories.get)
• TransportGetSettingsAction (org.elasticsearch.action.admin.indices.settings.get)
• TransportGetStatusAction (org.elasticsearch.xpack.ilm.action)
• TransportGetStoredScriptAction (org.elasticsearch.action.admin.cluster.storedscripts)
• TransportGetTrialStatusAction (org.elasticsearch.license)
• TransportGetWatcherSettingsAction (org.elasticsearch.xpack.watcher.transport.actions) Run TransportGetWatcherSettingsAction on local node #122857
• TransportIndicesShardStoresAction (org.elasticsearch.action.admin.indices.shards)
• TransportPendingClusterTasksAction (org.elasticsearch.action.admin.cluster.tasks)
• TransportPrevalidateNodeRemovalAction (org.elasticsearch.action.admin.cluster.node.shutdown)
• TransportSimulateIndexTemplateAction (org.elasticsearch.action.admin.indices.template.post) Run template simulation actions on local node #120038
• TransportSimulateTemplateAction (org.elasticsearch.action.admin.indices.template.post) Run template simulation actions on local node #120038
• TransportXPackUsageAction and all inheritors of XPackUsageFeatureTransportAction (various packages) Run XPack usage actions on local node #122933
• any other TransportMasterNodeReadAction implementations added since this list was created

Relates #77466

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[idegtiarenko]

I believe TransportGetDesiredBalanceAction is required to run on elected as it response contains DesiredBalanceStats that are computed in the allocator during execution. I think the same is true regarding ClusterInfo. Alternatively we can update the action to run anywhere in the cluster and read only the stats and clusterInfo from elected master.

[DaveCTurner]

I believe TransportGetDesiredBalanceAction is required to run on elected

Me too - please feel free to cross it off the list in the OP with a comment to that effect.

[nielsbauman]

I'm adding TransportGetStatusAction and TransportGetLifecycleAction (org.elasticsearch.xpack.ilm.action) to the list. They don't extend TransportMasterNodeReadAction (they extend TransportMasterNodeAction), but they're also pure functions of the cluster state.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)