[ML] Transition EIS auth polling to persistent task on a single node #136713

jonathan-buttner · 2025-10-16T18:32:51Z

~~This PR is based on: #136569~~ Already merged

This PR moves the EIS authorization polling logic to a persistent task on a single node.

Notable changes:

It removes the polling logic from occurring on each node
A cluster state listener is registered which checks to see if the task exists and if it doesn't, it creates the task
If a node running the task shuts down, the persistent task framework handles moving the task to a new node
If the EIS url is empty or null, the persistent task will not be created
If a cluster is no longer authorized to access certain preconfigured endpoints, the endpoints will remain instead of being removed
The polling logic compares the received authorized models with the preconfigured inference endpoints that are already stored in cluster state to determine if any are new. Only new preconfigured inference endpoints are stored
The polling logic uses a new action to send the new inference endpoints to the master node to be store. The master node must do this logic because it updates the cluster state

Testing

Start EIS

cd eis-gateway
make TLS_VERIFY_CLIENT_CERTS=false run

Start ES pointing at EIS

run-es -Dtests.es.xpack.inference.elastic.url=https://localhost:8443 -Dtests.es.xpack.inference.elastic.http.ssl.verification_mode=none -Dtests.es.xpack.inference.elastic.authorization_request_interval="5s" -Dtests.es.xpack.inference.elastic.max_authorization_request_jitter="1s"

Retrieve all the endpoints from the inference API should return some EIS endpoints now

GET _inference/_all

A task should be present in the list eis-authorization-poller[c]

GET _tasks

elasticsearchmachine · 2025-10-16T18:33:33Z

Hi @jonathan-buttner, I've created a changelog YAML for you.

DaveCTurner

Please don't use the master for admin tasks that don't actually need to run on the master. If you need a task to run approximately once in the cluster, use a persistent task instead.

…sticsearch into ml-eis-auth-polling

jonathan-buttner · 2025-10-30T18:15:46Z

.../org/elasticsearch/xpack/inference/integration/AuthorizationTaskExecutorMultipleNodesIT.java

+ */
+public class AuthorizationTaskExecutorMultipleNodesIT extends ESIntegTestCase {
+
+    private static final String AUTH_TASK_ACTION = AuthorizationPoller.TASK_NAME + "[c]";


I couldn't find exactly where [c] was being added in the code to include in this concatenation, but it always seems to be there.

jonathan-buttner · 2025-10-30T18:16:40Z

server/src/main/java/org/elasticsearch/inference/Model.java

 import java.util.Objects;

-public class Model {
+public class Model implements Writeable {


Model needs to be serialized now because it's going to be sent to the master node to be stored in the inference index by the ModelRegistry.

jonathan-buttner · 2025-10-30T18:20:07Z

...st/java/org/elasticsearch/xpack/inference/integration/InferenceRevokeDefaultEndpointsIT.java

@@ -1,357 +0,0 @@
-/*


I removed this because we are no longer revoking endpoints.

jonathan-buttner · 2025-10-30T18:20:36Z

.../internalClusterTest/java/org/elasticsearch/xpack/inference/integration/ModelRegistryIT.java


    public void testStoreModels_ReturnsEmptyList_WhenGivenNoModelsToStore() {
-        PlainActionFuture<List<ModelRegistry.ModelStoreResponse>> storeListener = new PlainActionFuture<>();
+        PlainActionFuture<List<ModelStoreResponse>> storeListener = new PlainActionFuture<>();


These changes should be from move ModelStoreResponse to core.

jonathan-buttner · 2025-10-30T19:02:53Z

...rc/main/java/org/elasticsearch/xpack/inference/services/elastic/ElasticInferenceService.java

-        );
-    }
-
-    private static Map<String, DefaultModelConfig> initDefaultEndpoints(


No longer needed because authorization is moved out of this class.

jonathan-buttner · 2025-10-30T19:03:38Z

...in/java/org/elasticsearch/xpack/inference/services/elastic/ElasticInferenceServiceModel.java

 import java.util.Objects;

-public abstract class ElasticInferenceServiceModel extends RateLimitGroupingModel {
+public class ElasticInferenceServiceModel extends RateLimitGroupingModel {


I'm using this to create a generic instance of the model so we can store them using the ModelRegistry

jonathan-buttner · 2025-10-30T19:04:12Z

.../java/org/elasticsearch/xpack/inference/services/elastic/InternalPreconfiguredEndpoints.java

+ * Represents the preconfigured endpoints that are included in Elasticsearch. EIS will support dynamic preconfigured endpoints which means
+ * it can provide new preconfigured endpoints that do not exist in the source here.
+ */
+public class InternalPreconfiguredEndpoints {


Mostly moved from the ElasticInferenceService.

jonathan-buttner · 2025-10-30T19:05:49Z

...ference/services/elastic/authorization/ElasticInferenceServiceAuthorizationHandlerTests.java

@@ -1,283 +0,0 @@
-/*


Class under test was removed so we don't need it anymore. Most of the functionality was moved to AuthorizationPoller and its test files.

…sticsearch into ml-eis-auth-polling

…uth-polling

…sticsearch into ml-eis-auth-polling

…uth-polling

jonathan-buttner added >enhancement :ml Machine learning Team:ML Meta label for the ML team v9.3.0 labels Oct 16, 2025

DaveCTurner requested changes Oct 16, 2025

View reviewed changes

jonathan-buttner mentioned this pull request Oct 20, 2025

[ML] Integrate calls to EIS for preconfigured inference endpoints for ModelRegistry functionality #136192

Closed

jonathan-buttner added the cloud-deploy Publish cloud docker image for Cloud-First-Testing label Oct 21, 2025

jonathan-buttner and others added 15 commits October 29, 2025 12:59

Creating new cluster state listener to kick off polling logic

c6fc960

Update docs/changelog/136713.yaml

ecfe885

[CI] Auto commit changes from spotless

b6928b6

Starting persistent tasks

59ce7d0

Switching to a persistent task, need to create the action though

273d551

Adding master action

458664e

Successful task creation

a7f0f91

Starting tests

2e3246c

More tests

9e5ed51

Even more tests

36deff5

[CI] Auto commit changes from spotless

83f2c65

Starting integration tests

d39d7ef

Adding test stub

02d4766

[CI] Auto commit changes from spotless

b41b1d4

Adding integration test

10ad8f2

jonathan-buttner force-pushed the ml-eis-auth-polling branch from b2cd14a to 10ad8f2 Compare October 29, 2025 17:09

jonathan-buttner and others added 4 commits October 29, 2025 17:12

Fixing relocation test

9762fc6

[CI] Auto commit changes from spotless

0b1551d

working test

6f2c27b

Merge branch 'ml-eis-auth-polling' of github.com:jonathan-buttner/ela…

9b764a6

…sticsearch into ml-eis-auth-polling

jonathan-buttner changed the title ~~[ML] Transition EIS auth polling to master node~~ [ML] Transition EIS auth polling to persistent task on a single node Oct 30, 2025

Some clean up

fd0b0cf

jonathan-buttner and others added 3 commits October 30, 2025 14:11

Removing unneeded tests

5f99fe2

[CI] Auto commit changes from spotless

75005b7

Refactoring tests

c34ac07

jonathan-buttner commented Oct 30, 2025

View reviewed changes

jonathan-buttner added 3 commits October 30, 2025 15:07

Merge branch 'ml-eis-auth-polling' of github.com:jonathan-buttner/ela…

acb0620

…sticsearch into ml-eis-auth-polling

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

a27d2f7

…uth-polling

updating transport version

df94ce9

jonathan-buttner requested a review from DaveCTurner October 30, 2025 19:09

elasticsearchmachine and others added 11 commits October 30, 2025 19:16

[CI] Auto commit changes from spotless

a7e67f8

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

8c2a40a

…uth-polling

Fixing transport version

4b2b33f

Fixing check for preconfigured endpoints

4b7e6cf

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

78c2a5c

…uth-polling

[CI] Auto commit changes from spotless

909ef5e

Fixing tests

f16b912

Merge branch 'ml-eis-auth-polling' of github.com:jonathan-buttner/ela…

4d23e3f

…sticsearch into ml-eis-auth-polling

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

fb2ac58

…uth-polling

Fixing text embedding test

45d167d

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

ad63742

…uth-polling

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ML] Transition EIS auth polling to persistent task on a single node #136713

[ML] Transition EIS auth polling to persistent task on a single node #136713

jonathan-buttner commented Oct 16, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Oct 16, 2025

Uh oh!

DaveCTurner left a comment

Uh oh!

jonathan-buttner Oct 30, 2025

Uh oh!

jonathan-buttner Oct 30, 2025

Uh oh!

jonathan-buttner Oct 30, 2025

Uh oh!

jonathan-buttner Oct 30, 2025

Uh oh!

jonathan-buttner Oct 30, 2025

Uh oh!

jonathan-buttner Oct 30, 2025

Uh oh!

jonathan-buttner Oct 30, 2025

Uh oh!

jonathan-buttner Oct 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[ML] Transition EIS auth polling to persistent task on a single node #136713

Are you sure you want to change the base?

[ML] Transition EIS auth polling to persistent task on a single node #136713

Conversation

jonathan-buttner commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

elasticsearchmachine commented Oct 16, 2025

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jonathan-buttner commented Oct 16, 2025 •

edited

Loading