Scheduler for regular search evaluation runs

.github/workflows/delete_backport_branch.yml

@@ -1,9 +1,9 @@
 name: Delete merged branch of the backport PRs
-on: 
+on:
   pull_request:
     types:
       - closed
-  
+
 jobs:
   delete-branch:
     runs-on: ubuntu-latest
@@ -19,4 +19,4 @@ jobs:
             owner: context.repo.owner,
             repo: context.repo.repo,
             ref: `heads/${context.payload.pull_request.head.ref}`,
-          })
+          })

CHANGELOG.md

@@ -7,6 +7,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 ### Breaking Changes
 
 ### Features
+* Added APIs and components to implement running scheduled experiments ([#220](https://github.com/opensearch-project/search-relevance/pull/220))
 
 ### Enhancements
 

DEVELOPER_GUIDE.md

@@ -294,3 +294,36 @@ The template is particularly valuable for:
 - Changes affecting security or backward compatibility
 
 Refer to the [design documentation guide](docs/README.md) for detailed instructions, examples, and best practices on using the template effectively.
+
+## Explanation on Scheduled Experiment Indices
+
+When running a scheduled experiment, the data is stored a little bit differently compared to running an individual experiment. The information of the job that runs the experiment is stored in a separate index called, `.search-relevance-scheduled-experiment-jobs`. The information stored is the schedule (in the cron job format) and whether the job is actually running or not.
+
+The diagram below shows how the scheduled experiment API interacts with the `.search-relevance-scheduled-experiment-jobs` index.
+
+```mermaid
+sequenceDiagram
+    PostScheduledExperimentTransportAction->>+.search-relevance-scheduled-experiment-jobs: puts schedule and experiment_id and other metadata into index
+    .search-relevance-scheduled-experiment-jobs->>+IndexOperationListener:The index operation listener activates the scheduler in the job scheduler plugin which submits the experiment to the job runner
+```
+
+The results of these runs are also stored in its own index called `.search-relevance-scheduled-experiment-history`. This helps track the status of the experiment, the results, and the id of the experiment that the scheduled run is based on.
+
+The diagram below shows the code flow behind how the data is stored into this index.
+
+```mermaid
+sequenceDiagram
+    SearchRelevanceJobRunner->>+ThreadPoolExecutor: Submits a scheduled experiment to be run into the threadpool executor
+    ThreadPoolExecutor->>+ScheduledExperimentRunnerManager:Runs the tasks which will invoke ScheduledExperimentRunnerManager to run the experiment
+    ScheduledExperimentRunnerManager->>+.search-relevance-scheduled-experiment-history: Submits result of that scheduled experiment
+```
+
+The evaluation results index `search-relevance-evaluation-result` will also be updated for viewing the scheduled experiment results in Search Relevance Dashboards. One nuance is that there will be an additional marker to indicate that the evaluation result was based on a scheduled experiment.
+
+Below is a diagram to show how evaluation results are submitted.
+
+```mermaid
+sequenceDiagram
+    PutExperimentTransportAction/ScheduledExperimentRunnerManager->>+Hybrid/Pointwise ExperimentProcessor: Starts running a pointwise or hybrid search experiment
+    Hybrid/Pointwise ExperimentProcessor->>+search-relevance-evaluation-result: Submits evaluation_result
+```

build.gradle

@@ -174,6 +174,7 @@ opensearchplugin {
     classname "${packagePath}.${pathToPlugin}.${pluginClassName}"
     licenseFile rootProject.file('LICENSE.txt')
     noticeFile rootProject.file('NOTICE.txt')
+    extendedPlugins = ['opensearch-job-scheduler']
 }
 
 dependencyLicenses.enabled = false
@@ -194,6 +195,7 @@ dependencies {
     implementation "org.opensearch:opensearch:${opensearch_version}"
     api "org.opensearch:opensearch:${opensearch_version}"
     api group: 'org.opensearch', name:'opensearch-ml-client', version: "${opensearch_build}"
+    compileOnly "org.opensearch:opensearch-job-scheduler-spi:${opensearch_build}"
     implementation group: 'com.google.guava', name: 'guava', version:'33.4.8-jre'
     compileOnly group: 'commons-lang', name: 'commons-lang', version: '2.6'
     implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.18.0'
@@ -229,6 +231,7 @@ dependencies {
     zipArchive group: 'org.opensearch.plugin', name:'neural-search', version: "${opensearch_build}"
     zipArchive group: 'org.opensearch.plugin', name:'opensearch-ubi', version: "${opensearch_build}"
     opensearchPlugin "org.opensearch.plugin:opensearch-security:${opensearch_build}@zip"
+    opensearchPlugin "org.opensearch.plugin:opensearch-job-scheduler:${opensearch_build}@zip"
 
     configurations.all {
         resolutionStrategy {
@@ -293,6 +296,7 @@ ext.resolvePluginFile = { pluginId ->
     }
 }
 def securityPluginFile = resolvePluginFile("opensearch-security")
+def jobSchedulerFile = resolvePluginFile("opensearch-job-scheduler")
 
 // === Setup security test ===
 // This flag indicates the existence of security plugin
@@ -372,6 +376,11 @@ testClusters.integTest {
     }
 }
 
+// Set up integration test clusters, installs all zipArchive dependencies
+testClusters.yamlRestTest {
+    plugin(provider(jobSchedulerFile))
+}
+
 // Re-write WaitForHttpResource with updated code to support security plugin use case
 class WaitForClusterYellow {