Skip to content

Conversation

cliu123
Copy link
Member

@cliu123 cliu123 commented Sep 10, 2025

Description

Currently, the Job Scheduler plugin stores job metadata in a system index on the local OpenSearch cluster. However, some deployments require more flexibility in how and where metadata is stored. There is no straightforward way to configure the plugin to store job metadata outside of the local OpenSearch cluster.
This PR does 2 things:

  1. introduces a configurable client wrapper that allows to configure a remote metadata store as a replacement of local system index. The
  2. migrates LockService.java from OpenSearch transport client to SdkClient. Note that the SdkClient does not support remote index or DynamoDB table creation for now, so the index creation will be skipped when remote metadata store is configured. isMultiTenancyEnabled is used as a proxy of enabling remote metadata store.

Related Issues

Resolves #828

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@cliu123 cliu123 force-pushed the add_configurable_client branch from a6a11ae to a49957b Compare September 10, 2025 23:42
@cliu123 cliu123 changed the title Add configurable client Introduce a configurable remote metadata client Sep 10, 2025
Copy link

codecov bot commented Sep 11, 2025

Codecov Report

❌ Patch coverage is 62.93103% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.51%. Comparing base (fd51d8f) to head (750960a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...opensearch/jobscheduler/utils/LockServiceImpl.java 61.53% 25 Missing and 10 partials ⚠️
...rg/opensearch/jobscheduler/JobSchedulerPlugin.java 60.00% 7 Missing and 1 partial ⚠️

❌ Your project status has failed because the head coverage (70.51%) is below the target coverage (75.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #831      +/-   ##
==========================================
+ Coverage   70.49%   70.51%   +0.02%     
==========================================
  Files          42       42              
  Lines        2057     2106      +49     
  Branches      165      174       +9     
==========================================
+ Hits         1450     1485      +35     
- Misses        495      505      +10     
- Partials      112      116       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cliu123 cliu123 force-pushed the add_configurable_client branch 3 times, most recently from e580318 to a07a360 Compare September 15, 2025 00:12
@cwperks
Copy link
Member

cwperks commented Sep 15, 2025

@cliu123 is it possible to spin up an external service for integration test purposes to simulate the remote store? i.e. let's run the integration tests of the sample plugin with remote store enabled.

Potentially DynamoDB local? https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html

Docker images located here: https://hub.docker.com/r/amazon/dynamodb-local

@cliu123 cliu123 force-pushed the add_configurable_client branch from 157d656 to 5a4cb62 Compare September 15, 2025 22:22
@cliu123 cliu123 marked this pull request as draft September 16, 2025 17:30
@cliu123 cliu123 force-pushed the add_configurable_client branch 6 times, most recently from 277df11 to dc655e6 Compare September 22, 2025 18:50
@cliu123 cliu123 marked this pull request as ready for review September 22, 2025 22:18
@kkhatua kkhatua added the v3.3.0 Issues targeting release v3.3.0 label Sep 23, 2025
@cliu123 cliu123 force-pushed the add_configurable_client branch 2 times, most recently from 1083c83 to 0e11b0a Compare September 25, 2025 06:15
@cliu123 cliu123 changed the title Introduce a configurable remote metadata client Introduce a configurable remote metadata client AND migrate LockService to the client Sep 25, 2025
@cliu123 cliu123 force-pushed the add_configurable_client branch from 0e11b0a to 2b935b4 Compare September 25, 2025 06:39
@cliu123 cliu123 force-pushed the add_configurable_client branch from 202df9c to 3a8754e Compare September 25, 2025 17:04
@cliu123 cliu123 force-pushed the add_configurable_client branch from 3a8754e to ad89150 Compare September 25, 2025 17:15
Signed-off-by: cliu123 <[email protected]>
@cwperks cwperks merged commit db49a15 into opensearch-project:main Sep 25, 2025
14 of 16 checks passed
@github-project-automation github-project-automation bot moved this from 👀 In Review to ✅ Done in Engineering Effectiveness Board Sep 25, 2025
cliu123 added a commit to cliu123/job-scheduler that referenced this pull request Sep 25, 2025
…ockService to the client (opensearch-project#831)"

This reverts commit db49a15 to avoid jarHell in downstream plugins.
@kkhatua kkhatua removed the v3.3.0 Issues targeting release v3.3.0 label Sep 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

[FEATURE] Introduce a Configurable Client Wrapper to Configure the Remote Metadata Store

4 participants