[BUG] Job Scheduler plugin can no longer access .opendistro-job-scheduler-lock index.

[msoler8785]

Describe the bug

After I upgraded OpenSearch from 2.9.1 to 3.3.2 I started observing errors about not being able to access the .opendistro-job-scheduler-lock system table.

I decided to reinstall and restore indices from snapshots. Then I restored my ISM policies. I started getting similar errors that the plugin cannot create the index.

Error

[2025-12-05T15:49:06,791][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node10] uncaught exception in thread [DefaultDispatcher-worker-2]
org.opensearch.OpenSearchSecurityException: no permissions for [] and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=[], requestedTenant=null]

Stacktrace

[2025-12-05T15:49:06,790][INFO ][o.o.j.t.PluginClient     ] [node10] Running transport action with subject: plugin:org.opensearch.jobscheduler.JobSchedulerPlugin
[2025-12-05T15:49:06,791][INFO ][o.o.s.p.SystemIndexAccessEvaluator] [node10] No indices:admin/create permission for user roles [] to System Indices .opendistro-job-scheduler-lock
[2025-12-05T15:49:06,791][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node10] uncaught exception in thread [DefaultDispatcher-worker-2]
org.opensearch.OpenSearchSecurityException: no permissions for [] and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=[], requestedTenant=null]
        at org.opensearch.security.filter.SecurityFilter.lambda$apply0$3(SecurityFilter.java:415) ~[?:?]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:509) ~[?:?]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:175) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.FilterClient.doExecute(FilterClient.java:83) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.jobscheduler.transport.PluginClient.lambda$doExecute$0(PluginClient.java:56) ~[?:?]
        at org.opensearch.security.identity.SecurePluginSubject.runAs(SecurePluginSubject.java:52) ~[?:?]
        at org.opensearch.jobscheduler.transport.PluginClient.doExecute(PluginClient.java:54) ~[?:?]
        at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1610) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.support.AbstractClient$IndicesAdmin.create(AbstractClient.java:1700) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.createLockIndex(LockServiceImpl.java:108) ~[opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLockWithId(LockServiceImpl.java:177) ~[opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLock(LockServiceImpl.java:138) ~[opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1.invokeSuspend$lambda$0(ManagedIndexRunner.kt:244) ~[opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:230) ~[opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1.invokeSuspend(ManagedIndexRunner.kt:244) ~[opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-2.2.0.jar:2.2.0-release-294]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException

I created the index manually based on the plugin settings:

PUT .opendistro-job-scheduler-lock
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "job_id": {
        "type": "keyword"
      },
      "job_index_name": {
        "type": "keyword"
      },
      "lock_duration_seconds": {
        "type": "long"
      },
      "lock_time": {
        "type": "date",
        "format": "epoch_second"
      },
      "released": {
        "type": "boolean"
      }
    }
  },
  "settings": {
    "index": {
      "number_of_shards": "1",
      "number_of_replicas": "1"
    }
  }
}

And I started getting the same error as before:

Error

[2025-12-05T15:59:15,241][ERROR][o.o.j.u.LockServiceImpl  ] [node10] Exception occurred finding lock
org.opensearch.OpenSearchSecurityException: no permissions for [] and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=[], requestedTenant=null]

Stacktrace

[2025-12-05T15:59:15,240][INFO ][o.o.j.t.PluginClient     ] [node10] Running transport action with subject: plugin:org.opensearch.jobscheduler.JobSchedulerPlugin
[2025-12-05T15:59:15,241][INFO ][o.o.s.p.SystemIndexAccessEvaluator] [node10] No indices:data/read/get permission for user roles [] to System Indices .opendistro-job-scheduler-lock
[2025-12-05T15:59:15,241][ERROR][o.o.j.u.LockServiceImpl  ] [node10] Exception occurred finding lock
org.opensearch.OpenSearchSecurityException: no permissions for [] and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=[], requestedTenant=null]
        at org.opensearch.security.filter.SecurityFilter.lambda$apply0$3(SecurityFilter.java:415) [opensearch-security-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:509) [opensearch-security-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:175) [opensearch-security-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81) [opensearch-performance-analyzer-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.FilterClient.doExecute(FilterClient.java:83) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.jobscheduler.transport.PluginClient.lambda$doExecute$0(PluginClient.java:56) [opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.security.identity.SecurePluginSubject.runAs(SecurePluginSubject.java:52) [opensearch-security-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.transport.PluginClient.doExecute(PluginClient.java:54) [opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.support.AbstractClient.get(AbstractClient.java:597) [opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.findLock(LockServiceImpl.java:282) [opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.lambda$acquireLockWithId$0(LockServiceImpl.java:180) [opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.3.2.jar:3.3.2]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.createLockIndex(LockServiceImpl.java:102) [opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLockWithId(LockServiceImpl.java:177) [opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLock(LockServiceImpl.java:138) [opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1.invokeSuspend$lambda$0(ManagedIndexRunner.kt:244) [opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:230) [opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1.invokeSuspend(ManagedIndexRunner.kt:244) [opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-2.2.0.jar:2.2.0-release-294]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
[2025-12-05T15:59:15,243][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node10] uncaught exception in thread [DefaultDispatcher-worker-2]
org.opensearch.OpenSearchSecurityException: no permissions for [] and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=[], requestedTenant=null]
        at org.opensearch.security.filter.SecurityFilter.lambda$apply0$3(SecurityFilter.java:415) ~[?:?]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:509) ~[?:?]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:175) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.FilterClient.doExecute(FilterClient.java:83) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.jobscheduler.transport.PluginClient.lambda$doExecute$0(PluginClient.java:56) ~[?:?]
        at org.opensearch.security.identity.SecurePluginSubject.runAs(SecurePluginSubject.java:52) ~[?:?]
        at org.opensearch.jobscheduler.transport.PluginClient.doExecute(PluginClient.java:54) ~[?:?]
        at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.transport.client.support.AbstractClient.get(AbstractClient.java:597) ~[opensearch-3.3.2.jar:3.3.2]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.findLock(LockServiceImpl.java:282) ~[opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.lambda$acquireLockWithId$0(LockServiceImpl.java:180) ~[opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-3.3.2.jar:3.3.2]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.createLockIndex(LockServiceImpl.java:102) ~[opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLockWithId(LockServiceImpl.java:177) ~[opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLock(LockServiceImpl.java:138) ~[opensearch-job-scheduler-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1.invokeSuspend$lambda$0(ManagedIndexRunner.kt:244) ~[opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:230) ~[opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1.invokeSuspend(ManagedIndexRunner.kt:244) ~[opensearch-index-management-3.3.2.0.jar:3.3.2.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-2.2.0.jar:2.2.0-release-294]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
        Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException

Additional Information

All the policy managed indices are in the initializing state.

Related component

Plugins

To Reproduce

1. Install OS 3.3.2
2. Create an index
3. Apply an ISM policy to the index
4. Observe the logs for the aforementioned error.

Expected behavior

The plugin is able to access the indices it needs.

Additional Details

Plugins
Default

Screenshots
NA

Host/Environment (please complete the following information):

• OS: Ubuntu
• Version 20.04.6 LTS

Additional context
Installed using Debian .dpkg.

[cwperks]

@msoler8785 do you have plugins.security.system_indices.permission.enabled set to true in opensearch.yml? This looks like a bug with that setting. I suspect if you set to false this would work as expected. I can raise a PR later in the week to fix this case when plugins.security.system_indices.permission.enabled is set to true.

[msoler8785]

Yes, I had enabled it during the upgrade to reindex some of the indices to a newer version. Setting plugins.security.system_indices.permission.enabled: false fixed the issue, and the plugin is now initializing the ISM policies.

[10000-ki]

same here

This is a highly critical bug. A quick fix is needed.

Changing the plugins.security.system_indices.permission.enabled setting to false is not advisable as it creates significant security vulnerabilities.

[msoler8785]

@10000-ki setting plugins.security.system_indices.permission.enabled to false disables the ability to use roles for users that can access the system index. This is actually more secure I think.

When set to false, the permission is disabled and only admins with an admin certificate can make changes to system indexes. By default, the permission is set to false in a new cluster.

https://docs.opensearch.org/latest/install-and-configure/configuring-opensearch/security-settings/#:~:text=plugins.security.system_indices.permission.enabled

I think you are thinking of the plugins.security.system_indices.enabled setting. Which is what enables the system indices. This can stay at true

[llilyy]

Hi, @cwperks
This issue is quite urgent for our team, so I went ahead and created a PR to fix it: opensearch-project/security#5858
Have you already started working on your fix? If not, would you be able to review my PR instead? I'd really appreciate your feedback.

[cwperks]

Thank you @llilyy, I will take a look ASAP.

Do you have plugins.security.system_indices.permission.enabled set to true as well? System index access is always granted to the user connecting with admin certificate regardless of this setting. See an example of making a request with the admin certificate here: https://docs.opensearch.org/latest/security/configuration/system-indices/

[cwperks]

Changing the plugins.security.system_indices.permission.enabled setting to false is not advisable as it creates significant security vulnerabilities.

This is actually more secure. When set to false only the user connecting with the admin certificate can access system indices.