Skip to content

New configurable service.idle_jobs_threshold #3375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

mattnowzari
Copy link
Contributor

Closes #2298

This PR creates a new configurable field

service.idle_jobs_threshold

which is the amount of time (in seconds) before a sync job is considered 'idle'.

We make this field configurable by grabbing it from the service configurations and passing the value to SyncJobIndex::idle_jobs() as an new argument - previously this was a hardcoded global variable.

TODO

  • Setting this value to zero should disable 'idle' jobs fully (?)
  • Possibly provide a log message as to why this timeout occurred

Checklists

Pre-Review Checklist

  • this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
  • this PR has a meaningful title
  • this PR links to all relevant github issues that it fixes or partially addresses
  • if there is no GH issue, please create it. Each PR should have a link to an issue
  • this PR has a thorough description
  • Covered the changes with automated tests
  • Tested the changes locally
  • Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
  • Considered corresponding documentation changes
  • Contributed any configuration settings changes to the configuration reference

@mattnowzari mattnowzari added enhancement New feature or request sdh-driven labels Apr 21, 2025
@artem-shelkovnikov
Copy link
Member

There's one thing to keep in mind here - Kibana has its own threshold where it'll also mark jobs as "Idle" in UI. Don't remember where exactly, but somewhere in our module.

Connectors fail the job once idle threshold is reached, while Kibana just signalises that it haven't seen any updates for a bit and job might be stuck - might be worth to re-visit this part too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve job idle timeout logic
3 participants