[BUG]: With high concurrency/ingest rate, job just sit in redis but never get picked up for execution

[veerkumar]

Version

26.1.2

Which installation method(s) does this occur on?

Docker

Describe the bug.

If the client issues too many api's call (in my case, 150 (docs from sample 767 dataset with pdf_split_page_count: 1), the job result never get return. On inspecting nv-ingest, we can see that jobs are sitting in Redis but never get picked up for execution. Finally, they expired aftera few hours. While client submited same set of jobs with only 50 concurrency, they got finished in 30 min.

Minimum reproducible example

Use nv-ingest api (not the ingestor api) and submit > 100 jobs from the 767 dataset in this repo.

Relevant log output

Other/Misc.

No response