Log retrieval using the Elasticsearch backend only supports up to 10,000 log lines and any additional logs beyond this limit are not retrieved

[sandeepbarnwal]

Jenkins and plugins versions report

Environment

Jenkins: 2.504.3
OS: Linux - 6.12.5-linuxkit
Java: 21.0.7 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
opentelemetry:3.1543.v8446b_92b_cd64
opentelemetry-api:1.49.0.59.vc32561e77cf3

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller and agent are running in docker containers using demo
OS: Linux - 6.12.5-linuxkit

Reproduction steps

1. create the jenkins environment with OTel collector, EDOT and elastic backend using make -C demos start from https://github.com/jenkinsci/opentelemetry-plugin/blob/main/demos/Makefile.

2. Add Elastic Observability and enable Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins

3. create a pipeline with below script

pipeline {
             agent any

             stages {
                 stage('Generate File') {
                     steps {
                         sh '''
                         set +x
                         I=0
                         while [ "$I" -lt 15000 ]; do
                             echo "This is line ${I} - Lorem ipsum adipiscing elit, sunt in culpa qui officia deserunt mollit anim id est laborum" >> test.txt
                             I=$(($I+1))
                         done
                         '''
                     }
                 }

                 stage('Print File') {
                     steps {
                         sh 'cat test.txt'
                     }
                 }

                 stage('Delete File') {
                     steps {
                         sh 'rm -f test.txt'
                     }
                 }

             }
         }

4. The pipeline script generates a high volume of log lines in a short period. Adjust the otel.blrp.max.queue.size setting to accommodate the increased log throughput.

5. Execute the pipeline and wait for it to succeed.

6. Navigate to console output

Expected Results

Console output should show the complete log

Actual Results

Console output shows logs up to 10000 lines.

Anything else?

No response

Are you interested in contributing a fix?

Yes. I know the fix but I'd need to investigate the performance impact.