Pod was idle during processing.

[cdangelo-kline]

Dear all,
During the POC, I have a source component that generates 4,190 records and a Mapper that processes each record. The Mapper is configured to scale from 0 to 5 instances.
After some time, Numaflow starts scaling up and eventually reaches the maximum of 5 instances (which is expected).
The issue is that one of these instances—specifically the 4th pod—never received any records to process. There are no errors in the 'numa' container, and the logs from the application container only show repeated IsReady calls (which always return true).
Is there any message dispatching logic that could prevent all pods from being utilized?
Interestingly, the 5th pod was actively processing records, while the 4th remained idle.

No partitions were configured.

Below my configurations of IBS and Pipeline.

kind: InterStepBufferService
apiVersion: numaflow.numaproj.io/v1alpha1
metadata:
  name: wot-ibs
  namespace: numaflow-system
  finalizers:
    - numaflow.numaproj.io/isbsvc-controller
spec:
  jetstream:
    version: latest
    replicas: 1
    persistence:
      volumeSize: 1Gi
    tls: false
    bufferConfig: |
      stream:
        storage: 1
        retention: 1
        maxAge: 1h
        replicas: 1

kind: Pipeline
apiVersion: numaflow.numaproj.io/v1alpha1
metadata:
  name: wot-cg
  namespace: numaflow-system
spec:
  interStepBufferServiceName: wot-ibs
  vertices:
    - name: in
      source:
        udsource:
          container:
            image: localhost/wotinputbae:0.0.1
            imagePullPolicy: Never
      containerTemplate:
        resources: {}
        env:
          - name: NUMAFLOW_DEBUG
            value: "true"
      limits:
        readTimeout: 1m0s
    - name: edmrpr
      udf:
        container:
          image: localhost/wotstepbae:0.0.1
          resources: {}
          imagePullPolicy: Never
        groupBy: null
      scale:
        min: 1
        max: 2
    - name: out
      sink:
        log: {}
        retryStrategy:
          onFailure: retry
  edges:
    - from: in
      to: edmrpr
      conditions: null
    - from: edmrpr
      to: out
      conditions: null
  limits:
    readBatchSize: 500
    bufferMaxLength: 30000
    bufferUsageLimit: 80
    readTimeout: 1s

[yhl25]

We don’t have any specific logic for message distribution across the pods. All messages are written to a common NATS stream, and each pod subscribes to the pull consumer to consume from it. Could you share the numa container logs from the pod that was idle?

[cdangelo-kline]

Hi all,
in attachment the numa log and the application log
wot-cg-edmrpr-4-xsoah-numa-logs.txt
wot-cg-edmrpr-4-xsoah-udf-logs.txt

[cdangelo-kline]

Update:
I modified the autoscale increasing to 25 instances, but only 4 instance worked.
It seems like to have 4 partitions

[vigith]

If you replace the udf with one of the builtins or quay.io/numaio/numaflow-go/map-cat:stable and see whether it works?

I do not see partitions set in the config.

[cdangelo-kline]

I believe I’ve identified the issue.
The source generates 4,190 records within a few seconds, and the UDF was using the default readBatchSize of 500. It seems that Numaflow sends these chunks of data to each available instance.

The problem is that, after a while, all chunks are distributed, but the number of pending messages in the ISB remains high because each step takes around 800 ms to process a single message. As a result, Numaflow starts upscaling, but there are no more messages left to distribute.

By setting readBatchSize = 10, all instances received messages, and both upscaling and downscaling worked correctly.