Polling projection occasionally misses 1–2 rows (once a month) despite low RPS — how to harden GapDetection / persistence config?

[lifinsky]

Hi!
We have a polling projection that builds an account statement (balance movement view). Roughly once per month (sometimes a few times/month) we observe that 1–2 rows are missing in the projection table (no duplicates, just missing rows).

Load characteristics

• Overall balance aggregate throughput: ~5 rps max
• Per single balance aggregate: usually <= 1 rps
• This still happens “randomly” (stable frequency, hard to reproduce).

Polling projection config

$gapDetection = new GapDetection(
    retryConfig: [0, 5, 50, 500, 800, 1500, 3000, 5000],
    detectionWindow: new GapDetection\DateInterval('PT120S')
);

$options = [
    ProjectionRunningConfiguration::OPTION_LOAD_COUNT => 50,
    ProjectionRunningConfiguration::OPTION_WAIT_BEFORE_CALLING_ES_WHEN_NO_EVENTS_FOUND => 1000000,
    ProjectionRunningConfiguration::OPTION_PERSIST_CHANGES_AFTER_AMOUNT_OF_OPERATIONS => 5,
    ProjectionRunningConfiguration::OPTION_GAP_DETECTION => $gapDetection,
];

ProjectionRunningConfiguration::createPolling(CustomerAccountStatementProjection::NAME)
    ->withOptions($options);

Questions

1. Does withOptions($options) definitely apply to this polling projection at runtime?
2. Is there a recommended way to verify/log the effective options used by the projection runner (or confirm they’re not overridden elsewhere)?
3. Any recommendations to make this setup more resilient against missed rows?

[lifinsky]

@dgafka @jlabedo @unixslayer
Hi, guys! Your expertise is really important here regarding projections for a Prooph event-sourced aggregate.

[unixslayer]

@lifinsky

withOptions apply options properly and they are not overwritten in runtime.

Are you facing this issue with most recent events? How do you run this projection (worker setup number of workers, etc.)?

[lifinsky]

@unixslayer The polling consumer is deployed as a separate Kubernetes deployment with a single pod.
The Deployment uses the Recreate strategy to ensure that only one polling consumer is running at any given time and to prevent concurrent message processing.

[dgafka]

Prooph's Projecting System is based on time checks, which freezes projection from consuming new events, and if gap timeout is reached, it simply moves forward. This creates scenarios, where potentially something may be missed.

Projecting v2 with global tracking will store the gaps, so it will be able to move forward, and catch up later without freezing the projection. There will be also a configurable limit, how many events in the past we want to track for gaps.

Secondly Projection v2 with partitioned tracking will not suffer from gap problems anymore. So meaning we will be able to move forward safely without any configuration for gap detections (because simply there won't be any).