Spring Boot Cloud Run - issues with Pub/Sub subscription after deployment

[lassschr]

Hi,

I am having issues with message acknowledgement when deploying a new revision of a Spring Boot application in Cloud Run. The issue occurs when messages are being pulled (refer to image "Creating subscriptions") after deployment of a new instance.

Following are some images displaying the logs from the application of the following case:

1. Publishing a message that will intentionally fail and triggering retries.
2. Deploying a new revision that stops the message from failing
3. New instance started and message getting acknowledged

The issue occurs in step 3 with the error messages failed to send operations and Permanent error invalid ack id message, will not resend. These error messages started after enabling exactlyOnceDelivery.

I enabled this setting to avoid handling and acknowledging the same message twice - this issue seems to be somewhat resolved as it does not happen every time in this case. However, it has been possible for me to replicate after enabling this setting (See last image of logs).

I want to make sure that I do not handle and acknowledge the same message more than once and also fix these errors occurring related to acknowledging.

Logs before deployment with failed message retrying as expected:

Logs after deployment:

Issues with duplicated message pull around deployment of new instances. Both for retrying failed messages and newly published message only when retried/published short after deployment:

Context:
Spring Boot application with Google Cloud Run (min 1 instance / max 5 instances).

Dependencies:

            <dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>spring-cloud-gcp-dependencies</artifactId>
                <version>6.2.3</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>

        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>spring-cloud-gcp-starter-pubsub</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

application.properties:

freddy.pubsub.enabled=true
spring.cloud.gcp.pubsub.publisher.enable-message-ordering=true
spring.cloud.gcp.pubsub.publisher.endpoint=europe-west1-pubsub.googleapis.com:443
spring.cloud.gcp.pubsub.subscriber.min-duration-per-ack-extension=300

Creating subscriptions:

*Note: This is executed via an endpoint that is called on-demand.

Pulling Pub/Sub messages:
*Note: This is executed once Spring Boot context event ApplicationReadyEvent is published.

Topic in Google Cloud:

Subscription in Google Cloud:

[jinseopkim0]

Thanks for opening an issue. If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.