Skip to content

Application Hangs on Startup Due to Deadlock in DefaultMigrationManager.runSql() with CompletableFuture.join() #932

@AlekseyMishanin

Description

@AlekseyMishanin

🛠 Environment

We’re using the following tech stack in our project:

  • Spring Boot: 3.4.10

  • Java: 21

  • Micrometer Observation: 1.15.4

  • Jetty: 12.0.12

⚙️ Configuration

We're configuring the TransactionOutbox bean via the following Spring config class:

@Configuration
@Import({SpringInstantiator.class})
public class TransactionOutboxConfig {

    @Bean
    TransactionOutboxMetrics transactionOutboxMetrics(MeterRegistry registry) {
        return new TransactionOutboxMetrics(registry);
    }

    @Bean
    TransactionOutbox transactionOutbox(DataSourceOutboxTransactionManager transactionManager,
                                        SpringInstantiator springInstantiator,
                                        TransactionOutboxMetrics transactionOutboxMetrics) {

        return TransactionOutbox.builder()
                .instantiator(springInstantiator)
                .transactionManager(transactionManager)
                .persistor(DefaultPersistor.builder()
                        .dialect(Dialect.POSTGRESQL_9)
                        .serializer(JacksonInvocationSerializer.builder()
                                .mapper(txOutboxObjectMapper())
                                .build())
                        .build())
                .listener(transactionOutboxMetrics)
                .build();
    }

    private ObjectMapper txOutboxObjectMapper() {
        return Jackson2ObjectMapperBuilder.json()
                .featuresToDisable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
                .build();
    }
}

🧨 Issue

When starting the application, the process hangs indefinitely. The last output in the logs is:

INFO [main] transactionoutbox.DefaultMigrationManager: Running migration 1: Create outbox table

Full log snapshot:

INFO [main] sentry.SpringBootDefaultSentryConfig: Sentry initialized with version: 0, environment: dev
INFO [main] jetty.JettyServletWebServerFactory: Server initialized with port: 8181
INFO [main] server.Server: jetty-12.0.12; jvm 21.0.8
INFO [main] ContextHandler.application: Initializing Spring embedded WebApplicationContext
INFO [main] context.ServletWebServerApplicationContext: Root WebApplicationContext: initialization completed
INFO [main] hikari.HikariDataSource: main-1 - Start completed.
INFO [main] hikari.HikariDataSource: vpn-1 - Start completed.
INFO [main] transactionoutbox.DefaultMigrationManager: No version record found. Attempting to create
INFO [main] transactionoutbox.DefaultMigrationManager: Created version record.
INFO [main] transactionoutbox.DefaultMigrationManager: Current version is 0, obtained lock
INFO [main] transactionoutbox.DefaultMigrationManager: Running migration 1: Create outbox table

At this point, the application freezes.

🔎 Analysis
Problem Summary

The application hangs during startup in DefaultMigrationManager.runSql() when executing:

CompletableFuture.runAsync(...).join();

Root Cause

A classic deadlock occurs between the Spring bean initialization thread and the asynchronous thread running the migration.

🧵 Thread Dump Summary
🧵 Main Thread (Spring Initialization)

  • Status: WAITING for CompletableFuture to complete

  • Holds: Spring's AbstractAutowireCapableBeanFactory lock

  • Stack:

CompletableFuture.join()
→ DefaultMigrationManager.runSql()
→ DefaultMigrationManager.migrate()
→ TransactionOutboxImpl.initialize()
→ Spring bean initialization

🧵 Thread-11 (Async Executor)

  • Status: BLOCKED waiting for Spring's BeanFactory lock

  • Wants: To complete transaction inside the migration

  • Stack:

AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck()
→ DataSourceObservationListener.handleGetConnectionBefore()
→ DataSourceTransactionManager.doBegin()
→ txm.inTransactionThrows()
→ DefaultMigrationManager.lambda$runSql$7()
→ CompletableFuture$AsyncRun.run()

🔁 Deadlock Diagram

Main Thread:  [Holds BeanFactory Lock] → [Waits for CompletableFuture]
                                              ↓
Thread-11:    [Waits for BeanFactory Lock] ← [Executing CompletableFuture]

❓ Why This Happens

  1. The migration is executed inside a CompletableFuture.runAsync().

  2. The main thread calls .join() on this future, while still holding the Spring BeanFactory lock during initialization.

  3. The async thread triggers Spring's transaction management (txm.inTransactionThrows()), which involves:

  • Micrometer's DataSourceObservationListener

  • Bean lookups for observation and tracing

  1. This results in an attempt to access the Spring context — which is still locked by the main thread → Deadlock.

✅ Confirmed Workaround

The issue was resolved by excluding DataSourceObservationAutoConfiguration, which prevents the observation system from activating during migration:

spring.autoconfigure.exclude:
  - org.springframework.boot.actuate.autoconfigure.observation.jdbc.DataSourceObservationAutoConfiguration

🙏 Request

Could you please take a look at this? It would be helpful to either:

  • Provide an option to disable async execution in DefaultMigrationManager

  • Or, delay transactional/observation configuration until after the migration phase

I am happy to provide more diagnostic details or test proposed fixes.

Thanks in advance for your help and the great work on this project!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions