-
Notifications
You must be signed in to change notification settings - Fork 53
Description
🛠 Environment
We’re using the following tech stack in our project:
-
Spring Boot: 3.4.10
-
Java: 21
-
Micrometer Observation: 1.15.4
-
Jetty: 12.0.12
⚙️ Configuration
We're configuring the TransactionOutbox bean via the following Spring config class:
@Configuration
@Import({SpringInstantiator.class})
public class TransactionOutboxConfig {
@Bean
TransactionOutboxMetrics transactionOutboxMetrics(MeterRegistry registry) {
return new TransactionOutboxMetrics(registry);
}
@Bean
TransactionOutbox transactionOutbox(DataSourceOutboxTransactionManager transactionManager,
SpringInstantiator springInstantiator,
TransactionOutboxMetrics transactionOutboxMetrics) {
return TransactionOutbox.builder()
.instantiator(springInstantiator)
.transactionManager(transactionManager)
.persistor(DefaultPersistor.builder()
.dialect(Dialect.POSTGRESQL_9)
.serializer(JacksonInvocationSerializer.builder()
.mapper(txOutboxObjectMapper())
.build())
.build())
.listener(transactionOutboxMetrics)
.build();
}
private ObjectMapper txOutboxObjectMapper() {
return Jackson2ObjectMapperBuilder.json()
.featuresToDisable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
.build();
}
}
🧨 Issue
When starting the application, the process hangs indefinitely. The last output in the logs is:
INFO [main] transactionoutbox.DefaultMigrationManager: Running migration 1: Create outbox table
Full log snapshot:
INFO [main] sentry.SpringBootDefaultSentryConfig: Sentry initialized with version: 0, environment: dev
INFO [main] jetty.JettyServletWebServerFactory: Server initialized with port: 8181
INFO [main] server.Server: jetty-12.0.12; jvm 21.0.8
INFO [main] ContextHandler.application: Initializing Spring embedded WebApplicationContext
INFO [main] context.ServletWebServerApplicationContext: Root WebApplicationContext: initialization completed
INFO [main] hikari.HikariDataSource: main-1 - Start completed.
INFO [main] hikari.HikariDataSource: vpn-1 - Start completed.
INFO [main] transactionoutbox.DefaultMigrationManager: No version record found. Attempting to create
INFO [main] transactionoutbox.DefaultMigrationManager: Created version record.
INFO [main] transactionoutbox.DefaultMigrationManager: Current version is 0, obtained lock
INFO [main] transactionoutbox.DefaultMigrationManager: Running migration 1: Create outbox table
At this point, the application freezes.
🔎 Analysis
Problem Summary
The application hangs during startup in DefaultMigrationManager.runSql() when executing:
CompletableFuture.runAsync(...).join();
Root Cause
A classic deadlock occurs between the Spring bean initialization thread and the asynchronous thread running the migration.
🧵 Thread Dump Summary
🧵 Main Thread (Spring Initialization)
-
Status: WAITING for CompletableFuture to complete
-
Holds: Spring's AbstractAutowireCapableBeanFactory lock
-
Stack:
CompletableFuture.join()
→ DefaultMigrationManager.runSql()
→ DefaultMigrationManager.migrate()
→ TransactionOutboxImpl.initialize()
→ Spring bean initialization
🧵 Thread-11 (Async Executor)
-
Status: BLOCKED waiting for Spring's BeanFactory lock
-
Wants: To complete transaction inside the migration
-
Stack:
AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck()
→ DataSourceObservationListener.handleGetConnectionBefore()
→ DataSourceTransactionManager.doBegin()
→ txm.inTransactionThrows()
→ DefaultMigrationManager.lambda$runSql$7()
→ CompletableFuture$AsyncRun.run()
🔁 Deadlock Diagram
Main Thread: [Holds BeanFactory Lock] → [Waits for CompletableFuture]
↓
Thread-11: [Waits for BeanFactory Lock] ← [Executing CompletableFuture]
❓ Why This Happens
-
The migration is executed inside a CompletableFuture.runAsync().
-
The main thread calls .join() on this future, while still holding the Spring BeanFactory lock during initialization.
-
The async thread triggers Spring's transaction management (txm.inTransactionThrows()), which involves:
-
Micrometer's DataSourceObservationListener
-
Bean lookups for observation and tracing
- This results in an attempt to access the Spring context — which is still locked by the main thread → Deadlock.
✅ Confirmed Workaround
The issue was resolved by excluding DataSourceObservationAutoConfiguration, which prevents the observation system from activating during migration:
spring.autoconfigure.exclude:
- org.springframework.boot.actuate.autoconfigure.observation.jdbc.DataSourceObservationAutoConfiguration
🙏 Request
Could you please take a look at this? It would be helpful to either:
-
Provide an option to disable async execution in DefaultMigrationManager
-
Or, delay transactional/observation configuration until after the migration phase
I am happy to provide more diagnostic details or test proposed fixes.
Thanks in advance for your help and the great work on this project!