java.lang.OutOfMemoryError: Java heap space

[JakeCube]

I've a self-hosted tolgee where setup via dockers
docker run -d -v tolgee_data:/data/ -p 8085:8080 tolgee/tolgee
When I tried to do mass translation around 2000+ string with 4 languages.
The application start became slow & not responding from UI & API level.
As result the application was hang & always return timeout now.

After a moment, I decided to inspect the docker. It throws error belows.

• I've tried to restart or reinstall the dockers, unable to resolve this.

2024-10-23 08:03:16.014  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-10-23 08:03:16.015  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2024-10-23 08:03:16.018  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 3 ms
2024-10-23 08:03:26.948  WARN 1 --- [MessageBroker-1] io.tolgee.batch.ScheduledJobCleaner      : Removing stuck job state 1000114122 using scheduled task
Exception in thread "MessageBroker-2" java.lang.OutOfMemoryError: Java heap space
2024-10-23 08:03:48.902 ERROR 1 --- [atcher-worker-1] i.t.batch.BatchJobConcurrentLauncher     : Error in batch job action service

java.lang.OutOfMemoryError: Java heap space

2024-10-23 08:03:48.906  WARN 1 --- [atcher-worker-1] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@7904fa55 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
Exception in thread "Catalina-utility-1" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Catalina-utility-2" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-nio-8080-Poller" java.lang.OutOfMemoryError: Java heap space
2024-10-23 08:04:03.490 ERROR 1 --- [atcher-worker-1] i.t.batch.BatchJobConcurrentLauncher     : Error in batch job action service

java.lang.OutOfMemoryError: Java heap space

2024-10-23 08:04:03.506  WARN 1 --- [atcher-worker-1] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@4c79242b (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-10-23 08:04:14.842  INFO 1 --- [MessageBroker-3] o.s.w.s.c.WebSocketMessageBrokerStats    : WebSocketSession[0 current WS(0)-HttpStream(0)-HttpPoll(0), 0 total, 0 closed abnormally (0 connect failure, 0 send limit, 0 transport error)], stompSubProtocol[processed CONNECT(0)-CONNECTED(0)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], outboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], sockJsScheduler[pool size = 2, active threads = 1, queued tasks = 5, completed tasks = 13]

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Catalina-utility-3"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Catalina-utility-4"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessageBroker-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DefaultDispatcher-worker-2"
2024-10-23 08:04:32.973 ERROR 1 --- [MessageBroker-3] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-nio-8080-Acceptor" java.lang.OutOfMemoryError: Java heap space
2024-10-23 08:04:32.973 ERROR 1 --- [atcher-worker-1] i.t.batch.BatchJobConcurrentLauncher     : Error in batch job action service

java.lang.OutOfMemoryError: Java heap space

2024-10-23 08:04:33.017  WARN 1 --- [atcher-worker-1] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@76ff1983 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-10-23 08:04:46.419 ERROR 1 --- [atcher-worker-1] i.t.batch.BatchJobConcurrentLauncher     : Error in batch job action service

2024-10-23 08:08:20.536  WARN 1 --- [MessageBroker-3] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 53200
2024-10-23 08:08:20.536 ERROR 1 --- [MessageBroker-3] o.h.engine.jdbc.spi.SqlExceptionHelper   : Ran out of memory retrieving query results.
2024-10-23 08:08:20.536  WARN 1 --- [MessageBroker-3] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 08006
2024-10-23 08:08:20.536 ERROR 1 --- [MessageBroker-3] o.h.engine.jdbc.spi.SqlExceptionHelper   : An I/O error occurred while sending to the backend.
2024-10-23 08:08:20.536  WARN 1 --- [atcher-worker-1] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@4a8861b9 (null). Possibly consider using a shorter maxLifetime value.
2024-10-23 08:08:20.574 ERROR 1 --- [MessageBroker-3] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1626) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.query.Query.getResultList(Query.java:165) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]
        at org.springframework.orm.jpa.SharedEntityManagerCreator$DeferredQueryInvocationHandler.invoke(SharedEntityManagerCreator.java:410) ~[spring-orm-5.3.28.jar:5.3.28]
        at com.sun.proxy.$Proxy228.getResultList(Unknown Source) ~[na:na]
        at io.tolgee.batch.BatchJobChunkExecutionQueue.populateQueue(BatchJobChunkExecutionQueue.kt:63) ~[data-v3.37.1-plain.jar:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.3.28.jar:5.3.28]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.28.jar:5.3.28]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]
Caused by: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:67) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.getResultSet(Loader.java:2322) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2075) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2037) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.doQuery(Loader.java:956) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:357) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.doList(Loader.java:2868) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.doList(Loader.java:2850) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2682) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.Loader.list(Loader.java:2677) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:540) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:218) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1459) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1649) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        ... 19 common frames omitted
Caused by: org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2336) ~[postgresql-42.3.8.jar:42.3.8]
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355) ~[postgresql-42.3.8.jar:42.3.8]
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490) ~[postgresql-42.3.8.jar:42.3.8]
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408) ~[postgresql-42.3.8.jar:42.3.8]
        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167) ~[postgresql-42.3.8.jar:42.3.8]
        at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:119) ~[postgresql-42.3.8.jar:42.3.8]
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) ~[HikariCP-4.0.3.jar:na]
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) ~[HikariCP-4.0.3.jar:na]
        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        ... 34 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space

2024-10-23 08:08:34.078 ERROR 1 --- [atcher-worker-1] i.t.batch.BatchJobConcurrentLauncher     : Error in batch job action service
java.lang.OutOfMemoryError: Java heap space

2024-10-23 08:08:34.089  WARN 1 --- [atcher-worker-1] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@7ff10395 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024-10-23 08:08:49.137 ERROR 1 --- [atcher-worker-1] i.t.batch.BatchJobConcurrentLauncher     : Error in batch job action service

[JakeCube]

Resolved by alternative way, by cleaning the queue task. Drop a solution here, hope can help others in future.

1. docker exec -it {docker_name} sh
2. psql -U postgres -d postgres
3. UPDATE tolgee_batch_job SET status = 'CANCELLED' WHERE ID = {id};
4. DELETE FROM tolgee_batch_job_chunk_execution WHERE status = 'PENDING';
5. Exit from psql & docker
6. docker restart {docker_name}

[JanCizmar]

Hey! Thanks for sharing this. What version of Tolgee Platform do you use?

We have recently published some fixes, which might resolve this.

[JakeCube]

Hey @JanCizmar, I believe it was latest version.
Cause i always install the apps via docker command from docs.