startup - avoid iterating over geOrchestra users during bootstrap

[pmauduit]

Description

I am not sure this is a bug or an improvement, but considering the scenario where mapstore2-georchestra is deployed into a geOrchestra instance with several hundreds of users registered in the LDAP, the webapp can take a lot of time to bootstrap itself.

For instance, at Rennes-Métropole, one of their instances contains 8k+ users in the LDAP, and with the following code being run at startup:
https://github.com/geosolutions-it/geostore/blob/master/src/web/app/src/main/java/it/geosolutions/geostore/init/GeoStoreInit.java#L115-L172
will iterate over the 8k users, which can take from 7/8 to 15 minutes to perform into their infrastructure.

We came along with the following patch:

diff --git a/src/web/app/src/main/java/it/geosolutions/geostore/init/GeoStoreInit.java b/src/web/app/src/main/java/it/geosolutions/geostore/init/GeoStoreInit.java
index 409c820..194aa7f 100644
--- a/src/web/app/src/main/java/it/geosolutions/geostore/init/GeoStoreInit.java
+++ b/src/web/app/src/main/java/it/geosolutions/geostore/init/GeoStoreInit.java
@@ -71,44 +71,49 @@ public class GeoStoreInit implements InitializingBean {
 
         LOGGER.info("===== Starting GeoStore services =====");
         // initialize password encoding
-        initPasswordEncoding();
-        long catCnt = categoryService.getCount(null);
-        if (catCnt == 0) {
-            LOGGER.warn("No category found.");
-            if (categoryListInitFile != null) {
-                LOGGER.warn("Initializing categories from file " + categoryListInitFile);
-                initCategories(categoryListInitFile);
+        if (new Boolean(System.getProperty("it.geosolutions.geostore.init.skipInit", "false"))
+                == false) {
+            initPasswordEncoding();
+            long catCnt = categoryService.getCount(null);
+            if (catCnt == 0) {
+                LOGGER.warn("No category found.");
+                if (categoryListInitFile != null) {
+                    LOGGER.warn("Initializing categories from file " + categoryListInitFile);
+                    initCategories(categoryListInitFile);
+                } else {
+                    LOGGER.info("No category initializer defined.");
+                }
             } else {
-                LOGGER.info("No category initializer defined.");
+                LOGGER.info("Categories already in db: " + catCnt);
             }
-        } else {
-            LOGGER.info("Categories already in db: " + catCnt);
-        }
 
-        long userGroupCnt = userGroupService.getAll(null, null).size();
-        if (userGroupCnt == 0) {
-            LOGGER.warn("No usersgroup found.");
-            if (userGroupListInitFile != null) {
-                LOGGER.warn("Initializing users from file " + userGroupListInitFile);
-                initUsersGroup(userGroupListInitFile);
+            long userGroupCnt = userGroupService.getAll(null, null).size();
+            if (userGroupCnt == 0) {
+                LOGGER.warn("No usersgroup found.");
+                if (userGroupListInitFile != null) {
+                    LOGGER.warn("Initializing users from file " + userGroupListInitFile);
+                    initUsersGroup(userGroupListInitFile);
+                } else {
+                    LOGGER.info("No usersgroup initializer defined.");
+                }
             } else {
-                LOGGER.info("No usersgroup initializer defined.");
+                LOGGER.info("UsersGroup already in db: " + userGroupCnt);
             }
-        } else {
-            LOGGER.info("UsersGroup already in db: " + userGroupCnt);
-        }
 
-        long userCnt = userService.getCount(null);
-        if (userCnt == 0) {
-            LOGGER.warn("No user found.");
-            if (userListInitFile != null) {
-                LOGGER.warn("Initializing users from file " + userListInitFile);
-                initUsers(userListInitFile);
+            long userCnt = userService.getCount(null);
+            if (userCnt == 0) {
+                LOGGER.warn("No user found.");
+                if (userListInitFile != null) {
+                    LOGGER.warn("Initializing users from file " + userListInitFile);
+                    initUsers(userListInitFile);
+                } else {
+                    LOGGER.info("No user initializer defined.");
+                }
             } else {
-                LOGGER.info("No user initializer defined.");
+                LOGGER.info("Users already in db: " + userCnt);
             }
         } else {
-            LOGGER.info("Users already in db: " + userCnt);
+            LOGGER.info("it.geosolutions.geostore.init.skipInit = true. Skipping init procedure.");
         }
     }

so that we could skip the password encoding logic which is outside the scope of mapstore2 or any other geOrchestra webapps, in a geOrchestra context.

What kind of improvement you want to add? (check one with "x", remove the others)

• Minor changes to existing features
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Mapstore2georchestra improvement
• Mapstore2 improvement
• Other... Please describe:

Other useful information

The provided patch above targets geostore, but since this looks like being very geOrchestra-specific, I preferred to report it on this repository.

[jusabatier]

We also found huge stratup time for Mapstore due to our LDAP size (~1.4k accounts)

[catmorales]

@tdipisa , do you have an improvement in your roadmap for this issue?
Are there other users with the same problem ?

[tdipisa]

Hi all, I remember we have encountered a similar problem in the past, raised by #541, that we have handled with geosolutions-it/geostore#290 (comment), where the PR was merged 3 years ago (more or less). Since the problem was solved last time, the first thing I'm wondering is why an issue like this is raised today after all this time. My point is mostly conceptual, I know :-)
Saying that, @pmauduit at a very first look the patch sounds logic to me but I would like to have one of our team members to evaluate it better. Is it maybe possible for you to raise a PR to GeoStore or at least an issue there to let us to handle that?

[pmauduit]

Is it maybe possible for you to raise a PR to GeoStore or at least an issue there to let us to handle that?

Sure, I'll do

the first thing I'm wondering is why an issue like this is raised today after all this time

There are several things: we are building the new test line into a kubernetes environment which will not reflect the coming production one (nor the former rancher one) in term of resources, so it is probably more resource-constrained than the former rancher environment.
Also, it was not particularly fast on the former rancher infrastructure, I just did a try and it took ~ 2 mintes 38 sec to bootstrap:

INFO 2025-03-24 09:19:28.303 org.springframework.security.ldap.DefaultSpringSecurityContextSource::<init>:80 -  URL 'ldap://external-db.georchestra-database:389/dc=georchestra-test,dc=org', root DN is 'dc=georchestra-test,dc=org'
INFO 2025-03-24 09:19:28.695 it.geosolutions.geostore.init.GeoStoreInit::afterPropertiesSet:83 - ===== Starting GeoStore services =====
INFO 2025-03-24 09:19:28.695 it.geosolutions.geostore.init.GeoStoreInit::initPasswordEncoding:128 - === Set up the security system   ====
INFO 2025-03-24 09:19:28.695 it.geosolutions.geostore.init.GeoStoreInit::initPasswordEncoding:129 - Encoding Type:DIGEST
WARN 2025-03-24 09:21:03.159 it.geosolutions.geostore.init.GeoStoreInit::initPasswordEncoding:142 - =======================================================================================
WARN 2025-03-24 09:21:03.161 it.geosolutions.geostore.init.GeoStoreInit::initPasswordEncoding:143 -    WARNING: USERS PASSWORDS ARE NOT SYNCRONIZED WITH THE CONFIGURED PASSWORD ENCODER   
WARN 2025-03-24 09:21:03.162 it.geosolutions.geostore.init.GeoStoreInit::initPasswordEncoding:144 - =======================================================================================
INFO 2025-03-24 09:21:04.082 it.geosolutions.geostore.init.GeoStoreInit::afterPropertiesSet:96 - Categories already in db: 9
INFO 2025-03-24 09:21:04.177 it.geosolutions.geostore.init.GeoStoreInit::afterPropertiesSet:109 - UsersGroup already in db: 53
INFO 2025-03-24 09:22:31.897 it.geosolutions.geostore.init.GeoStoreInit::afterPropertiesSet:123 - Users already in db: 5344
INFO 2025-03-24 09:22:33.467 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /extjs
INFO 2025-03-24 09:22:34.453 org.springframework.security.web.DefaultSecurityFilterChain::<init>:43 - Creating filter chain: any request, [org.springframework.security.web.context.SecurityContextPersistenceFilter@34077851, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@2c0c6398, org.springframework.security.web.header.HeaderWriterFilter@423ed07f, org.springframework.security.web.authentication.logout.LogoutFilter@3f850b9e, it.geosolutions.geostore.services.rest.security.HeadersAuthenticationFilter@7a5773a, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@17ea7d9c, it.geosolutions.geostore.services.rest.security.SessionTokenAuthenticationFilter@55ad5c9e, org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter@5c37f9a7, org.springframework.security.web.authentication.ui.DefaultLogoutPageGeneratingFilter@43f6cedd, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@62c66988, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@66233c9f, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@6896867a, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@1f711954, org.springframework.security.web.session.SessionManagementFilter@4f6d2e54, org.springframework.security.web.access.ExceptionTranslationFilter@627f32b0, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@19754ef8]
INFO 2025-03-24 09:22:34.895 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /data
INFO 2025-03-24 09:22:34.936 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /resources
INFO 2025-03-24 09:22:34.972 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /categories
INFO 2025-03-24 09:22:35.003 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /users
INFO 2025-03-24 09:22:35.038 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /usergroups
INFO 2025-03-24 09:22:35.068 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /misc
INFO 2025-03-24 09:22:35.085 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /backup
INFO 2025-03-24 09:22:35.102 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /session
INFO 2025-03-24 09:22:35.118 org.apache.cxf.endpoint.ServerImpl::initDestination:95 - Setting the server's publish address to be /openid
INFO 2025-03-24 09:22:35.175 org.springframework.web.context.ContextLoader::initWebApplicationContext:307 - Root WebApplicationContext initialized in 198975 ms
INFO 2025-03-24 09:22:35.550 org.springframework.web.servlet.DispatcherServlet::initServletBean:525 - Initializing Servlet 'loadAssets'
INFO 2025-03-24 09:22:35.647 org.springframework.security.config.SecurityNamespaceHandler::<init>:79 - Spring Security 'config' module version is 5.3.10.RELEASE
INFO 2025-03-24 09:22:37.515 org.springframework.web.servlet.DispatcherServlet::initServletBean:547 - Completed initialization in 1965 ms
INFO 2025-03-24 09:22:37.518 org.springframework.web.servlet.DispatcherServlet::initServletBean:525 - Initializing Servlet 'configs'
INFO 2025-03-24 09:22:37.598 org.springframework.security.config.SecurityNamespaceHandler::<init>:79 - Spring Security 'config' module version is 5.3.10.RELEASE
INFO 2025-03-24 09:22:37.851 org.springframework.web.servlet.DispatcherServlet::initServletBean:547 - Completed initialization in 332 ms
INFO 2025-03-24 09:22:37.855 org.springframework.web.servlet.DispatcherServlet::initServletBean:525 - Initializing Servlet 'extensions'
INFO 2025-03-24 09:22:37.920 org.springframework.security.config.SecurityNamespaceHandler::<init>:79 - Spring Security 'config' module version is 5.3.10.RELEASE
INFO 2025-03-24 09:22:38.111 org.springframework.web.servlet.DispatcherServlet::initServletBean:547 - Completed initialization in 255 ms
24-Mar-2025 09:22:38.154 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/mapstore] has finished in [218,472] ms
24-Mar-2025 09:22:38.160 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
24-Mar-2025 09:22:38.238 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [218732] milliseconds

The geOrchestra helm chart expects the webapp to bootstrap in at most 1 minute 15 sec. On rancher, the initializing timeout was set to 5 minutes.

[landryb]

fwiw, on our prod instance the mapstore webapp takes 50s to start with ~2500 ldap users.

Le déploiement de l'archive de l'application web [/data/tomcat/georchestra/webapps/mapstore.war]
     s'est terminé en [50 579] ms

INFO   2025-03-18 20:22:56.680   GeoStoreInit.afterPropertiesSet() - Categories already in db: 9 
INFO   2025-03-18 20:22:56.707   GeoStoreInit.afterPropertiesSet() - UsersGroup already in db: 23 
INFO   2025-03-18 20:23:12.855   GeoStoreInit.afterPropertiesSet() - Users already in db: 2415