fix: declare synchronized entity classes on native writes DHIS2-21963 [42] (#24847)

* fix: declare synchronized entity classes on native writes DHIS2-21963 (#24803)

* fix: declare synchronized entity classes on native writes

Native queries that run executeUpdate() without declaring which tables
they touch make Hibernate conservatively evict EVERY L2 entity region,
not just the affected one.

NativeSQLQueryPlan.performExecuteUpdate calls coordinateSharedCacheCleanup
with getCustomQuery().getQuerySpaces(). SQLCustomQuery leaves that set
empty unless addSynchronizedEntityClass/QuerySpace was called, and
BulkOperationCleanupAction.affectedEntity treats an empty set as "affects
everything", so every persister that can write to cache gets an
EntityCleanup. EntityCleanup.release() calls unlockRegion(), which is
evictAll() -> evictData() on the whole region.

Note AbstractReadWriteAccess.removeAll IS a no-op under read-write, which
is why this is easy to miss; the eviction comes from unlockRegion, not
removeAll. Both stores here use stateless sessions, where
isEventSource() is false, so the cleanup runs immediately rather than
being deferred to the ActionQueue.

Draft to confirm the mechanism: measured on a Sierra Leone instance, a
single data value write with a new period emptied the whole DataElement
region (0 hits / 3037 logical misses on the next read), while the same
write with an existing period left it fully warm. No eviction or removal
counter moves and nothing is logged, so this is invisible without
instrumenting Hibernate.

HousekeepingJob runs every 20s by default and drives the jobconfiguration
writes here, so on a stock instance every cached entity region is dropped
on that cadence.

* fix: declare query spaces on the remaining native writes

* fix: synchronize entities not join tables so cached collections are evicted

* docs: explain query spaces in nativeSynchronizedQuery javadoc

* refactor: add stateless session variant of nativeSynchronizedQuery

* refactor: use nativeSynchronizedQuery in complete data set registration store

* refactor: run audit trigger DDL via jdbcTemplate

* fix: declare element entities for native writes to collection tables

A write to a collection table needs the collection's element entity
declared, not the owning entity. An entity's query spaces are its own
table(s) only (SingleTableEntityPersister), never the tables of the
collections it owns, and BulkOperationCleanupAction resolves collection
regions via getCollectionRolesByEntityParticipant, which is keyed by the
collection's element entity.

So synchronizing on the owner leaves its cached collection stale:

* categories_categoryoptions needs CategoryOption, not Category
* users_catdimensionconstraints needs Category, not User

Confirmed against a running instance. With a category collection warm and
in steady state (+1 hit, 0 misses per read), a merge of an unrelated pair
of categories left a bystander category's cached collection reloading cold
(+2 misses, +1 put), while an unrelated write left it fully warm.

Also correct the comments in the category combo and data set stores, which
reached the right regions but described the wrong reason, and expand the
nativeSynchronizedQuery javadoc with the element-entity rule.

* test: expect OptionSet region to survive housekeeping DHIS2-21963

The housekeeping job no longer wipes every L2 entity region, so the
OptionSet region keeps its entries across the job and the query after it
hits the cache instead of reloading cold.

This is the same assertion change master and 2.43 already carry from
"fix: Replicate users with JDBC" (#23154), which is not on 2.42.

Author: teleivo

dhis-2/dhis-services/dhis-service-core/src/main/java/org/hisp/dhis/datavalue/hibernate/HibernateDataValueTrimStore.java

@@ -87,8 +87,7 @@ public int updateDeletedIfNotZeroIsSignificant() {
           AND de.zeroissignificant = false
           AND dv.dataelementid = de.dataelementid
           AND (dv.value IS NULL OR dv.value = '')""";
-    return getSession()
-        .createNativeQuery(sql)
+    return nativeSynchronizedQuery(sql)
         .setLockOptions(new LockOptions(PESSIMISTIC_WRITE).setTimeOut(1000))
         .executeUpdate();
   }

dhis-2/dhis-services/dhis-service-core/src/main/java/org/hisp/dhis/program/hibernate/HibernateEventStore.java

@@ -155,6 +155,6 @@ where attributeoptioncomboid in (%s)
         """
             .formatted(coc, cocs.stream().map(String::valueOf).collect(Collectors.joining(",")));
 
-    entityManager.createNativeQuery(sql).executeUpdate();
+    nativeSynchronizedQuery(sql).executeUpdate();
   }
 }

dhis-2/dhis-services/dhis-service-core/src/main/java/org/hisp/dhis/scheduling/HibernateJobConfigurationStore.java

@@ -50,6 +50,7 @@
 import org.hibernate.query.NativeQuery;
 import org.hisp.dhis.common.UID;
 import org.hisp.dhis.common.hibernate.HibernateIdentifiableObjectStore;
+import org.hisp.dhis.fileresource.FileResource;
 import org.hisp.dhis.hibernate.jsonb.type.JsonJobParametersType;
 import org.hisp.dhis.security.acl.AclService;
 import org.springframework.context.ApplicationEventPublisher;
@@ -385,7 +386,10 @@ public boolean tryExecuteNow(@Nonnull UID jobId) {
         and (schedulingtype != 'ONCE_ASAP' or lastfinished is null)
         """;
     return runWriteInStatelessSession(
-            q -> q.createNativeQuery(sql).setParameter("id", jobId.getValue()).executeUpdate())
+            q ->
+                nativeSynchronizedQuery(q, sql)
+                    .setParameter("id", jobId.getValue())
+                    .executeUpdate())
         > 0;
   }
 
@@ -413,7 +417,10 @@ and not exists (
         )
         """;
     return runWriteInStatelessSession(
-            q -> q.createNativeQuery(sql).setParameter("id", jobId.getValue()).executeUpdate())
+            q ->
+                nativeSynchronizedQuery(q, sql)
+                    .setParameter("id", jobId.getValue())
+                    .executeUpdate())
         > 0;
   }
 
@@ -444,7 +451,10 @@ public boolean tryCancel(@Nonnull UID jobId) {
           )
         """;
     return runWriteInStatelessSession(
-            q -> q.createNativeQuery(sql).setParameter("id", jobId.getValue()).executeUpdate())
+            q ->
+                nativeSynchronizedQuery(q, sql)
+                    .setParameter("id", jobId.getValue())
+                    .executeUpdate())
         > 0;
   }
 
@@ -475,7 +485,7 @@ public boolean tryFinish(@Nonnull UID jobId, JobStatus status) {
         """;
     return runWriteInStatelessSession(
             q ->
-                q.createNativeQuery(sql)
+                nativeSynchronizedQuery(q, sql)
                     .setParameter("id", jobId.getValue())
                     .setParameter("status", status.name())
                     .executeUpdate())
@@ -503,7 +513,7 @@ public boolean trySkip(@Nonnull String queue) {
           or lastexecuted < (select lastexecuted from jobconfiguration where queuename = :queue and queueposition = 0 limit 1))
         """;
     return runWriteInStatelessSession(
-            q -> q.createNativeQuery(sql).setParameter("queue", queue).executeUpdate())
+            q -> nativeSynchronizedQuery(q, sql).setParameter("queue", queue).executeUpdate())
         > 0;
   }
 
@@ -521,7 +531,7 @@ public void updateProgress(
         """;
     runWriteInStatelessSession(
         q ->
-            q.createNativeQuery(sql)
+            nativeSynchronizedQuery(q, sql)
                 .setParameter("id", jobId.getValue())
                 .setParameter("json", progressJson)
                 .setParameter("errors", errorCodes)
@@ -539,7 +549,7 @@ public int updateDisabledJobs() {
         where jobstatus = 'SCHEDULED'
         and enabled = false
         """;
-    return runWriteInStatelessSession(q -> q.createNativeQuery(sql).executeUpdate());
+    return runWriteInStatelessSession(q -> nativeSynchronizedQuery(q, sql).executeUpdate());
   }
 
   @Override
@@ -557,7 +567,7 @@ and now() > lastfinished + :ttl * interval '1 minute'
     int deletedCount =
         runWriteInStatelessSession(
             q ->
-                q.createNativeQuery(sql)
+                nativeSynchronizedQuery(q, sql)
                     .setLockOptions(new LockOptions(LockMode.PESSIMISTIC_WRITE).setTimeOut(2000))
                     .setParameter("ttl", max(1, ttlMinutes))
                     .executeUpdate());
@@ -575,6 +585,7 @@ and uid not in (select uid from jobconfiguration where schedulingtype = 'ONCE_AS
     runWriteInStatelessSession(
         q ->
             q.createNativeQuery(sql2)
+                .addSynchronizedEntityClass(FileResource.class)
                 .setLockOptions(new LockOptions(LockMode.PESSIMISTIC_WRITE).setTimeOut(2000))
                 .executeUpdate());
     return deletedCount;
@@ -606,7 +617,7 @@ and now() > lastalive + :timeout * interval '1 minute'
         """;
     return runWriteInStatelessSession(
         q ->
-            q.createNativeQuery(sql)
+            nativeSynchronizedQuery(q, sql)
                 .setParameter("timeout", max(1, timeoutMinutes))
                 .executeUpdate());
   }
@@ -637,7 +648,10 @@ public boolean tryRevertNow(@Nonnull UID jobId) {
         and now() > jobconfiguration.lastalive + interval '1 minute'
       """;
     return runWriteInStatelessSession(
-            q -> q.createNativeQuery(sql).setParameter("id", jobId.getValue()).executeUpdate())
+            q ->
+                nativeSynchronizedQuery(q, sql)
+                    .setParameter("id", jobId.getValue())
+                    .executeUpdate())
         > 0;
   }
 

dhis-2/dhis-support/dhis-support-hibernate/src/main/java/org/hisp/dhis/HibernateNativeStore.java

@@ -37,6 +37,7 @@
 import javax.annotation.Nonnull;
 import lombok.extern.slf4j.Slf4j;
 import org.hibernate.Session;
+import org.hibernate.StatelessSession;
 import org.hibernate.metamodel.spi.MetamodelImplementor;
 import org.hibernate.persister.entity.EntityPersister;
 import org.hibernate.persister.entity.SingleTableEntityPersister;
@@ -97,8 +98,22 @@ protected final Session getSession() {
    * current class of the store. Use this to avoid all Hibernate second level caches from being
    * invalidated.
    *
-   * <p>Be aware that it is only correct to use this if and only if the only table touched by the
-   * native query is the one table belonging to the store.
+   * <p>Every native write triggers a cache cleanup driven only by its declared query spaces (the
+   * tables it affects). Declare none and Hibernate reads that as "unknown" and evicts
+   * <em>every</em> entity and collection region, so always declare, even when this entity is not
+   * cached.
+   *
+   * <p>Correct only when the sole table this statement <em>writes</em> is the store's own; tables
+   * merely read by a join or subquery need no declaration. Otherwise call {@code
+   * addSynchronizedEntityClass} directly, once per entity written.
+   *
+   * <p>Beware collection tables. An entity's query spaces are its own table(s) only, never the
+   * tables of the collections it owns, so declaring the owner does <em>not</em> evict its cached
+   * collections. Hibernate resolves collection regions through {@code
+   * getCollectionRolesByEntityParticipant}, which is keyed by the collection's <em>element</em>
+   * entity. So for a write against a join or child table, declare the entity on the far side of the
+   * association: {@code categories_categoryoptions} needs {@code CategoryOption}, not {@code
+   * Category}.
    *
    * @param sql the SQL query to execute
    * @return the {@link NativeQuery} instance
@@ -108,6 +123,17 @@ protected NativeQuery nativeSynchronizedQuery(@Language("SQL") String sql) {
     return getSession().createNativeQuery(sql).addSynchronizedEntityClass(getClazz());
   }
 
+  /**
+   * Same as {@link #nativeSynchronizedQuery(String)} but on a {@link StatelessSession}, which is a
+   * different session from the store's own. Pass the session that opened the surrounding
+   * transaction, otherwise the query runs outside it.
+   */
+  @SuppressWarnings("rawtypes")
+  protected NativeQuery nativeSynchronizedQuery(
+      StatelessSession session, @Language("SQL") String sql) {
+    return session.createNativeQuery(sql).addSynchronizedEntityClass(getClazz());
+  }
+
   /**
    * Same as {@link #nativeSynchronizedQuery(String)} just with the return type being specified as
    * the store entity type. Use only when the result is a of the store entity type or a list of it.

dhis-2/dhis-test-integration/src/test/java/org/hisp/dhis/cache/HibernateQueryCacheTest.java

@@ -121,16 +121,17 @@ void testQueryCache() {
   void testHouseKeepingJobWithCache() {
     setUpData();
     createSelectQuery(10);
-    assertEquals(9, sessionFactory.getStatistics().getQueryCacheHitCount());
+    long queryCacheHitCountBefore = sessionFactory.getStatistics().getQueryCacheHitCount();
+    assertEquals(9, queryCacheHitCountBefore);
     housekeepingJob.execute(null, JobProgress.noop());
     createSelectQuery(1);
     assertEquals(
-        9,
+        10,
         sessionFactory
             .getStatistics()
             .getCacheRegionStatistics(OptionSet.class.getName())
             .getHitCount());
-    assertTrue(sessionFactory.getStatistics().getQueryCacheHitCount() > 10);
+    assertTrue(sessionFactory.getStatistics().getQueryCacheHitCount() > queryCacheHitCountBefore);
   }
 
   private void createSelectQuery(int numberOfQueries) {