Release/13.3 (#339)

* Release/13.2 (#332)

* Update GrantMandatoryQuestionFundingLocation enum to replace 'OUTSIDE_UK' with 'INTERNATIONAL'

* Update GrantMandatoryQuestionFundingLocation enum to replace 'INTERNATIONAL' with 'OUTSIDE_UK'

* Add submission anonymisation feature

- Introduced SubmissionAnonymisationConfigProperties for configuration settings.
- Added EXPIRED status to SubmissionStatus enum.
- Enhanced GrantAttachmentRepository and GrantMandatoryQuestionRepository with delete methods for submissions.
- Updated SubmissionRepository with methods for anonymising submissions and deleting related data.
- Implemented SubmissionAnonymisationScheduler to handle scheduled anonymisation of submissions.
- Created SubmissionAnonymisationService to manage the anonymisation process, including S3 object deletion and database cleanup.
- Added application properties for submission anonymisation configuration.
- Created database migration to document the new EXPIRED status in the submission table.

* Refactor SubmissionAnonymisationConfigProperties

- Removed Lombok annotations: @builder, @AllArgsConstructor, and @NoArgsConstructor.
- Simplified the class by retaining only @Getter and @Setter annotations.
- Adjusted the default value for daysBeforeExpiry to 90, ensuring clarity in configuration settings.

* Enhance S3Service to support deletion of attachments using S3 URI

- Updated deleteAttachment method to accept an S3 URI instead of just the object key.
- Extracted bucket name and key from the S3 URI for improved flexibility.
- Added logging to indicate which bucket and object are being deleted.
- Ensured deletion from both the specified bucket and the attachments bucket.

* Refactor SubmissionAnonymisationService to improve S3 deletion handling

- Updated the S3 deletion logic to abort anonymisation if any deletion fails, ensuring the submission remains in IN_PROGRESS for retry.
- Enhanced logging to provide clearer context on failures during S3 object deletion, improving traceability and error handling.

* Release/13.3 (#334)

* Add submission anonymisation feature

- Introduced SubmissionAnonymisationConfigProperties for configuration settings.
- Added EXPIRED status to SubmissionStatus enum.
- Enhanced GrantAttachmentRepository and GrantMandatoryQuestionRepository with delete methods for submissions.
- Updated SubmissionRepository with methods for anonymising submissions and deleting related data.
- Implemented SubmissionAnonymisationScheduler to handle scheduled anonymisation of submissions.
- Created SubmissionAnonymisationService to manage the anonymisation process, including S3 object deletion and database cleanup.
- Added application properties for submission anonymisation configuration.
- Created database migration to document the new EXPIRED status in the submission table.

* Refactor SubmissionAnonymisationConfigProperties

- Removed Lombok annotations: @builder, @AllArgsConstructor, and @NoArgsConstructor.
- Simplified the class by retaining only @Getter and @Setter annotations.
- Adjusted the default value for daysBeforeExpiry to 90, ensuring clarity in configuration settings.

* Enhance S3Service to support deletion of attachments using S3 URI

- Updated deleteAttachment method to accept an S3 URI instead of just the object key.
- Extracted bucket name and key from the S3 URI for improved flexibility.
- Added logging to indicate which bucket and object are being deleted.
- Ensured deletion from both the specified bucket and the attachments bucket.

* Refactor SubmissionAnonymisationService to improve S3 deletion handling

- Updated the S3 deletion logic to abort anonymisation if any deletion fails, ensuring the submission remains in IN_PROGRESS for retry.
- Enhanced logging to provide clearer context on failures during S3 object deletion, improving traceability and error handling.

* Add batch size configuration and update repository methods for pagination

- Introduced a new `batchSize` property in `SubmissionAnonymisationConfigProperties` with a default value of 500.
- Updated `SubmissionRepository` to include pagination support in the `findByStatusAndLastUpdatedBefore` method.
- Modified `SubmissionAnonymisationScheduler` to utilize the new pagination feature when retrieving submissions for anonymisation.

* Add batch size configuration for submission anonymisation scheduler

- Introduced a new property `submission-anonymisation-scheduler.batchSize` with a value of 500 in the application properties file to enhance the configuration of the submission anonymisation process.

* Release/13.3 (#336)

* Add submission anonymisation feature

- Introduced SubmissionAnonymisationConfigProperties for configuration settings.
- Added EXPIRED status to SubmissionStatus enum.
- Enhanced GrantAttachmentRepository and GrantMandatoryQuestionRepository with delete methods for submissions.
- Updated SubmissionRepository with methods for anonymising submissions and deleting related data.
- Implemented SubmissionAnonymisationScheduler to handle scheduled anonymisation of submissions.
- Created SubmissionAnonymisationService to manage the anonymisation process, including S3 object deletion and database cleanup.
- Added application properties for submission anonymisation configuration.
- Created database migration to document the new EXPIRED status in the submission table.

* Refactor SubmissionAnonymisationConfigProperties

- Removed Lombok annotations: @builder, @AllArgsConstructor, and @NoArgsConstructor.
- Simplified the class by retaining only @Getter and @Setter annotations.
- Adjusted the default value for daysBeforeExpiry to 90, ensuring clarity in configuration settings.

* Enhance S3Service to support deletion of attachments using S3 URI

- Updated deleteAttachment method to accept an S3 URI instead of just the object key.
- Extracted bucket name and key from the S3 URI for improved flexibility.
- Added logging to indicate which bucket and object are being deleted.
- Ensured deletion from both the specified bucket and the attachments bucket.

* Refactor SubmissionAnonymisationService to improve S3 deletion handling

- Updated the S3 deletion logic to abort anonymisation if any deletion fails, ensuring the submission remains in IN_PROGRESS for retry.
- Enhanced logging to provide clearer context on failures during S3 object deletion, improving traceability and error handling.

* Add batch size configuration and update repository methods for pagination

- Introduced a new `batchSize` property in `SubmissionAnonymisationConfigProperties` with a default value of 500.
- Updated `SubmissionRepository` to include pagination support in the `findByStatusAndLastUpdatedBefore` method.
- Modified `SubmissionAnonymisationScheduler` to utilize the new pagination feature when retrieving submissions for anonymisation.

* Add batch size configuration for submission anonymisation scheduler

- Introduced a new property `submission-anonymisation-scheduler.batchSize` with a value of 500 in the application properties file to enhance the configuration of the submission anonymisation process.

* Update SubmissionRepository and SubmissionAnonymisationScheduler for advert closure handling

- Modified the SubmissionRepository to add a new method that retrieves submissions based on status, last updated date, and advert closure.
- Updated the SubmissionAnonymisationScheduler to utilize the new repository method for fetching submissions due for anonymisation.

Author: rmohammed-goaco

src/main/java/gov/cabinetoffice/gap/adminbackend/repositories/SubmissionRepository.java

@@ -39,8 +39,16 @@ void updateLastRequiredChecksExportByGrantApplicationIdAndStatus(Instant lastReq
     void updateLastRequiredChecksExportBySchemeIdAndStatus(Instant lastRequiredChecksExport, Integer id,
             SubmissionStatus status);
 
-    List<Submission> findByStatusAndLastUpdatedBefore(SubmissionStatus status, LocalDateTime cutoff,
-            Pageable pageable);
+    @Query(value = """
+            SELECT gs.* FROM grant_submission gs
+            JOIN grant_advert ga ON ga.scheme_id = gs.scheme_id
+            WHERE gs.status = :status
+              AND gs.last_updated < :cutoff
+              AND ga.closing_date < NOW()
+            LIMIT :#{#pageable.pageSize}
+            """, nativeQuery = true)
+    List<Submission> findByStatusAndLastUpdatedBeforeAndAdvertClosed(@Param("status") String status,
+            @Param("cutoff") LocalDateTime cutoff, Pageable pageable);
 
     @Transactional
     @Modifying

src/main/java/gov/cabinetoffice/gap/adminbackend/schedulers/SubmissionAnonymisationScheduler.java

@@ -39,7 +39,7 @@ public void anonymiseInactiveSubmissions() {
                 cutoff);
 
         final List<Submission> dueForAnonymisation = submissionRepository
-                .findByStatusAndLastUpdatedBefore(SubmissionStatus.IN_PROGRESS, cutoff,
+                .findByStatusAndLastUpdatedBeforeAndAdvertClosed(SubmissionStatus.IN_PROGRESS.name(), cutoff,
                         PageRequest.of(0, config.getBatchSize()));
 
         log.info("Found {} submission(s) to anonymise", dueForAnonymisation.size());