Feature/ultip 5215 (#232)

* Add support for "Common Lands" processing

Introduced functionality to handle "Common Lands" data, including:
- Added `SamCommonLandsEnabled` flag in `appsettings.json`.
- Updated `SamDailyScanContext` and `SamHoldingImportContext` to include "Common Lands".
- Enhanced `SamHoldingImportAggregationStep` to fetch and process "Common Lands".
- Added mapping logic for "Common Lands" in Silver and Gold models.
- Introduced new API routes and queries for "Common Lands".
- Extended `IDataBridgeClient` to support "Common Lands" API calls.
- Updated `SiteDto`, `SamHoldingDocument`, and `SiteDocument` to include "Common Lands" fields.
- Added anonymization support for "Common Lands" in `PiiAnonymizerHelper`.
- Enhanced `FakeDataBridgeClient` to simulate "Common Lands" data.
- Added `SamCommonLandDailyScanStep` for daily scanning of "Common Lands".
- Introduced `SamCommonLandMapper` for mapping "Common Lands" data.
- Added new classes for "Common Lands" relationships and identifiers.

These changes enable the application to retrieve, process, and map "Common Lands" data while ensuring proper anonymization and testing support.

* Add support for SamCommonLand data handling

Introduced nullability support in `ToAssociatedCommonLands` to handle null or empty lists. Added `PermanentLandHoldingIdentifier` to `SiteDocumentExtensions` for enhanced data mapping.

Updated `SamHoldingImportOrchestratorTests` and `SamBulkImportWithAccurateRawDataTests` to include `commonLands` in test setups, mocking, and verification. Modified `GetAllQueryUris` to support `commonLandsUri`. Added mock setup for `GetByCodeAsync` to return "Common Land" site type.

Enhanced `SamTestScenarios` with `RawCommonLandsByCommonCph` for test data. Added `SamCommonLandMapperTests` to validate mapping logic and `SamCommonLandTests` to ensure proper behavior of `SamCommonLand`.

These changes ensure robust handling, mapping, and testing of `SamCommonLand` data.

* Replace null with string.Empty for MAIN_CPH in tests

Updated test cases in `SamCommonLandMapperTests.cs` and
`SamCommonLandTests.cs` to replace `null` values for the
`MAIN_CPH` property with `string.Empty`. This aligns with
the updated behavior where `MAIN_CPH` no longer supports
`null` values.

Removed a test case in `SamCommonLandTests.cs` that
validated `null` `MAIN_CPH`, as it is no longer relevant.
Other test cases remain to validate empty and whitespace
values for `MAIN_CPH`.

* Clean up redundant braces and improve test coverage

* Ensure culture-invariant date parsing and formatting

Updated the `NormaliseDate` method to use `CultureInfo.InvariantCulture` for both `DateTime.TryParse` and `ToString` to ensure consistent, culture-independent behavior. Added `System.Globalization` namespace to support these changes.

* Add tests for Sam Common Lands functionality

Enhanced `DataBridgeClientTests` with new unit tests to validate
the behavior of `DataBridgeClient` methods for fetching "Sam
Common Lands" data. Added tests for both enabled and disabled
feature flag scenarios, covering paged responses and filtering
by `COMMON_CPH`.

Introduced helper methods in `MockSamData` to generate mock
responses for "Sam Common Lands" data, ensuring realistic and
consistent test data. These changes improve test coverage and
verify functionality under various configurations.

* Add unit tests for PII anonymization in SamCommonLand

Added a new test class `PiiAnonymizerHelperTests` to validate
the `PiiAnonymizerHelper` class. Introduced tests to ensure
PII fields in `SamCommonLand` are anonymized correctly,
including `ADDRESS_LINE_1`, `POSTCODE`, `PREMISES_NAME`,
`EASTING`, and `NORTHING`. Verified deterministic behavior
based on `COMMON_CPH` and ensured non-PII fields remain
unchanged.

Added tests for edge cases such as null or empty fields,
placeholder values, and empty `DataBridgeResponse` objects.
Included a comprehensive test to validate anonymization of
all PII fields in a single call.

* Refine filters and update object references

Updated `VerifySilverDataTypesAsync` in both `SamImportHoldingAnonMessageTests` and `SamImportHoldingMessageTests` to add a filter excluding `SiteTypeCode` equal to "CL".

Replaced `_localStackFixture` with `localStackFixture` in `SamImportHoldingMessageTests` for `CreateMessage` and `SendMessageAsync` calls to ensure consistent object usage.

Updated `DisposeAsync` in `SamImportHoldingMessageTests` to use `mongoDbFixture` instead of `_mongoDbFixture` for purging data tables.

These changes improve query precision, naming consistency, and maintainability.

* Refactor tests and add SamCommonLandDailyScan tests

Refactored `SamImportHoldingAnonMessageTests` and
`SamImportHoldingMessageTests` to use private readonly fields for
dependencies, improving encapsulation and maintainability. Updated
method calls to align with the new structure.

Added `SamCommonLandDailyScanStepTests` to test the functionality of
`SamCommonLandDailyScanStep`. This includes mocking dependencies,
configuring in-memory feature flags, and writing unit tests for
various scenarios such as handling feature flags, API responses,
pagination, and error handling.

Introduced `FluentAssertions` for expressive assertions and improved
test readability. Enhanced code consistency and maintainability
through better naming conventions and encapsulation.

* Replace null with empty string in COMMON_CPH property

* Add tests for anonymization and edge case handling

Added multiple test methods to validate anonymization logic:
- `GetSamCommonLandsAsync_Generic_ShouldAnonymizeAddressAndLocationFields`: Ensures PII fields are anonymized while non-PII fields remain unchanged.
- `GetSamCommonLandsByCommonCphAsync_ShouldAnonymizeAllCommonLands`: Verifies anonymization of all common lands for a given `COMMON_CPH`.
- `GetSamCommonLandsByCommonCphAsync_ShouldHandleNullFields`: Confirms proper handling of null values in PII fields.
- `GetSamCommonLandsByCommonCphAsync_ShouldNotAnonymizePlaceholderPremisesName`: Ensures placeholder values like `"-"` are not anonymized.
- `GetSamCommonLandsAsync_ShouldProduceDeterministicResults`: Validates deterministic anonymization for the same `COMMON_CPH`.
- `GetSamCommonLandsAsync_ShouldPreserveNonPiiFields`: Confirms non-PII fields are preserved while PII fields are anonymized.
- `GetSamCommonLandsAsync_ShouldReturnNull_WhenInnerReturnsNull`: Ensures `null` is returned when the inner client returns `null`.
- `GetSamCommonLandsByCommonCphAsync_ShouldHandleEmptyList`: Verifies proper handling of empty list responses.

---------

Co-authored-by: Graham McVea <graham.mcvea@bartonkeys.com>

Author: grahammcvea-defra

src/KeeperData.Api/appsettings.json

@@ -66,7 +66,8 @@
     "SamHoldingsEnabled": true,
     "SamHoldersEnabled": true,
     "SamHerdsEnabled": true,
-    "SamPartiesEnabled": true
+    "SamPartiesEnabled": true,
+    "SamCommonLandsEnabled": true
   },
   "DataBridgeScanConfiguration": {
     "QueryPageSize": 100,

src/KeeperData.Application/Orchestration/ChangeScanning/Sam/Daily/SamDailyScanContext.cs

@@ -9,4 +9,5 @@ public class SamDailyScanContext : ScanContext
     public EntityScanContext Holders { get; init; } = new();
     public EntityScanContext Herds { get; init; } = new();
     public EntityScanContext Parties { get; init; } = new();
+    public EntityScanContext CommonLands { get; init; } = new();
 }

src/KeeperData.Application/Orchestration/ChangeScanning/Sam/Daily/Steps/SamCommonLandDailyScanStep.cs

@@ -0,0 +1,63 @@
+using KeeperData.Application.Orchestration.ChangeScanning.BaseClasses;
+using KeeperData.Core.ApiClients.DataBridgeApi;
+using KeeperData.Core.ApiClients.DataBridgeApi.Configuration;
+using KeeperData.Core.ApiClients.DataBridgeApi.Contracts;
+using KeeperData.Core.Attributes;
+using KeeperData.Core.Messaging.Contracts.V1.Sam;
+using KeeperData.Core.Messaging.MessagePublishers;
+using KeeperData.Core.Messaging.MessagePublishers.Clients;
+using KeeperData.Core.Providers;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.Logging;
+
+namespace KeeperData.Application.Orchestration.ChangeScanning.Sam.Daily.Steps;
+
+[StepOrder(5)]
+public class SamCommonLandDailyScanStep(
+    IDataBridgeClient dataBridgeClient,
+    IMessagePublisher<IntakeEventsQueueClient> intakeMessagePublisher,
+    DataBridgeScanConfiguration dataBridgeScanConfiguration,
+    IDelayProvider delayProvider,
+    IConfiguration configuration,
+    ILogger<SamCommonLandDailyScanStep> logger)
+    : DailyScanStepBase<SamDailyScanContext, SamScanCommonLandIdentifier>(dataBridgeClient, intakeMessagePublisher, dataBridgeScanConfiguration,
+        delayProvider, configuration, logger)
+{
+    private const string SelectFields = "COMMON_CPH";
+    private const string OrderBy = "COMMON_CPH asc";
+
+    protected override bool IsEntityEnabled()
+        => Configuration.GetValue<bool>("DataBridgeCollectionFlags:SamCommonLandsEnabled");
+
+    protected override EntityScanContext GetScanContext(SamDailyScanContext context)
+        => context.CommonLands;
+
+    protected override async Task<DataBridgeResponse<SamScanCommonLandIdentifier>?> QueryDataAsync(
+        SamDailyScanContext context,
+        CancellationToken cancellationToken)
+        => await DataBridgeClient.GetSamCommonLandsAsync<SamScanCommonLandIdentifier>(
+            context.CommonLands.CurrentTop,
+            context.CommonLands.CurrentSkip,
+            SelectFields,
+            context.UpdatedSinceDateTime,
+            OrderBy,
+            cancellationToken);
+
+    protected override async Task PublishMessagesAsync(
+        DataBridgeResponse<SamScanCommonLandIdentifier> queryResponse,
+        CancellationToken cancellationToken)
+    {
+        var identifiers = queryResponse.Data
+            .Select(x => x.COMMON_CPH)
+            .Where(x => !string.IsNullOrWhiteSpace(x))
+            .Distinct()
+            .ToList();
+
+        foreach (var id in identifiers)
+        {
+            var message = new SamUpdateHoldingMessage { Id = Guid.NewGuid(), Identifier = id };
+
+            await IntakeMessagePublisher.PublishAsync(message, cancellationToken);
+        }
+    }
+}

src/KeeperData.Application/Orchestration/Imports/Sam/Holdings/SamHoldingImportContext.cs

@@ -15,6 +15,7 @@ public class SamHoldingImportContext
     public List<SamHerd> RawHerds { get; set; } = [];
     public List<SamCphHolder> RawHolders { get; set; } = [];
     public List<SamParty> RawParties { get; set; } = [];
+    public List<SamCommonLand> RawCommonLandsByCommonCph { get; set; } = [];
 
     public List<SamHoldingDocument> SilverHoldings { get; set; } = [];
     public List<SamPartyDocument> SilverParties { get; set; } = [];

src/KeeperData.Application/Orchestration/Imports/Sam/Holdings/Steps/SamHoldingImportAggregationStep.cs

@@ -18,18 +18,22 @@ protected override async Task ExecuteCoreAsync(SamHoldingImportContext context,
         var getHoldingsTask = _dataBridgeClient.GetSamHoldingsAsync(context.Cph, cancellationToken);
         var getHoldersTask = _dataBridgeClient.GetSamHoldersByCphAsync(context.Cph, cancellationToken);
         var getHerdsTask = _dataBridgeClient.GetSamHerdsAsync(context.Cph, cancellationToken);
+        var getCommonLandsByCommonCphTask = _dataBridgeClient.GetSamCommonLandsByCommonCphAsync(context.Cph, cancellationToken);
 
         await Task.WhenAll(
             getHoldingsTask,
             getHoldersTask,
-            getHerdsTask);
+            getHerdsTask,
+            getCommonLandsByCommonCphTask);
 
         context.RawHoldings = getHoldingsTask.Result;
 
         context.RawHerds = getHerdsTask.Result;
 
         context.RawHolders = getHoldersTask.Result;
 
+        context.RawCommonLandsByCommonCph = getCommonLandsByCommonCphTask.Result;
+
         var parties = await GetSamPartiesAsync(context, cancellationToken);
         context.RawParties = SamPartyMapper.AggregatePartyAndHolder(parties, context.RawHolders);
     }

src/KeeperData.Application/Orchestration/Imports/Sam/Holdings/Steps/SamHoldingImportGoldMappingStep.cs

@@ -1,6 +1,7 @@
 using KeeperData.Application.Orchestration.Imports.Sam.Mappings;
 using KeeperData.Core.Attributes;
 using KeeperData.Core.Documents;
+using KeeperData.Core.Documents.Silver;
 using KeeperData.Core.Domain.Enums;
 using KeeperData.Core.Extensions;
 using KeeperData.Core.Repositories;
@@ -68,6 +69,8 @@ protected override async Task ExecuteCoreAsync(SamHoldingImportContext context,
                 siteTypeDerivedCodeLookupService,
                 cancellationToken);
 
+            EnrichWithCommonLandData(context.GoldSite, representative);
+
             context.GoldSitePartyRoles = SitePartyRoleMapper.ToGold(
                 context.GoldParties,
                 context.GoldSiteGroupMarks,
@@ -79,4 +82,31 @@ protected override async Task ExecuteCoreAsync(SamHoldingImportContext context,
                 context.GoldSite);
         }
     }
+
+    private static void EnrichWithCommonLandData(SiteDocument? goldSite, SamHoldingDocument representative)
+    {
+        if (goldSite == null) return;
+
+        goldSite.LocalAuthorityName = representative.LocalAuthorityName;
+
+        goldSite.AssociatedMainHoldings = representative.AssociatedMainHoldings
+            .Select(r => new AssociatedHoldingDocument
+            {
+                HoldingIdentifier = r.HoldingIdentifier,
+                ContiguousFlag = r.ContiguousFlag,
+                StartDate = r.StartDate,
+                EndDate = r.EndDate
+            })
+            .ToList();
+
+        goldSite.AssociatedCommonLands = representative.AssociatedCommonLands
+            .Select(r => new AssociatedHoldingDocument
+            {
+                HoldingIdentifier = r.HoldingIdentifier,
+                ContiguousFlag = r.ContiguousFlag,
+                StartDate = r.StartDate,
+                EndDate = r.EndDate
+            })
+            .ToList();
+    }
 }

src/KeeperData.Application/Orchestration/Imports/Sam/Holdings/Steps/SamHoldingImportSilverMappingStep.cs

@@ -25,6 +25,12 @@ protected override async Task ExecuteCoreAsync(SamHoldingImportContext context,
             countryIdentifierLookupService.FindAsync,
             cancellationToken);
 
+        var commonLandHoldings = SamCommonLandMapper.ToSilver(context.RawCommonLandsByCommonCph);
+        if (commonLandHoldings.Count > 0)
+        {
+            context.SilverHoldings.AddRange(commonLandHoldings);
+        }
+
         context.SilverParties = [
             .. await SamPartyMapper.ToSilver(
                 context.Cph,

src/KeeperData.Application/Orchestration/Imports/Sam/Mappings/SamCommonLandMapper.cs

@@ -0,0 +1,125 @@
+using KeeperData.Core.ApiClients.DataBridgeApi.Contracts;
+using KeeperData.Core.Documents.Silver;
+using KeeperData.Core.Domain.Sites.Formatters;
+using System.Globalization;
+
+namespace KeeperData.Application.Orchestration.Imports.Sam.Mappings;
+
+public static class SamCommonLandMapper
+{
+    private const string CommonLandSiteTypeCode = "CL";
+
+    public static List<SamHoldingDocument> ToSilver(List<SamCommonLand> rawCommonLands)
+    {
+        if (rawCommonLands == null || rawCommonLands.Count == 0)
+            return [];
+
+        var definitionRecords = rawCommonLands
+            .Where(r => r.IsDefinitionRecord && !string.IsNullOrWhiteSpace(r.COMMON_CPH))
+            .GroupBy(r => r.COMMON_CPH)
+            .ToList();
+
+        var relationshipRecords = rawCommonLands
+            .Where(r => r.IsRelationshipRecord)
+            .ToList();
+
+        var result = new List<SamHoldingDocument>();
+
+        foreach (var group in definitionRecords)
+        {
+            var representative = group.OrderByDescending(r => r.UpdatedAtUtc).First();
+            var commonCph = group.Key;
+
+            var associatedMainHoldings = relationshipRecords
+                .Where(r => r.COMMON_CPH == commonCph)
+                .Select(r => new AssociatedHoldingRelationship
+                {
+                    HoldingIdentifier = r.MAIN_CPH,
+                    ContiguousFlag = string.Equals(r.CONTIGUOUS_COMMON, "Yes", StringComparison.OrdinalIgnoreCase),
+                    StartDate = NormaliseDate(r.START_DATE),
+                    EndDate = NormaliseDate(r.END_DATE)
+                })
+                .ToList();
+
+            var holding = new SamHoldingDocument
+            {
+                LastUpdatedBatchId = representative.BATCH_ID,
+                CreatedDate = representative.CreatedAtUtc ?? DateTime.UtcNow,
+                LastUpdatedDate = representative.UpdatedAtUtc ?? DateTime.UtcNow,
+                Deleted = representative.IsDeleted ?? false,
+
+                CountyParishHoldingNumber = commonCph,
+                CphTypeIdentifier = string.Empty,
+                LocationName = UnwrapPlaceholder(representative.PREMISES_NAME),
+
+                HoldingStartDate = default,
+                HoldingEndDate = null,
+                HoldingStatus = HoldingStatusFormatters.FormatHoldingStatus(representative.IsDeleted ?? false),
+
+                SiteTypeCode = CommonLandSiteTypeCode,
+
+                LocalAuthorityName = representative.LOCAL_AUTH_NAME,
+                AssociatedMainHoldings = associatedMainHoldings,
+
+                Location = new LocationDocument
+                {
+                    IdentifierId = Guid.NewGuid().ToString(),
+                    Easting = ParseNullableDouble(representative.EASTING),
+                    Northing = ParseNullableDouble(representative.NORTHING),
+                    Address = new AddressDocument
+                    {
+                        IdentifierId = Guid.NewGuid().ToString(),
+                        AddressLine = representative.ADDRESS_LINE_1,
+                        AddressLocality = representative.ADDRESS_LINE_2,
+                        AddressStreet = representative.ADDRESS_LINE_3,
+                        AddressPostCode = representative.POSTCODE,
+                        CountryCode = representative.COUNTRY
+                    }
+                }
+            };
+
+            result.Add(holding);
+        }
+
+        return result;
+    }
+
+    public static List<AssociatedHoldingRelationship> ToAssociatedCommonLands(List<SamCommonLand>? relationshipRecords)
+    {
+        if (relationshipRecords == null || relationshipRecords.Count == 0)
+            return [];
+
+        return relationshipRecords
+            .Where(r => r.IsRelationshipRecord && !string.IsNullOrWhiteSpace(r.COMMON_CPH))
+            .Select(r => new AssociatedHoldingRelationship
+            {
+                HoldingIdentifier = r.COMMON_CPH,
+                ContiguousFlag = string.Equals(r.CONTIGUOUS_COMMON, "Yes", StringComparison.OrdinalIgnoreCase),
+                StartDate = NormaliseDate(r.START_DATE),
+                EndDate = NormaliseDate(r.END_DATE)
+            })
+            .ToList();
+    }
+
+    private static string? NormaliseDate(string? date)
+    {
+        if (string.IsNullOrWhiteSpace(date))
+            return null;
+
+        if (DateTime.TryParse(date, CultureInfo.InvariantCulture, DateTimeStyles.None, out var parsed))
+        {
+            if (parsed.Year >= 2999)
+                return null;
+
+            return parsed.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture);
+        }
+
+        return date;
+    }
+
+    private static string? UnwrapPlaceholder(string? value)
+        => string.IsNullOrWhiteSpace(value) || value == "-" ? null : value;
+
+    private static double? ParseNullableDouble(string? value)
+        => double.TryParse(value, out var result) ? result : null;
+}

src/KeeperData.Core/ApiClients/DataBridgeApi/Contracts/SamCommonLand.cs

@@ -0,0 +1,61 @@
+using KeeperData.Core.ApiClients.DataBridgeApi.Converters;
+using System.Text.Json.Serialization;
+
+namespace KeeperData.Core.ApiClients.DataBridgeApi.Contracts;
+
+public class SamCommonLand : BronzeBase
+{
+    [JsonPropertyName("COMMON_LAND_PREMISE_ID")]
+    public string? COMMON_LAND_PREMISE_ID { get; set; }
+
+    [JsonPropertyName("MAIN_CPH")]
+    public string MAIN_CPH { get; set; } = string.Empty;
+
+    [JsonPropertyName("COMMON_CPH")]
+    public string COMMON_CPH { get; set; } = string.Empty;
+
+    [JsonPropertyName("BUSINESS_USAGE")]
+    public string? BUSINESS_USAGE { get; set; }
+
+    [JsonPropertyName("PREMISES_NAME")]
+    public string? PREMISES_NAME { get; set; }
+
+    [JsonPropertyName("ADDRESS_LINE_1")]
+    public string? ADDRESS_LINE_1 { get; set; }
+
+    [JsonPropertyName("ADDRESS_LINE_2")]
+    public string? ADDRESS_LINE_2 { get; set; }
+
+    [JsonPropertyName("ADDRESS_LINE_3")]
+    public string? ADDRESS_LINE_3 { get; set; }
+
+    [JsonPropertyName("LOCAL_AUTH_NAME")]
+    public string? LOCAL_AUTH_NAME { get; set; }
+
+    [JsonPropertyName("COUNTRY")]
+    public string? COUNTRY { get; set; }
+
+    [JsonPropertyName("POSTCODE")]
+    public string? POSTCODE { get; set; }
+
+    [JsonPropertyName("EASTING")]
+    public string? EASTING { get; set; }
+
+    [JsonPropertyName("NORTHING")]
+    public string? NORTHING { get; set; }
+
+    [JsonPropertyName("LINK_ID")]
+    public string? LINK_ID { get; set; }
+
+    [JsonPropertyName("CONTIGUOUS_COMMON")]
+    public string? CONTIGUOUS_COMMON { get; set; }
+
+    [JsonPropertyName("START_DATE")]
+    public string? START_DATE { get; set; }
+
+    [JsonPropertyName("END_DATE")]
+    public string? END_DATE { get; set; }
+
+    public bool IsDefinitionRecord => MAIN_CPH == "-" || string.IsNullOrWhiteSpace(MAIN_CPH);
+    public bool IsRelationshipRecord => !IsDefinitionRecord;
+}

src/KeeperData.Core/ApiClients/DataBridgeApi/Contracts/SamScanCommonLandIdentifier.cs

@@ -0,0 +1,9 @@
+using System.Text.Json.Serialization;
+
+namespace KeeperData.Core.ApiClients.DataBridgeApi.Contracts;
+
+public class SamScanCommonLandIdentifier
+{
+    [JsonPropertyName("COMMON_CPH")]
+    public string COMMON_CPH { get; set; } = string.Empty;
+}