Skip to content

Commit 041c820

Browse files
samgibsonmojsamgibsonmoj
authored
develop -> main (#18)
* File Sync service (#17) * WIP: S3 File Sync * Remove FileStorage project. Move interface into environment setup * Remove redundant if statement * Enforce patterns for file searches Previously, both the Offloc Cleaner and Delius Parser would attempt to process any files in the input directories. * Removes redundant docker file processing service Also tightens some file path concatenation * Cleanup + implementation of S3FileSource service * Removes Parallel parsing (and other related redundant code) + hooks in FileSync * Fix for Delius post-parser encoding and line endings * MinIO integration * Bump packages (excludes Aspire) Issues remain with bumping Aspire to .NET 10 with the community toolkit packages * Enables S3 file synchronization and processing Refactors the file synchronization process to support S3 as a file source. Introduces configurable file sources for Delius and Offloc files (FileSystem, MinIO, S3). Adds database interaction to track file processing status and associate Offloc files with archive names. Enhances file processing workflow with new database messaging and processing start events. Updates stored procedures to handle file processing status and archive associations. * Configures dotnet environment * Configures and initialises MinIO bucket * Clarifies Visualiser setup requirement * Fixes S3 file source path resolution * Orders files by date + handles ddMMyyyy and dMMyyyy formats * Ensures `ArchiveFileName` is nullable in queries * Adds AWS Security Token package * Enables pre-kickoff tasks execution * Enable periodic file sync * Adds safe guard for file processing in Delius and Offloc * Adds Windows x64 runtime identifier for publishing * Removes extracted offloc file from input directory if already processed * Fixes issue with change tracker displaying node id instead of upci/group (#19) * Adds README to Matching.Core project (#21) Adds a README file to the Matching.Core project, providing Also, it disables package generation on build. * Upgrade aspire (#20) * Updates Aspire dependencies to v13 * Configures parameters for various services * Updates CommunityToolkit packages to stable version * Removes known issue from README (#22) Removes the known issue section regarding potential SQL project startup timing issues, as it was fixed in CommunityToolkit's v13. * Guarding for dates (#23) * Overwrite files when extracting archive * Adds logging for unavailable files * Clears input directories before download * Adds file date guarding Implements logic to prevent processing of files older than the last processed files. * Prevents concurrent file processing Adds a semaphore to prevent concurrent executions of the file processing logic. * Adds additional logging to file sync process * Dependabot & package cleanup (#24) * Removes unused package versions * Bump Rebus * Remove unused Visual Studio packages * Adds Dependabot configuration file
1 parent 6a36c42 commit 041c820

116 files changed

Lines changed: 1873 additions & 1030 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/dependabot.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: "nuget"
4+
directory: "/"
5+
schedule:
6+
interval: "weekly"
7+
day: "sunday"
8+
time: "04:00"
9+
target-branch: "develop"
10+
commit-message:
11+
prefix: "deps"
12+
open-pull-requests-limit: 1
13+
groups:
14+
all-dependencies:
15+
patterns:
16+
- "*"

‎.github/workflows/build-artifacts.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ jobs:
2323
run: dotnet restore
2424

2525
- name: Publish
26-
run: dotnet publish -c Release --artifacts-path ./output
26+
run: dotnet publish -c Release -r win-x64 --artifacts-path ./output
2727

2828
- name: Upload build artifacts
2929
uses: actions/upload-artifact@v4

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -280,6 +280,9 @@ LocalFiles/
280280
*.dat
281281
*~lock.*
282282
*.pubxml
283+
cfoextract*
284+
C_NOMIS_OFFENDER*
285+
**/DMS_STAGING/
283286

284287
# Mac
285288
.DS_STORE

‎Directory.Packages.props‎

Lines changed: 31 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -3,86 +3,69 @@
33
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
44
</PropertyGroup>
55
<ItemGroup>
6-
<PackageVersion Include="Aspire.Hosting.AppHost" Version="9.5.2" />
7-
<PackageVersion Include="Aspire.Hosting.RabbitMQ" Version="9.5.2" />
8-
<PackageVersion Include="Aspire.Hosting.Redis" Version="9.5.2" />
9-
<PackageVersion Include="Aspire.Hosting.SqlServer" Version="9.5.2" />
10-
<PackageVersion Include="Aspire.Microsoft.EntityFrameworkCore.SqlServer" Version="9.5.2" />
11-
<PackageVersion Include="Aspire.RabbitMQ.Client" Version="9.5.2" />
12-
<PackageVersion Include="CommunityToolkit.Aspire.Hosting.SqlDatabaseProjects" Version="9.8.1-beta.424" />
6+
<PackageVersion Include="Aspire.Hosting.AppHost" Version="13.0.0" />
7+
<PackageVersion Include="Aspire.Hosting.RabbitMQ" Version="13.0.0" />
8+
<PackageVersion Include="Aspire.Hosting.Redis" Version="13.0.0" />
9+
<PackageVersion Include="Aspire.Hosting.SqlServer" Version="13.0.0" />
10+
<PackageVersion Include="Aspire.RabbitMQ.Client" Version="13.0.0" />
11+
<PackageVersion Include="AWSSDK.Extensions.NETCore.Setup" Version="4.0.3.11" />
12+
<PackageVersion Include="AWSSDK.SecurityToken" Version="4.0.4.1" />
13+
<PackageVersion Include="CommunityToolkit.Aspire.Hosting.Minio" Version="13.0.0" />
14+
<PackageVersion Include="CommunityToolkit.Aspire.Hosting.SqlDatabaseProjects" Version="13.0.0" />
1315
<PackageVersion Include="Autofac" Version="8.4.0" />
1416
<PackageVersion Include="Autofac.Extensions.DependencyInjection" Version="10.0.0" />
15-
<PackageVersion Include="AWSSDK.S3" Version="4.0.9.1" />
16-
<PackageVersion Include="AutoMapper" Version="12.0.1" />
17-
<PackageVersion Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="12.0.1" />
17+
<PackageVersion Include="AWSSDK.S3" Version="4.0.11.1" />
18+
<PackageVersion Include="CommunityToolkit.Aspire.Minio.Client" Version="13.0.0" />
1819
<PackageVersion Include="coverlet.collector" Version="6.0.4">
1920
<PrivateAssets>all</PrivateAssets>
2021
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
2122
</PackageVersion>
2223
<PackageVersion Include="Dapper" Version="2.1.66" />
23-
<PackageVersion Include="DevExpress.Data" Version="23.2.3" />
2424
<PackageVersion Include="DocumentFormat.OpenXml" Version="3.3.0" />
2525
<PackageVersion Include="DotNetEnv" Version="3.1.1" />
26-
<PackageVersion Include="FluentValidation" Version="11.8.0" />
27-
<PackageVersion Include="HtmlAgilityPack" Version="1.11.57" />
28-
<PackageVersion Include="Microsoft.AspNet.WebApi.Core" Version="5.3.0" />
29-
<PackageVersion Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.0-rc.2.25502.107" NoWarn="NU1605" />
30-
<PackageVersion Include="Microsoft.AspNetCore.Authentication.OpenIdConnect" Version="10.0.0-rc.2.25502.107" NoWarn="NU1605" />
31-
<PackageVersion Include="Microsoft.AspNetCore.Blazor" Version="9.0.10" />
32-
<PackageVersion Include="Microsoft.AspNetCore.Components" Version="10.0.0-rc.2.25502.107" />
33-
<PackageVersion Include="Microsoft.AspNetCore.DataProtection" Version="8.0.0" />
34-
<PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="10.0.0-rc.2.25502.107" />
26+
<PackageVersion Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.0" />
27+
<PackageVersion Include="Microsoft.AspNetCore.Authentication.OpenIdConnect" Version="10.0.0" />
28+
<PackageVersion Include="Microsoft.AspNetCore.Components" Version="10.0.0" />
29+
<PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="10.0.0" />
3530
<PackageVersion Include="Microsoft.Data.SqlClient" Version="7.0.0-preview2.25289.6" />
36-
<PackageVersion Include="Microsoft.EntityFrameworkCore" Version="10.0.0-rc.2.25502.107" />
37-
<PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.0-rc.2.25502.107">
31+
<PackageVersion Include="Microsoft.EntityFrameworkCore" Version="10.0.0" />
32+
<PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.0">
3833
<PrivateAssets>all</PrivateAssets>
3934
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
4035
</PackageVersion>
41-
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.0-rc.2.25502.107" />
42-
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.0-rc.2.25502.107" />
43-
<PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="8.0.0" />
44-
<PackageVersion Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="10.0.0-rc.2.25502.107" />
45-
<PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="8.0.0" />
46-
<PackageVersion Include="Microsoft.Extensions.Configuration.UserSecrets" Version="10.0.0-rc.2.25502.107" />
47-
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.0-rc.2.25502.107" />
48-
<PackageVersion Include="Microsoft.Extensions.Hosting" Version="10.0.0-rc.2.25502.107" />
49-
<PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.0-rc.2.25502.107" />
50-
<PackageVersion Include="Microsoft.Extensions.Hosting.WindowsServices" Version="10.0.0-rc.2.25502.107" />
51-
<PackageVersion Include="Microsoft.Extensions.Http.Resilience" Version="9.10.0" />
52-
<PackageVersion Include="Microsoft.Extensions.ServiceDiscovery" Version="9.5.2" />
36+
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.0" />
37+
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.0" />
38+
<PackageVersion Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="10.0.0" />
39+
<PackageVersion Include="Microsoft.Extensions.Configuration.UserSecrets" Version="10.0.0" />
40+
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.0" />
41+
<PackageVersion Include="Microsoft.Extensions.Hosting" Version="10.0.0" />
42+
<PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.0" />
43+
<PackageVersion Include="Microsoft.Extensions.Hosting.WindowsServices" Version="10.0.0" />
44+
<PackageVersion Include="Microsoft.Extensions.Http.Resilience" Version="10.0.0" />
45+
<PackageVersion Include="Microsoft.Extensions.ServiceDiscovery" Version="10.0.0" />
5346
<PackageVersion Include="Microsoft.Identity.Web" Version="4.0.1" />
5447
<PackageVersion Include="Microsoft.Identity.Web.DownstreamApi" Version="4.0.1" />
5548
<PackageVersion Include="Microsoft.Identity.Web.UI" Version="4.0.1" />
56-
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="18.0.0" />
57-
<PackageVersion Include="Microsoft.PowerShell.SDK" Version="7.5.4" />
58-
<PackageVersion Include="Microsoft.SqlServer.Dac" Version="1.0.1" />
59-
<PackageVersion Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.23.0-preview.4" />
60-
<PackageVersion Include="Microsoft.VisualStudio.Shell.Framework" Version="17.14.40264" />
61-
<PackageVersion Include="MimeTypes" Version="2.4.1" />
62-
<PackageVersion Include="NSwag.AspNetCore" Version="14.6.1" />
49+
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />
50+
<PackageVersion Include="NSwag.AspNetCore" Version="14.6.2" />
6351
<PackageVersion Include="Newtonsoft.Json" Version="13.0.4" />
6452
<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.14.0-rc.1" />
6553
<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.14.0-rc.1" />
6654
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.13.0" />
6755
<PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.13.0" />
6856
<PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="1.13.0" />
6957
<PackageVersion Include="Phonix" Version="1.0.2" />
70-
<PackageVersion Include="Polly" Version="8.6.1" />
71-
<PackageVersion Include="Pythonnet" Version="3.0.3" />
7258
<PackageVersion Include="RabbitMQ.Client" Version="6.8.1" />
7359
<PackageVersion Include="Rebus" Version="8.9.0" />
7460
<PackageVersion Include="Rebus.RabbitMq" Version="10.1.0" />
75-
<PackageVersion Include="Rebus.ServiceProvider" Version="10.6.0" />
61+
<PackageVersion Include="Rebus.ServiceProvider" Version="10.7.0" />
7662
<PackageVersion Include="Sentry.Serilog" Version="6.0.0-preview.2-prerelease" />
7763
<PackageVersion Include="Serilog.Extensions.Hosting" Version="9.0.1-dev-02307" />
7864
<PackageVersion Include="Serilog.Settings.Configuration" Version="9.0.1-dev-02319" />
79-
<PackageVersion Include="Serilog.Sinks.Console" Version="6.0.1-dev-00953" />
65+
<PackageVersion Include="Serilog.Sinks.Console" Version="6.1.1" />
8066
<PackageVersion Include="Serilog.Sinks.File" Version="7.0.0" />
8167
<PackageVersion Include="SimMetrics.Net" Version="1.0.5" />
82-
<PackageVersion Include="Swashbuckle.AspNetCore" Version="6.5.0" />
8368
<PackageVersion Include="System.Data.SqlClient" Version="4.9.0" />
84-
<PackageVersion Include="System.Management.Automation" Version="7.6.0-preview.5" />
85-
<PackageVersion Include="WebDAVClient" Version="2.2.1" />
8669
<PackageVersion Include="xunit" Version="2.9.3" />
8770
<PackageVersion Include="xunit.runner.visualstudio" Version="3.1.5">
8871
<PrivateAssets>all</PrivateAssets>

‎README.md‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,7 @@ Any queries, please contact andrew.grocott@justice.gov.uk or visit our slack cha
1212
# Development Setup and Execution Guide
1313

1414
## Setup (development)
15-
1. From the **project root**, run the setup script. This will automatically create the required folder structure in your devices home directory:
16-
```sh
17-
./setup.sh
18-
```
19-
2. Configure secret(s) for applications in the *src* directory:
15+
1. To use the Visualiser app, you must configure secret(s) for applications in the *src* directory:
2016
- *Visualiser.csproj* → Manage User Secrets
2117
```json
2218
{
@@ -28,7 +24,3 @@ Any queries, please contact andrew.grocott@justice.gov.uk or visit our slack cha
2824
The recommended way to run and debug these apps is using .NET Aspire.
2925
- **Using Visual Studio Code**: open the project and press `F5`, selecting the *Default Configuration*.
3026
- **Using Visual Studio or other IDEs**: From the debug configuration dropdown, select `Aspire.AppHost` and start the application.
31-
32-
### Known issues
33-
1. Due to a [known issue](https://github.com/CommunityToolkit/Aspire/issues/942) in the Aspire Community Toolkit, some SQL projects may start too early.
34-
- You may need to re-run the project for database changes to apply correctly.

‎base.development.env‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,3 @@
11
DOTNET_ENVIRONMENT=Development
22

3-
OFFLOC_USERNAME=
4-
OFFLOC_PASSWORD=
5-
DELIUS_USERNAME=
6-
DELIUS_PASSWORD=
7-
Staging__DownloadOfflocFiles=false
8-
Staging__DownloadDeliusFiles=false
9-
10-
Parallel=false
113
RedundantOfflocFields=11,20,32,33,34,35,36,37,38,41,43,44,45,46,53,54,55,56,57,58,59,60,61,62,74,75,76,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,142,144

‎development.docker.env‎

Lines changed: 0 additions & 37 deletions
This file was deleted.

‎development.local.env‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,6 @@
22

33
DOTNET_ENVIRONMENT=Development
44

5-
# Core
6-
OFFLOC_USERNAME=
7-
OFFLOC_PASSWORD=
8-
DELIUS_USERNAME=
9-
DELIUS_PASSWORD=
10-
NEXTCLOUD_ADDRESS=
11-
Staging__DownloadOfflocFiles=false
12-
Staging__DownloadDeliusFiles=false
13-
14-
Parallel=false
155
RedundantOfflocFields= Age, Occupation Description, Check Hold Governor, Check Hold General (to be left blank), Check Hold Discipline (to be left blank), Check Hold Allocation, Check Hold Security, Check Hold Medical, Check Hold Parole, ACCT Status (F2052), Status Rank (to be left blank), Pending Transfers (Full Establishment Name), Received From, Vulnerable Prisoner Alert, Height (metres), Complexion, Hair Colour, Left Eye, Right Eye, Build, Facial Shape, Facial Hair, Physical Mark Head, Physical Mark Body, Rule 45/YOI Rule 49, ACCT (Self Harm) Status, ACCT (Self Harm) Start Date, Remark Type Allocation, Remarks Allocation, Remark Type Security, Remarks Security, Remark Type Medical, Remarks Medical, Remark Type Parole, Remarks Parole, Remark Type Discipline, Remarks Discipline, Remark Type General, Remarks General, Remark Type Reception, Remarks Reception, Remark Type Labour, Remarks Labour, Date Of First Movement, Diary Details
166

177
# Rabbit
@@ -62,8 +52,5 @@ Serilog__Enrich__1=WithMachineName
6252
Serilog__Enrich__2=WithProcessId
6353
Serilog__Enrich__3=WithThreadId
6454

65-
DMSFilesBasePath=~/DMS/
66-
DeliusInputMount=${DMSFilesBasePath}\Delius\Input
67-
DeliusOutputMount=${DMSFilesBasePath}\Delius\Output
68-
OfflocInputMount=${DMSFilesBasePath}\Offloc\Input
69-
OfflocOutputMount=${DMSFilesBasePath}\Offloc\Output
55+
# e.g. /app/
56+
#DMSFilesBasePath=~/DMS/

0 commit comments

Comments
 (0)