Skip to content

Commit c6cb6ee

Browse files
samgibsonmojdependabot[bot]
samgibsonmoj
and
dependabot[bot]
authored
merge develop -> main (#71)
* deps: Bump the all-dependencies group with 9 updates (#62) Bumps Aspire.Hosting.AppHost from 13.0.1 to 13.0.2 Bumps Aspire.Hosting.RabbitMQ from 13.0.1 to 13.0.2 Bumps Aspire.Hosting.Redis from 13.0.1 to 13.0.2 Bumps Aspire.Hosting.SqlServer from 13.0.1 to 13.0.2 Bumps Aspire.RabbitMQ.Client from 13.0.1 to 13.0.2 Bumps AWSSDK.Extensions.NETCore.Setup from 4.0.3.14 to 4.0.3.15 Bumps AWSSDK.S3 from 4.0.13.1 to 4.0.14.1 Bumps AWSSDK.SecurityToken from 4.0.5.1 to 4.0.5.2 Bumps Sentry.Serilog from 6.0.0-preview.2-prerelease to 6.0.0-rc.2-prerelease --- updated-dependencies: - dependency-name: Aspire.Hosting.AppHost dependency-version: 13.0.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: Aspire.Hosting.RabbitMQ dependency-version: 13.0.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: Aspire.Hosting.Redis dependency-version: 13.0.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: Aspire.Hosting.SqlServer dependency-version: 13.0.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: Aspire.RabbitMQ.Client dependency-version: 13.0.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: AWSSDK.Extensions.NETCore.Setup dependency-version: 4.0.3.15 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: AWSSDK.S3 dependency-version: 4.0.14.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: AWSSDK.SecurityToken dependency-version: 4.0.5.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: Sentry.Serilog dependency-version: 6.0.0-rc.2-prerelease dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Enforce dedicated scopes for Visualiser (#69) Enforces dedicated visualiser scopes. Updates the API to use dedicated scopes for the visualiser application. This change improves security by isolating the permissions required for visualiser functionality from the core data management system (DMS). * Update readme (#68) * Updates README to more accurately reflect architecture * Updates Entra app registration instructions Clarifies that Implicit grant and hybrid flows do not need to be configured during Entra app registration. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1 parent 4704e3b commit c6cb6ee

7 files changed

Lines changed: 41 additions & 24 deletions

File tree

‎Directory.Packages.props‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3,18 +3,18 @@
33
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
44
</PropertyGroup>
55
<ItemGroup>
6-
<PackageVersion Include="Aspire.Hosting.AppHost" Version="13.0.1" />
7-
<PackageVersion Include="Aspire.Hosting.RabbitMQ" Version="13.0.1" />
8-
<PackageVersion Include="Aspire.Hosting.Redis" Version="13.0.1" />
9-
<PackageVersion Include="Aspire.Hosting.SqlServer" Version="13.0.1" />
10-
<PackageVersion Include="Aspire.RabbitMQ.Client" Version="13.0.1" />
11-
<PackageVersion Include="AWSSDK.Extensions.NETCore.Setup" Version="4.0.3.14" />
12-
<PackageVersion Include="AWSSDK.SecurityToken" Version="4.0.5.1" />
6+
<PackageVersion Include="Aspire.Hosting.AppHost" Version="13.0.2" />
7+
<PackageVersion Include="Aspire.Hosting.RabbitMQ" Version="13.0.2" />
8+
<PackageVersion Include="Aspire.Hosting.Redis" Version="13.0.2" />
9+
<PackageVersion Include="Aspire.Hosting.SqlServer" Version="13.0.2" />
10+
<PackageVersion Include="Aspire.RabbitMQ.Client" Version="13.0.2" />
11+
<PackageVersion Include="AWSSDK.Extensions.NETCore.Setup" Version="4.0.3.15" />
12+
<PackageVersion Include="AWSSDK.SecurityToken" Version="4.0.5.2" />
1313
<PackageVersion Include="CommunityToolkit.Aspire.Hosting.Minio" Version="13.0.0" />
1414
<PackageVersion Include="CommunityToolkit.Aspire.Hosting.SqlDatabaseProjects" Version="13.0.0" />
1515
<PackageVersion Include="Autofac" Version="9.0.0" />
1616
<PackageVersion Include="Autofac.Extensions.DependencyInjection" Version="10.0.0" />
17-
<PackageVersion Include="AWSSDK.S3" Version="4.0.13.1" />
17+
<PackageVersion Include="AWSSDK.S3" Version="4.0.14.1" />
1818
<PackageVersion Include="CommunityToolkit.Aspire.Minio.Client" Version="13.0.0" />
1919
<PackageVersion Include="coverlet.collector" Version="6.0.4">
2020
<PrivateAssets>all</PrivateAssets>
@@ -59,7 +59,7 @@
5959
<PackageVersion Include="Rebus" Version="8.9.0" />
6060
<PackageVersion Include="Rebus.RabbitMq" Version="10.1.0" />
6161
<PackageVersion Include="Rebus.ServiceProvider" Version="10.7.0" />
62-
<PackageVersion Include="Sentry.Serilog" Version="6.0.0-preview.2-prerelease" />
62+
<PackageVersion Include="Sentry.Serilog" Version="6.0.0-rc.2-prerelease" />
6363
<PackageVersion Include="Serilog.Extensions.Hosting" Version="10.0.0" />
6464
<PackageVersion Include="Serilog.Settings.Configuration" Version="10.0.0" />
6565
<PackageVersion Include="Serilog.Sinks.Console" Version="6.1.1" />

‎README.md‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -11,19 +11,24 @@ HMPPS CFO DMS
1111
HMPPS Creating Future Opportunities (CFO) - Data Management System (DMS). It is intended for internal use only and is used to process PNOMIS and NDelius offender data to supply CATS (Case Assessment and Tracking System - also used by HMPPS CFO) with accurate offender movements and updates.
1212

1313
## Architecture
14-
CFO DMS is built as a microservices architecture using .NET Aspire for orchestration. Data flows through the following pipeline:
14+
CFO DMS is built as a distributed microservices architecture. Data flows through the following pipeline:
1515

16-
**File Ingestion → Parsing/Cleaning → Staging → Import → Running Picture → Blocking/Matching → Clustering**
16+
**File Ingestion → Parsing/Cleaning → Staging → Import → Running Picture → Blocking/Matching → Clustering → Data Consumption**
1717

18-
1. **FileSync** monitors MinIO/S3/FileSystem storage and syncs incoming files
19-
2. **Parsers/Cleaners** (Offloc, Delius) transform raw PNOMIS and NDelius files into structured records in staging databases
20-
3. **Import** validates and migrates data from staging to running picture databases
21-
4. **Matching Engine** identifies and links related offender records across systems
22-
5. **Cluster database** maintains grouped offender data
23-
6. **API** exposes the processed data via REST endpoints for downstream consumers (e.g., CATS)
24-
7. **Visualiser** provides a web UI for exploring and visualising relationships between offender data
18+
### Pipeline Applications
19+
1. **File Ingestion** - [**FileSync**](src/FileSync) monitors MinIO/S3/FileSystem storage and syncs incoming files
20+
2. **Parsing/Cleaning** - [**Offloc.Parser**](src/Offloc.Parser), [**Offloc.Cleaner**](src/Offloc.Cleaner), [**Delius.Parser**](src/Delius.Parser) transform raw p-NOMIS and nDelius files into structured records
21+
3. **Staging/Import/Running Picture** - [**Import**](src/Import) validates and migrates data from staging to running picture databases
22+
4. **Blocking/Matching** - [**Blocking**](src/Blocking) generates candidate record pairs, [**Matching.Engine**](src/Matching.Engine) identifies and links related offender records across systems
23+
5. **Clustering** - [**Matching.Engine**](src/Matching.Engine) groups related records into clusters representing unique individuals
24+
6. **Data Consumption** - [**API**](src/API) exposes the processed data via REST endpoints for downstream consumers (e.g., CATS), [**Visualiser**](src/Visualiser) provides a web UI for exploring and visualising relationships between offender data
2525

26-
Supporting services include **DbInteractions** (complex database operations), **Blocking** (matching rules), **Cleanup** (data maintenance), and **Logging**. Services communicate asynchronously via RabbitMQ message queues.
26+
### Supporting Applications
27+
- [**Cleanup**](src/Cleanup) - Performs data maintenance tasks
28+
- [**DbInteractions**](src/DbInteractions) handles complex database operations
29+
- [**Logging**](src/Logging) - Centralised logging service
30+
31+
Services communicate asynchronously via RabbitMQ message queues. See the Message Flow Diagram below for detailed service interactions.
2732

2833
# Development Setup and Execution Guide
2934

‎src/API/Endpoints/VisualisationEndpoints.cs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,9 @@ public static IEndpointRouteBuilder RegisterVisualisationEndpoints(this IEndpoin
2222

2323
group.MapPost("/Save", SaveNetworkAsync)
2424
.ProducesProblem(StatusCodes.Status500InternalServerError)
25-
.RequireAuthorization("write");
25+
.RequireAuthorization("visualisation-write");
2626

27-
group.RequireAuthorization("read");
27+
group.RequireAuthorization("visualisation-read");
2828

2929
return routes;
3030
}

‎src/API/Program.cs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,17 @@
7373
{
7474
options.AddPolicy("read", policy => policy.RequireScope("dms.read"));
7575
options.AddPolicy("write", policy => policy.RequireScope("dms.write"));
76+
77+
options.AddPolicy("visualisation-read", policy =>
78+
policy.RequireAuthenticatedUser()
79+
.RequireScope("visualiser.read")
80+
.AddAuthenticationSchemes(JwtBearerDefaults.AuthenticationScheme));
81+
82+
options.AddPolicy("visualisation-write", policy =>
83+
policy.RequireAuthenticatedUser()
84+
.RequireScope("visualiser.write")
85+
.AddAuthenticationSchemes(JwtBearerDefaults.AuthenticationScheme));
86+
7687
options.FallbackPolicy = options.DefaultPolicy;
7788
});
7889

‎src/Visualiser/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
1. New app registration in Entra
44
- Supported account types: Single tenant
55
- Redirect URI: Platform = web, URI = https://localhost:7123/signin-oidc
6+
- NOTE: You do not need to configure Implicit grant and hybrid flows
67
2. Register permissions under "API Permissions" section. Configure scopes:
78
- `email`
89
- `openid`

‎src/Visualiser/appsettings.Development.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@
1111
},
1212
"API": {
1313
"Scopes": [
14-
"api://916ace49-a3db-4b11-84c5-6c4bd20260ef/dms.read",
15-
"api://916ace49-a3db-4b11-84c5-6c4bd20260ef/dms.write"
14+
"api://916ace49-a3db-4b11-84c5-6c4bd20260ef/visualiser.read",
15+
"api://916ace49-a3db-4b11-84c5-6c4bd20260ef/visualiser.write"
1616
]
1717
}
1818
}

‎src/Visualiser/appsettings.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ For more info see https://aka.ms/dotnet-template-ms-identity-platform
1717
"API": {
1818
"BaseUrl": "https://localhost:7013",
1919
"Scopes": [
20-
// E.g. "api://{api_client_id}/dms.read" and "api://{api_client_id}/dms.write"
20+
// E.g. "api://{api_client_id}/visualiser.read" and "api://{api_client_id}/visualiser.write"
2121
]
2222
},
2323
"Serilog": {

0 commit comments

Comments
 (0)