Skip to content

Commit 442f95c

Browse files
AndrewThienAndyRae
AndrewThien
and
AndyRae
authored
Add more C4 diagram and some tutorials (#25)
* add system architechture * add more content * add more content * update naming * fix egress c4 * merge what is 5st to introduction * update name * add tutorial pages and fix them * fix colors tags * add link * update linkl * fix links * Update app/introduction/page.mdx Co-authored-by: Andy Rae <1127507+AndyRae@users.noreply.github.com> * Update app/introduction/page.mdx Co-authored-by: Andy Rae <1127507+AndyRae@users.noreply.github.com> * Update app/introduction/page.mdx Co-authored-by: Andy Rae <1127507+AndyRae@users.noreply.github.com> * Update app/explanations/how-5s-tes-works/egress/page.mdx Co-authored-by: Andy Rae <1127507+AndyRae@users.noreply.github.com> * Update app/explanations/how-5s-tes-works/architecture/page.mdx Co-authored-by: Andy Rae <1127507+AndyRae@users.noreply.github.com> * fix link and add presi --------- Co-authored-by: Andy Rae <1127507+AndyRae@users.noreply.github.com>
1 parent 1a4e3a2 commit 442f95c

35 files changed

Lines changed: 645 additions & 117 deletions

File tree

‎.prettierignore‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
1-
app/connect/setup-tre/deployment-steps/page.mdx
2-
app/co-ordinate/setup-sub-layer/deployment-steps/page.mdx
3-
app/co-ordinate/entity-management/add-realm-admin/page.mdx
4-
app/co-ordinate/entity-management/assign-tre-user-to-project/page.mdx
5-
app/connect/controlling-tre-data-access/page.mdx
6-
app/connect/components/page.mdx
7-
app/co-ordinate/entity-management/add-keycloak-user/page.mdx
1+
app/guides/manage-tre/deployment-steps/page.mdx
2+
app/guides/manage-fed/deployment-steps/page.mdx
3+
app/guides/manage-fed/add-realm-admin/page.mdx
4+
app/guides/manage-fed/assign-tre-user-to-project/page.mdx
5+
app/guides/manage-tre/control-tre-data-access/page.mdx
6+
app/explanations/how-5s-tes-works/tre-agent/page.mdx
7+
app/co-ordinate/entity-management/add-keycloak-user/page.mdx
8+
app/tutorials/collecting-results/page.mdx

‎app/_meta.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ export default {
55
introduction: "Introduction",
66
explanations: "Explanations",
77
guides: "How-to Guides",
8+
tutorials: "Tutorials",
89
references: "References",
910
journeys: {
1011
// exclude routes until content published

‎app/explanations/_meta.js‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
export default {
2-
"what-is-five-safes-tes": "What is Five Safes TES?",
3-
"how-5st-works": "How Five Safes TES works",
42
personas: "Personas",
3+
"how-5s-tes-works": "How Five Safes TES works",
54
};

‎app/explanations/how-5s-tes-works/_meta.js‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
export default {
2+
architecture: "System Architecture",
3+
"submission-layer": "Submission Layer",
4+
"tre-agent": "TRE Agent",
5+
egress: "Egress",
6+
};

‎app/explanations/how-5s-tes-works/architecture/page.mdx‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
readingTime: PT10M
3+
audiences: everyone
4+
docType: explanation
5+
title: System Architecture
6+
---
7+
8+
import { Callout } from "nextra/components";
9+
import { DocMetadata } from "@/components/doc-metadata/DocMetadata";
10+
import Diagram from "@/components/c4_diagram";
11+
12+
<DocMetadata />
13+
14+
## Prerequisites
15+
16+
- Understanding of [Five Safes TES](/introduction).
17+
18+
## System Architecture
19+
20+
The Five Safes TES (Task Execution Service) is designed to handle sensitive data analysis within a secure and controlled environment. The architecture followed by a Fan-out pattern for the implementation and is organized into multiple layers and application components.
21+
22+
Aggregation is supported via external tools and after 5S-TES completes analysis tasks, users can run a dedicated aggregation tool to collect and summarise results.
23+
24+
{/* TODO: link to an example of aggregation */}
25+
26+
<figure>
27+
<Diagram
28+
alt_image="/images/likec4/deployments/parl.png"
29+
alt_text="Illustrative context diagram for Five Safes TES architecture. Researcher interacts with submission layer, behind which three TREs are shown, each of which interact with the submission layer separately."
30+
viewId="parl"
31+
/>
32+
<figcaption
33+
style={{ fontStyle: "italic", fontSize: "0.7em", marginTop: "8px" }}
34+
>
35+
Figure 1: Illustration of the context diagram for Five Safes TES
36+
architecture. Click for interactive mode.
37+
</figcaption>
38+
</figure>
39+
40+
## Components Architecture
41+
42+
The Five Safes TES is divided into multiple layers of process and multiple app stack as discussed below.
43+
44+
**The layers of the architecture are:**
45+
46+
- **Submission Layer**: Entrypoint for user requests, authentication, and authorisation.
47+
- **TRE Layer**: Core processing layer for data analysis.
48+
49+
**The Five Safes TES app stack includes:**
50+
51+
- **Submission App:** Handles user submissions.
52+
- **TRE Agent App:** Manages task execution within the TRE for data analytics.
53+
- **Egress App:** Checks and approve/reject the results of the analysis.
54+
55+
<figure>
56+
<Diagram
57+
alt_image="/images/likec4/logic.png"
58+
alt_text="Data flow through the Five Safe TES architecture."
59+
viewId="logic"
60+
/>
61+
<figcaption
62+
style={{ fontStyle: "italic", fontSize: "0.7em", marginTop: "8px" }}
63+
>
64+
Figure 2: App structure and data flow within Five Safe TES. Click for
65+
interactive mode, including data flow walkthrough.
66+
</figcaption>
67+
</figure>
68+
<Callout type="info">
69+
The data flow diagram above is interactive. Click twice on the diagram to
70+
enter interactive mode, and then click on the blue 'Start' button at the top
71+
of the diagram to enter the data flow walkthrough.
72+
</Callout>
73+
74+
## Operational Overview
75+
76+
The high level overview of working of the Five Safes TES is as follows:
77+
78+
<u style={{ display: 'block', marginTop: '20px' }}>
79+
<strong>Submission App</strong>
80+
</u>
81+
<div>
82+
83+
1. The process begins with the end user or researcher submitting a request for the desired analysis through the Trusted Research Environments. Then, authentication and authorisation checks are performed to ensure the user is assigned to the relevant project.
84+
85+
2. The request is then placed in a queue as a task for the corresponding nodes in the federated network and the researcher can track the stages of the analysis.
86+
87+
</div>
88+
89+
<u style={{ display: 'block', marginTop: '20px' }}>
90+
<strong>TRE Agent App</strong>
91+
</u>
92+
<div>
93+
1. The TRE Agent in the TRE Layer monitors the queue for new jobs (tasks). Once identified, TRE agent get the task and start the pre-processing of the task.
94+
95+
2. A tool called Camunda will dynamically inject the ephemeral credentials into the environment variables of the TES message, which Executor will use to connect to the database.
96+
97+
3. The standardised payload is sent to the Executor, then the analysis will be executed against TRE's restricted database.
98+
99+
4. The results of this analysis are then saved in storage at the TRE Layer.
100+
101+
</div>
102+
103+
<u style={{ display: 'block', marginTop: '20px' }}>
104+
<strong>Egress App</strong>
105+
</u>
106+
<div>
107+
1. After the analysis results are stored, the TRE notifies the data owner (a designated human reviewer) to approve the release of results.
108+
109+
2. The owner can now reviews the output and, if approved, authorises the egress.
110+
111+
3. Upon approval, the results are copied over to the Submission layer's storage, and finally, the researcher receives the analysis results.
112+
113+
</div>

‎app/explanations/how-5s-tes-works/egress/page.mdx‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
readingTime: PT10M
3+
audiences: [tre-operator, contributor]
4+
title: Egress
5+
---
6+
7+
import { Steps, Callout, Table } from "nextra/components";
8+
import { DocMetadata } from "@/components/doc-metadata/DocMetadata";
9+
import Diagram from "@/components/c4_diagram";
10+
11+
<DocMetadata />
12+
13+
## Prerequisites
14+
15+
- Understanding of [Five Safes TES System Architecture](/explanations/how-5st-works/architecture).
16+
17+
## Overview
18+
19+
The Egress application handles data egress requests. Once a task is finished executing, the results are submitted to it for approval where a Trusted Research Environment (TRE) administrator can check them. Once results are approved or rejected, Egress then sends an update to the [TRE Agent](/software/agent) API.
20+
21+
The code for Egress is open source, and can be found on [Github](https://github.com/SwanseaUniversityMedical/DARE-Control).
22+
23+
<figure>
24+
<Diagram
25+
alt_image="/images/likec4/egress/egress_app.png"
26+
alt_text="Egress architecture diagram"
27+
viewId="egress_app"
28+
/>
29+
<figcaption
30+
style={{ fontStyle: "italic", fontSize: "0.7em", marginTop: "8px" }}
31+
>
32+
Figure 1: Egress components. Click for interactive mode.
33+
</figcaption>
34+
</figure>
35+
36+
## Core Functionality
37+
38+
- Accepts submitted results from completed TRE tasks for review
39+
- Provides an approval and rejection workflow for TRE administrators
40+
- Tracks the status of egress requests and their outcomes
41+
- Notifies the TRE Agent when results are approved or rejected
42+
- Maintains an auditable record of egress decisions
43+
- Provides a API that can be used to integrate and build against
44+
45+
## Ideal Uses Cases
46+
47+
- Provide egress checking functionality to a Trusted Research Environment

app/explanations/how-5st-works/sub-layer-components/page.mdx renamed to app/explanations/how-5s-tes-works/submission-layer/page.mdx

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,27 @@ import Diagram from "@/components/c4_diagram";
1313

1414
## Prerequisites
1515

16-
- Understanding of [Five Safes TES Context C4 diagram](/context-c4-diagram).
16+
- Understanding of [Five Safes TES System Architecture](/explanations/how-5st-works/architecture).
17+
18+
## Overview
19+
20+
This is the entry point for a researcher. Approved researchers can submit tasks to a Project by interacting with the user interface, or through the API.
21+
22+
The code for Submission is open source, and can be found on [Github](https://github.com/SwanseaUniversityMedical/5s-Tes).
23+
24+
### Core Functionality
25+
26+
- Provides an API and user interface for researchers to submit tasks to the TREs.
27+
- Authenticates and authorises approved researchers
28+
- Forwards validated tasks to the Trusted Research Environment agent for execution
29+
- Tracks the status of submitted tasks
30+
- Provides a API that can be used to integrate and build against.
31+
32+
### Ideal Uses Cases
33+
34+
- Entry point for researchers into a federated TRE system
35+
- Submitting and monitoring analysis tasks
36+
- Integrating external tools with the TRE task workflow
1737

1838
## Core Components
1939

app/explanations/how-5st-works/tre-components/page.mdx renamed to app/explanations/how-5s-tes-works/tre-agent/page.mdx

Lines changed: 29 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,31 @@ import Diagram from "@/components/c4_diagram";
1212

1313
## Prerequisites
1414

15-
{/* TODO: add link to the context diagram */}
15+
- Understanding of [Five Safes TES System Architecture](/explanations/how-5st-works/architecture).
1616

17-
- Understanding of [Five Safes TES Context C4 diagram](/context-c4-diagram).
17+
## Overview
18+
19+
The TRE Agent handles the communication between the [Submission Layer](/explanations/how-5st-works/submission-layer) and a Trusted Research Environment.
20+
The TRE Agent polls out to the Submission Layer, gets a task, then sends the task to an implementation of the [GA4GH Task Execution Service](/standards/#task-execution-service-tes).
21+
Once the task is finished executing the TRE Agent sends the results to [Egress](/explanations/how-5st-works/egress).
22+
23+
The code for TRE Agent is open source, and can be found on [Github](https://github.com/SwanseaUniversityMedical/5s-Tes).
24+
25+
## Core Functionality
26+
27+
- Polls the Submission Layer to retrieve new tasks
28+
- Submits tasks to a GA4GH TES implementation
29+
- Monitors task execution and collects results
30+
- Sends task outputs to the Egress service for approval
31+
- Maintains communication flow between the Submission Layer, TES, and Egress
32+
- Provides a API that can be used to integrate and build against.
33+
34+
## Ideal Uses Cases
35+
36+
- Orchestrating task execution inside a TRE
37+
- Connecting submission workflows to TES backends
38+
- Enabling controlled communication between TRE components
39+
- Supporting federated task execution across TREs
1840

1941
## Core Components
2042

@@ -82,12 +104,15 @@ A TRE database is the database which the analysis query runs against.
82104
If a TRE database is deployed in cloud managed instances (for example, AWS RDS, Azure SQL DB Managed Instance, etc.), this 5 Safes TES implementation requires that you set the TRE Agent with the `admin` credentials of the TRE database, in order to create and revoke ephemeral credentials. This can be done by setting the environment variables `TRE_DATA_USER` and `TRE_DATA_PASSWORD` with the credentials of the user, more details [here](/5s-tes-docs/connect/setup-tre/configuration#general-app-configuration) .
83105

84106
<Callout type="info">
85-
- `admin` credentials are the ones used when creating or setting up the TRE DB instance.
107+
- `admin` credentials are the ones used when creating or setting up the TRE DB
108+
instance.
86109
- If the TRE decides to use PostgreSQL as the database, version 16
87110
or later is required. Because, in Postgres 16+, to revoke a user/role, either
88111
`superuser` or `admin` privilege is required. More details
89112
[here](https://www.postgresql.org/docs/16/sql-droprole.html).
90-
- A user created with the `CREATEROLE` permission can create ephemeral credentials but cannot revoke them.
113+
- A user created
114+
with the `CREATEROLE` permission can create ephemeral credentials but cannot
115+
revoke them.
91116
- In the case where a TRE cannot not use `admin` credentials, the
92117
TRE database can be hosted in the same `postgresql` database instance, which
93118
keeps the local state of TRE Agent, and superuser credentials of the

‎app/explanations/how-5st-works/_meta.js‎

Lines changed: 0 additions & 4 deletions
This file was deleted.

‎app/explanations/what-is-five-safes-tes/page.mdx‎

Lines changed: 0 additions & 61 deletions
This file was deleted.

0 commit comments

Comments
 (0)