Skip to content

Commit 148615b

Browse files
bp1183bp1183
committed
Merge branch 'main' into connect/controlling-tre-data-access
2 parents 5c0d1db + a0fdb95 commit 148615b

20 files changed

Lines changed: 437 additions & 56 deletions

File tree

‎app/_meta.js‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
export default {
22
connect: "Connect",
33
"co-ordinate": "Co-ordinate",
4+
5+
// Welcome Report URL redirect, never show in nav
6+
five_safes_tes: {
7+
display: "hidden"
8+
}
49
};

‎app/connect/_meta.js‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
11
export default {
2+
"components": "TRE Layer Components",
23
"setup-tre": "Setup TRE Agent",
34
"connect-submission": "Connecting a TRE",
45
approvals: "Approve Projects",
56
"controlling-tre-data-access": "Controlling TRE Data Access",
6-
"approve-outputs":"Review Outputs"
7+
"connect-submission": "Connecting a TRE",
8+
"connect-egress": "Connect TRE to Data Egress",
9+
"approve-outputs":"Review Outputs",
10+
"tes-backend": "TES Backend",
711
};

‎app/connect/components/page.mdx‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
readingTime: PT10M
3+
audiences: [tre-operator, contributor]
4+
---
5+
6+
import { Steps, Callout, Table } from "nextra/components";
7+
import { DocMetadata } from "@/components/doc-metadata/DocMetadata";
8+
9+
<DocMetadata />
10+
11+
# TRE Layer Components
12+
13+
14+
## Prerequisites
15+
{/* TODO: add link to the context diagram */}
16+
- Understanding of [Five Safes TES Context C4 diagram](/context-c4-diagram).
17+
18+
## Core Components
19+
20+
### Summary
21+
22+
![TRE layer components](/tre-components.png)
23+
24+
Here is a summary of the required components which make up the TRE Layer:
25+
26+
| Component | Notes |
27+
| ------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------- |
28+
| **KeyCloak** | The TRE Agent and Egress apps each require a Keycloak realm containing users and clients authorised to access them |
29+
| **S3 (MinIO or RustFS)** | An Amazon S3 compatible TRE Storage service |
30+
| **RabbitMQ** | A message broker used for queueing |
31+
| **PostgreSQL** | A database for the TRE Agent and Egress apps to keep local state |
32+
| **Hashicorp Vault** | Used for accessing ephemeral credentials as secrets |
33+
| **OpenLDAP** | Used to provide ephemeral credentials for Trino (if in use as a datasource) |
34+
| **Camunda Connectors** | A REST API allowing other services (e.g. TRE Agent) to integrate with Camunda |
35+
| **Camunda Orchestration** | A consolidated service that combines Camunda functionality via [Zeebe + Operate + Tasklist](https://camunda.com/process-orchestration/) |
36+
| **ElasticSearch** | Used for Camunda's local datastore such as workflow state |
37+
| **TRE-Camunda** | A service defining the handlers for ephemeral credentials management. Registers the handlers with Camunda, via Zeebe |
38+
| **TRE Agent API** | A REST API for TRE Agent functionality |
39+
| **TRE Agent GUI** | A Web Frontend for TRE Admins to log into and interact with the TRE Agent |
40+
| **Egress API** | A REST API for Egress Portal functionality |
41+
| **Egress GUI** | A Web Frontend for Egress Officers to log into and interact with the Egress Portal |
42+
| **TES Backend** | A standard GA4GH TES implementation to execute the analysis |
43+
44+
You may choose how to distribute your own deployment of these components, as long as they are able to communicate with each other over the network.
45+
46+
For some components, such as KeyCloak or S3 (MinIO or RustFS), you may wish to use your own existing deployments.
47+
48+
### User accessible components
49+
50+
A subset of the components will need to be accessible by users, outside of the TRE Layer environment, through a reverse proxy or port forwarding:
51+
52+
| Component | Reason | Sample Port * |
53+
| ----------------- | :------------------------------------------------------------------------------------------------------------------- | :------|
54+
| **KeyCloak** | So that users can authenticate and admins can manage the TRE Agent and Egress KeyCloak Realms (if you are using the internal, built-in KeyCloak) | 8085 |
55+
| **TRE Agent GUI** | So that TRE Admins can log into and interact with the TRE Agent | 8989 |
56+
| **TRE Agent API** | So that TRE Agent GUI browser functionality, and optionally other services, can interact with the TRE Agent REST API | 8072 |
57+
| **Egress GUI** | So that Egress Officers can log into and interact with the Egress Portal | 8100 |
58+
| **S3 UI** | So that Egress Officers can interact with the S3 storage through a web interface | 9003 |
59+
60+
**Note**: Sample ports are provided for reference only. It can be modified to reflect the your own
61+
infrastructure and configuration choices.
62+
63+
<Callout>
64+
If you are setting up a reverse proxy for S3 (MinIO or RustFS), the `Websockets Support` should be
65+
enabled for S3 GUI Proxy host.
66+
</Callout>
67+
68+
### TRE Database
69+
70+
A TRE database is the database which the analysis query runs against.
71+
72+
If a TRE database is deployed in cloud managed instances (for example, AWS RDS, Azure SQL DB Managed Instance, etc.), this 5 Safes TES implementation requires that you set the TRE Agent with the `admin` credentials of the TRE database, in order to create and revoke ephemeral credentials. This can be done by setting the environment variables `TRE_DATA_USER` and `TRE_DATA_PASSWORD` with the credentials of the user, more details [here](/5s-tes-docs/connect/setup-tre/configuration#general-app-configuration) .
73+
74+
<Callout type="info">
75+
- `admin` credentials are the ones used when creating or setting up the TRE DB instance.
76+
- If the TRE decides to use PostgreSQL as the database, version 16 or later is required. Because, in Postgres 16+, to revoke a user/role, either `superuser` or `admin` privilege is required. More details [here](https://www.postgresql.org/docs/16/sql-droprole.html).
77+
- A user created with the `CREATEROLE` permission can create ephemeral credentials but cannot revoke them.
78+
- In the case where a TRE cannot not use `admin` credentials, the TRE database can be hosted in the same `postgresql` database instance, which keeps the local state of TRE Agent, and superuser credentials of the `postgresql` can be used in the environment variables `TRE_DATA_USER` and `TRE_DATA_PASSWORD`.
79+
</Callout>
80+
81+
<Callout>
82+
Remember the TES Backend's environment will also need network access to
83+
project data sources or TRE database, in order for analysis to run against them.
84+
</Callout>

‎app/connect/connect-egress/page.mdx‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
readingTime: PT15M
3+
audiences: tre-operator
4+
---
5+
6+
import { Steps, Callout } from "nextra/components";
7+
import { DocMetadata } from "@/components/doc-metadata/DocMetadata";
8+
9+
<DocMetadata />
10+
11+
# Connect TRE to Data Egress Layer
12+
13+
## Prerequisites
14+
15+
- A deployed TRE Layer. Deployment steps can be found [here](/connect/setup-tre/deployment-steps).
16+
- Access to the TRE UI.
17+
- Access to the Egress UI.
18+
- Keycloak Admin access to the `Dare-TRE` and `Data-Egress` Keycloak realms.
19+
{/* TODO: add link to the guide of creating a Keycloak user */}
20+
21+
## Steps
22+
23+
<Steps>
24+
### Create TRE and Egress admin users
25+
26+
To establish a connection between a TRE Agent and Egress, two users need to be created in the Egress Layer and the TRE Agent Keycloak realms by TRE Layer's Keycloak Admin.
27+
28+
**1.1 Create and add a user to access the Egress from the TRE Agent**
29+
30+
- Login to Keycloak server of TRE Layer as an admin user.
31+
- Navigate to the `Data-Egress` realm.
32+
- Create a user with the `dare-tre-admin` role.
33+
34+
**1.2 Create and add a user to access the TRE Layer from Egress**
35+
36+
- Login to Keycloak server of TRE Layer as an admin user.
37+
- Navigate to the `Dare-TRE` realm.
38+
- Create a user with the `dare-egress-admin` role.
39+
40+
### Setup the connection between TRE and Egress
41+
42+
**2.1 TRE Admin task**
43+
44+
- Login to TRE UI with your admin credentials
45+
- Navigate to the `Update Data Egress Credentials` page
46+
- Add the username and password of the user created in the step `1.1` above.
47+
48+
**2.2 Egress Admin task**
49+
50+
- Login to Egress UI with your admin credentials.
51+
- Navigate to the `Update TRE Credentials` page.
52+
- Add the username and password of the user created in the step `1.2` above.
53+
54+
<Callout type="info">
55+
TRE Admin can now confirm that the Connection between Data Egress and TRE Agent is successful through the health check tab in the TRE UI under `Deployment Configuration`.
56+
</Callout>
57+
</Steps>
58+
59+
## What's next?
60+
61+
- [Set up TRE data access](/connect/controlling-tre-data-access)

‎app/connect/connect-submission/page.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ The current health status is displayed in the **Deployment Admin** page under th
144144

145145
Once any configuration or connectivity issues have been resolved, the Synchronisation can be restarted from the **Deployment Admin** page by selecting **Enable Sync**.
146146

147-
### What’s next? :
147+
### What's next?
148148

149149
- [Approving Projects/Memberships](/connect/approvals)
150150
- [Review & Approve Outputs](/connect/approve-outputs)

‎app/connect/controlling-tre-data-access/page.mdx‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,14 @@ import { DocMetadata } from '@/components/doc-metadata/DocMetadata'
1010
<DocMetadata />
1111

1212
# Controlling TRE Data Access
13+
## Prerequisites
14+
- TRE agent deployed
15+
- Access to the TRE UI
16+
- TRE admin credentials
17+
- TRE data source connection details
18+
- Knowledge of how the TRE grants access to its DB
1319

20+
## Configuring DMN Rules
1421
Log in to the TRE Layer UI and navigate to the “DMN Rules” section.
1522

1623
This page helps to set the values which are used by the TES backend (e.g., Funnel) to connect to the TRE DB using short-lived, ephemeral credentials rather than static, long-lived ones - reducing the risk of credential leakage or reuse.
@@ -34,10 +41,3 @@ Find the rules with the following tags and change them to match the values of th
3441

3542
- Any string value put into an output value should be inside double quotes. For example: `"public"`. If the output values refer to input variables, double quotes are not required, for example, `string(project)`. See [Camunda DMN rule convention](https://docs.camunda.io/docs/components/modeler/feel/language-guide/feel-expressions-introduction/) for more details.
3643
- The value of `postgresSchema` defaults to the project name in the Submission Layer. If the OMOP CDM is under a different schema, change the value to the actual lowercase schema name in double quotes.
37-
38-
### Prerequisites
39-
- TRE agent deployed
40-
- Access to the TRE UI
41-
- TRE admin credentials
42-
- TRE data source connection details
43-
- Knowledge of how the TRE grants access to its DB

‎app/connect/setup-tre/deployment-steps/page.mdx‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,16 @@ import { DocMetadata } from "@/components/doc-metadata/DocMetadata"
1010

1111
# Deployment Steps
1212

13+
## Prerequisites
14+
15+
- [Docker](https://docker.com/) installed. For Linux/Ubuntu VMs, you can follow this [guide](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) to install Docker.
16+
- [Git](https://git-scm.com/about) installed on your machine, if you want to clone the repository.
17+
- TRE restrictive database connection details.
18+
19+
## Steps
1320
<Steps>
1421

15-
## Download the deployment files
22+
### Download the deployment files
1623

1724
There are a few ways to download the deployment files.
1825

@@ -60,7 +67,7 @@ There are a few ways to download the deployment files.
6067
![Get deployment files](/get-deployment-files.png)
6168

6269

63-
## Prepare the necessary files
70+
### Prepare the necessary files
6471

6572
If you would like to modify the services while keeping the original files structure (to make Git tracking easier, for example), you can follow the steps below.
6673
- Create a new `compose.yml` file next to `DeploymentStack` and `ServiceStack` directories.
@@ -124,7 +131,7 @@ For example, the `CONFIG_PATH` for the scenario below is `../../../config` becau
124131
- Since the directory structure is changed, you need to update the relative paths in the `docker-compose.yml` and `CONFIG_PATH` in the `.env` file accordingly.
125132

126133

127-
## Configure the stack
134+
### Configure the stack
128135

129136
__3.1 Create a new .env file__
130137

@@ -413,7 +420,9 @@ A guide to set up SSL certs and subdomains is coming soon.
413420

414421
__3.8 Configure TES Backend__
415422

416-
To complete the setup, you need to put the URL for the TES backend in the `.env` file. If you have not set this up yet, follow this guide to [Setup a TES backend](/connect/setup-TES/backend) and configure it to use the TRE S3 storage of this deployment.
423+
To complete the setup, you need to put the URL for the TES backend in the `.env` file.
424+
425+
If you have not set this up yet, setup a TES backend, e.g., Funnel (with the installation guide [here](/connect/tes-backend/install-funnel)), and configure it to use the TRE S3 storage of this deployment.
417426

418427

419428
<Table>
@@ -431,7 +440,7 @@ To complete the setup, you need to put the URL for the TES backend in the `.env`
431440
</tbody>
432441
</Table>
433442

434-
## Run the Docker compose
443+
### Run the Docker compose
435444

436445
At the directory where `docker-compose.yml` or `compose.yml` and `.env` files are located, run the following command to start the TRE Agent and Data Egress:
437446

@@ -443,4 +452,4 @@ docker compose up -d
443452

444453
## What's next?
445454

446-
- Connect TRE to Data Egress
455+
- [Connect TRE to Data Egress](/5s-tes-docs/connect/connect-egress)

‎app/connect/tes-backend/_meta.js‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
export default {
2+
"tes-implementations": "TES Implementations",
3+
"install-funnel": "Install Funnel",
4+
};

0 commit comments

Comments
 (0)