Skip to content

Commit c082a60

Browse files
AndrewThienAndrewThien
authored
Add Submission layer components page (#17)
* inital add components page * improve TRE components * update info for Sub components * fix metadata
1 parent 0f88acf commit c082a60

4 files changed

Lines changed: 103 additions & 39 deletions

File tree

‎app/co-ordinate/_meta.js‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
export default {
2+
components: "Submission Layer Components",
3+
};

‎app/co-ordinate/components/page.mdx‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
readingTime: PT30M
3+
audiences: tre-operator
4+
docType: reference
5+
title: Submission Layer Components
6+
---
7+
8+
import { Callout } from "nextra/components";
9+
import { DocMetadata } from "@/components/doc-metadata/DocMetadata";
10+
11+
<DocMetadata />
12+
13+
## Prerequisites
14+
15+
- Understanding of [Five Safes TES Context C4 diagram](/context-c4-diagram).
16+
17+
## Core Components
18+
19+
![Submission layer components](/sub-layer-components.png)
20+
21+
Here is a summary of the required components which make up the Submission Layer:
22+
23+
| Component | Notes |
24+
| ---------------------------------------------------- | :------------------------------------------------------------------------------------------------------- |
25+
| **KeyCloak** (only if you use the built-in KeyCloak) | The Submission apps require a Keycloak realm containing users and clients authorised to access them |
26+
| **S3 storage** (e.g., MinIO, RustFS) | An Amazon S3 compatible Submission Storage service |
27+
| **RabbitMQ** | A message broker for queueing submissions for target TREs |
28+
| **PostgreSQL** | A database for the Submission apps to keep local state |
29+
| **Submission API** | A REST API for Submission Layer functionality, including the GA4GH TES API |
30+
| **Submission GUI** | A Web Frontend for users to log into and interact with the Submission Layer |
31+
| **Hashicorp Vault** | Identity-based secrets and encryption management system used for storing TRE service account credentials |
32+
33+
You may choose how to distribute your own deployment of these components, as long as they are able to communicate with each other over the network.
34+
35+
For some components, such as KeyCloak or S3 (e.g., MinIO, RustFS), you may wish to use your own existing deployments.
36+
37+
### User accessible components
38+
39+
A subset of the components will need to be accessible by users, outside of the TRE Layer environment, through a reverse proxy or port forwarding:
40+
41+
| Component | Reason | Sample Port \* |
42+
| ------------------ | :-------------------------------------------------------------------------------------------------------------------------- | :------------- |
43+
| **KeyCloak** | So that users can authenticate and admins can manage the Submission KeyCloak Realm (if you are using the built-in KeyCloak) | 8085 |
44+
| **S3 GUI** | So that Researchers can upload and download Submission data (inputs or outputs) through a web interface | 9001 |
45+
| **S3 API** | So that TREs can put the analytics results into S3 storage through a REST API | 9000 |
46+
| **Submission GUI** | So that users can log into and interact with the Submission Layer | 7220 |
47+
| **Submission API** | So that Researchers and TREs can interact with the Submission Layer via a REST API | 5034 |
48+
49+
**Note**: Sample ports are provided for reference only. It can be modified to reflect the your own
50+
infrastructure and configuration choices.
51+
52+
<Callout>
53+
If you are setting up a reverse proxy for S3 (e.g., MinIO, RustFS), the
54+
`Websockets Support` should be enabled for S3 GUI Proxy host.
55+
</Callout>

‎app/connect/components/page.mdx‎

Lines changed: 45 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,68 +1,66 @@
11
---
22
readingTime: PT10M
33
audiences: [tre-operator, contributor]
4+
title: TRE Layer Components
45
---
56

67
import { Steps, Callout, Table } from "nextra/components";
78
import { DocMetadata } from "@/components/doc-metadata/DocMetadata";
89

910
<DocMetadata />
1011

11-
# TRE Layer Components
12-
13-
1412
## Prerequisites
13+
1514
{/* TODO: add link to the context diagram */}
15+
1616
- Understanding of [Five Safes TES Context C4 diagram](/context-c4-diagram).
1717

1818
## Core Components
1919

20-
### Summary
21-
2220
![TRE layer components](/tre-components.png)
2321

2422
Here is a summary of the required components which make up the TRE Layer:
2523

26-
| Component | Notes |
27-
| ------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------- |
28-
| **KeyCloak** | The TRE Agent and Egress apps each require a Keycloak realm containing users and clients authorised to access them |
29-
| **S3 (MinIO or RustFS)** | An Amazon S3 compatible TRE Storage service |
30-
| **RabbitMQ** | A message broker used for queueing |
31-
| **PostgreSQL** | A database for the TRE Agent and Egress apps to keep local state |
32-
| **Hashicorp Vault** | Used for accessing ephemeral credentials as secrets |
33-
| **OpenLDAP** | Used to provide ephemeral credentials for Trino (if in use as a datasource) |
34-
| **Camunda Connectors** | A REST API allowing other services (e.g. TRE Agent) to integrate with Camunda |
35-
| **Camunda Orchestration** | A consolidated service that combines Camunda functionality via [Zeebe + Operate + Tasklist](https://camunda.com/process-orchestration/) |
36-
| **ElasticSearch** | Used for Camunda's local datastore such as workflow state |
37-
| **TRE-Camunda** | A service defining the handlers for ephemeral credentials management. Registers the handlers with Camunda, via Zeebe |
38-
| **TRE Agent API** | A REST API for TRE Agent functionality |
39-
| **TRE Agent GUI** | A Web Frontend for TRE Admins to log into and interact with the TRE Agent |
40-
| **Egress API** | A REST API for Egress Portal functionality |
41-
| **Egress GUI** | A Web Frontend for Egress Officers to log into and interact with the Egress Portal |
42-
| **TES Backend** | A standard GA4GH TES implementation to execute the analysis |
24+
| Component | Notes |
25+
| ---------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------- |
26+
| **KeyCloak** (only if you use the built-in KeyCloak) | The TRE Agent and Egress apps each require a Keycloak realm containing users and clients authorised to access them |
27+
| **S3 storage** (e.g., MinIO, RustFS) | An Amazon S3 compatible TRE Storage service |
28+
| **RabbitMQ** | A message broker used for queueing |
29+
| **PostgreSQL** | A database for the TRE Agent and Egress apps to keep local state |
30+
| **Hashicorp Vault** | Used for accessing ephemeral credentials as secrets |
31+
| **OpenLDAP** | Used to provide ephemeral credentials for Trino (if in use as a datasource) |
32+
| **Camunda Connectors** | A REST API allowing other services (e.g. TRE Agent) to integrate with Camunda |
33+
| **Camunda Orchestration** | A consolidated service that combines Camunda functionality via [Zeebe + Operate + Tasklist](https://camunda.com/process-orchestration/) |
34+
| **ElasticSearch** | Used for Camunda's local datastore such as workflow state |
35+
| **TRE-Camunda** | A service defining the handlers for ephemeral credentials management. Registers the handlers with Camunda, via Zeebe |
36+
| **TRE Agent API** | A REST API for TRE Agent functionality |
37+
| **TRE Agent GUI** | A Web Frontend for TRE Admins to log into and interact with the TRE Agent |
38+
| **Egress API** | A REST API for Egress Portal functionality |
39+
| **Egress GUI** | A Web Frontend for Egress Officers to log into and interact with the Egress Portal |
40+
| **TES Backend** | A standard GA4GH TES implementation to execute the analysis |
4341

4442
You may choose how to distribute your own deployment of these components, as long as they are able to communicate with each other over the network.
4543

46-
For some components, such as KeyCloak or S3 (MinIO or RustFS), you may wish to use your own existing deployments.
44+
For some components, such as KeyCloak or S3 (e.g., MinIO, RustFS), you may wish to use your own existing deployments.
4745

4846
### User accessible components
4947

5048
A subset of the components will need to be accessible by users, outside of the TRE Layer environment, through a reverse proxy or port forwarding:
5149

52-
| Component | Reason | Sample Port * |
53-
| ----------------- | :------------------------------------------------------------------------------------------------------------------- | :------|
54-
| **KeyCloak** | So that users can authenticate and admins can manage the TRE Agent and Egress KeyCloak Realms (if you are using the internal, built-in KeyCloak) | 8085 |
55-
| **TRE Agent GUI** | So that TRE Admins can log into and interact with the TRE Agent | 8989 |
56-
| **TRE Agent API** | So that TRE Agent GUI browser functionality, and optionally other services, can interact with the TRE Agent REST API | 8072 |
57-
| **Egress GUI** | So that Egress Officers can log into and interact with the Egress Portal | 8100 |
58-
| **S3 UI** | So that Egress Officers can interact with the S3 storage through a web interface | 9003 |
50+
| Component | Reason | Sample Port \* |
51+
| ----------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- | :------------- |
52+
| **KeyCloak** | So that users can authenticate and admins can manage the TRE Agent and Egress KeyCloak Realms (if you are using the internal, built-in KeyCloak) | 8085 |
53+
| **TRE Agent GUI** | So that TRE Admins can log into and interact with the TRE Agent | 8989 |
54+
| **TRE Agent API** | So that TRE Agent GUI browser functionality, and optionally other services, can interact with the TRE Agent REST API | 8072 |
55+
| **Egress GUI** | So that Egress Officers can log into and interact with the Egress Portal | 8100 |
56+
| **S3 UI** | So that Egress Officers can interact with the S3 storage through a web interface | 9003 |
5957

6058
**Note**: Sample ports are provided for reference only. It can be modified to reflect the your own
61-
infrastructure and configuration choices.
59+
infrastructure and configuration choices.
6260

6361
<Callout>
64-
If you are setting up a reverse proxy for S3 (MinIO or RustFS), the `Websockets Support` should be
65-
enabled for S3 GUI Proxy host.
62+
If you are setting up a reverse proxy for S3 (MinIO or RustFS), the
63+
`Websockets Support` should be enabled for S3 GUI Proxy host.
6664
</Callout>
6765

6866
### TRE Database
@@ -72,13 +70,21 @@ A TRE database is the database which the analysis query runs against.
7270
If a TRE database is deployed in cloud managed instances (for example, AWS RDS, Azure SQL DB Managed Instance, etc.), this 5 Safes TES implementation requires that you set the TRE Agent with the `admin` credentials of the TRE database, in order to create and revoke ephemeral credentials. This can be done by setting the environment variables `TRE_DATA_USER` and `TRE_DATA_PASSWORD` with the credentials of the user, more details [here](/5s-tes-docs/connect/setup-tre/configuration#general-app-configuration) .
7371

7472
<Callout type="info">
75-
- `admin` credentials are the ones used when creating or setting up the TRE DB instance.
76-
- If the TRE decides to use PostgreSQL as the database, version 16 or later is required. Because, in Postgres 16+, to revoke a user/role, either `superuser` or `admin` privilege is required. More details [here](https://www.postgresql.org/docs/16/sql-droprole.html).
77-
- A user created with the `CREATEROLE` permission can create ephemeral credentials but cannot revoke them.
78-
- In the case where a TRE cannot not use `admin` credentials, the TRE database can be hosted in the same `postgresql` database instance, which keeps the local state of TRE Agent, and superuser credentials of the `postgresql` can be used in the environment variables `TRE_DATA_USER` and `TRE_DATA_PASSWORD`.
73+
- `admin` credentials are the ones used when creating or setting up the TRE DB
74+
instance. - If the TRE decides to use PostgreSQL as the database, version 16
75+
or later is required. Because, in Postgres 16+, to revoke a user/role, either
76+
`superuser` or `admin` privilege is required. More details
77+
[here](https://www.postgresql.org/docs/16/sql-droprole.html). - A user created
78+
with the `CREATEROLE` permission can create ephemeral credentials but cannot
79+
revoke them. - In the case where a TRE cannot not use `admin` credentials, the
80+
TRE database can be hosted in the same `postgresql` database instance, which
81+
keeps the local state of TRE Agent, and superuser credentials of the
82+
`postgresql` can be used in the environment variables `TRE_DATA_USER` and
83+
`TRE_DATA_PASSWORD`.
7984
</Callout>
8085

8186
<Callout>
8287
Remember the TES Backend's environment will also need network access to
83-
project data sources or TRE database, in order for analysis to run against them.
84-
</Callout>
88+
project data sources or TRE database, in order for analysis to run against
89+
them.
90+
</Callout>

‎public/sub-layer-components.png‎

160 KB
Loading

0 commit comments

Comments
 (0)