Skip to content

TES backend #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndrewThien merged 15 commits into from
Jul 28, 2026
Merged

TES backend #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
c047baf
initial add tes backend
AndrewThien Jul 27, 2026
a33b571
add metadata
AndrewThien Jul 27, 2026
db25113
improve
AndrewThien Jul 27, 2026
4e08a74
remove outdated methods
AndrewThien Jul 27, 2026
ad218d6
more time
AndrewThien Jul 27, 2026
31d0112
linking
AndrewThien Jul 27, 2026
eccbeee
fix path
AndrewThien Jul 27, 2026
773ce15
Merge branch 'main' into tes-backend
AndrewThien Jul 27, 2026
985ccce
update naming
AndrewThien Jul 27, 2026
6260d74
add Prerequisites
AndrewThien Jul 27, 2026
3a716b6
more details
AndrewThien Jul 27, 2026
83734e5
add link
AndrewThien Jul 27, 2026
44d38f4
Merge branch 'main' into tes-backend
AndrewThien Jul 27, 2026
d445ae0
add comma
AndrewThien Jul 27, 2026
aaf724a
Merge branch 'main' into tes-backend
AndrewThien Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion app/connect/_meta.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,7 @@ export default {
"setup-tre": "Setup TRE Agent",
"connect-submission": "Connecting a TRE",
approvals: "Approve Projects",
"approve-outputs":"Review Outputs"
"connect-submission": "Connecting a TRE",
"approve-outputs":"Review Outputs",
"tes-backend": "TES Backend",
};
21 changes: 15 additions & 6 deletions app/connect/setup-tre/deployment-steps/page.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,16 @@ import { DocMetadata } from "@/components/doc-metadata/DocMetadata"

# Deployment Steps

## Prerequisites

- [Docker](https://docker.com/) installed. For Linux/Ubuntu VMs, you can follow this [guide](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) to install Docker.
- [Git](https://git-scm.com/about) installed on your machine, if you want to clone the repository.
- TRE restrictive database connection details.

## Steps
<Steps>

## Download the deployment files
### Download the deployment files

There are a few ways to download the deployment files.

Expand Down Expand Up @@ -60,7 +67,7 @@ There are a few ways to download the deployment files.
![Get deployment files](/get-deployment-files.png)


## Prepare the necessary files
### Prepare the necessary files

If you would like to modify the services while keeping the original files structure (to make Git tracking easier, for example), you can follow the steps below.
- Create a new `compose.yml` file next to `DeploymentStack` and `ServiceStack` directories.
Expand Down Expand Up @@ -124,7 +131,7 @@ For example, the `CONFIG_PATH` for the scenario below is `../../../config` becau
- Since the directory structure is changed, you need to update the relative paths in the `docker-compose.yml` and `CONFIG_PATH` in the `.env` file accordingly.


## Configure the stack
### Configure the stack

__3.1 Create a new .env file__

Expand Down Expand Up @@ -413,7 +420,9 @@ A guide to set up SSL certs and subdomains is coming soon.

__3.8 Configure TES Backend__

To complete the setup, you need to put the URL for the TES backend in the `.env` file. If you have not set this up yet, follow this guide to [Setup a TES backend](/connect/setup-TES/backend) and configure it to use the TRE S3 storage of this deployment.
To complete the setup, you need to put the URL for the TES backend in the `.env` file.

If you have not set this up yet, setup a TES backend, e.g., Funnel (with the installation guide [here](/connect/tes-backend/install-funnel)), and configure it to use the TRE S3 storage of this deployment.


<Table>
Expand All @@ -431,7 +440,7 @@ To complete the setup, you need to put the URL for the TES backend in the `.env`
</tbody>
</Table>

## Run the Docker compose
### Run the Docker compose

At the directory where `docker-compose.yml` or `compose.yml` and `.env` files are located, run the following command to start the TRE Agent and Data Egress:

Expand All @@ -443,4 +452,4 @@ docker compose up -d

## What's next?

- Connect TRE to Data Egress
- [Connect TRE to Data Egress](/5s-tes-docs/connect/connect-egress)
4 changes: 4 additions & 0 deletions app/connect/tes-backend/_meta.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
export default {
"tes-implementations": "TES Implementations",
"install-funnel": "Install Funnel",
};
143 changes: 143 additions & 0 deletions app/connect/tes-backend/install-funnel/page.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
---
readingTime: PT45M
audiences: tre-operator
---

import { Steps, Callout, Tabs } from "nextra/components";
import { DocMetadata } from "@/components/doc-metadata/DocMetadata";

<DocMetadata />

# Install Funnel TES Backend

[Funnel](https://github.com/calypr/funnel) is an implementation of the GA4GH Task Execution Schemas. It provides an API server,
multiple storage backends, multiple compute backends and a web UI.

This guide will walk you through the process of installing Funnel and configuring it to use the TRE Layer S3 instance.

<Steps>
## Set up S3 Access key

This step is required to configure Funnel to use the TRE Layer S3 instance.

- Navigate to the S3 instance of the TRE Layer. It will be on the same host as the TRE Agent, at port `9002`, if you followed the example configuration. If you've configured it to be different, you'll need to use the correct port.

- Login with S3 TRE root credentials (which are set up in the environment variables in the `.env` file).

- To create the access keys, in the S3 (currently RustFS) console navigate to **_Access Keys -> Add Access Key_**

<Callout type="info">
While you can log in with your KeyCloak credentials, using the option `Log in
with KeyCloak`, to create access keys with efficient privileges, you need to
log in with S3 root credentials.
</Callout>

![RustFS](/S3-RustFS.png)

## Install Funnel

You can find the latest releases with installation instructions [here](https://github.com/calypr/funnel/releases).

<Callout type="info">
Funnel is currently only available on linux and macOS.
</Callout>

<Tabs items={['Install Script', 'Build from Source']}>
<Tabs.Tab>
This installs the latest release candidate at the time of writing `v0.11.12`.
```bash copy
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/calypr/funnel/develop/install.sh)" -- "v0.11.12"
```
<Callout type="info">
You can add the destination of Funnel by adding it in the end of the command. For example, to install Funnel to `/opt/funnel`:
```bash copy
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/calypr/funnel/develop/install.sh)" -- "v0.11.12" "/opt/funnel"
```
</Callout>
</Tabs.Tab>

</Tabs>

## Check if funnel has been successfully installed

```bash copy
funnel --help
```

## Funnel config file

Funnel will need access to the TRE layer S3 storage. Once funnel is installed, create a `config.yml` file.

- `Endpoint` is where the instance of S3 in the TRE Layer is hosted.
- `Access key` & `Secret` are created in the [previous step](#set-up-s3-access-key).

```yaml filename="config.yml" copy
GenericS3:
- Disabled: false
Endpoint: "localhost:9002" # URL to S3 API. Change to match your environment. `9002` is the default port in our deployment samples.
Key: "<access key>"
Secret: "<secret>"
```

You may also wish to configure other elements of the Funnel installation, e.g.:

```yaml filename="config.yml" copy
Server:
HTTPPort: "80" # Control which port Funnel binds on, default is 8000
Worker:
WorkDir: "/var/opt/funnel" # explicitly specify Funnel's working directory
# ...
```

<Callout>

[Funnel's documentation](https://calypr.org/tools/funnel/docs/) offers configuration details in the relevant pages for each feature area.

Funnel's own [deployment documentation](https://calypr.org/tools/funnel/docs/compute/deployment/) also offers suggested approaches to deployment.

</Callout>
## Run Funnel

Run the funnel server using the config created above.

```bash copy
funnel server run -c config.yml
```

<Callout type="info">
You can decide the location of the config file, but the command above will
need to be changed to point to the correct config file location. For example,
if the config file is located in `/etc/funnel/`: ```bash copy funnel server
run -c /etc/funnel/config.yml ```
</Callout>

Optionally, to run Funnel in the background on a typical linux server, you may wish to setup a systemd service.

For example, if Funnel is installed manually to `/opt/funnel/funnel`:

```inf filename="funnel.service" copy
[Unit]
Description=Funnel Server

[Service]
ExecStart=/opt/funnel/funnel server run --config /etc/funnel/config.yml
SyslogIdentifier=funnel
LimitNOFILE=1000000
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target
```

## Access to Funnel

By default, Funnel Web Dashboard can be accessed at host and port specified in the config file (`Server.HTTPPort`) with a default value of `8000`.

<Callout type="warning">
It is strongly recommended NOT to open public internet access to Funnel
Dashboard and to limit the Funnel network access to specific IP addresses,
e.g., IP of developer's machine for debugging purposes, only.
</Callout>

</Steps>
53 changes: 53 additions & 0 deletions app/connect/tes-backend/tes-implementations/page.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
readingTime: PT20M
audiences: tre-operator
---

import { Steps, Callout, Tabs } from "nextra/components";
import { DocMetadata } from "@/components/doc-metadata/DocMetadata";

<DocMetadata />

# TES Implementations

Task Execution Service (or [TES](https://www.ga4gh.org/product/task-execution-service-tes/)) is a GA4GH standard for executing tasks. Details about its schema can be found [here](https://ga4gh.github.io/task-execution-schemas/docs/#tag/TaskService/operation/CreateTask).

There are a few implementations of the TES standard, including:

- [TES-K](https://github.com/elixir-cloud-aai/TESK): A Kubernetes-native TES implementation where each job runs as a Kubernetes pod. This makes it straightforward to enforce pod-level security policies via Kubernetes admission controller.
However, TESK has the operational overhead of running a Kubernetes cluster. The [director-wfs](https://github.com/SwanseaUniversityMedical/director-wfs) from Swansea University Medical School can help mitigate this by providing a full-stack deployment of TESK on [KIND (Kubernetes IN Docker)](https://kind.sigs.k8s.io/), with hardening enforced by [Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/) admission control, runtime detection via [Falco](https://falco.org/), network policies, monitoring, and ingress management all active by default. A single bootstrap script stands up the whole cluster on macOS or Ubuntu.

- [Funnel](https://github.com/calypr/funnel) is a Go implementation of TES that translates task messages directly into `docker run` commands. It comprises a server, worker, and a collection of compute, storage, and database backends, making it a TES compatible execution backend.
Given a task description, Funnel will locate a worker to carry out the task, download inputs, launch a number of (Docker) containers, upload outputs, record logs, and monitor the entire process.

Below is a comparison between Funnel (default Docker backend) and TESK as deployed via director-wfs. TRE operators considering TESK for production should be aware that director-wfs is one possible deployment pattern - TESK itself supports any Kubernetes cluster. This comparison is best understood as a demonstration of what a security-conscious TESK deployment can look like in practice.

## Comparison between TES-K and Funnel

Below is a comparison between TES-K and Funnel. The purpose of this comparison is to help TRE operators decide which implementation to use for their production TRE.

| Category | Funnel (default Docker backend) | TESK via director-wfs |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Implementation language** | Go | Java (tesk-api) + Python (tesk-core) |
| **Underlying runtime** | Docker (direct `docker run`) | Kubernetes pods (via KIND on a single host) |
| **Deployment complexity** | Low — single binary, runs anywhere Docker is available | Higher — full KIND cluster, but bootstrapped via a [single script](<(https://github.com/SwanseaUniversityMedical/director-wfs)>) |
| **Horizontal scalability** | Yes (also supports K8s backend) | No — director-wfs targets single-server deployments only |
| **Host OS support** | Any Docker host | macOS or Ubuntu (per director-wfs bootstrap) |
| **Container root filesystem** | Read-only, with a writable `/tmp` provided by Funnel | Read-only, enforced by Gatekeeper admission control |
| **`/tmp` writability** | Writable (Funnel mounts a host dir), so diagnostic socket creates fine, for example. | Read-only — can break runtimes that write to `/tmp` at startup. So `DOTNET_EnableDiagnostics=0` needs to be set in the task payload, for example. |
| **Container user** | Whatever the image specifies (usually root) | Non-root by default, enforced by Gatekeeper |
| **Linux capabilities** | Docker's full default set granted | All dropped by default, enforced by Gatekeeper |
| **Privileged containers** | Possible if worker config permits | Rejected by Gatekeeper before they run |
| **Host filesystem access** | Whatever volumes the worker is configured to mount | Restricted to a single storage path, enforced by Gatekeeper |
| **Host Docker socket exposure** | Required by the Funnel worker (root-equivalent reach on host) | Not used |
| **Security enforcement model** | Relies entirely on correct operator configuration | Enforced at cluster level — insecure pod specs are rejected |
| **Hardening defaults** | None — `--user`, `--cap-drop`, `--security-opt`, `user-namespace` flags absent by default | Hardened by default via Gatekeeper policies |
| **Runtime threat monitoring** | None by default | Falco watches container activity at runtime |
| **Network policies** | None built-in | Active by default |
| **Admission control** | None | Gatekeeper (OPA) |
| **Container ENTRYPOINT behaviour** | Honoured — command passed as args to image ENTRYPOINT. This is diverging from spec of [TES schema](<(https://ga4gh.github.io/task-execution-schemas/docs/#tag/TaskService/operation/CreateTask)>). | Ignored — `command[0]` is exec'd directly; full command must be in the task payload, per the TES/argv spec |
| **Migration impact** | Tolerant of images that rely on ENTRYPOINT | Requires TES payloads to include program name as `command[0]` |
| **Logging** | All logs in one place (simple) | Per-pod/job logs via kubectl |
| **Ingress / TLS** | Operator-managed | Managed by director-wfs; SSL can be disabled for private networks |
| **Storage backend (tested)** | Operator-defined | MinIO (S3-compatible), configured at install time |
| **Operator burden** | High — all hardening must be configured deliberately per deployment | Low — secure defaults enforced; operator configures exceptions, not rules |
2 changes: 1 addition & 1 deletion package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,4 @@
"tailwindcss": "^4.3.3",
"typescript": "^6"
}
}
}
Binary file added public/S3-RustFS.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.