Skip to content

Commit ea8f330

Browse files
jgwestjannfis
jgwest
and
jannfis
authored
chore: Add support for developing on non-local K8s via reverse tunnel (#399)
Signed-off-by: Jonathan West <jonwest@redhat.com> Signed-off-by: Jonathan West <jgwest@users.noreply.github.com> Co-authored-by: Jann Fischer <jann@mistrust.net>
1 parent bfea10b commit ea8f330

12 files changed

Lines changed: 363 additions & 0 deletions

File tree

‎hack/dev-env/reverse-tunnel/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Reverse-proxy via rathole
2+
3+
This folder contains shell scripts and Kubernetes resources to establish a reverse tunnel from the K8s cluster, back to your local development environment.
4+
5+
Primarily this enables the resource-proxy feature when developing on remote K8s clusters, as Argo CD (running on vcluster-control-plane) is now able to tunnel to principal running locally.
6+
7+
This also allows all the E2E tests to pass in this configuration. Previously, the resource proxy E2E tests were failing when running against a remote cluster.
8+
9+
10+
**NOTE**: The current implementation requires Service LoadBalancer support from K8s (which is available from, for example, Red Hat's 'clusterbot' service).
11+
12+
The tunnel that is enabled is:
13+
* Argo CD 'server' workload (on vcluster-control-plane) -> managed-agent cluster Secret (on vcluster-control-plane) -> rathole Deployment (on vcluster) -> rathole Container (running on local machine) -> argocd-agent 'principal' OS process (local machine)
14+
15+
16+
17+
### How to enable
18+
19+
To enable the configuration:
20+
21+
```
22+
# Setup environment as usual
23+
make setup-e2e2
24+
25+
# Installs Rathole Deployment/Services on vcluster, update 'managed-agent' cluster Secret to point to Rathole Deployment, and start local rathole container
26+
./hack/dev-env/reverse-tunnel/setup.sh
27+
28+
```
29+
30+
To test the configuration:
31+
32+
```
33+
# Start E2E tests
34+
make start-e2e2
35+
```
36+
37+
38+
39+
### Configuration
40+
41+
Running 'setup-e2e2', followed by 'reverse-tunnel/setup.sh', enables this configuration:
42+
43+
44+
'vcluster-control-plane' vcluster (on k8s cluster):
45+
- argocd namespace:
46+
- Argo CD workloads
47+
- Argo CD Server (no change)
48+
- Argo CD 'agent-managed' Cluster Secret
49+
- `.data.server` field points to Service, `https://rathole-container-internal:9090?agentName=agent-managed`
50+
- `.data.config` field is updated to enable insecure mode, and remove `caData` from TLS config
51+
- K8s Service
52+
- `rathole-container-internal`: internal service that directs traffic on port 9090 to `rathole` Deployment (below)
53+
54+
- Rathole:
55+
- Deployment `rathole`:
56+
- redirects traffic from argocd (above) to local rathole container (below)
57+
- Service `rathole-container-external`: Redirect LoadBalancer to `rathole` Deployment (allows local machine to connect to cluster rathole)
58+
59+
Running Locally:
60+
- Principal process (no change)
61+
- resource-proxy: traffic is redirect from cluster rathole Deployment to here
62+
- `rathole` container: Ultimately redirects traffic from remote argocd server (via cluster rathole) to local resource-proxy in principal process

‎hack/dev-env/reverse-tunnel/client/client.toml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# client.toml
2+
[client]
3+
remote_addr = "EXTERNAL-HOSTNAME:2333" # The address of the server. The port must be the same with the port in `server.bind_addr`
4+
5+
[client.services.one]
6+
token = "AUTHENTICATION_TOKEN" # Must be the same with the server to pass the validation
7+
local_addr = "127.0.0.1:9090" # The address of the service that needs to be forwarded
8+
9+
# Client Side Configuration
10+
[client.transport]
11+
type = "noise"
12+
13+
[client.transport.noise]
14+
remote_public_key = "REMOTE_PUBLIC_KEY"
15+

‎hack/dev-env/reverse-tunnel/client/run.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/bash
2+
3+
SCRIPTPATH="$(
4+
cd -- "$(dirname "$0")" >/dev/null 2>&1 || exit
5+
pwd -P
6+
)"
7+
8+
docker run --network host --name rathole-proxy -it --rm -v "$SCRIPTPATH/client.toml:/app/config.toml" "quay.io/jgwest-redhat/rathole:latest@sha256:53999f80b69f9a5020e19e9c9be90fc34b973d9bd822d4fd44b968f2ebe0845f" --client /app/config.toml
9+
# Container image is built from 'rathole-image' directory
10+
11+
# or, without docker:
12+
#
13+
# rathole --client $SCRIPTPATH/config.toml
14+

‎hack/dev-env/reverse-tunnel/rathole-image/artifacts/Dockerfile.new‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
FROM rust:1.79-bookworm AS builder
2+
RUN apt update && apt install -y libssl-dev
3+
WORKDIR /home/rust/src
4+
COPY . .
5+
ARG FEATURES
6+
RUN cargo build --locked --release --features ${FEATURES:-default}
7+
RUN mkdir -p build-out/
8+
RUN cp target/release/rathole build-out/
9+
10+
11+
12+
FROM gcr.io/distroless/cc-debian12
13+
WORKDIR /app
14+
COPY --from=builder /home/rust/src/build-out/rathole .
15+
USER 1000:1000
16+
ENTRYPOINT ["./rathole"]

‎hack/dev-env/reverse-tunnel/rathole-image/build-and-push.sh‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/bin/bash
2+
3+
# Build and push a multi-arch docker container containing latest rathole release built from source.
4+
5+
# I needed to run this first:
6+
# - docker buildx create --name mybuilder --driver docker-container --use
7+
# I'm on Linux x64 (Fedora)
8+
9+
REPO=quay.io/jgwest-redhat
10+
11+
set -ex
12+
13+
SCRIPTPATH="$(
14+
cd -- "$(dirname "$0")" >/dev/null 2>&1 || exit
15+
pwd -P
16+
)"
17+
18+
TEMP_DIR=$(mktemp -d)
19+
20+
cd $TEMP_DIR
21+
git clone https://github.com/yujqiao/rathole
22+
23+
cd rathole
24+
25+
git checkout ebb764ae53d7ffe4fcb45f83f7563bec5c74199d # v0.5.0
26+
27+
cp $SCRIPTPATH/artifacts/Dockerfile.new $TEMP_DIR/rathole/Dockerfile
28+
29+
docker buildx build -t $REPO/rathole:latest --platform linux/amd64,linux/arm64 --push .
30+

‎hack/dev-env/reverse-tunnel/server/deployment.yaml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: rathole-container
5+
spec:
6+
replicas: 1
7+
revisionHistoryLimit: 3
8+
selector:
9+
matchLabels:
10+
app: rathole-container
11+
template:
12+
metadata:
13+
labels:
14+
app: rathole-container
15+
spec:
16+
containers:
17+
- image: 'quay.io/jgwest-redhat/rathole:latest@sha256:53999f80b69f9a5020e19e9c9be90fc34b973d9bd822d4fd44b968f2ebe0845f' # built by rathole-image directory
18+
name: rathole-container
19+
ports:
20+
- containerPort: 2333
21+
protocol: TCP
22+
- containerPort: 9090
23+
protocol: TCP
24+
command:
25+
- /app/rathole
26+
args:
27+
- '--server'
28+
- /config/server.toml
29+
volumeMounts:
30+
- name: config-volume
31+
mountPath: /config
32+
volumes:
33+
- name: config-volume
34+
secret:
35+
secretName: rathole-container

‎hack/dev-env/reverse-tunnel/server/kustomization.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
resources:
5+
- deployment.yaml
6+
- service.yaml
7+
- service-internal.yaml
8+
9+
secretGenerator:
10+
- name: rathole-container
11+
files:
12+
- server.toml

‎hack/dev-env/reverse-tunnel/server/server.toml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# server.toml
2+
[server]
3+
bind_addr = "0.0.0.0:2333" # `2333` specifies the port that rathole listens for clients
4+
5+
[server.services.one]
6+
token = "AUTHENTICATION_TOKEN" # Token that is used to authenticate the client for the service. Change to a arbitrary value.
7+
bind_addr = "0.0.0.0:9090" # `9090` specifies the port that exposes `one` to the Internet
8+
9+
# Server Side Configuration
10+
[server.transport]
11+
type = "noise"
12+
13+
[server.transport.noise]
14+
local_private_key = "LOCAL_PRIVATE_KEY"

‎hack/dev-env/reverse-tunnel/server/service-internal.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: rathole-container-internal
5+
spec:
6+
ports:
7+
- name: "9090"
8+
port: 9090
9+
targetPort: 9090
10+
protocol: TCP
11+
selector:
12+
app: rathole-container

‎hack/dev-env/reverse-tunnel/server/service.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: rathole-container-external
5+
spec:
6+
selector:
7+
app: rathole-container
8+
# type: NodePort
9+
type: LoadBalancer
10+
ports:
11+
- protocol: TCP
12+
name: "2333"
13+
port: 2333 # The port the service listens on inside the cluster
14+
targetPort: 2333 # The port your application is listening on in the pod
15+
# nodePort: 30000 # The static port exposed on each node (optional, Kubernetes assigns if omitted)

0 commit comments

Comments
 (0)