Skip to content

Commit 5e3f58e

Browse files
elijah-rouelijah-rou
committed
feat: Add DaemonSet mount mode for shared StorageClass mount pods
## Summary Introduces a new DaemonSet deployment mode for JuiceFS mount pods when using StorageClass with mount sharing enabled. This feature provides better resource management by deploying mount pods as DaemonSets instead of individual shared pods, with configurable node affinity control through ConfigMaps. ## Key Features - **Three mount modes**: per-pvc (default), shared-pod, and daemonset - **ConfigMap-based configuration**: Control mount mode and node affinity via `juicefs-mount-config` ConfigMap - **Node affinity support**: Restrict DaemonSet deployment to specific nodes using standard Kubernetes node affinity - **Automatic fallback**: Falls back to shared-pod mode if DaemonSet cannot schedule on a node - **Seamless transition**: Works with existing StorageClasses without modification ## Implementation Details ### Core Components - `pkg/juicefs/mount/mount_selector.go`: Dynamic mount type selection based on configuration - `pkg/juicefs/mount/daemonset_mount.go`: DaemonSet mount implementation with scheduling error handling - `pkg/config/mount_config.go`: ConfigMap parsing for mount mode and node affinity - `pkg/config/mount_config_helper.go`: Helper functions for DaemonSet configuration - `pkg/juicefs/mount/builder/daemonset.go`: DaemonSet resource builder ### Configuration Mount modes are configured via ConfigMap in kube-system namespace: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: juicefs-mount-config namespace: kube-system data: default: | mode: shared-pod # Options: per-pvc, shared-pod, daemonset my-storageclass: | mode: daemonset nodeAffinity: # Required for daemonset mode requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node.kubernetes.io/workload operator: In values: ["compute"] ``` ### Changes to Existing Components - **pod_driver.go**: Skip DaemonSet pods in deletion/recreation handlers - **juicefs.go**: Add MountSelector for dynamic mount type selection and AuthFs serialization - **k8sclient**: Add DaemonSet CRUD operations - **RBAC**: Add DaemonSet permissions to CSI node service account ### Documentation & Examples - `docs/en/guide/daemonset-mount.md`: Comprehensive usage documentation - `docs/en/guide/mount-pod-configuration.md`: Mount pod configuration guide - `deploy/kubernetes/csi-daemonset-mount/`: Example DaemonSet configurations - `deploy/kubernetes/mount-config/`: Example ConfigMap configurations ### Testing - Unit tests for mount selector logic - Unit tests for DaemonSet mount implementation - ConfigMap parsing and validation tests ## Benefits 1. **Better resource utilization**: One mount pod per node instead of per PVC 2. **Improved control**: Node affinity ensures mount pods only run where needed 3. **Simplified operations**: Easier to manage and monitor as DaemonSets 4. **Automatic lifecycle**: DaemonSet controller handles pod creation/deletion 5. **Backward compatible**: Works with existing StorageClasses and mount sharing ## Migration Path 1. Enable `STORAGE_CLASS_SHARE_MOUNT` in CSI Controller and Node 2. Grant DaemonSet RBAC permissions (included in updated manifests) 3. Create ConfigMap with desired mount mode configuration 4. New PVCs will automatically use configured mount mode ## Deployment Updates - Updated `deploy/k8s.yaml` and `deploy/k8s_before_v1_18.yaml` with DaemonSet RBAC - Added DaemonSet resource configurations - Updated Docker build process for consistency This feature enhances the JuiceFS CSI Driver's flexibility in managing mount pods, particularly beneficial for large-scale deployments where mount pod proliferation can become a resource management challenge.
1 parent 81d23bf commit 5e3f58e

30 files changed

+4859
-80
lines changed

‎.github/workflows/ci_release.yml‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
name: JuiceFS CSI Release
2+
on:
3+
push:
4+
branches:
5+
- master
6+
- release-*
7+
8+
jobs:
9+
release:
10+
runs-on: ubuntu-latest
11+
outputs:
12+
version: ${{ steps.version.outputs.version }}
13+
steps:
14+
- name: "Checkout Code"
15+
uses: actions/checkout@v4
16+
with:
17+
fetch-depth: 0
18+
19+
- name: Determine version
20+
id: version
21+
run: |
22+
# Get the latest tag that matches v* pattern
23+
LATEST_TAG=$(git describe --tags --match 'v*' --abbrev=0 2>/dev/null || echo "v0.0.0")
24+
25+
# Extract major, minor, patch
26+
VERSION=${LATEST_TAG#v}
27+
MAJOR=$(echo $VERSION | cut -d. -f1)
28+
MINOR=$(echo $VERSION | cut -d. -f2)
29+
PATCH=$(echo $VERSION | cut -d. -f3)
30+
31+
# Increment patch version
32+
PATCH=$((PATCH + 1))
33+
NEXT_VERSION="v${MAJOR}.${MINOR}.${PATCH}"
34+
35+
echo "version=${NEXT_VERSION}" >> $GITHUB_OUTPUT
36+
echo "Next version: ${NEXT_VERSION}"
37+
38+
- name: Create tag
39+
run: |
40+
git config --local user.email "action@github.com"
41+
git config --local user.name "GitHub Action"
42+
git tag -a ${{ steps.version.outputs.version }} -m "Release ${{ steps.version.outputs.version }}"
43+
git push origin ${{ steps.version.outputs.version }}
44+
45+
build-csi:
46+
needs: release
47+
runs-on: ubuntu-latest
48+
strategy:
49+
matrix:
50+
region:
51+
- name: us-east-1
52+
identifier: virginia
53+
- name: eu-west-2
54+
identifier: london
55+
steps:
56+
- name: "Checkout"
57+
uses: actions/checkout@v4
58+
59+
- name: Configure AWS credentials
60+
uses: aws-actions/configure-aws-credentials@v4
61+
with:
62+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
63+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
64+
aws-region: ${{ matrix.region.name }}
65+
66+
- name: Login to ECR
67+
id: login-ecr
68+
uses: aws-actions/amazon-ecr-login@v2
69+
70+
- uses: depot/setup-action@v1
71+
72+
- name: Build and Push CSI Driver
73+
uses: depot/build-push-action@v1
74+
with:
75+
platforms: linux/amd64,linux/arm64
76+
file: docker/csi.Dockerfile
77+
context: "{{defaultContext}}"
78+
build-args: |
79+
JFSCHAN=stable
80+
JUICEFS_CE_MOUNT_IMAGE=juicedata/mount:ce-latest
81+
JUICEFS_EE_MOUNT_IMAGE=juicedata/mount:ee-5.2.8-1305e8c
82+
provenance: false
83+
token: ${{ secrets.DEPOT_TOKEN }}
84+
push: true
85+
tags: |
86+
${{ steps.login-ecr.outputs.registry }}/juicefs-csi-driver-${{ matrix.region.identifier }}:${{ needs.release.outputs.version }}
87+
${{ steps.login-ecr.outputs.registry }}/juicefs-csi-driver-${{ matrix.region.identifier }}:latest
88+
89+
# build-mount-images:
90+
# needs: release
91+
# runs-on: ubuntu-latest
92+
# strategy:
93+
# matrix:
94+
# region:
95+
# - name: us-east-1
96+
# identifier: virginia
97+
# - name: eu-west-2
98+
# identifier: london
99+
# edition:
100+
# - name: ce
101+
# dockerfile: ce.juicefs.Dockerfile
102+
# build-args: |
103+
# CEJUICEFS_VERSION=latest
104+
# - name: ee
105+
# dockerfile: ee.juicefs.Dockerfile
106+
# build-args: |
107+
# EEJUICEFS_VERSION=5.2.8-1305e8c
108+
# steps:
109+
# - name: "Checkout"
110+
# uses: actions/checkout@v4
111+
#
112+
# - name: Configure AWS credentials
113+
# uses: aws-actions/configure-aws-credentials@v4
114+
# with:
115+
# aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
116+
# aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
117+
# aws-region: ${{ matrix.region.name }}
118+
#
119+
# - name: Login to ECR
120+
# id: login-ecr
121+
# uses: aws-actions/amazon-ecr-login@v2
122+
#
123+
# - uses: depot/setup-action@v1
124+
#
125+
# - name: Build and Push Mount Image (${{ matrix.edition.name }})
126+
# uses: depot/build-push-action@v1
127+
# with:
128+
# platforms: linux/amd64,linux/arm64
129+
# context: docker
130+
# file: docker/${{ matrix.edition.dockerfile }}
131+
# build-args: ${{ matrix.edition.build-args }}
132+
# provenance: false
133+
# token: ${{ secrets.DEPOT_TOKEN }}
134+
# push: true
135+
# tags: |
136+
# ${{ steps.login-ecr.outputs.registry }}/juicefs-mount-${{ matrix.edition.name }}-${{ matrix.region.identifier }}:${{ needs.release.outputs.version }}
137+
# ${{ steps.login-ecr.outputs.registry }}/juicefs-mount-${{ matrix.edition.name }}-${{ matrix.region.identifier }}:latest
138+
#
139+
# build-dashboard:
140+
# needs: release
141+
# runs-on: ubuntu-latest
142+
# strategy:
143+
# matrix:
144+
# region:
145+
# - name: us-east-1
146+
# identifier: virginia
147+
# - name: eu-west-2
148+
# identifier: london
149+
# steps:
150+
# - name: "Checkout"
151+
# uses: actions/checkout@v4
152+
#
153+
# - uses: pnpm/action-setup@v4
154+
# name: Install pnpm
155+
# with:
156+
# version: 9
157+
# run_install: |
158+
# cwd: dashboard-ui-v2
159+
#
160+
# - name: Build dashboard
161+
# run: make dashboard-dist
162+
#
163+
# - name: Configure AWS credentials
164+
# uses: aws-actions/configure-aws-credentials@v4
165+
# with:
166+
# aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
167+
# aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
168+
# aws-region: ${{ matrix.region.name }}
169+
#
170+
# - name: Login to ECR
171+
# id: login-ecr
172+
# uses: aws-actions/amazon-ecr-login@v2
173+
#
174+
# - uses: depot/setup-action@v1
175+
#
176+
# - name: Build and Push Dashboard
177+
# uses: depot/build-push-action@v1
178+
# with:
179+
# platforms: linux/amd64,linux/arm64
180+
# context: .
181+
# file: docker/dashboard.Dockerfile
182+
# build-contexts: |
183+
# project=.
184+
# ui=./dashboard-ui-v2
185+
# provenance: false
186+
# token: ${{ secrets.DEPOT_TOKEN }}
187+
# push: true
188+
# tags: |
189+
# ${{ steps.login-ecr.outputs.registry }}/juicefs-csi-dashboard-${{ matrix.region.identifier }}:${{ needs.release.outputs.version }}
190+
# ${{ steps.login-ecr.outputs.registry }}/juicefs-csi-dashboard-${{ matrix.region.identifier }}:latest
191+
192+
notify:
193+
needs: [release, build-csi]
194+
runs-on: ubuntu-latest
195+
if: always() && needs.build-csi.result == 'success' && needs.build-mount-images.result == 'success' && needs.build-dashboard.result == 'success'
196+
steps:
197+
- name: Notify Slack!
198+
uses: someimportantcompany/github-actions-slack-message@v1
199+
with:
200+
webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
201+
text: ":rocket: JuiceFS CSI Driver ${{ needs.release.outputs.version }} images built and pushed to all ECR registries! :rocket:"

‎.github/workflows/ci_update_dev.yml‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
name: JuiceFS CSI Dev CI/CD
2+
on:
3+
pull_request:
4+
branches:
5+
- master
6+
- release-*
7+
push:
8+
branches:
9+
- development
10+
- dev
11+
12+
jobs:
13+
build-csi:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: "Checkout"
17+
uses: actions/checkout@v4
18+
19+
- name: Configure AWS credentials
20+
uses: aws-actions/configure-aws-credentials@v4
21+
with:
22+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
23+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
24+
aws-region: us-east-1
25+
26+
- name: Login to ECR
27+
id: login-ecr
28+
uses: aws-actions/amazon-ecr-login@v2
29+
30+
- uses: depot/setup-action@v1
31+
32+
- name: Build and Push CSI Driver
33+
uses: depot/build-push-action@v1
34+
with:
35+
platforms: linux/amd64,linux/arm64
36+
context: .
37+
file: docker/csi.Dockerfile
38+
build-contexts: |
39+
project=.
40+
ui=./dashboard-ui-v2
41+
build-args: |
42+
TARGETARCH=amd64
43+
JFSCHAN=stable
44+
JUICEFS_CE_MOUNT_IMAGE=juicedata/mount:ce-latest
45+
JUICEFS_EE_MOUNT_IMAGE=juicedata/mount:ee-5.2.8-1305e8c
46+
provenance: false
47+
token: ${{ secrets.DEPOT_TOKEN }}
48+
push: true
49+
tags: |
50+
${{ steps.login-ecr.outputs.registry }}/juicefs-csi-driver-dev-virginia:dev
51+
${{ steps.login-ecr.outputs.registry }}/juicefs-csi-driver-dev-virginia:pr-${{ github.event.pull_request.number || 'push' }}
52+
53+
build-mount-images:
54+
runs-on: ubuntu-latest
55+
strategy:
56+
matrix:
57+
edition:
58+
- name: ce
59+
dockerfile: ce.juicefs.Dockerfile
60+
build-args: |
61+
CEJUICEFS_VERSION=latest
62+
- name: ee
63+
dockerfile: ee.juicefs.Dockerfile
64+
build-args: |
65+
EEJUICEFS_VERSION=5.2.8-1305e8c
66+
steps:
67+
- name: "Checkout"
68+
uses: actions/checkout@v4
69+
70+
- name: Configure AWS credentials
71+
uses: aws-actions/configure-aws-credentials@v4
72+
with:
73+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
74+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
75+
aws-region: us-east-1
76+
77+
- name: Login to ECR
78+
id: login-ecr
79+
uses: aws-actions/amazon-ecr-login@v2
80+
81+
- uses: depot/setup-action@v1
82+
83+
- name: Build and Push Mount Image (${{ matrix.edition.name }})
84+
uses: depot/build-push-action@v1
85+
with:
86+
platforms: linux/amd64,linux/arm64
87+
context: docker
88+
file: docker/${{ matrix.edition.dockerfile }}
89+
build-args: ${{ matrix.edition.build-args }}
90+
provenance: false
91+
token: ${{ secrets.DEPOT_TOKEN }}
92+
push: true
93+
tags: |
94+
${{ steps.login-ecr.outputs.registry }}/juicefs-mount-${{ matrix.edition.name }}-dev-virginia:dev
95+
${{ steps.login-ecr.outputs.registry }}/juicefs-mount-${{ matrix.edition.name }}-dev-virginia:pr-${{ github.event.pull_request.number || 'push' }}
96+
97+
build-dashboard:
98+
runs-on: ubuntu-latest
99+
steps:
100+
- name: "Checkout"
101+
uses: actions/checkout@v4
102+
103+
- uses: pnpm/action-setup@v4
104+
name: Install pnpm
105+
with:
106+
version: 9
107+
run_install: |
108+
cwd: dashboard-ui-v2
109+
110+
- name: Build dashboard
111+
run: make dashboard-dist
112+
113+
- name: Configure AWS credentials
114+
uses: aws-actions/configure-aws-credentials@v4
115+
with:
116+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
117+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
118+
aws-region: us-east-1
119+
120+
- name: Login to ECR
121+
id: login-ecr
122+
uses: aws-actions/amazon-ecr-login@v2
123+
124+
- uses: depot/setup-action@v1
125+
126+
- name: Build and Push Dashboard
127+
uses: depot/build-push-action@v1
128+
with:
129+
platforms: linux/amd64,linux/arm64
130+
context: .
131+
file: docker/dashboard.Dockerfile
132+
build-contexts: |
133+
project=.
134+
ui=./dashboard-ui-v2
135+
provenance: false
136+
token: ${{ secrets.DEPOT_TOKEN }}
137+
push: true
138+
tags: |
139+
${{ steps.login-ecr.outputs.registry }}/juicefs-csi-dashboard-dev-virginia:dev
140+
${{ steps.login-ecr.outputs.registry }}/juicefs-csi-dashboard-dev-virginia:pr-${{ github.event.pull_request.number || 'push' }}
141+
142+
notify:
143+
needs: [build-csi, build-mount-images, build-dashboard]
144+
runs-on: ubuntu-latest
145+
if: always() && needs.build-csi.result == 'success' && needs.build-mount-images.result == 'success' && needs.build-dashboard.result == 'success'
146+
steps:
147+
- name: Notify Slack!
148+
uses: someimportantcompany/github-actions-slack-message@v1
149+
with:
150+
webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
151+
text: ":construction: JuiceFS CSI Driver dev images built and pushed to ECR! :construction:"

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,4 @@ cov2.out
2020
yalc.lock
2121
.ropeproject
2222
dist/
23+
*.test

‎.markdownlint-cli2.jsonc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@
4747
},
4848
"link-fragments": false,
4949
"no-trailing-slash-in-links": true,
50+
"MD029": false,
5051
"enhanced-proper-names": {
5152
"code_blocks": false,
5253
"html_elements": false,

‎deploy/k8s.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,7 +255,19 @@ rules:
255255
resources:
256256
- nodes
257257
verbs:
258+
- get
259+
- list
260+
- apiGroups:
261+
- apps
262+
resources:
263+
- daemonsets
264+
verbs:
265+
- get
258266
- list
267+
- watch
268+
- create
269+
- update
270+
- patch
259271
---
260272
apiVersion: rbac.authorization.k8s.io/v1
261273
kind: ClusterRole

0 commit comments

Comments
 (0)