diff --git a/.github/workflows/buildAndTest.yaml b/.github/workflows/buildAndTest.yaml index bd9cb2d..16c9cec 100644 --- a/.github/workflows/buildAndTest.yaml +++ b/.github/workflows/buildAndTest.yaml @@ -7,7 +7,7 @@ on: branches: [ "master" ] jobs: - build: + build-and-test: runs-on: ubuntu-latest diff --git a/charts/kubeocean/templates/crds/cloud.tencent.com_clusterbindings.yaml b/charts/kubeocean/templates/crds/cloud.tencent.com_clusterbindings.yaml index 51bd602..52114e8 100644 --- a/charts/kubeocean/templates/crds/cloud.tencent.com_clusterbindings.yaml +++ b/charts/kubeocean/templates/crds/cloud.tencent.com_clusterbindings.yaml @@ -17,6 +17,9 @@ spec: scope: Cluster versions: - additionalPrinterColumns: + - jsonPath: .spec.clusterID + name: ClusterID + type: string - jsonPath: .status.phase name: Phase type: string diff --git a/docs/quick-start.md b/docs/quick-start.md index b3e54f3..eaf4e65 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -1,6 +1,11 @@ +--- +cwd: ../ +--- + # Quick Start This document introduces: + - Deploying kubeocean components in a local KIND (kubernetes in docker) cluster - Binding two worker clusters into kubeocean and extracting computing resources to form virtual computing nodes - Creating Pods on computing nodes that can work normally @@ -16,42 +21,53 @@ This document introduces: ## Build Environment and Deploy kubeocean Components 1. Clone the repository and enter the directory -``` + +```sh git clone https://github.com/gocrane/kubeocean cd kubeocean ``` 2. Modify inotify kernel parameters to support KIND multi-cluster -``` + +```sh sudo sysctl fs.inotify.max_user_watches=524288 sudo sysctl fs.inotify.max_user_instances=512 ``` 3. Build 3 KIND clusters locally -``` + +```sh make kind-create-all ``` + The above command will create 3 k8s clusters locally, named kubeocean-manager, kubeocean-worker1 and kubeocean-worker2. You can use the following command to switch between different cluster contexts: -``` -# can be kubeocean-manager, kubeocean-worker1 and kubeocean-worker2 -kubectl config use-context kind- + +```sh +# CLUSTER_NAME 可为 kubeocean-manager,kubeocean-worker1 和 kubeocean-worker2 +export CLUSTER_NAME=kubeocean-worker1 +kubectl config use-context kind-$CLUSTER_NAME ``` 4. Deploy kubernetes-intranet and kube-dns-intranet Services -``` + +```sh make kind-deploy-pre ``` + The above command will deploy kubernetes-intranet and kube-dns-intranet Services in the created kubeocean-manager cluster to prepare for kubeocean component deployment and usage. 5. Deploy kubeocean components in kubeocean-manager cluster -``` + +```sh # Load images KIND_CLUSTER_NAME=kubeocean-manager make kind-load-images # Switch to manager cluster and deploy components kubectl config use-context kind-kubeocean-manager +# Get current version +version=$(git describe --tags --always --dirty)-amd64 + # Install components using helm -version=$(git describe --tags --always --dirty) helm upgrade --install kubeocean charts/kubeocean \ --set global.imageRegistry="ccr.ccs.tencentyun.com/tke-eni-test" \ --set manager.image.tag=${version} \ @@ -59,16 +75,23 @@ helm upgrade --install kubeocean charts/kubeocean \ --set proxier.image.tag=${version} \ --wait # Or use preset make command to install -make install-manager +INSTALL_IMG_TAG=${version} make install-manager ``` ## Bind Worker Clusters and Extract Computing Nodes -**Note: Replace kubeocean-worker1 with kubeocean-worker2 in the following commands to complete worker2 cluster binding** +0. Set environment variables -1. Deploy kubeocean-worker in worker cluster +```sh +export CLUSTER_NAME=kubeocean-worker1 +export CLUSTERID=cls-worker1 +# Set CLUSTER_NAME to kubeocean-worker2 and CLUSTERID to cls-worker2, then re-execute to complete the second worker cluster registration ``` -kubectl config use-context kind-kubeocean-worker1 + +1. Deploy kubeocean-worker in worker cluster + +```sh +kubectl config use-context kind-$CLUSTER_NAME # Install using helm helm upgrade --install kubeocean-worker charts/kubeocean-worker --wait # Or use preset make command to install @@ -76,30 +99,33 @@ make install-worker ``` 2. Extract kubeconfig from kubeocean-worker -``` + +```sh # Use script to extract kubeconfig -bash hack/kubeconfig.sh kubeocean-syncer kubeocean-worker /tmp/kubeconfig-worker1 +bash hack/kubeconfig.sh kubeocean-syncer kubeocean-worker /tmp/kubeconfig-$CLUSTER_NAME # Replace APIServer's localhost address with corresponding docker container address -WORKER1_IP=$(docker inspect kubeocean-worker1-control-plane --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}') -sed -i "s|server:.*|server: \"https://${WORKER1_IP}:6443\"|" /tmp/kubeconfig-worker1 +WORKER1_IP=$(docker inspect $CLUSTER_NAME-control-plane --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}') +sed -i "s|server:.*|server: \"https://${WORKER1_IP}:6443\"|" /tmp/kubeconfig-$CLUSTER_NAME ``` 3. Create related secrets in manager cluster -``` + +```sh kubectl config use-context kind-kubeocean-manager -kubectl -nkubeocean-system create secret generic worker1-cluster-kubeconfig --from-file=kubeconfig=/tmp/kubeconfig-worker1 +kubectl -nkubeocean-system create secret generic $CLUSTER_NAME-kubeconfig --from-file=kubeconfig=/tmp/kubeconfig-$CLUSTER_NAME ``` 4. Bind worker cluster -``` -# cb1.yaml + +```sh +cat > cb.yaml << EOF apiVersion: cloud.tencent.com/v1beta1 kind: ClusterBinding metadata: - name: cb-worker1 + name: cb-$CLUSTER_NAME namespace: kubeocean-system spec: - clusterID: cls-worker1 + clusterID: $CLUSTERID mountNamespace: kubeocean-worker nodeSelector: nodeSelectorTerms: @@ -109,37 +135,48 @@ spec: values: - worker secretRef: - name: worker1-cluster-kubeconfig + name: $CLUSTER_NAME-kubeconfig namespace: kubeocean-system +EOF + ``` + Create the above clusterbinding object in manager cluster: -``` + +```sh kubectl config use-context kind-kubeocean-manager -kubectl apply -f cb1.yaml +kubectl apply -f cb.yaml ``` + After the above command is executed, you can check if the corresponding clusterbinding status is Ready: + +```sh +kubectl get cb cb-$CLUSTER_NAME ``` -kubectl get cb cb-worker1 -``` + Expected execution result: + +```sh +NAME PHASE AGE +cb-kubeocean-worker1 Ready Xs ``` -NAME CLUSTERID PHASE -cb-worker1 cls-worker1 Ready -``` + At the same time, after cluster binding, corresponding worker and proxier pods will be synchronously created in the kubeocean-system namespace, which can be viewed with the following command: -``` + +```sh kubectl -nkubeocean-system get po -owide ``` 5. Extract computing resources to form virtual nodes -``` -# rlp1.yaml + +```sh +cat > rlp.yaml << EOF apiVersion: cloud.tencent.com/v1beta1 kind: ResourceLeasingPolicy metadata: - name: rlp-worker1 + name: rlp-$CLUSTER_NAME spec: - cluster: cb-worker1 + cluster: cb-$CLUSTER_NAME forceReclaim: true nodeSelector: nodeSelectorTerms: @@ -156,19 +193,26 @@ spec: percent: 80 # Take the smaller of 4 CPUs or 80% of available CPUs - resource: memory percent: 90 # Take 90% of available memory +EOF ``` + Create the above ResourceLeasingPolicy object in worker1 cluster to extract computing nodes: + +```sh +kubectl config use-context kind-$CLUSTER_NAME +kubectl apply -f rlp.yaml ``` -kubectl config use-context kind-kubeocean-worker1 -kubectl apply -f rlp1.yaml -``` + After the above command is executed, you can observe in the manager cluster whether computing nodes are extracted normally: -``` + +```sh kubectl config use-context kind-kubeocean-manager kubectl get node ``` + If nodes starting with vnode are created, it means computing resource extraction is successful: -``` + +```sh NAME STATUS ROLES AGE VERSION kubeocean-manager-control-plane Ready control-plane 92m v1.28.0 kubeocean-manager-worker Ready 91m v1.28.0 @@ -179,8 +223,8 @@ vnode-cls-worker1-kubeocean-worker1-worker2 Ready 5m v1.2 ## Create and Deploy Sample Pod -``` -# job.yaml +```sh +cat > job.yaml << EOF kind: Job apiVersion: batch/v1 metadata: @@ -199,9 +243,12 @@ spec: tolerations: - operator: Exists key: kubeocean.io/vnode +EOF ``` + Deploy the above job in manager cluster. You can cordon non-virtual nodes for better verification effect: -``` + +```sh # Pull image docker pull busybox:latest bin/kind load docker-image busybox:latest --name kubeocean-worker1 @@ -211,8 +258,16 @@ kubectl config use-context kind-kubeocean-manager kubectl cordon kubeocean-manager-control-plane kubeocean-manager-worker kubeocean-manager-worker2 kubectl create -f job.yaml ``` -Use `kubectl get po -owide -w` to view the results. You can observe that the job can run and complete normally: + +After deployment, use `kubectl` to view the results + +```sh +kubectl get po -owide -w ``` + +You can observe that the job can run and complete normally: + +```sh NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES test-job-9ln8m 0/1 ContainerCreating 0 3s vnode-cls-worker1-kubeocean-worker1-worker2 test-job-9ln8m 1/1 Running 0 8s 10.242.1.2 vnode-cls-worker1-kubeocean-worker1-worker2 diff --git a/docs/quick-start_zh.md b/docs/quick-start_zh.md index 9e07202..6260ee0 100644 --- a/docs/quick-start_zh.md +++ b/docs/quick-start_zh.md @@ -1,6 +1,11 @@ +--- +cwd: ../ +--- + # 快速开始 本文档介绍了: + - 在本地的 KIND(kubernetes in docker)集群中部署 kubeocean 组件 - 将两个业务集群绑定进 kubeocean,并抽取算力资源形成虚拟算力节点 - 在算力节点上创建 Pod,并能正常工作 @@ -10,45 +15,59 @@ - git - kubectl v1.28+ - docker - - go v1.24+ - helm v3 ## 构建环境并部署 kubeocean 组件 1. 克隆代码仓库并进入目录 -``` + +```sh git clone https://github.com/gocrane/kubeocean cd kubeocean ``` + 2. 修改 inotify 内核参数以支持 KIND 多集群 -``` + +```sh sudo sysctl fs.inotify.max_user_watches=524288 sudo sysctl fs.inotify.max_user_instances=512 ``` + 3. 在本地构建3个 KIND 集群 -``` + +```sh make kind-create-all ``` + 以上命令会在本地创建3个 k8s 集群,名称分别为 kubeocean-manager,kubeocean-worker1 和 kubeocean-worker2。 可以使用以下命令切换不同集群的 context + +```sh +# CLUSTER_NAME 可为 kubeocean-manager,kubeocean-worker1 和 kubeocean-worker2 +export CLUSTER_NAME=kubeocean-worker1 +kubectl config use-context kind-$CLUSTER_NAME ``` -# 可为 kubeocean-manager,kubeocean-worker1 和 kubeocean-worker2 -kubectl config use-context kind- -``` + 4. 部署 kubernetes-intranet 和 kube-dns-intranet Service -``` + +```sh make kind-deploy-pre ``` + 以上命令会在上述创建的 kubeocean-manager 集群中部署 kubernetes-intranet 和 kube-dns-intranet Service,为 kubeocean 组件部署和使用做准备。 + 5. 在 kubeocean-manager 集群部署 kubeocean 组件 -``` + +```sh # 加载镜像 KIND_CLUSTER_NAME=kubeocean-manager make kind-load-images # 切换到 manager 集群并部署组件 kubectl config use-context kind-kubeocean-manager +# 获取当前版本 +version=$(git describe --tags --always --dirty)-amd64 + # 使用 helm 安装组件 -version=$(git describe --tags --always --dirty) helm upgrade --install kubeocean charts/kubeocean \ --set global.imageRegistry="ccr.ccs.tencentyun.com/tke-eni-test" \ --set manager.image.tag=${version} \ @@ -56,43 +75,57 @@ helm upgrade --install kubeocean charts/kubeocean \ --set proxier.image.tag=${version} \ --wait # 或者使用预置 make 命令安装 -make install-manager +INSTALL_IMG_TAG=${version} make install-manager ``` ## 绑定业务集群并抽取算力节点 -**注:下述命令将 kubeocean-worker1 替换成 kubeocean-worker2 即可完成 worker2 集群的绑定** -1. 在 worker 集群部署 kubeocean-worker +0. 设置环境变量 + +```sh +export CLUSTER_NAME=kubeocean-worker1 +export CLUSTERID=cls-worker1 +# 将CLUSTER_NAME设为kubeocean-worker2,CLUSTERID设为cls-worker2,再重新执行即可完成第二个业务集群注册 ``` -kubectl config use-context kind-kubeocean-worker1 + +1. 在 worker 集群部署 kubeocean-worker + +```sh +kubectl config use-context kind-$CLUSTER_NAME # 使用 helm 安装 helm upgrade --install kubeocean-worker charts/kubeocean-worker --wait # 或使用预置 make 命令安装 make install-worker ``` + 2. 提取 kubeocean-worker 的 kubeconfig -``` + +```sh # 使用脚本提取 kubeconfig -bash hack/kubeconfig.sh kubeocean-syncer kubeocean-worker /tmp/kubeconfig-worker1 +bash hack/kubeconfig.sh kubeocean-syncer kubeocean-worker /tmp/kubeconfig-$CLUSTER_NAME # 用对应 docker 容器的地址替换 APIServer 的 localhost 地址 -WORKER1_IP=$(docker inspect kubeocean-worker1-control-plane --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}') -sed -i "s|server:.*|server: \"https://${WORKER1_IP}:6443\"|" /tmp/kubeconfig-worker1 +WORKER1_IP=$(docker inspect $CLUSTER_NAME-control-plane --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}') +sed -i "s|server:.*|server: \"https://${WORKER1_IP}:6443\"|" /tmp/kubeconfig-$CLUSTER_NAME ``` + 3. 在 manager 集群中创建相关 secret -``` + +```sh kubectl config use-context kind-kubeocean-manager -kubectl -nkubeocean-system create secret generic worker1-cluster-kubeconfig --from-file=kubeconfig=/tmp/kubeconfig-worker1 +kubectl -nkubeocean-system create secret generic $CLUSTER_NAME-kubeconfig --from-file=kubeconfig=/tmp/kubeconfig-$CLUSTER_NAME ``` + 4. 绑定 worker 集群 -``` -# cb1.yaml + +```sh +cat > cb.yaml << EOF apiVersion: cloud.tencent.com/v1beta1 kind: ClusterBinding metadata: - name: cb-worker1 + name: cb-$CLUSTER_NAME namespace: kubeocean-system spec: - clusterID: cls-worker1 + clusterID: $CLUSTERID mountNamespace: kubeocean-worker nodeSelector: nodeSelectorTerms: @@ -102,36 +135,48 @@ spec: values: - worker secretRef: - name: worker1-cluster-kubeconfig + name: $CLUSTER_NAME-kubeconfig namespace: kubeocean-system +EOF + ``` + 在 manager 集群中创建上述 clusterbinding 对象: -``` + +```sh kubectl config use-context kind-kubeocean-manager -kubectl apply -f cb1.yaml +kubectl apply -f cb.yaml ``` + 上述命令执行完成后,可查看对应 clusterbinding 的状态是否为 Ready: + +```sh +kubectl get cb cb-$CLUSTER_NAME ``` -kubectl get cb cb-worker1 -``` + 执行结果预期为: + +```sh +NAME PHASE AGE +cb-kubeocean-worker1 Ready Xs ``` -NAME CLUSTERID PHASE -cb-worker1 cls-worker1 Ready -``` + 同时,集群绑定后,kubeocean-system namespace 下会同步创建对应的 worker 和 proxier pod,可以通过以下命令查看: -``` + +```sh kubectl -nkubeocean-system get po -owide ``` + 5. 抽取算力资源,形成虚拟节点 -``` -# rlp1.yaml + +```sh +cat > rlp.yaml << EOF apiVersion: cloud.tencent.com/v1beta1 kind: ResourceLeasingPolicy metadata: - name: rlp-worker1 + name: rlp-$CLUSTER_NAME spec: - cluster: cb-worker1 + cluster: cb-$CLUSTER_NAME forceReclaim: true nodeSelector: nodeSelectorTerms: @@ -148,19 +193,26 @@ spec: percent: 80 # Take the smaller of 4 CPUs or 80% of available CPUs - resource: memory percent: 90 # Take 90% of available memory +EOF ``` + 在 worker1 集群中创建上述 ResourceLeasingPolicy 对象,抽取算力节点 + +```sh +kubectl config use-context kind-$CLUSTER_NAME +kubectl apply -f rlp.yaml ``` -kubectl config use-context kind-kubeocean-worker1 -kubectl apply -f rlp1.yaml -``` + 上述命令执行完后,可在 manager 集群中观察算力节点是否正常抽取: -``` + +```sh kubectl config use-context kind-kubeocean-manager kubectl get node ``` + 若有 vnode 开头的节点创建,则说明算力资源抽取成功: -``` + +```sh NAME STATUS ROLES AGE VERSION kubeocean-manager-control-plane Ready control-plane 92m v1.28.0 kubeocean-manager-worker Ready 91m v1.28.0 @@ -171,8 +223,8 @@ vnode-cls-worker1-kubeocean-worker1-worker2 Ready 5m v1.2 ## 创建和部署样例 Pod -``` -# job.yaml +```sh +cat > job.yaml << EOF kind: Job apiVersion: batch/v1 metadata: @@ -191,9 +243,12 @@ spec: tolerations: - operator: Exists key: kubeocean.io/vnode +EOF ``` + 在 manager 集群中部署上述 job,可封锁非虚拟节点从而达到更好的验证效果: -``` + +```sh # 拉取镜像 docker pull busybox:latest bin/kind load docker-image busybox:latest --name kubeocean-worker1 @@ -203,8 +258,16 @@ kubectl config use-context kind-kubeocean-manager kubectl cordon kubeocean-manager-control-plane kubeocean-manager-worker kubeocean-manager-worker2 kubectl create -f job.yaml ``` -使用 `kubectl get po -owide -w` 查看结果,可以观察到该 job 能够正常运行和结束: + +部署完之后,可用 `kubectl` 查看结果 + +```sh +kubectl get po -owide -w ``` + +可以观察到该 job 能够正常运行和结束: + +```sh NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES test-job-9ln8m 0/1 ContainerCreating 0 3s vnode-cls-worker1-kubeocean-worker1-worker2 test-job-9ln8m 1/1 Running 0 8s 10.242.1.2 vnode-cls-worker1-kubeocean-worker1-worker2 @@ -212,4 +275,4 @@ test-job-9ln8m 0/1 Completed 0 28s 10.242.1.2 vnode test-job-9ln8m 0/1 Completed 0 29s vnode-cls-worker1-kubeocean-worker1-worker2 test-job-9ln8m 0/1 Completed 0 30s 10.242.1.2 vnode-cls-worker1-kubeocean-worker1-worker2 test-job-9ln8m 0/1 Completed 0 30s 10.242.1.2 vnode-cls-worker1-kubeocean-worker1-worker2 -``` \ No newline at end of file +``` diff --git a/docs/tutorials/bindCluster.md b/docs/tutorials/bindCluster.md index 229d092..e7dd00d 100644 --- a/docs/tutorials/bindCluster.md +++ b/docs/tutorials/bindCluster.md @@ -1,18 +1,34 @@ # Binding Worker Clusters This document covers: + - Binding worker clusters to virtual computing clusters where kubeocean components are deployed - Extracting computing resources and generating virtual nodes in virtual computing clusters +## Environment Variables Setup + +First, set some environment variables to simplify subsequent operations: + +```sh +# Set cluster identifiers +export CLUSTER_NAME=example-cluster +export CLUSTER_ID=cls-example +# Set kubeconfig path +export WORKER_KUBECONFIG=/tmp/kubeconfig-worker +``` + ## Deploy kubeocean-worker in Worker Cluster 1. Clone the repository and enter the directory -``` + +```sh git clone https://github.com/gocrane/kubeocean cd kubeocean ``` + 2. Deploy kubeocean-worker in the worker cluster -``` + +```sh # Install using helm helm upgrade --install kubeocean-worker charts/kubeocean-worker ``` @@ -20,34 +36,37 @@ helm upgrade --install kubeocean-worker charts/kubeocean-worker ## Extract kubeconfig from kubeocean-worker 1. Extract kubeconfig -``` + +```sh # Use script to extract kubeconfig # Ensure current kubectl accesses the worker cluster # This script will use the APIServer address configured in the current default kubeconfig -bash hack/kubeconfig.sh kubeocean-syncer kubeocean-worker /tmp/kubeconfig-worker +bash hack/kubeconfig.sh kubeocean-syncer kubeocean-worker $WORKER_KUBECONFIG ``` + 2. Create related secret in the computing cluster -``` + +```sh # Switch kubectl to computing cluster -kubectl config use-context -kubectl -nkubeocean-system create secret generic worker-cluster-kubeconfig --from-file=kubeconfig=/tmp/kubeconfig-worker +kubectl config use-context +kubectl -nkubeocean-system create secret generic ${CLUSTER_NAME}-kubeconfig --from-file=kubeconfig=$WORKER_KUBECONFIG ``` ## Create ClusterBinding Object to Bind Worker Cluster 1. Create ClusterBinding object -Reference example: -``` -# cb.yaml +```sh +# Use heredoc to create ClusterBinding YAML +cat > cb.yaml < rlp.yaml < kubectl apply -f rlp.yaml # Add a label to nodes that expect to extract resources kubectl label node kubeocean.io/role=worker ``` -After executing the above command, observe in the manager cluster whether computing nodes are extracted normally: -``` + +2. Verify resource extraction result + +After executing the above command, observe in the computing cluster whether computing nodes are extracted normally: + +```sh # Switch kubectl to virtual computing cluster +kubectl config use-context kubectl get node ``` -If nodes starting with vnode are created, it indicates successful resource extraction, as follows: + +If nodes starting with vnode are created, it indicates successful resource extraction: + ``` NAME STATUS ROLES AGE VERSION -vnode-cls-example-node1 Ready 5m v1.28.0 -vnode-cls-example-node2 Ready 5m v1.28.0 +vnode-cls-example-node1 Ready 5m v1.28.0 +vnode-cls-example-node2 Ready 5m v1.28.0 ``` -Key features: -1. ResourceLeasingPolicy needs to be created in the worker cluster to take effect. -2. Only one ResourceLeasingPolicy object takes effect per worker cluster. If multiple are created, the earliest ResourceLeasingPolicy object takes effect. -3. nodeSelector can be empty, if empty then matches all nodes. -4. timeWindows can be empty, if empty then effective full-time. -5. resourceLimits can be empty, resources not appearing in the list are extracted at 100% of remaining resources. +## Key Features + +1. **ResourceLeasingPolicy Location**: Needs to be created in the worker cluster to take effect +2. **Uniqueness**: Only one ResourceLeasingPolicy object takes effect per worker cluster. If multiple are created, the earliest ResourceLeasingPolicy object takes effect +3. **nodeSelector**: Can be empty, if empty then matches all nodes +4. **timeWindows**: Can be empty, if empty then effective full-time +5. **resourceLimits**: Can be empty, resources not appearing in the list are extracted at 100% of remaining resources diff --git a/docs/tutorials/bindCluster_zh.md b/docs/tutorials/bindCluster_zh.md index 825e324..df56692 100644 --- a/docs/tutorials/bindCluster_zh.md +++ b/docs/tutorials/bindCluster_zh.md @@ -1,18 +1,34 @@ # 绑定工作集群 本文档介绍了: + - 绑定工作集群到已部署 kubeocean 组件的虚拟算力集群中 - 抽取算力资源,并在虚拟算力集群中生成虚拟节点 +## 环境变量设置 + +首先设置一些环境变量,简化后续操作: + +```sh +# 设置集群标识 +export CLUSTER_NAME=example-cluster +export CLUSTER_ID=cls-example +# 设置 kubeconfig 路径 +export WORKER_KUBECONFIG=/tmp/kubeconfig-worker +``` + ## 在 worker 集群部署 kubeocean-worker 1. 克隆代码仓库并进入目录 -``` + +```sh git clone https://github.com/gocrane/kubeocean cd kubeocean ``` + 2. 在 worker 集群中部署 kubeocean-worker -``` + +```sh # 使用 helm 安装 helm upgrade --install kubeocean-worker charts/kubeocean-worker ``` @@ -20,34 +36,37 @@ helm upgrade --install kubeocean-worker charts/kubeocean-worker ## 提取 kubeocean-worker 的 kubeconfig 1. 提取 kubeconfig -``` + +```sh # 使用脚本提取 kubeconfig # 需要确保当前 kubectl 访问的是 worker 集群 # 该脚本会使用当前默认 kubeconfig 里配置的 APIServer 地址 -bash hack/kubeconfig.sh kubeocean-syncer kubeocean-worker /tmp/kubeconfig-worker +bash hack/kubeconfig.sh kubeocean-syncer kubeocean-worker $WORKER_KUBECONFIG ``` + 2. 在算力集群中创建相关 secret -``` + +```sh # kubectl 切换到算力集群 -kubectl config use-context -kubectl -nkubeocean-system create secret generic worker-cluster-kubeconfig --from-file=kubeconfig=/tmp/kubeconfig-worker +kubectl config use-context +kubectl -nkubeocean-system create secret generic ${CLUSTER_NAME}-kubeconfig --from-file=kubeconfig=$WORKER_KUBECONFIG ``` ## 创建 ClusterBinding 对象以绑定 worker 集群 1. 创建 ClusterBinding 对象 -参考样例: -``` -# cb.yaml +```sh +# 使用 heredoc 创建 ClusterBinding YAML +cat > cb.yaml < rlp.yaml < kubectl apply -f rlp.yaml # 给期望抽取资源的节点添加 label kubectl label node kubeocean.io/role=worker ``` -上述命令执行完后,可在 manager 集群中观察算力节点是否正常抽取: -``` + +2. 验证资源抽取结果 + +上述命令执行完后,可在算力集群中观察算力节点是否正常抽取: + +```sh # kubectl 切换到虚拟算力集群 +kubectl config use-context kubectl get node ``` -若有 vnode 开头的节点创建,则说明算力资源抽取成功,如下: + +若有 vnode 开头的节点创建,则说明算力资源抽取成功: + ``` NAME STATUS ROLES AGE VERSION -vnode-cls-example-node1 Ready 5m v1.28.0 -vnode-cls-example-node2 Ready 5m v1.28.0 +vnode-cls-example-node1 Ready 5m v1.28.0 +vnode-cls-example-node2 Ready 5m v1.28.0 ``` -几个关键特性: -1. ResourceLeasingPolicy 需要在 worker 集群中创建才能生效。 -2. 每个 worker 集群只运行一个 ResourceLeasingPolicy 对象生效,若创建了多个,则最早的 ResourceLeasingPolicy 对象生效。 -3. nodeSelector 可为空,若为空则匹配所有节点。 -4. timeWindows 可为空,若为空则全时生效。 -5. resourceLimits 可为空,列表中未出现的资源按照剩余资源 100% 抽取。 \ No newline at end of file +## 关键特性说明 + +1. **ResourceLeasingPolicy 位置**:需要在 worker 集群中创建才能生效 +2. **唯一性**:每个 worker 集群只有一个 ResourceLeasingPolicy 对象生效,若创建了多个,则最早的 ResourceLeasingPolicy 对象生效 +3. **nodeSelector**:可为空,若为空则匹配所有节点 +4. **timeWindows**:可为空,若为空则全时生效 +5. **resourceLimits**:可为空,列表中未出现的资源按照剩余资源 100% 抽取 \ No newline at end of file diff --git a/docs/tutorials/installation.md b/docs/tutorials/installation.md index b2d3408..7596b91 100644 --- a/docs/tutorials/installation.md +++ b/docs/tutorials/installation.md @@ -9,12 +9,15 @@ ## Install kubeocean Components with Helm 1. Clone the repository and enter the directory -``` + +```sh git clone https://github.com/gocrane/kubeocean cd kubeocean ``` + 2. Deploy kubeocean components in the computing cluster -``` + +```sh helm upgrade --install kubeocean charts/kubeocean ``` @@ -27,18 +30,36 @@ In [TKE (Tencent Kubernetes Engine)](https://cloud.tencent.com/product/tke) clus ![k8s-svc](../images/k8s-svc.png) +Or use Tencent Cloud CLI tool `tccli` to call cloud API to enable intranet access + +```sh +# Set the region, cluster ID and subnet ID +export REGION=ap-guangzhou +export CLUSTER_ID=cls-abcdefgh +export SUBNET_ID=subnet-abcdefgh +TENCENTCLOUD_REGION="$REGION" tccli tke CreateClusterEndpoint \ + --ClusterId "$CLUSTER_ID" \ + --SubnetId "$SUBNET_ID" \ + --IsExtranet false +``` + ### Deploy kube-dns-intranet kubeocean components require the virtual computing cluster's `kube-dns` to provide intranet access to other worker clusters. A LoadBalancer type service named `kube-dns-intranet` needs to be deployed. -In [TKE (Tencent Kubernetes Engine)](https://cloud.tencent.com/product/tke) clusters, you can use the following YAML to create this service, need to fill in and replace the ``: -``` -# k8s-dns-svc.yaml +In [TKE (Tencent Kubernetes Engine)](https://cloud.tencent.com/product/tke) clusters, you can use the following YAML to create this service, need to set the subnet for the intranet load balancer: + +```sh +# Set the subnet ID in the VPC where the cluster is located for the intranet load balancer +export SUBNET_ID=subnet-abcdefgh +cat > tke-dns-svc.yaml < + service.cloud.tencent.com/direct-access: "true" + service.cloud.tencent.com/pass-to-target: "true" + service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: $SUBNET_ID name: kube-dns-intranet namespace: kube-system spec: @@ -61,11 +82,12 @@ spec: k8s-app: kube-dns sessionAffinity: None type: LoadBalancer +EOF ``` + Create and deploy the above YAML in TKE cluster: -``` -# Fill in the subnet in the VPC where the cluster is located to replace -sed -i "s||subnet-xxxxxxxx|" k8s-dns-svc.yaml + +```sh # Create and deploy Service -kubectl create -f k8s-dns-svc.yaml +kubectl create -f tke-dns-svc.yaml ``` diff --git a/docs/tutorials/installation_zh.md b/docs/tutorials/installation_zh.md index 09fbbf2..76748fa 100644 --- a/docs/tutorials/installation_zh.md +++ b/docs/tutorials/installation_zh.md @@ -9,12 +9,15 @@ ## 用 helm 安装 kubeocean 组件 1. 克隆代码仓库并进入目录 -``` + +```sh git clone https://github.com/gocrane/kubeocean cd kubeocean ``` + 2. 在算力集群中部署 kubeocean 组件 -``` + +```sh helm upgrade --install kubeocean charts/kubeocean ``` @@ -27,18 +30,36 @@ kubeocean 组件要求虚拟算力集群的 apiserver 给其他工作集群提 ![k8s-svc](../images/k8s-svc.png) +或者使用腾讯云 CLI 工具 `tccli` 调用云 API 开启内网访问 + +```sh +# 设置调用集群的地域、集群ID和子网ID +export REGION=ap-guangzhou +export CLUSTER_ID=cls-abcdefgh +export SUBNET_ID=subnet-abcdefgh +TENCENTCLOUD_REGION="$REGION" tccli tke CreateClusterEndpoint \ + --ClusterId "$CLUSTER_ID" \ + --SubnetId "$SUBNET_ID" \ + --IsExtranet false +``` + ### 部署 kube-dns-intranet kubeocean 组件要求虚拟算力集群的 `kube-dns` 给其他工作集群提供内网访问,需要部署名为 `kube-dns-intranet` 的 LoadBalancer 类型的服务。 -在 [TKE(Tencent Kubernetes Engine)](https://cloud.tencent.com/product/tke) 集群中,可以使用下述 YAML 创建该服务,需要填写替换其中的 ``: -``` -# k8s-dns-svc.yaml +在 [TKE(Tencent Kubernetes Engine)](https://cloud.tencent.com/product/tke) 集群中,可以使用下述 YAML 创建该服务,需要设置内网负载均衡使用的子网`: + +```sh +# 用集群所在 VPC 的子网ID设置内网负载均衡所在的子网 +export SUBNET_ID=subnet-abcdefgh +cat > tke-dns-svc.yaml < + service.cloud.tencent.com/direct-access: "true" + service.cloud.tencent.com/pass-to-target: "true" + service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: $SUBNET_ID name: kube-dns-intranet namespace: kube-system spec: @@ -61,11 +82,12 @@ spec: k8s-app: kube-dns sessionAffinity: None type: LoadBalancer +EOF ``` + 在 TKE 集群中创建部署上述 YAML: -``` -# 用集群所在 VPC 中的子网填写替换 -sed -i "s||subnet-xxxxxxxx|" k8s-dns-svc.yaml + +```sh # 创建部署 Service -kubectl create -f k8s-dns-svc.yaml +kubectl create -f tke-dns-svc.yaml ``` \ No newline at end of file diff --git a/examples/cookbook/installation-tke/README.md b/examples/cookbook/installation-tke/README.md new file mode 100644 index 0000000..f9b3a53 --- /dev/null +++ b/examples/cookbook/installation-tke/README.md @@ -0,0 +1,517 @@ +# Kubeocean One-Click Deployment Scripts for TKE Clusters + +> English | [中文](README_zh.md) + +This directory provides one-click deployment scripts for Kubeocean components and cluster bindings, enabling a seamless experience with Kubeocean features on Tencent Cloud TKE clusters. + +## Features + +- Complete cluster environment configuration: Enable APIServer intranet access, create kube-dns-intranet +- Deploy Kubeocean components on compute and worker clusters +- Bind worker clusters to compute clusters and configure simple resource leasing policies + +## Prerequisites + +- At least one TKE standard cluster as the virtual compute cluster, and one TKE standard cluster as the worker cluster +- Direct connectivity between compute and worker cluster Pod networks and node networks +- Ability to access clusters using `kubectl` +- Local environment with `helm` installed (version v3) +- Installed and configured [Tencent Cloud CLI Tool (tccli)](https://cloud.tencent.com/document/product/440/6176) +- Installed [jq](https://jqlang.org/download/) JSON processing tool +- TKE cluster region, cluster ID, and subnet ID prepared +- Other requirements refer to: [Requirements](../../../docs/requirements.md) + +## Basic Usage + +### One-Click Installation (Recommended) + +#### 1. Create Configuration File (config.env) + +```bash +cat > config.env < config.env < prod.env < config.env < \ + --IsExtranet false \ + --region + ``` + + Confirm the returned status is `"Status": "Running"` + +2. **Wait for internal load balancer to be ready** + + When cluster intranet access is just enabled, the load balancer may need a few seconds to initialize. Suggestions: + - Wait 5-10 seconds and retry + - Script has built-in wait mechanism, but some scenarios may require longer time + +3. **Check network connectivity** + ```bash + # Get internal network access address + tccli tke DescribeClusterSecurity \ + --ClusterId \ + --region | jq -r '.PgwEndpoint' + + # Test connectivity (example address) + curl -k https://:443 + ``` + +4. **Check subnet configuration** + + Ensure the provided `SUBNET_ID` is within the same network reachable range as the machine running the script. + +#### Error Message 2: `Cannot connect to Kubernetes cluster. Please check kubeconfig configuration` + +**Scenario**: kubectl cannot connect to the cluster when deploying components. + +**Possible Causes**: +- kubectl context not properly set +- kubeconfig file corrupted or incomplete +- Cluster API Server temporarily unavailable +- Network connection interrupted + +**Solutions**: + +1. **Check current context** + ```bash + kubectl config current-context + kubectl config get-contexts + ``` + +2. **Test cluster connection** + ```bash + kubectl cluster-info + kubectl get nodes + ``` + +3. **Switch to the correct context** + ```bash + # Worker cluster + kubectl config use-context worker-admin- + + # Manager cluster + kubectl config use-context manager-admin- + ``` + +4. **Re-obtain kubeconfig** + + If kubeconfig is corrupted, you can rerun the installation script to obtain it: + ```bash + # Rerunning will automatically detect and reuse or update context + ./install-worker-tke.sh -c config.env + ``` + +5. **View detailed error information** + ```bash + kubectl get nodes -v=8 + ``` + +## References + +- [Kubeocean Documentation](../../../README.md) +- [System Requirements](../../../docs/requirements.md) +- [Tencent Cloud CLI Tool Documentation](https://cloud.tencent.com/document/product/440/6176) + diff --git a/examples/cookbook/installation-tke/README_zh.md b/examples/cookbook/installation-tke/README_zh.md new file mode 100644 index 0000000..e19245b --- /dev/null +++ b/examples/cookbook/installation-tke/README_zh.md @@ -0,0 +1,516 @@ +# TKE 集群上 Kubeocean 一键部署脚本 + +> [English](README.md) | 中文 + +本目录提供了 Kubeocean 组件和集群绑定的一键部署脚本,支持在腾讯云 TKE 集群上一站式体验 Kubeocean 相关功能。 + +## 功能 + +- 完成集群环境配置:开启 APIServer 内网访问,创建 kube-dns-intranet +- 在算力集群和工作集群部署 Kubeocean 组件 +- 绑定工作集群到算力集群,并配置简单资源抽取策略 + +## 前置准备 + +- 需要至少一个 TKE 标准集群作为虚拟算力集群,以及一个 TKE 标准集群作为工作集群 +- 算力集群和业务集群 Pod 网络直接互通、节点网络互通 +- 可以使用 `kubectl` 并通过内网访问集群 +- 本地环境已安装 `helm`,且版本为 v3 +- 安装并配置 [腾讯云 CLI 工具 (tccli)](https://cloud.tencent.com/document/product/440/6176) +- 安装 [jq](https://jqlang.org/download/) JSON 处理工具 +- 准备 TKE 集群所在的地域、集群 ID 和子网 ID +- 其他要求参考:[要求](../../../docs/requirements_zh.md) + +## 基础使用 + +### 一键安装(推荐) + +#### 1. 创建配置文件 (config.env) + +```bash +cat > config.env < config.env < prod.env < config.env < \ + --IsExtranet false \ + --region + ``` + + 确认返回状态为 `"Status": "Running"` + +2. **等待内网负载均衡就绪** + + 集群内网访问刚开启时,负载均衡可能需要几秒时间初始化。建议: + - 等待 5-10 秒后重试 + - 脚本已内置等待机制,但某些场景可能需要更长时间 + +3. **检查网络连通性** + ```bash + # 获取内网访问地址 + tccli tke DescribeClusterSecurity \ + --ClusterId \ + --region | jq -r '.PgwEndpoint' + + # 测试连通性(示例地址) + curl -k https://:443 + ``` + +4. **检查子网配置** + + 确保提供的 `SUBNET_ID` 与运行脚本的机器在同一网络可达范围内。 + +#### 错误信息 2:`Cannot connect to Kubernetes cluster. Please check kubeconfig configuration` + +**场景**:在部署组件时 kubectl 无法连接到集群。 + +**可能原因**: +- kubectl context 未正确设置 +- kubeconfig 文件损坏或不完整 +- 集群 API Server 临时不可用 +- 网络连接中断 + +**解决方法**: + +1. **检查当前 context** + ```bash + kubectl config current-context + kubectl config get-contexts + ``` + +2. **测试集群连接** + ```bash + kubectl cluster-info + kubectl get nodes + ``` + +3. **切换到正确的 context** + ```bash + # Worker 集群 + kubectl config use-context worker-admin- + + # Manager 集群 + kubectl config use-context manager-admin- + ``` + +4. **重新获取 kubeconfig** + + 如果 kubeconfig 已损坏,可以重新运行安装脚本获取: + ```bash + # 重新运行会自动检测并复用或更新 context + ./install-worker-tke.sh -c config.env + ``` + +5. **查看详细错误信息** + ```bash + kubectl get nodes -v=8 + ``` + +## 参考资料 + +- [Kubeocean 文档](../../../README.md) +- [系统要求](../../../docs/requirements_zh.md) +- [腾讯云 CLI 工具文档](https://cloud.tencent.com/document/product/440/6176) diff --git a/examples/cookbook/installation-tke/config.env.example b/examples/cookbook/installation-tke/config.env.example new file mode 100644 index 0000000..1ccaebe --- /dev/null +++ b/examples/cookbook/installation-tke/config.env.example @@ -0,0 +1,15 @@ +# Kubeocean TKE Installation Configuration Example + +# Manager Cluster Settings +MANAGER_REGION="ap-guangzhou" +MANAGER_CLUSTER_ID="cls-xxx1" +MANAGER_SUBNET_ID="subnet-xxx1" +SKIP_MANAGER_UNINSTALL="false" + +# Worker Cluster Settings +WORKER_REGION="ap-guangzhou" +WORKER_CLUSTER_ID="cls-xxx1" +WORKER_SUBNET_ID="subnet-xxx2" +WORKER_KUBECONFIG="/tmp/kubeconfig-worker" +WORKER_CLUSTER_NAME="example-cluster" + diff --git a/examples/cookbook/installation-tke/config.env.template b/examples/cookbook/installation-tke/config.env.template new file mode 100644 index 0000000..a313038 --- /dev/null +++ b/examples/cookbook/installation-tke/config.env.template @@ -0,0 +1,50 @@ +# Kubeocean TKE Installation Configuration Template +# Copy this file to config.env and modify the values +# +# Usage: +# cp config.env.template config.env +# vim config.env +# ./install-worker-tke.sh -c config.env +# ./install-manager-tke.sh -c config.env + +# ============================================ +# Manager Cluster Settings +# (for install-manager-tke.sh) +# ============================================ + +# Manager cluster region (Required for manager installation) +MANAGER_REGION="ap-guangzhou" + +# Manager cluster ID (Required for manager installation) +MANAGER_CLUSTER_ID="cls-xxx1" + +# Manager cluster subnet ID (Required for manager installation) +MANAGER_SUBNET_ID="subnet-xxx1" + +# Skip manager uninstallation (Optional, default: false) +# - Set to "true" to only delete bindings and secrets without uninstalling manager components +SKIP_MANAGER_UNINSTALL="false" + +# ============================================ +# Worker Cluster Settings +# (for install-worker-tke.sh) +# ============================================ + +# Worker cluster region (Required for worker installation) +WORKER_REGION="ap-guangzhou" + +# Worker cluster ID (Required for both worker and manager installation) +WORKER_CLUSTER_ID="cls-xxx2" + +# Worker cluster subnet ID (Required for worker installation) +WORKER_SUBNET_ID="subnet-xxx2" + +# Worker kubeconfig output/path (Optional, default: /tmp/kubeconfig-worker) +# - For install-worker-tke.sh: output path for generated kubeconfig +# - For install-manager-tke.sh: path to existing worker kubeconfig +WORKER_KUBECONFIG="/tmp/kubeconfig-worker" + +# Worker cluster name (Optional, default: example-cluster) +# - Used for ResourceLeasingPolicy in install-worker-tke.sh +WORKER_CLUSTER_NAME="example-cluster" + diff --git a/examples/playbook/install-manager-tke.sh b/examples/cookbook/installation-tke/install-manager-tke.sh similarity index 71% rename from examples/playbook/install-manager-tke.sh rename to examples/cookbook/installation-tke/install-manager-tke.sh index 7e0fcf2..18b421b 100755 --- a/examples/playbook/install-manager-tke.sh +++ b/examples/cookbook/installation-tke/install-manager-tke.sh @@ -52,16 +52,22 @@ log_error() { # Error handling trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR -# Script root directory (assuming script is in examples/playbook directory) +# Script root directory (assuming script is in examples/playbook/installation directory) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" -# Configuration variables -REGION="" -CLUSTER_ID="" -SUBNET_ID="" +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Manager Cluster Configuration variables +MANAGER_REGION="" +MANAGER_CLUSTER_ID="" +MANAGER_SUBNET_ID="" + +# Worker Cluster Configuration variables WORKER_CLUSTER_ID="" WORKER_KUBECONFIG="/tmp/kubeconfig-worker" +WORKER_CLUSTER_NAME="example-cluster" # VPC information (will be extracted from cluster info) VPC_ID="" @@ -75,11 +81,17 @@ Usage: $0 [options] Options: -h, --help Display this help message - -r, --region REGION TKE cluster region (required) - -c, --cluster-id ID TKE cluster ID (required) - -s, --subnet-id ID Subnet ID for internal network access (required for enabling internal access) - -w, --worker-kubeconfig FILE Worker cluster kubeconfig path (default: /tmp/kubeconfig-worker) - -i, --worker-cluster-id ID Worker cluster ID (required) + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + MANAGER_REGION Manager cluster region + MANAGER_CLUSTER_ID Manager cluster ID + MANAGER_SUBNET_ID Manager cluster subnet ID for internal network access + WORKER_CLUSTER_ID Worker cluster ID + +Optional Configuration Variables: + WORKER_KUBECONFIG Worker cluster kubeconfig path (default: /tmp/kubeconfig-worker) + WORKER_CLUSTER_NAME Worker cluster name for binding (default: example-cluster) Prerequisites: tccli Tencent Cloud CLI tool (must be installed and configured) @@ -88,20 +100,35 @@ Prerequisites: helm v3.x Helm package manager (version 3.x required) Examples: - # Install TKE manager cluster with default worker kubeconfig path - $0 --region ap-guangzhou --cluster-id cls-xxxxxxxx --subnet-id subnet-xxxxxxxx \\ - --worker-cluster-id cls-worker-xxx + # Use default configuration file (./config.env) + $0 + + # Use custom configuration file + $0 --config /path/to/my-config.env - # Install with custom worker kubeconfig path - $0 --region ap-guangzhou --cluster-id cls-xxxxxxxx --subnet-id subnet-xxxxxxxx \\ - --worker-kubeconfig /tmp/my-kubeconfig --worker-cluster-id cls-worker-xxx + # Use environment variables to override config file + WORKER_CLUSTER_ID="cls-new-worker" $0 --config prod.env - # Short form - $0 -r ap-guangzhou -c cls-xxxxxxxx -s subnet-xxxxxxxx -i cls-worker-xxx +Configuration File Example: + cp config.env.template config.env + # Edit config.env with your values + vim config.env EOF } +# Load configuration from file +load_config() { + if [[ -f "$CONFIG_FILE" ]]; then + log_info "Loading configuration from: $CONFIG_FILE" + set -a + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + set +a + log_success "Configuration loaded" + fi +} + # Parse command line arguments parse_args() { while [[ $# -gt 0 ]]; do @@ -110,28 +137,14 @@ parse_args() { show_help exit 0 ;; - -r|--region) - REGION="$2" - shift 2 - ;; - -c|--cluster-id) - CLUSTER_ID="$2" - shift 2 - ;; - -s|--subnet-id) - SUBNET_ID="$2" - shift 2 - ;; - -w|--worker-kubeconfig) - WORKER_KUBECONFIG="$2" - shift 2 - ;; - -i|--worker-cluster-id) - WORKER_CLUSTER_ID="$2" + -c|--config) + CONFIG_FILE="$2" shift 2 ;; *) log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + log_info "All other configurations should be provided in config file" show_help exit 1 ;; @@ -144,13 +157,13 @@ check_prerequisites() { log_info "Checking prerequisites..." # Check required parameters - if [ -z "$REGION" ]; then - log_error "Region is required. Please specify with --region" + if [ -z "$MANAGER_REGION" ]; then + log_error "MANAGER_REGION is required. Please set it in config file" exit 1 fi - if [ -z "$CLUSTER_ID" ]; then - log_error "Cluster ID is required. Please specify with --cluster-id" + if [ -z "$MANAGER_CLUSTER_ID" ]; then + log_error "MANAGER_CLUSTER_ID is required. Please set it in config file" exit 1 fi @@ -165,9 +178,10 @@ check_prerequisites() { exit 1 fi - log_info "Region: $REGION" - log_info "Cluster ID: $CLUSTER_ID" + log_info "Manager Region: $MANAGER_REGION" + log_info "Manager Cluster ID: $MANAGER_CLUSTER_ID" log_info "Worker Cluster ID: $WORKER_CLUSTER_ID" + log_info "Worker Cluster Name: $WORKER_CLUSTER_NAME" log_info "Worker Kubeconfig: $WORKER_KUBECONFIG" # Check tccli installation @@ -200,7 +214,7 @@ check_prerequisites() { log_info "Checking cluster existence and getting VPC info..." local cluster_info ret=0 - cluster_info=$(TENCENTCLOUD_REGION="$REGION" tccli tke DescribeClusters --ClusterIds "[\"$CLUSTER_ID\"]" 2>&1) || ret=$? + cluster_info=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusters --ClusterIds "[\"$MANAGER_CLUSTER_ID\"]" 2>&1) || ret=$? if [ $ret -ne 0 ]; then log_error "Failed to query cluster information" @@ -213,7 +227,7 @@ check_prerequisites() { total_count=$(echo "$cluster_info" | jq -r '.TotalCount // 0') if [ "$total_count" -eq 0 ]; then - log_error "Cluster $CLUSTER_ID not found in region $REGION" + log_error "Cluster $MANAGER_CLUSTER_ID not found in region $MANAGER_REGION" exit 1 fi @@ -243,7 +257,7 @@ enable_cluster_internal_access() { # Check if internal access is already enabled local endpoint_status ret=0 - endpoint_status=$(TENCENTCLOUD_REGION="$REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$CLUSTER_ID" 2>&1) || ret=$? + endpoint_status=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$MANAGER_CLUSTER_ID" 2>&1) || ret=$? if [ $ret -ne 0 ]; then log_error "Failed to query cluster endpoint status" @@ -263,17 +277,17 @@ enable_cluster_internal_access() { log_info "Enabling cluster internal network access..." # Check subnet ID parameter - if [ -z "$SUBNET_ID" ]; then - log_error "Subnet ID is required for enabling internal access. Please specify with --subnet-id" + if [ -z "$MANAGER_SUBNET_ID" ]; then + log_error "MANAGER_SUBNET_ID is required for enabling internal access. Please set it in config file" exit 1 fi # Enable internal access local enable_result ret=0 - enable_result=$(TENCENTCLOUD_REGION="$REGION" tccli tke CreateClusterEndpoint \ - --ClusterId "$CLUSTER_ID" \ - --SubnetId "$SUBNET_ID" \ + enable_result=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke CreateClusterEndpoint \ + --ClusterId "$MANAGER_CLUSTER_ID" \ + --SubnetId "$MANAGER_SUBNET_ID" \ --IsExtranet false 2>&1) || ret=$? if [ $ret -ne 0 ]; then @@ -292,7 +306,7 @@ enable_cluster_internal_access() { sleep 2 ret=0 - endpoint_status=$(TENCENTCLOUD_REGION="$REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$CLUSTER_ID" 2>&1) || ret=$? + endpoint_status=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$MANAGER_CLUSTER_ID" 2>&1) || ret=$? if [ $ret -eq 0 ]; then status=$(echo "$endpoint_status" | jq -r '.Status // empty') @@ -327,8 +341,8 @@ enable_kube_dns_internal_access() { log_info "Creating kube-dns-intranet service..." # Check subnet ID parameter - if [ -z "$SUBNET_ID" ]; then - log_error "Subnet ID is required for creating kube-dns-intranet service. Please specify with --subnet-id" + if [ -z "$MANAGER_SUBNET_ID" ]; then + log_error "MANAGER_SUBNET_ID is required for creating kube-dns-intranet service. Please set it in config file" exit 1 fi @@ -338,7 +352,9 @@ apiVersion: v1 kind: Service metadata: annotations: - service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: $SUBNET_ID + service.cloud.tencent.com/direct-access: "true" + service.cloud.tencent.com/pass-to-target: "true" + service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: $MANAGER_SUBNET_ID name: kube-dns-intranet namespace: kube-system spec: @@ -376,10 +392,25 @@ EOF get_cluster_kubeconfig() { log_info "Getting cluster kubeconfig..." + # Check if context already exists + local context_name="manager-admin-$MANAGER_CLUSTER_ID" + if kubectl config get-contexts "$context_name" &> /dev/null; then + log_info "Context '$context_name' already exists, reusing it" + kubectl config use-context "$context_name" + log_success "Switched to existing context: $context_name" + return 0 + fi + + log_info "Context not found, fetching kubeconfig from TKE..." + + # Wait for internal load balancer information to update + log_info "Waiting 5 seconds for internal load balancer information to update..." + sleep 5 + # Get cluster security info (including kubeconfig) local security_info ret=0 - security_info=$(TENCENTCLOUD_REGION="$REGION" tccli tke DescribeClusterSecurity --ClusterId "$CLUSTER_ID" 2>&1) || ret=$? + security_info=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusterSecurity --ClusterId "$MANAGER_CLUSTER_ID" 2>&1) || ret=$? if [ $ret -ne 0 ]; then log_error "Failed to get cluster security information" @@ -408,7 +439,7 @@ get_cluster_kubeconfig() { log_info "PgwEndpoint: $pgw_endpoint" # Save kubeconfig to temporary file - local temp_kubeconfig="/tmp/tke-kubeconfig-$CLUSTER_ID" + local temp_kubeconfig="/tmp/tke-kubeconfig-$MANAGER_CLUSTER_ID" echo "$kubeconfig_content" > "$temp_kubeconfig" # Replace server address with PgwEndpoint @@ -424,8 +455,18 @@ get_cluster_kubeconfig() { exit 1 fi + # Verify connection before merging + log_info "Verifying cluster connection..." + if ! kubectl --kubeconfig="$temp_kubeconfig" cluster-info &> /dev/null; then + log_error "Failed to connect to cluster using the kubeconfig" + log_error "Please check if the cluster internal network access is properly configured" + rm -f "$temp_kubeconfig" + exit 1 + fi + log_success "Successfully verified cluster connection" + # Set kubectl context name - local context_name="manager-admin-$CLUSTER_ID" + local context_name="manager-admin-$MANAGER_CLUSTER_ID" log_info "Merging kubeconfig to default kubectl config" log_info "Context name: $context_name" @@ -462,33 +503,27 @@ get_cluster_kubeconfig() { log_success "Kubeconfig merged to: $default_kubeconfig" log_success "Context name: $context_name" - - # Verify connection - if ! kubectl cluster-info &> /dev/null; then - log_error "Failed to connect to cluster using the kubeconfig" - exit 1 - fi - - log_success "Successfully connected to cluster" + log_success "Cluster is ready for use" } # Install kubeocean-manager install_kubeocean_manager() { log_info "Installing kubeocean-manager..." - # Call install-manager.sh script - local install_manager_script="${SCRIPT_DIR}/install-manager.sh" + # Call install-manager.sh script (in parent installation directory) + local install_manager_script="${SCRIPT_DIR}/../installation/install-manager.sh" if [ ! -f "$install_manager_script" ]; then log_error "install-manager.sh script not found: $install_manager_script" exit 1 fi - log_info "Calling install-manager.sh with worker cluster: $WORKER_CLUSTER_ID" + log_info "Calling install-manager.sh with worker cluster: $WORKER_CLUSTER_ID (name: $WORKER_CLUSTER_NAME)" if bash "$install_manager_script" \ --worker-kubeconfig "$WORKER_KUBECONFIG" \ - --cluster-id "$WORKER_CLUSTER_ID"; then + --cluster-id "$WORKER_CLUSTER_ID" \ + --cluster-name "$WORKER_CLUSTER_NAME"; then log_success "kubeocean-manager installed successfully" else log_error "Failed to install kubeocean-manager" @@ -504,12 +539,13 @@ show_summary() { echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" echo "" echo "📋 Installation Information:" - echo " • Region: $REGION" - echo " • Cluster ID: $CLUSTER_ID" + echo " • Manager Region: $MANAGER_REGION" + echo " • Manager Cluster ID: $MANAGER_CLUSTER_ID" echo " • VPC ID: $VPC_ID" - echo " • Subnet ID: $SUBNET_ID" - echo " • Context: manager-admin-$CLUSTER_ID (merged to ~/.kube/config)" + echo " • Manager Subnet ID: $MANAGER_SUBNET_ID" + echo " • Context: manager-admin-$MANAGER_CLUSTER_ID (merged to ~/.kube/config)" echo " • Worker Cluster ID: $WORKER_CLUSTER_ID" + echo " • Worker Cluster Name: $WORKER_CLUSTER_NAME" echo " • Worker Kubeconfig: $WORKER_KUBECONFIG" echo "" echo "🔍 Next Steps:" @@ -520,7 +556,7 @@ show_summary() { echo " kubectl get clusterbindings" echo "" echo " 3. Switch to this context anytime:" - echo " kubectl config use-context manager-admin-$CLUSTER_ID" + echo " kubectl config use-context manager-admin-$MANAGER_CLUSTER_ID" echo "" echo " 4. Check synced resources in worker cluster:" echo " # Switch to worker cluster context and check" @@ -535,7 +571,20 @@ main() { echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" echo "" - # Parse arguments + # Step 0: Parse config file path first + for arg in "$@"; do + if [[ "$arg" == "-c" || "$arg" == "--config" ]]; then + shift + CONFIG_FILE="$1" + break + fi + shift + done + + # Load configuration file + load_config + + # Parse arguments (command line overrides config file) parse_args "$@" # Step 1: Check prerequisites diff --git a/examples/cookbook/installation-tke/install-tke.sh b/examples/cookbook/installation-tke/install-tke.sh new file mode 100755 index 0000000..7dcae7f --- /dev/null +++ b/examples/cookbook/installation-tke/install-tke.sh @@ -0,0 +1,270 @@ +#!/bin/bash + +# Copyright 2025 The Kubeocean Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Kubeocean One-Click Installation Script for TKE +# Features: +# 1. Install and configure worker cluster +# 2. Install and configure manager cluster +# 3. Create cluster binding + +set -e +set -o pipefail + +# Color definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Logging functions +log_info() { + echo -e "${BLUE}ℹ️ $1${NC}" +} + +log_success() { + echo -e "${GREEN}✅ $1${NC}" +} + +log_warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +log_error() { + echo -e "${RED}❌ $1${NC}" +} + +# Error handling +trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR + +# Script root directory +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Display help information +show_help() { + cat << EOF +Kubeocean One-Click Installation Script for TKE + +This script automates the complete installation process by sequentially executing: +1. install-worker-tke.sh - Install worker cluster components +2. install-manager-tke.sh - Install manager cluster components and create binding + +Usage: $0 [options] + +Options: + -h, --help Display this help message + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + MANAGER_REGION Manager cluster region + MANAGER_CLUSTER_ID Manager cluster ID + MANAGER_SUBNET_ID Manager cluster subnet ID + WORKER_REGION Worker cluster region + WORKER_CLUSTER_ID Worker cluster ID + WORKER_SUBNET_ID Worker cluster subnet ID + +Optional Configuration Variables: + WORKER_KUBECONFIG Worker kubeconfig path (default: /tmp/kubeconfig-worker) + WORKER_CLUSTER_NAME Worker cluster name for RLP (default: example-cluster) + +Prerequisites: + tccli Tencent Cloud CLI tool (must be installed and configured) + jq Command-line JSON processor + kubectl Kubernetes command-line tool + helm v3.x Helm package manager (version 3.x required) + +Examples: + # Use default configuration file (./config.env) + $0 + + # Use custom configuration file + $0 -c /path/to/my-config.env + + # Quick setup + cp config.env.template config.env + vim config.env + $0 + +EOF +} + +# Parse command line arguments +parse_args() { + while [[ $# -gt 0 ]]; do + case $1 in + -h|--help) + show_help + exit 0 + ;; + -c|--config) + CONFIG_FILE="$2" + shift 2 + ;; + *) + log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + show_help + exit 1 + ;; + esac + done +} + +# Check if config file exists +check_config_file() { + if [ ! -f "$CONFIG_FILE" ]; then + log_error "Configuration file not found: $CONFIG_FILE" + log_info "Please create config file first:" + log_info " cp config.env.template config.env" + log_info " vim config.env" + exit 1 + fi + + log_info "Using configuration file: $CONFIG_FILE" +} + +# Load configuration from config file +load_config() { + log_info "Loading configuration from: $CONFIG_FILE" + + # Source the config file + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + + # Export all configuration variables so they are available to child scripts + export MANAGER_REGION + export MANAGER_CLUSTER_ID + export MANAGER_SUBNET_ID + export WORKER_REGION + export WORKER_CLUSTER_ID + export WORKER_SUBNET_ID + export WORKER_KUBECONFIG + export WORKER_CLUSTER_NAME + + log_success "Configuration loaded and exported as environment variables" +} + +# Install worker cluster +install_worker() { + local install_worker_script="${SCRIPT_DIR}/install-worker-tke.sh" + + if [ ! -f "$install_worker_script" ]; then + log_error "install-worker-tke.sh not found: $install_worker_script" + exit 1 + fi + + log_info "Executing install-worker-tke.sh..." + log_info "Configuration passed via environment variables" + + if bash "$install_worker_script"; then + log_success "Worker cluster installation completed" + else + log_error "Worker cluster installation failed" + exit 1 + fi +} + +# Install manager cluster +install_manager() { + local install_manager_script="${SCRIPT_DIR}/install-manager-tke.sh" + + if [ ! -f "$install_manager_script" ]; then + log_error "install-manager-tke.sh not found: $install_manager_script" + exit 1 + fi + + log_info "Executing install-manager-tke.sh..." + log_info "Configuration passed via environment variables" + + if bash "$install_manager_script"; then + log_success "Manager cluster installation completed" + else + log_error "Manager cluster installation failed" + exit 1 + fi +} + +# Display final summary +show_final_summary() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + log_success "Kubeocean One-Click Installation Complete!" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "📋 Installation Summary:" + echo " • Configuration: $CONFIG_FILE" + echo " • Worker cluster components installed ✓" + echo " • Manager cluster components installed ✓" + echo " • Cluster binding created ✓" + echo "" + echo "🔍 Quick Verification:" + echo " # Check worker cluster (switch context if needed)" + echo " kubectl get nodes -l kubeocean.io/role=worker" + echo " kubectl get all -n kubeocean-worker" + echo " kubectl get resourceleasingpolicies" + echo "" + echo " # Check manager cluster (switch context if needed)" + echo " kubectl get all -n kubeocean-system" + echo " kubectl get clusterbindings" + echo "" + echo "🎉 Kubeocean is ready to use!" + echo "" +} + +# Main function +main() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo " 🚀 Kubeocean One-Click Installation for TKE" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + + # Parse arguments + parse_args "$@" + + # Check config file + check_config_file + + # Load configuration and export as environment variables + load_config + echo "" + + # Step 1: Install worker cluster + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "📦 Phase 1/2: Installing Worker Cluster" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + install_worker + echo "" + + # Step 2: Install manager cluster + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "📦 Phase 2/2: Installing Manager Cluster" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + install_manager + echo "" + + # Display final summary + show_final_summary +} + +# Execute main function +main "$@" + diff --git a/examples/playbook/install-worker-tke.sh b/examples/cookbook/installation-tke/install-worker-tke.sh similarity index 61% rename from examples/playbook/install-worker-tke.sh rename to examples/cookbook/installation-tke/install-worker-tke.sh index 0223ddc..a39497f 100755 --- a/examples/playbook/install-worker-tke.sh +++ b/examples/cookbook/installation-tke/install-worker-tke.sh @@ -51,15 +51,19 @@ log_error() { # Error handling trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR -# Script root directory (assuming script is in examples/playbook directory) +# Script root directory (assuming script is in examples/playbook/installation directory) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" -# Configuration variables -REGION="" -CLUSTER_ID="" -SUBNET_ID="" -OUTPUT_KUBECONFIG="/tmp/kubeconfig-worker" +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Worker Cluster Configuration variables +WORKER_REGION="" +WORKER_CLUSTER_ID="" +WORKER_SUBNET_ID="" +WORKER_KUBECONFIG="/tmp/kubeconfig-worker" +WORKER_CLUSTER_NAME="" # Optional, will be passed to install-worker.sh for RLP # VPC information (will be extracted from cluster info) VPC_ID="" @@ -73,10 +77,16 @@ Usage: $0 [options] Options: -h, --help Display this help message - -r, --region REGION TKE cluster region (required) - -c, --cluster-id ID TKE cluster ID (required) - -s, --subnet-id ID Subnet ID for internal network access (required for enabling internal access) - -o, --output FILE Kubeconfig output file path (default: /tmp/kubeconfig-worker) + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + WORKER_REGION Worker cluster region + WORKER_CLUSTER_ID Worker cluster ID + WORKER_SUBNET_ID Worker cluster subnet ID for internal network access + +Optional Configuration Variables: + WORKER_KUBECONFIG Kubeconfig output file path (default: /tmp/kubeconfig-worker) + WORKER_CLUSTER_NAME Worker cluster name for ResourceLeasingPolicy (default: example-cluster) Prerequisites: tccli Tencent Cloud CLI tool (must be installed and configured) @@ -85,15 +95,35 @@ Prerequisites: helm v3.x Helm package manager (version 3.x required) Examples: - # Install TKE worker cluster - $0 --region ap-guangzhou --cluster-id cls-xxxxxxxx --subnet-id subnet-xxxxxxxx + # Use default configuration file (./config.env) + $0 - # Specify output path - $0 -r ap-guangzhou -c cls-xxxxxxxx -s subnet-xxxxxxxx -o /tmp/my-kubeconfig + # Use custom configuration file + $0 --config /path/to/my-config.env + + # Use environment variables to override config file + REGION="ap-shanghai" $0 --config prod.env + +Configuration File Example: + cp config.env.template config.env + # Edit config.env with your values + vim config.env EOF } +# Load configuration from file +load_config() { + if [[ -f "$CONFIG_FILE" ]]; then + log_info "Loading configuration from: $CONFIG_FILE" + set -a + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + set +a + log_success "Configuration loaded" + fi +} + # Parse command line arguments parse_args() { while [[ $# -gt 0 ]]; do @@ -102,24 +132,14 @@ parse_args() { show_help exit 0 ;; - -r|--region) - REGION="$2" - shift 2 - ;; - -c|--cluster-id) - CLUSTER_ID="$2" - shift 2 - ;; - -s|--subnet-id) - SUBNET_ID="$2" - shift 2 - ;; - -o|--output) - OUTPUT_KUBECONFIG="$2" + -c|--config) + CONFIG_FILE="$2" shift 2 ;; *) log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + log_info "All other configurations should be provided in config file" show_help exit 1 ;; @@ -132,18 +152,18 @@ check_prerequisites() { log_info "Checking prerequisites..." # Check required parameters - if [ -z "$REGION" ]; then - log_error "Region is required. Please specify with --region" + if [ -z "$WORKER_REGION" ]; then + log_error "WORKER_REGION is required. Please set it in config file" exit 1 fi - if [ -z "$CLUSTER_ID" ]; then - log_error "Cluster ID is required. Please specify with --cluster-id" + if [ -z "$WORKER_CLUSTER_ID" ]; then + log_error "WORKER_CLUSTER_ID is required. Please set it in config file" exit 1 fi - log_info "Region: $REGION" - log_info "Cluster ID: $CLUSTER_ID" + log_info "Worker Region: $WORKER_REGION" + log_info "Worker Cluster ID: $WORKER_CLUSTER_ID" # Check tccli installation if ! command -v tccli &> /dev/null; then @@ -175,7 +195,7 @@ check_prerequisites() { log_info "Checking cluster existence and getting VPC info..." local cluster_info ret=0 - cluster_info=$(TENCENTCLOUD_REGION="$REGION" tccli tke DescribeClusters --ClusterIds "[\"$CLUSTER_ID\"]" 2>&1) || ret=$? + cluster_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusters --ClusterIds "[\"$WORKER_CLUSTER_ID\"]" 2>&1) || ret=$? if [ $ret -ne 0 ]; then log_error "Failed to query cluster information" @@ -188,7 +208,7 @@ check_prerequisites() { total_count=$(echo "$cluster_info" | jq -r '.TotalCount // 0') if [ "$total_count" -eq 0 ]; then - log_error "Cluster $CLUSTER_ID not found in region $REGION" + log_error "Cluster $WORKER_CLUSTER_ID not found in region $WORKER_REGION" exit 1 fi @@ -218,7 +238,7 @@ enable_cluster_internal_access() { # Check if internal access is already enabled local endpoint_status ret=0 - endpoint_status=$(TENCENTCLOUD_REGION="$REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$CLUSTER_ID" 2>&1) || ret=$? + endpoint_status=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$WORKER_CLUSTER_ID" 2>&1) || ret=$? if [ $ret -ne 0 ]; then log_error "Failed to query cluster endpoint status" @@ -238,17 +258,17 @@ enable_cluster_internal_access() { log_info "Enabling cluster internal network access..." # Check subnet ID parameter - if [ -z "$SUBNET_ID" ]; then - log_error "Subnet ID is required for enabling internal access. Please specify with --subnet-id" + if [ -z "$WORKER_SUBNET_ID" ]; then + log_error "WORKER_SUBNET_ID is required for enabling internal access. Please set it in config file" exit 1 fi # Enable internal access local enable_result ret=0 - enable_result=$(TENCENTCLOUD_REGION="$REGION" tccli tke CreateClusterEndpoint \ - --ClusterId "$CLUSTER_ID" \ - --SubnetId "$SUBNET_ID" \ + enable_result=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke CreateClusterEndpoint \ + --ClusterId "$WORKER_CLUSTER_ID" \ + --SubnetId "$WORKER_SUBNET_ID" \ --IsExtranet false 2>&1) || ret=$? if [ $ret -ne 0 ]; then @@ -267,7 +287,7 @@ enable_cluster_internal_access() { sleep 2 ret=0 - endpoint_status=$(TENCENTCLOUD_REGION="$REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$CLUSTER_ID" 2>&1) || ret=$? + endpoint_status=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$WORKER_CLUSTER_ID" 2>&1) || ret=$? if [ $ret -eq 0 ]; then status=$(echo "$endpoint_status" | jq -r '.Status // empty') @@ -293,10 +313,25 @@ enable_cluster_internal_access() { get_cluster_kubeconfig() { log_info "Getting cluster kubeconfig..." + # Check if context already exists + local context_name="worker-admin-$WORKER_CLUSTER_ID" + if kubectl config get-contexts "$context_name" &> /dev/null; then + log_info "Context '$context_name' already exists, reusing it" + kubectl config use-context "$context_name" + log_success "Switched to existing context: $context_name" + return 0 + fi + + log_info "Context not found, fetching kubeconfig from TKE..." + + # Wait for internal load balancer information to update + log_info "Waiting 5 seconds for internal load balancer information to update..." + sleep 5 + # Get cluster security info (including kubeconfig) local security_info ret=0 - security_info=$(TENCENTCLOUD_REGION="$REGION" tccli tke DescribeClusterSecurity --ClusterId "$CLUSTER_ID" 2>&1) || ret=$? + security_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusterSecurity --ClusterId "$WORKER_CLUSTER_ID" 2>&1) || ret=$? if [ $ret -ne 0 ]; then log_error "Failed to get cluster security information" @@ -325,7 +360,7 @@ get_cluster_kubeconfig() { log_info "PgwEndpoint: $pgw_endpoint" # Save kubeconfig to temporary file - local temp_kubeconfig="/tmp/tke-kubeconfig-$CLUSTER_ID" + local temp_kubeconfig="/tmp/tke-kubeconfig-$WORKER_CLUSTER_ID" echo "$kubeconfig_content" > "$temp_kubeconfig" # Replace server address with PgwEndpoint @@ -341,8 +376,18 @@ get_cluster_kubeconfig() { exit 1 fi + # Verify connection before merging + log_info "Verifying cluster connection..." + if ! kubectl --kubeconfig="$temp_kubeconfig" cluster-info &> /dev/null; then + log_error "Failed to connect to cluster using the kubeconfig" + log_error "Please check if the cluster internal network access is properly configured" + rm -f "$temp_kubeconfig" + exit 1 + fi + log_success "Successfully verified cluster connection" + # Set kubectl context name - local context_name="worker-admin-$CLUSTER_ID" + local context_name="worker-admin-$WORKER_CLUSTER_ID" log_info "Merging kubeconfig to default kubectl config" log_info "Context name: $context_name" @@ -379,31 +424,31 @@ get_cluster_kubeconfig() { log_success "Kubeconfig merged to: $default_kubeconfig" log_success "Context name: $context_name" - - # Verify connection - if ! kubectl cluster-info &> /dev/null; then - log_error "Failed to connect to cluster using the kubeconfig" - exit 1 - fi - - log_success "Successfully connected to cluster" + log_success "Cluster is ready for use" } # Install kubeocean-worker install_kubeocean_worker() { log_info "Installing kubeocean-worker..." - # Call install-worker.sh script - local install_worker_script="${SCRIPT_DIR}/install-worker.sh" + # Call install-worker.sh script (in parent installation directory) + local install_worker_script="${SCRIPT_DIR}/../installation/install-worker.sh" if [ ! -f "$install_worker_script" ]; then log_error "install-worker.sh script not found: $install_worker_script" exit 1 fi - log_info "Calling install-worker.sh with output path: $OUTPUT_KUBECONFIG" + log_info "Calling install-worker.sh with output path: $WORKER_KUBECONFIG" + + # Prepare environment variables for install-worker.sh + export OUTPUT_KUBECONFIG="$WORKER_KUBECONFIG" + if [ -n "$WORKER_CLUSTER_NAME" ]; then + export WORKER_CLUSTER_NAME="$WORKER_CLUSTER_NAME" + log_info "Using worker cluster name: $WORKER_CLUSTER_NAME" + fi - if bash "$install_worker_script" --output "$OUTPUT_KUBECONFIG"; then + if bash "$install_worker_script"; then log_success "kubeocean-worker installed successfully" else log_error "Failed to install kubeocean-worker" @@ -411,6 +456,55 @@ install_kubeocean_worker() { fi } +# Label all worker nodes +label_worker_nodes() { + log_info "Labeling worker nodes..." + + # Get all nodes + log_info "Getting all nodes in the cluster..." + ret=0 + local nodes=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to get nodes: $nodes" + exit 1 + fi + + if [ -z "$nodes" ]; then + log_warning "No nodes found in the cluster" + return 0 + fi + + # Display nodes + log_info "Found nodes:" + kubectl get nodes -o wide + echo "" + + # Label each node + local node_array=($nodes) + local success_count=0 + local fail_count=0 + ret=0 + + for node in "${node_array[@]}"; do + log_info "Labeling node: $node" + if kubectl label node "$node" kubeocean.io/role=worker --overwrite &> /dev/null; then + log_success " ✓ Labeled node: $node" + ((success_count++)) || ret=$? + else + log_warning " ✗ Failed to label node: $node" + ((fail_count++)) || ret=$? + fi + done + + echo "" + log_success "Node labeling completed: $success_count succeeded, $fail_count failed" + + # Display labeled nodes + log_info "Nodes with label 'kubeocean.io/role=worker':" + kubectl get nodes -l kubeocean.io/role=worker || log_warning "Unable to display labeled nodes" +} + # Display summary information show_summary() { echo "" @@ -419,12 +513,12 @@ show_summary() { echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" echo "" echo "📋 Installation Information:" - echo " • Region: $REGION" - echo " • Cluster ID: $CLUSTER_ID" + echo " • Region: $WORKER_REGION" + echo " • Cluster ID: $WORKER_CLUSTER_ID" echo " • VPC ID: $VPC_ID" - echo " • Subnet ID: $SUBNET_ID" - echo " • Context: worker-admin-$CLUSTER_ID (merged to ~/.kube/config)" - echo " • Worker Kubeconfig (for manager binding): $OUTPUT_KUBECONFIG" + echo " • Subnet ID: $WORKER_SUBNET_ID" + echo " • Context: worker-admin-$WORKER_CLUSTER_ID (merged to ~/.kube/config)" + echo " • Worker Kubeconfig (for manager binding): $WORKER_KUBECONFIG" echo "" echo "🔍 Next Steps:" echo " 1. Verify installation:" @@ -434,10 +528,10 @@ show_summary() { echo " kubectl label node kubeocean.io/role=worker" echo "" echo " 3. Switch to this context anytime:" - echo " kubectl config use-context worker-admin-$CLUSTER_ID" + echo " kubectl config use-context worker-admin-$WORKER_CLUSTER_ID" echo "" echo " 4. Use worker kubeconfig for manager cluster binding:" - echo " # Copy $OUTPUT_KUBECONFIG to manager cluster" + echo " # Copy $WORKER_KUBECONFIG to manager cluster" echo " # Then run install-manager.sh on manager cluster" echo "" } @@ -450,29 +544,47 @@ main() { echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" echo "" - # Parse arguments + # Step 0: Parse config file path first + for arg in "$@"; do + if [[ "$arg" == "-c" || "$arg" == "--config" ]]; then + shift + CONFIG_FILE="$1" + break + fi + shift + done + + # Load configuration file + load_config + + # Parse arguments (command line overrides config file) parse_args "$@" # Step 1: Check prerequisites - echo "📋 Step 1/4: Checking Prerequisites" + echo "📋 Step 1/5: Checking Prerequisites" check_prerequisites echo "" # Step 2: Enable cluster internal network access - echo "🌐 Step 2/4: Enabling Cluster Internal Network Access" + echo "🌐 Step 2/5: Enabling Cluster Internal Network Access" enable_cluster_internal_access echo "" # Step 3: Get cluster kubeconfig - echo "🔐 Step 3/4: Getting Cluster Kubeconfig" + echo "🔐 Step 3/5: Getting Cluster Kubeconfig" get_cluster_kubeconfig echo "" # Step 4: Install kubeocean-worker - echo "📦 Step 4/4: Installing Kubeocean-Worker" + echo "📦 Step 4/5: Installing Kubeocean-Worker" install_kubeocean_worker echo "" + # Step 5: Label worker nodes + echo "🏷️ Step 5/5: Labeling Worker Nodes" + label_worker_nodes + echo "" + # Display summary show_summary } diff --git a/examples/cookbook/installation-tke/uninstall-manager-tke.sh b/examples/cookbook/installation-tke/uninstall-manager-tke.sh new file mode 100755 index 0000000..a92f8f3 --- /dev/null +++ b/examples/cookbook/installation-tke/uninstall-manager-tke.sh @@ -0,0 +1,508 @@ +#!/bin/bash + +# Copyright 2025 The Kubeocean Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Kubeocean Manager Cluster Uninstallation Script for TKE +# Features: +# 1. Get cluster kubeconfig +# 2. Delete ClusterBinding resources +# 3. Delete worker kubeconfig secrets +# 4. Uninstall kubeocean-manager components + +set -e +set -o pipefail + +# Color definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Logging functions +log_info() { + echo -e "${BLUE}ℹ️ $1${NC}" +} + +log_success() { + echo -e "${GREEN}✅ $1${NC}" +} + +log_warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +log_error() { + echo -e "${RED}❌ $1${NC}" +} + +# Error handling +trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR + +# Script root directory (assuming script is in examples/playbook/installation-tke directory) +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" + +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Manager Cluster Configuration variables +MANAGER_REGION="" +MANAGER_CLUSTER_ID="" +SKIP_MANAGER_UNINSTALL="${SKIP_MANAGER_UNINSTALL:-false}" + +# Worker Cluster Configuration variables +WORKER_CLUSTER_NAME="example-cluster" + +# Display help information +show_help() { + cat << EOF +Kubeocean Manager Cluster Uninstallation Script for TKE + +Usage: $0 [options] + +Options: + -h, --help Display this help message + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + MANAGER_REGION Manager cluster region + MANAGER_CLUSTER_ID Manager cluster ID + +Optional Configuration Variables: + SKIP_MANAGER_UNINSTALL Skip manager uninstallation (default: false) + WORKER_CLUSTER_NAME Worker cluster name for binding deletion (default: example-cluster) + +Prerequisites: + tccli Tencent Cloud CLI tool (must be installed and configured) + jq Command-line JSON processor + kubectl Kubernetes command-line tool + helm v3.x Helm package manager (version 3.x required) + +Examples: + # Use default configuration file (./config.env) + $0 + + # Use custom configuration file + $0 -c /path/to/my-config.env + +EOF +} + +# Load configuration from file +load_config() { + if [[ -f "$CONFIG_FILE" ]]; then + log_info "Loading configuration from: $CONFIG_FILE" + set -a + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + set +a + log_success "Configuration loaded" + fi +} + +# Parse command line arguments +parse_args() { + while [[ $# -gt 0 ]]; do + case $1 in + -h|--help) + show_help + exit 0 + ;; + -c|--config) + CONFIG_FILE="$2" + shift 2 + ;; + *) + log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + log_info "All other configurations should be provided in config file" + show_help + exit 1 + ;; + esac + done +} + +# Check prerequisites +check_prerequisites() { + log_info "Checking prerequisites..." + + # Check required parameters + if [ -z "$MANAGER_REGION" ]; then + log_error "MANAGER_REGION is required. Please set it in config file" + exit 1 + fi + + if [ -z "$MANAGER_CLUSTER_ID" ]; then + log_error "MANAGER_CLUSTER_ID is required. Please set it in config file" + exit 1 + fi + + log_info "Manager Region: $MANAGER_REGION" + log_info "Manager Cluster ID: $MANAGER_CLUSTER_ID" + log_info "Worker Cluster Name: $WORKER_CLUSTER_NAME" + + # Check tccli installation + if ! command -v tccli &> /dev/null; then + log_error "tccli is not installed. Please install Tencent Cloud CLI first" + log_error "Visit: https://cloud.tencent.com/document/product/440/6176" + exit 1 + fi + + log_success "Found tccli: $(which tccli)" + + # Check jq installation + if ! command -v jq &> /dev/null; then + log_error "jq is not installed. Please install jq first" + log_error "Visit: https://jqlang.org/download/" + exit 1 + fi + + log_success "Found jq: $(which jq)" + + # Check kubectl + if ! command -v kubectl &> /dev/null; then + log_error "kubectl is not installed. Please install kubectl first" + exit 1 + fi + + log_success "Found kubectl: $(which kubectl)" + + # Check Helm installation + if ! command -v helm &> /dev/null; then + log_error "Helm is not installed. Please install Helm first" + log_error "Visit: https://helm.sh/docs/intro/install/" + exit 1 + fi + + # Check Helm version (must be v3) + HELM_VERSION_OUTPUT=$(helm version --short 2>/dev/null || echo "") + if [ -z "$HELM_VERSION_OUTPUT" ]; then + log_error "Failed to get Helm version" + exit 1 + fi + + if echo "$HELM_VERSION_OUTPUT" | grep -q "^v3\."; then + log_success "Helm version check passed: $HELM_VERSION_OUTPUT" + else + log_error "Helm version must be v3.x, but found: $HELM_VERSION_OUTPUT" + exit 1 + fi + + log_success "Prerequisites check passed" +} + +# Get cluster kubeconfig +get_cluster_kubeconfig() { + log_info "Getting cluster kubeconfig..." + + # Check if context already exists + local context_name="manager-admin-$MANAGER_CLUSTER_ID" + if kubectl config get-contexts "$context_name" &> /dev/null; then + log_info "Context '$context_name' already exists, reusing it" + kubectl config use-context "$context_name" + log_success "Switched to existing context: $context_name" + return 0 + fi + + log_info "Context not found, fetching kubeconfig from TKE..." + + # Check if cluster exists and get cluster info + local ret=0 + cluster_info=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusters --ClusterIds "[\"$MANAGER_CLUSTER_ID\"]" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to get cluster info: $cluster_info" + exit 1 + fi + + # Check if cluster exists + local cluster_count=$(echo "$cluster_info" | jq -r '.Clusters | length') + if [ "$cluster_count" -eq 0 ]; then + log_error "Cluster $MANAGER_CLUSTER_ID not found in region $MANAGER_REGION" + exit 1 + fi + + log_success "Cluster found: $MANAGER_CLUSTER_ID" + + # Get cluster security info (including kubeconfig) + log_info "Getting cluster security credentials..." + local security_info + ret=0 + security_info=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusterSecurity --ClusterId "$MANAGER_CLUSTER_ID" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to get cluster security information" + log_error "$security_info" + exit 1 + fi + + # Extract internal kubeconfig + local kubeconfig_content + kubeconfig_content=$(echo "$security_info" | jq -r '.Kubeconfig // empty') + + if [ -z "$kubeconfig_content" ]; then + log_error "Failed to extract kubeconfig from cluster security information" + exit 1 + fi + + # Extract PgwEndpoint (internal network endpoint) + local pgw_endpoint + pgw_endpoint=$(echo "$security_info" | jq -r '.PgwEndpoint // empty') + + if [ -z "$pgw_endpoint" ]; then + log_error "Failed to extract PgwEndpoint from cluster security information" + exit 1 + fi + + log_info "PgwEndpoint: $pgw_endpoint" + + # Save kubeconfig to temporary file + local temp_kubeconfig="/tmp/tke-kubeconfig-$MANAGER_CLUSTER_ID" + echo "$kubeconfig_content" > "$temp_kubeconfig" + + # Replace server address with PgwEndpoint + log_info "Replacing server address with PgwEndpoint..." + local cluster_name + cluster_name=$(kubectl --kubeconfig="$temp_kubeconfig" config view -o jsonpath='{.clusters[0].name}' 2>/dev/null) + + if [ -n "$cluster_name" ]; then + kubectl --kubeconfig="$temp_kubeconfig" config set-cluster "$cluster_name" --server="https://$pgw_endpoint" &> /dev/null + log_success "Server address updated to: $pgw_endpoint" + else + log_error "Failed to get cluster name from kubeconfig" + exit 1 + fi + + # Verify connection before merging + log_info "Verifying cluster connection..." + if ! kubectl --kubeconfig="$temp_kubeconfig" cluster-info &> /dev/null; then + log_error "Failed to connect to cluster using the kubeconfig" + log_error "Please check if the cluster internal network access is properly configured" + rm -f "$temp_kubeconfig" + exit 1 + fi + log_success "Successfully verified cluster connection" + + # Set kubectl context name + local context_name="manager-admin-$MANAGER_CLUSTER_ID" + + log_info "Merging kubeconfig to default kubectl config" + log_info "Context name: $context_name" + + # Get default KUBECONFIG path + local default_kubeconfig="${HOME}/.kube/config" + + # Create .kube directory if it doesn't exist + mkdir -p "${HOME}/.kube" + + # If default kubeconfig doesn't exist, create an empty one + if [ ! -f "$default_kubeconfig" ]; then + touch "$default_kubeconfig" + fi + + # Merge the TKE kubeconfig into default kubeconfig + KUBECONFIG="$default_kubeconfig:$temp_kubeconfig" kubectl config view --flatten > "${temp_kubeconfig}.merged" + mv "${temp_kubeconfig}.merged" "$default_kubeconfig" + + # Get the original context name from temp kubeconfig + local original_context + original_context=$(kubectl --kubeconfig="$temp_kubeconfig" config current-context 2>/dev/null) + + # Rename context to our desired name + if [ -n "$original_context" ]; then + kubectl config rename-context "$original_context" "$context_name" &> /dev/null || true + fi + + # Set as current context + kubectl config use-context "$context_name" &> /dev/null + + # Clean up temporary file + rm -f "$temp_kubeconfig" + + log_success "Kubeconfig merged to: $default_kubeconfig" + log_success "Context name: $context_name" + log_success "Cluster is ready for use" +} + +# Uninstall kubeocean-manager +uninstall_kubeocean_manager() { + log_info "Uninstalling kubeocean-manager..." + + # Call uninstall-manager.sh script (in parent installation directory) + local uninstall_manager_script="${SCRIPT_DIR}/../installation/uninstall-manager.sh" + + if [ ! -f "$uninstall_manager_script" ]; then + log_error "uninstall-manager.sh script not found: $uninstall_manager_script" + exit 1 + fi + + log_info "Calling uninstall-manager.sh with worker cluster: $WORKER_CLUSTER_NAME" + + # Prepare environment variables for uninstall-manager.sh + export SKIP_MANAGER_UNINSTALL="$SKIP_MANAGER_UNINSTALL" + + if [ "$SKIP_MANAGER_UNINSTALL" = "true" ]; then + log_info "SKIP_MANAGER_UNINSTALL is set to true, only deleting bindings and secrets" + fi + + if bash "$uninstall_manager_script" --cluster-name "$WORKER_CLUSTER_NAME"; then + log_success "kubeocean-manager uninstalled successfully" + else + log_error "Failed to uninstall kubeocean-manager" + exit 1 + fi +} + +# Cleanup kubectl config +cleanup_kubectl_config() { + # Skip cleanup if SKIP_MANAGER_UNINSTALL is true + if [ "$SKIP_MANAGER_UNINSTALL" = "true" ]; then + log_info "Skipping kubectl config cleanup (SKIP_MANAGER_UNINSTALL=true)" + log_info "Manager context will be preserved for future use" + return 0 + fi + + log_info "Cleaning up kubectl config..." + + local context_name="manager-admin-$MANAGER_CLUSTER_ID" + + # Check if context exists + if ! kubectl config get-contexts "$context_name" &> /dev/null; then + log_info "Context '$context_name' not found, skipping cleanup" + return 0 + fi + + # Get cluster and user names associated with this context + local cluster_name + local user_name + + cluster_name=$(kubectl config view -o jsonpath="{.contexts[?(@.name=='$context_name')].context.cluster}" 2>/dev/null) + user_name=$(kubectl config view -o jsonpath="{.contexts[?(@.name=='$context_name')].context.user}" 2>/dev/null) + + log_info "Found context: $context_name" + if [ -n "$cluster_name" ]; then + log_info " Cluster: $cluster_name" + fi + if [ -n "$user_name" ]; then + log_info " User: $user_name" + fi + + # Delete context + if kubectl config delete-context "$context_name" &> /dev/null; then + log_success "Deleted context: $context_name" + else + log_warning "Failed to delete context: $context_name" + fi + + # Delete cluster + if [ -n "$cluster_name" ]; then + if kubectl config delete-cluster "$cluster_name" &> /dev/null; then + log_success "Deleted cluster: $cluster_name" + else + log_warning "Failed to delete cluster: $cluster_name" + fi + fi + + # Delete user + if [ -n "$user_name" ]; then + if kubectl config unset "users.$user_name" &> /dev/null; then + log_success "Deleted user: $user_name" + else + log_warning "Failed to delete user: $user_name" + fi + fi + + log_success "Kubectl config cleanup completed" +} + +# Display summary information +show_summary() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + log_success "Kubeocean Manager Cluster Uninstallation Complete!" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "📋 Uninstallation Information:" + echo " • Manager Region: $MANAGER_REGION" + echo " • Manager Cluster ID: $MANAGER_CLUSTER_ID" + if [ "$SKIP_MANAGER_UNINSTALL" = "true" ]; then + echo " • Context: manager-admin-$MANAGER_CLUSTER_ID (preserved)" + else + echo " • Context: manager-admin-$MANAGER_CLUSTER_ID (cleaned up)" + fi + echo " • Worker Cluster Name: $WORKER_CLUSTER_NAME" + echo "" + echo "🔍 Verification Commands:" + echo " • Check if namespace deleted:" + echo " kubectl get ns kubeocean-system" + echo "" + echo " • Check if ClusterBindings deleted:" + echo " kubectl get clusterbindings" + echo "" +} + +# Main function +main() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo " 🗑️ TKE Manager Cluster Uninstallation Script" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + + # Step 0: Parse config file path first + for arg in "$@"; do + if [[ "$arg" == "-c" || "$arg" == "--config" ]]; then + shift + CONFIG_FILE="$1" + break + fi + shift + done + + # Load configuration file + load_config + + # Parse arguments (command line overrides config file) + parse_args "$@" + + # Step 1: Check prerequisites + echo "📋 Step 1/4: Checking Prerequisites" + check_prerequisites + echo "" + + # Step 2: Get cluster kubeconfig + echo "🔐 Step 2/4: Getting Cluster Kubeconfig" + get_cluster_kubeconfig + echo "" + + # Step 3: Uninstall kubeocean-manager + echo "🗑️ Step 3/4: Uninstalling Kubeocean-Manager" + uninstall_kubeocean_manager + echo "" + + # Step 4: Cleanup kubectl config + echo "🧹 Step 4/4: Cleaning Up Kubectl Config" + cleanup_kubectl_config + echo "" + + # Display summary + show_summary +} + +# Execute main function +main "$@" + diff --git a/examples/cookbook/installation-tke/uninstall-tke.sh b/examples/cookbook/installation-tke/uninstall-tke.sh new file mode 100755 index 0000000..7988ae4 --- /dev/null +++ b/examples/cookbook/installation-tke/uninstall-tke.sh @@ -0,0 +1,266 @@ +#!/bin/bash + +# Copyright 2025 The Kubeocean Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Kubeocean One-Click Uninstallation Script for TKE +# Features: +# 1. Uninstall manager cluster and remove cluster binding +# 2. Uninstall worker cluster components + +set -e +set -o pipefail + +# Color definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Logging functions +log_info() { + echo -e "${BLUE}ℹ️ $1${NC}" +} + +log_success() { + echo -e "${GREEN}✅ $1${NC}" +} + +log_warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +log_error() { + echo -e "${RED}❌ $1${NC}" +} + +# Error handling +trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR + +# Script root directory +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Display help information +show_help() { + cat << EOF +Kubeocean One-Click Uninstallation Script for TKE + +This script automates the complete uninstallation process by sequentially executing: +1. uninstall-manager-tke.sh - Uninstall manager cluster components and remove binding +2. uninstall-worker-tke.sh - Uninstall worker cluster components + +Usage: $0 [options] + +Options: + -h, --help Display this help message + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + MANAGER_REGION Manager cluster region + MANAGER_CLUSTER_ID Manager cluster ID + WORKER_REGION Worker cluster region + WORKER_CLUSTER_ID Worker cluster ID + +Optional Configuration Variables: + WORKER_CLUSTER_NAME Worker cluster name (default: example-cluster) + SKIP_MANAGER_UNINSTALL Skip manager uninstallation (default: false) + +Prerequisites: + tccli Tencent Cloud CLI tool (must be installed and configured) + jq Command-line JSON processor + kubectl Kubernetes command-line tool + +Examples: + # Use default configuration file (./config.env) + $0 + + # Use custom configuration file + $0 -c /path/to/my-config.env + + # Quick uninstallation + $0 + +Note: + - Manager cluster will be uninstalled first, then worker cluster + - Kubectl config contexts will be automatically cleaned up + - Set SKIP_MANAGER_UNINSTALL=true to only remove cluster binding without uninstalling manager components + +EOF +} + +# Parse command line arguments +parse_args() { + while [[ $# -gt 0 ]]; do + case $1 in + -h|--help) + show_help + exit 0 + ;; + -c|--config) + CONFIG_FILE="$2" + shift 2 + ;; + *) + log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + show_help + exit 1 + ;; + esac + done +} + +# Check if config file exists +check_config_file() { + if [ ! -f "$CONFIG_FILE" ]; then + log_error "Configuration file not found: $CONFIG_FILE" + log_info "Please provide a valid configuration file" + exit 1 + fi + + log_info "Using configuration file: $CONFIG_FILE" +} + +# Load configuration from config file +load_config() { + log_info "Loading configuration from: $CONFIG_FILE" + + # Source the config file + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + + # Export all configuration variables so they are available to child scripts + export MANAGER_REGION + export MANAGER_CLUSTER_ID + export WORKER_REGION + export WORKER_CLUSTER_ID + export WORKER_CLUSTER_NAME + export SKIP_MANAGER_UNINSTALL + + log_success "Configuration loaded and exported as environment variables" +} + +# Uninstall manager cluster +uninstall_manager() { + local uninstall_manager_script="${SCRIPT_DIR}/uninstall-manager-tke.sh" + + if [ ! -f "$uninstall_manager_script" ]; then + log_error "uninstall-manager-tke.sh not found: $uninstall_manager_script" + exit 1 + fi + + log_info "Executing uninstall-manager-tke.sh..." + log_info "Configuration passed via environment variables" + + if bash "$uninstall_manager_script"; then + log_success "Manager cluster uninstallation completed" + else + log_error "Manager cluster uninstallation failed" + exit 1 + fi +} + +# Uninstall worker cluster +uninstall_worker() { + local uninstall_worker_script="${SCRIPT_DIR}/uninstall-worker-tke.sh" + + if [ ! -f "$uninstall_worker_script" ]; then + log_error "uninstall-worker-tke.sh not found: $uninstall_worker_script" + exit 1 + fi + + log_info "Executing uninstall-worker-tke.sh..." + log_info "Configuration passed via environment variables" + + if bash "$uninstall_worker_script"; then + log_success "Worker cluster uninstallation completed" + else + log_error "Worker cluster uninstallation failed" + exit 1 + fi +} + +# Display final summary +show_final_summary() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + log_success "Kubeocean One-Click Uninstallation Complete!" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "📋 Uninstallation Summary:" + echo " • Configuration: $CONFIG_FILE" + echo " • Manager cluster components uninstalled ✓" + echo " • Worker cluster components uninstalled ✓" + echo " • Kubectl config contexts cleaned up ✓" + echo "" + echo "🔍 Verification:" + echo " # Verify manager namespace removed (switch context if needed)" + echo " kubectl get ns kubeocean-system" + echo "" + echo " # Verify worker namespace removed (switch context if needed)" + echo " kubectl get ns kubeocean-worker" + echo "" + echo " # Check kubectl contexts" + echo " kubectl config get-contexts" + echo "" + echo "🎉 Kubeocean has been completely uninstalled!" + echo "" +} + +# Main function +main() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo " 🗑️ Kubeocean One-Click Uninstallation for TKE" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + + # Parse arguments + parse_args "$@" + + # Check config file + check_config_file + + # Load configuration and export as environment variables + load_config + echo "" + + # Step 1: Uninstall manager cluster + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🗑️ Phase 1/2: Uninstalling Manager Cluster" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + uninstall_manager + echo "" + + # Step 2: Uninstall worker cluster + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🗑️ Phase 2/2: Uninstalling Worker Cluster" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + uninstall_worker + echo "" + + # Display final summary + show_final_summary +} + +# Execute main function +main "$@" + + diff --git a/examples/cookbook/installation-tke/uninstall-worker-tke.sh b/examples/cookbook/installation-tke/uninstall-worker-tke.sh new file mode 100755 index 0000000..3137432 --- /dev/null +++ b/examples/cookbook/installation-tke/uninstall-worker-tke.sh @@ -0,0 +1,480 @@ +#!/bin/bash + +# Copyright 2025 The Kubeocean Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Kubeocean Worker Cluster Uninstallation Script for TKE +# Features: +# 1. Get cluster kubeconfig +# 2. Uninstall kubeocean-worker components +# 3. Delete ResourceLeasingPolicy resources + +set -e +set -o pipefail + +# Color definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Logging functions +log_info() { + echo -e "${BLUE}ℹ️ $1${NC}" +} + +log_success() { + echo -e "${GREEN}✅ $1${NC}" +} + +log_warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +log_error() { + echo -e "${RED}❌ $1${NC}" +} + +# Error handling +trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR + +# Script root directory (assuming script is in examples/playbook/installation-tke directory) +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" + +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Worker Cluster Configuration variables +WORKER_REGION="" +WORKER_CLUSTER_ID="" + +# Display help information +show_help() { + cat << EOF +Kubeocean Worker Cluster Uninstallation Script for TKE + +Usage: $0 [options] + +Options: + -h, --help Display this help message + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + WORKER_REGION Worker cluster region + WORKER_CLUSTER_ID Worker cluster ID + +Prerequisites: + tccli Tencent Cloud CLI tool (must be installed and configured) + jq Command-line JSON processor + kubectl Kubernetes command-line tool + helm v3.x Helm package manager (version 3.x required) + +Examples: + # Use default configuration file (./config.env) + $0 + + # Use custom configuration file + $0 -c /path/to/my-config.env + +EOF +} + +# Load configuration from file +load_config() { + if [[ -f "$CONFIG_FILE" ]]; then + log_info "Loading configuration from: $CONFIG_FILE" + set -a + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + set +a + log_success "Configuration loaded" + fi +} + +# Parse command line arguments +parse_args() { + while [[ $# -gt 0 ]]; do + case $1 in + -h|--help) + show_help + exit 0 + ;; + -c|--config) + CONFIG_FILE="$2" + shift 2 + ;; + *) + log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + log_info "All other configurations should be provided in config file" + show_help + exit 1 + ;; + esac + done +} + +# Check prerequisites +check_prerequisites() { + log_info "Checking prerequisites..." + + # Check required parameters + if [ -z "$WORKER_REGION" ]; then + log_error "WORKER_REGION is required. Please set it in config file" + exit 1 + fi + + if [ -z "$WORKER_CLUSTER_ID" ]; then + log_error "WORKER_CLUSTER_ID is required. Please set it in config file" + exit 1 + fi + + log_info "Worker Region: $WORKER_REGION" + log_info "Worker Cluster ID: $WORKER_CLUSTER_ID" + + # Check tccli installation + if ! command -v tccli &> /dev/null; then + log_error "tccli is not installed. Please install Tencent Cloud CLI first" + log_error "Visit: https://cloud.tencent.com/document/product/440/6176" + exit 1 + fi + + log_success "Found tccli: $(which tccli)" + + # Check jq installation + if ! command -v jq &> /dev/null; then + log_error "jq is not installed. Please install jq first" + log_error "Visit: https://jqlang.org/download/" + exit 1 + fi + + log_success "Found jq: $(which jq)" + + # Check kubectl + if ! command -v kubectl &> /dev/null; then + log_error "kubectl is not installed. Please install kubectl first" + exit 1 + fi + + log_success "Found kubectl: $(which kubectl)" + + # Check Helm installation + if ! command -v helm &> /dev/null; then + log_error "Helm is not installed. Please install Helm first" + log_error "Visit: https://helm.sh/docs/intro/install/" + exit 1 + fi + + # Check Helm version (must be v3) + HELM_VERSION_OUTPUT=$(helm version --short 2>/dev/null || echo "") + if [ -z "$HELM_VERSION_OUTPUT" ]; then + log_error "Failed to get Helm version" + exit 1 + fi + + if echo "$HELM_VERSION_OUTPUT" | grep -q "^v3\."; then + log_success "Helm version check passed: $HELM_VERSION_OUTPUT" + else + log_error "Helm version must be v3.x, but found: $HELM_VERSION_OUTPUT" + exit 1 + fi + + log_success "Prerequisites check passed" +} + +# Get cluster kubeconfig +get_cluster_kubeconfig() { + log_info "Getting cluster kubeconfig..." + + # Check if context already exists + local context_name="worker-admin-$WORKER_CLUSTER_ID" + if kubectl config get-contexts "$context_name" &> /dev/null; then + log_info "Context '$context_name' already exists, reusing it" + kubectl config use-context "$context_name" + log_success "Switched to existing context: $context_name" + return 0 + fi + + log_info "Context not found, fetching kubeconfig from TKE..." + + # Check if cluster exists and get cluster info + local ret=0 + cluster_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusters --ClusterIds "[\"$WORKER_CLUSTER_ID\"]" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to get cluster info: $cluster_info" + exit 1 + fi + + # Check if cluster exists + local cluster_count=$(echo "$cluster_info" | jq -r '.Clusters | length') + if [ "$cluster_count" -eq 0 ]; then + log_error "Cluster $WORKER_CLUSTER_ID not found in region $WORKER_REGION" + exit 1 + fi + + log_success "Cluster found: $WORKER_CLUSTER_ID" + + # Get cluster security info (kubeconfig) + log_info "Getting cluster security credentials..." + # Get cluster security info (including kubeconfig) + local security_info + ret=0 + security_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusterSecurity --ClusterId "$WORKER_CLUSTER_ID" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to get cluster security information" + log_error "$security_info" + exit 1 + fi + + # Extract internal kubeconfig + local kubeconfig_content + kubeconfig_content=$(echo "$security_info" | jq -r '.Kubeconfig // empty') + + if [ -z "$kubeconfig_content" ]; then + log_error "Failed to extract kubeconfig from cluster security information" + exit 1 + fi + + # Extract PgwEndpoint (internal network endpoint) + local pgw_endpoint + pgw_endpoint=$(echo "$security_info" | jq -r '.PgwEndpoint // empty') + + if [ -z "$pgw_endpoint" ]; then + log_error "Failed to extract PgwEndpoint from cluster security information" + exit 1 + fi + + log_info "PgwEndpoint: $pgw_endpoint" + + # Save kubeconfig to temporary file + local temp_kubeconfig="/tmp/tke-kubeconfig-$WORKER_CLUSTER_ID" + echo "$kubeconfig_content" > "$temp_kubeconfig" + + # Replace server address with PgwEndpoint + log_info "Replacing server address with PgwEndpoint..." + local cluster_name + cluster_name=$(kubectl --kubeconfig="$temp_kubeconfig" config view -o jsonpath='{.clusters[0].name}' 2>/dev/null) + + if [ -n "$cluster_name" ]; then + kubectl --kubeconfig="$temp_kubeconfig" config set-cluster "$cluster_name" --server="https://$pgw_endpoint" &> /dev/null + log_success "Server address updated to: $pgw_endpoint" + else + log_error "Failed to get cluster name from kubeconfig" + exit 1 + fi + + # Verify connection before merging + log_info "Verifying cluster connection..." + if ! kubectl --kubeconfig="$temp_kubeconfig" cluster-info &> /dev/null; then + log_error "Failed to connect to cluster using the kubeconfig" + log_error "Please check if the cluster internal network access is properly configured" + rm -f "$temp_kubeconfig" + exit 1 + fi + log_success "Successfully verified cluster connection" + + # Set kubectl context name + local context_name="worker-admin-$WORKER_CLUSTER_ID" + + log_info "Merging kubeconfig to default kubectl config" + log_info "Context name: $context_name" + + # Get default KUBECONFIG path + local default_kubeconfig="${HOME}/.kube/config" + + # Create .kube directory if it doesn't exist + mkdir -p "${HOME}/.kube" + + # If default kubeconfig doesn't exist, create an empty one + if [ ! -f "$default_kubeconfig" ]; then + touch "$default_kubeconfig" + fi + + # Merge the TKE kubeconfig into default kubeconfig + KUBECONFIG="$default_kubeconfig:$temp_kubeconfig" kubectl config view --flatten > "${temp_kubeconfig}.merged" + mv "${temp_kubeconfig}.merged" "$default_kubeconfig" + + # Get the original context name from temp kubeconfig + local original_context + original_context=$(kubectl --kubeconfig="$temp_kubeconfig" config current-context 2>/dev/null) + + # Rename context to our desired name + if [ -n "$original_context" ]; then + kubectl config rename-context "$original_context" "$context_name" &> /dev/null || true + fi + + # Set as current context + kubectl config use-context "$context_name" &> /dev/null + + # Clean up temporary file + rm -f "$temp_kubeconfig" + + log_success "Kubeconfig merged to: $default_kubeconfig" + log_success "Context name: $context_name" + log_success "Cluster is ready for use" +} + +# Uninstall kubeocean-worker +uninstall_kubeocean_worker() { + log_info "Uninstalling kubeocean-worker..." + + # Call uninstall-worker.sh script (in parent installation directory) + local uninstall_worker_script="${SCRIPT_DIR}/../installation/uninstall-worker.sh" + + if [ ! -f "$uninstall_worker_script" ]; then + log_error "uninstall-worker.sh script not found: $uninstall_worker_script" + exit 1 + fi + + log_info "Calling uninstall-worker.sh" + + if bash "$uninstall_worker_script"; then + log_success "kubeocean-worker uninstalled successfully" + else + log_error "Failed to uninstall kubeocean-worker" + exit 1 + fi +} + +# Cleanup kubectl config +cleanup_kubectl_config() { + log_info "Cleaning up kubectl config..." + + local context_name="worker-admin-$WORKER_CLUSTER_ID" + + # Check if context exists + if ! kubectl config get-contexts "$context_name" &> /dev/null; then + log_info "Context '$context_name' not found, skipping cleanup" + return 0 + fi + + # Get cluster and user names associated with this context + local cluster_name + local user_name + + cluster_name=$(kubectl config view -o jsonpath="{.contexts[?(@.name=='$context_name')].context.cluster}" 2>/dev/null) + user_name=$(kubectl config view -o jsonpath="{.contexts[?(@.name=='$context_name')].context.user}" 2>/dev/null) + + log_info "Found context: $context_name" + if [ -n "$cluster_name" ]; then + log_info " Cluster: $cluster_name" + fi + if [ -n "$user_name" ]; then + log_info " User: $user_name" + fi + + # Delete context + if kubectl config delete-context "$context_name" &> /dev/null; then + log_success "Deleted context: $context_name" + else + log_warning "Failed to delete context: $context_name" + fi + + # Delete cluster + if [ -n "$cluster_name" ]; then + if kubectl config delete-cluster "$cluster_name" &> /dev/null; then + log_success "Deleted cluster: $cluster_name" + else + log_warning "Failed to delete cluster: $cluster_name" + fi + fi + + # Delete user + if [ -n "$user_name" ]; then + if kubectl config unset "users.$user_name" &> /dev/null; then + log_success "Deleted user: $user_name" + else + log_warning "Failed to delete user: $user_name" + fi + fi + + log_success "Kubectl config cleanup completed" +} + +# Display summary information +show_summary() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + log_success "Kubeocean Worker Cluster Uninstallation Complete!" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "📋 Uninstallation Information:" + echo " • Worker Region: $WORKER_REGION" + echo " • Worker Cluster ID: $WORKER_CLUSTER_ID" + echo " • Context: worker-admin-$WORKER_CLUSTER_ID (cleaned up)" + echo "" + echo "🔍 Verification Commands:" + echo " • Check if namespace deleted:" + echo " kubectl get ns kubeocean-worker" + echo "" + echo " • Check if ResourceLeasingPolicy deleted:" + echo " kubectl get resourceleasingpolicies" + echo "" +} + +# Main function +main() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo " 🗑️ TKE Worker Cluster Uninstallation Script" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + + # Step 0: Parse config file path first + for arg in "$@"; do + if [[ "$arg" == "-c" || "$arg" == "--config" ]]; then + shift + CONFIG_FILE="$1" + break + fi + shift + done + + # Load configuration file + load_config + + # Parse arguments (command line overrides config file) + parse_args "$@" + + # Step 1: Check prerequisites + echo "📋 Step 1/4: Checking Prerequisites" + check_prerequisites + echo "" + + # Step 2: Get cluster kubeconfig + echo "🔐 Step 2/4: Getting Cluster Kubeconfig" + get_cluster_kubeconfig + echo "" + + # Step 3: Uninstall kubeocean-worker + echo "🗑️ Step 3/4: Uninstalling Kubeocean-Worker" + uninstall_kubeocean_worker + echo "" + + # Step 4: Cleanup kubectl config + echo "🧹 Step 4/4: Cleaning Up Kubectl Config" + cleanup_kubectl_config + echo "" + + # Display summary + show_summary +} + +# Execute main function +main "$@" + diff --git a/examples/cookbook/installation/README.md b/examples/cookbook/installation/README.md new file mode 100644 index 0000000..6ff6c2e --- /dev/null +++ b/examples/cookbook/installation/README.md @@ -0,0 +1,81 @@ +# Kubeocean One-Click Deployment Playbook + +> English | [中文](README_zh.md) + +This directory provides one-click deployment scripts for Kubeocean components and cluster binding, supporting a one-stop experience of Kubeocean-related features. + +## Features + +- Deploy Kubeocean components on compute clusters and worker clusters +- Bind worker clusters to compute clusters and configure simple resource leasing policies + +## Prerequisites (Environment Requirements) + +- At least one Kubernetes cluster is required as a virtual compute cluster, and another Kubernetes cluster as a worker cluster +- Pod network and node network are directly interconnected between compute cluster and worker cluster +- Can access clusters using `kubectl` +- Local environment has `helm` installed with version v3 +- Both clusters have APIServer internal network access enabled, i.e., there is a service named `kubernetes-intranet` with type `LoadBalancer` in the `default` namespace. For TKE standard clusters, refer to the following image to enable it in the cluster console: +![k8s-svc](../../docs/images/k8s-svc.png) +- Other requirements refer to: [Requirements](../../docs/requirements.md) + +## Basic Usage + +### Install and Bind +```bash +# Switch kubectl to worker cluster +bash install-worker.sh +# Add label to nodes that need resource extraction +kubectl label node kubeocean.io/role=worker + +# Switch kubectl to compute cluster, and copy the generated /tmp/kubeconfig-worker to local +bash install-manager.sh +``` +*Note: Script execution order cannot be changed* + +### Uninstall +```bash +# Switch kubectl to compute cluster +bash uninstall-manager.sh + +# Switch kubectl to worker cluster +bash uninstall-worker.sh +``` +*Note: Script execution order cannot be changed* + +## Advanced Usage + +### Install and Bind +```bash +# Specify output path for worker kubeconfig +bash install-worker.sh -o /tmp/my-kubeconfig +bash install-worker.sh --output /tmp/my-kubeconfig + +# Skip ResourceLeasingPolicy deployment +bash install-worker.sh --skip-rlp + +# Install manager with specified worker cluster ID and name +bash install-manager.sh -i cls-prod -n prod-cluster +bash install-manager.sh --cluster-id cls-prod --cluster-name prod-cluster + +# Install manager with specified worker kubeconfig input path +bash install-manager.sh -w /tmp/kubeconfig-worker1 +bash install-manager.sh --worker-kubeconfig /tmp/kubeconfig-worker1 + +# Only bind cluster, skip manager installation, and specify cluster ID and name +bash install-manager.sh -i cls-prod -n prod-cluster --skip-manager +``` + +### Uninstall +```bash +# Unbind specific worker cluster by name +bash uninstall-manager.sh -n worker1 +bash uninstall-manager.sh --cluster-name worker1 + +# Only unbind worker cluster without uninstalling manager components +bash uninstall-manager.sh -n worker1 --skip-manager + +# Uninstall and clean up specific RLP object by name +bash uninstall-worker.sh -r my-policy +bash uninstall-worker.sh --rlp-name my-policy +``` diff --git a/examples/cookbook/installation/README_zh.md b/examples/cookbook/installation/README_zh.md new file mode 100644 index 0000000..33c3127 --- /dev/null +++ b/examples/cookbook/installation/README_zh.md @@ -0,0 +1,81 @@ +# Kubeocean 一键部署脚本箱 + +> [English](README.md) | 中文 + +本目录提供了 Kubeocean 组件和集群绑定的一键部署脚本箱,支持一站式体验 Kubeocean 相关功能的能力。 + +## 功能 + +- 在算力集群和工作集群部署 Kubeocean 组件 +- 绑定工作集群到算力集群,并配置简单资源抽取策略 + +## 前置准备(环境要求) + +- 需要至少一个 kubernetes 集群作为虚拟算力集群,以及一个 kubernetes 集群作为工作集群。 +- 算力集群和业务集群 Pod 网络直接互通、节点网络互通 +- 可以使用 `kubectl` 访问集群 +- 本地环境已安装 `helm`,且版本为 v3 +- 集群都已开启 APIServer 内网访问,即在 `default` namespace 下存在名为 `kubernetes-intranet` 且类型为 `LoadBalancer` 的服务。若集群为 TKE 标准集群,可参考下图在集群控制台开启: +![k8s-svc](../../../docs/images/k8s-svc.png) +- 其他要求参考:[要求](../../../docs/requirements_zh.md) + +## 基础使用 + +### 安装并绑定 +``` +# kubectl 切换到工作集群 +bash install-worker.sh +# 给期望抽取资源的节点添加 label +kubectl label node kubeocean.io/role=worker + +# 将 kubectl 切换到算力集群,并将上一步生成的 /tmp/kubeconfig-worker 拷贝到本地 +bash install-manager.sh +``` +*注意:脚本执行顺序不能修改* + +### 卸载 +``` +# kubectl 切换到算力集群 +bash uninstall-manager.sh + +# kubectl 切换到工作集群 +bash uninstall-worker.sh +``` +*注意:脚本执行顺序不能修改* + +## 进阶使用 + +### 安装并绑定 +``` +# 指定 worker kubeconfig 的输出路径 +bash install-worker.sh -o /tmp/my-kubeconfig +bash install-worker.sh --output /tmp/my-kubeconfig + +# 跳过 ResourceLeasingPolicy 的部署 +bash install-worker.sh --skip-rlp + +# 安装 manager 指定工作集群的 ID 和名称 +bash install-manager.sh -i cls-prod -n prod-cluster +bash install-manager.sh --cluster-id cls-prod --cluster-name prod-cluster + +# 安装 manager 指定 worker kubeconfig 的输入路径 +bash install-manager.sh -w /tmp/kubeconfig-worker1 +bash install-manager.sh --worker-kubeconfig /tmp/kubeconfig-worker1 + +# 只绑定集群,跳过安装 manager 步骤,并指定集群ID和名称 +bash install-manager.sh -i cls-prod -n prod-cluster --skip-manager +``` + +### 卸载 +``` +# 解绑特定名称的工作集群 +bash uninstall-manager.sh -n worker1 +bash uninstall-manager.sh --cluster-name worker1 + +# 只解绑工作集群,不卸载 manager 组件 +bash uninstall-manager.sh -n worker1 --skip-manager + +# 卸载并清理特定名称的 rlp 对象 +bash uninstall-worker.sh -r my-policy +bash uninstall-worker.sh --rlp-name my-policy +``` diff --git a/examples/playbook/install-manager.sh b/examples/cookbook/installation/install-manager.sh similarity index 99% rename from examples/playbook/install-manager.sh rename to examples/cookbook/installation/install-manager.sh index 0d32f47..1dc71e3 100755 --- a/examples/playbook/install-manager.sh +++ b/examples/cookbook/installation/install-manager.sh @@ -50,9 +50,9 @@ log_error() { # Error handling trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR -# Script root directory (assuming script is in examples/playbook directory) +# Script root directory (assuming script is in examples/playbook/installation directory) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" # Configuration variables LOCALBIN="${PROJECT_ROOT}/bin" diff --git a/examples/playbook/install-worker.sh b/examples/cookbook/installation/install-worker.sh similarity index 89% rename from examples/playbook/install-worker.sh rename to examples/cookbook/installation/install-worker.sh index fa2c8aa..5615b8c 100755 --- a/examples/playbook/install-worker.sh +++ b/examples/cookbook/installation/install-worker.sh @@ -50,15 +50,14 @@ log_error() { # Error handling trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR -# Script root directory (assuming script is in examples/playbook directory) +# Script root directory (assuming script is in examples/playbook/installation directory) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" # Configuration variables LOCALBIN="${PROJECT_ROOT}/bin" HELM="${LOCALBIN}/helm" KUBECONFIG_SCRIPT="${PROJECT_ROOT}/hack/kubeconfig.sh" -RLP_SAMPLE="${PROJECT_ROOT}/examples/resourceleasingpolicy_sample.yaml" # Fixed parameters (use default values) SERVICEACCOUNT_NAME="kubeocean-syncer" @@ -68,6 +67,7 @@ CLUSTER_NAME="" # Will be extracted from kubeconfig # Configurable parameters (can be overridden via environment variables) OUTPUT_KUBECONFIG="${OUTPUT_KUBECONFIG:-/tmp/kubeconfig-worker}" SKIP_RLP="${SKIP_RLP:-false}" +WORKER_CLUSTER_NAME="${WORKER_CLUSTER_NAME:-example-cluster}" # Display help information show_help() { @@ -84,6 +84,7 @@ Options: Environment Variables: OUTPUT_KUBECONFIG Kubeconfig output file path SKIP_RLP Skip RLP deployment (true/false) + WORKER_CLUSTER_NAME Worker cluster name for ResourceLeasingPolicy (default: example-cluster) Prerequisites: kubectl Kubernetes command-line tool @@ -102,6 +103,9 @@ Examples: # Use environment variables OUTPUT_KUBECONFIG=/tmp/my-kubeconfig $0 + # Specify worker cluster name for ResourceLeasingPolicy + WORKER_CLUSTER_NAME=prod-cluster $0 + EOF } @@ -292,21 +296,45 @@ deploy_rlp() { fi log_info "Deploying ResourceLeasingPolicy..." + log_info "Using worker cluster name: $WORKER_CLUSTER_NAME" + + # Create ResourceLeasingPolicy from template + cat < /dev/null +apiVersion: cloud.tencent.com/v1beta1 +kind: ResourceLeasingPolicy +metadata: + name: example-policy +spec: + cluster: ${WORKER_CLUSTER_NAME} + forceReclaim: true + nodeSelector: + nodeSelectorTerms: + - matchExpressions: + - key: kubeocean.io/role + operator: In + values: ["worker"] + timeWindows: + - start: "18:00" + end: "08:00" + days: ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"] + - start: "00:00" + end: "23:59" + days: ["Saturday", "Sunday"] + resourceLimits: + - resource: cpu + quantity: "96" + percent: 90 + - resource: memory + percent: 95 +EOF - if [ ! -f "$RLP_SAMPLE" ]; then - log_error "ResourceLeasingPolicy sample file not found: $RLP_SAMPLE" - exit 1 - fi - - log_info "Applying ResourceLeasingPolicy: $RLP_SAMPLE" - - if ! kubectl apply -f "$RLP_SAMPLE"; then + if [ $? -eq 0 ]; then + log_success "ResourceLeasingPolicy deployed successfully!" + else log_error "Failed to deploy ResourceLeasingPolicy" exit 1 fi - log_success "ResourceLeasingPolicy deployed successfully!" - # Display deployment status log_info "ResourceLeasingPolicy status:" kubectl get resourceleasingpolicies -o wide || log_warning "Unable to get ResourceLeasingPolicy status" @@ -324,6 +352,7 @@ show_summary() { echo " • Namespace: $NAMESPACE" echo " • ServiceAccount: $SERVICEACCOUNT_NAME" echo " • Kubeconfig: $OUTPUT_KUBECONFIG" + echo " • Worker Cluster Name (RLP): $WORKER_CLUSTER_NAME" echo "" echo "🔍 Verification Commands:" echo " • Check worker resources:" diff --git a/examples/playbook/uninstall-manager.sh b/examples/cookbook/installation/uninstall-manager.sh similarity index 99% rename from examples/playbook/uninstall-manager.sh rename to examples/cookbook/installation/uninstall-manager.sh index 1cb3a8a..a1a6329 100755 --- a/examples/playbook/uninstall-manager.sh +++ b/examples/cookbook/installation/uninstall-manager.sh @@ -50,9 +50,9 @@ log_error() { # Error handling trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR -# Script root directory (assuming script is in examples/playbook directory) +# Script root directory (assuming script is in examples/playbook/installation directory) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" # Configuration variables LOCALBIN="${PROJECT_ROOT}/bin" diff --git a/examples/playbook/uninstall-worker.sh b/examples/cookbook/installation/uninstall-worker.sh similarity index 99% rename from examples/playbook/uninstall-worker.sh rename to examples/cookbook/installation/uninstall-worker.sh index 68c2166..2e016a6 100755 --- a/examples/playbook/uninstall-worker.sh +++ b/examples/cookbook/installation/uninstall-worker.sh @@ -49,9 +49,9 @@ log_error() { # Error handling trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR -# Script root directory (assuming script is in examples/playbook directory) +# Script root directory (assuming script is in examples/playbook/installation directory) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" # Configuration variables LOCALBIN="${PROJECT_ROOT}/bin" diff --git a/examples/cookbook/llm-train-and-infer-cross-clusters/README.md b/examples/cookbook/llm-train-and-infer-cross-clusters/README.md new file mode 100644 index 0000000..5011732 --- /dev/null +++ b/examples/cookbook/llm-train-and-infer-cross-clusters/README.md @@ -0,0 +1,370 @@ +# LLM Training and Inference Cross-Cluster Deployment Example + +> English | [中文](README_zh.md) + +This directory contains a complete example of using Kubeocean for LLM training and inference across clusters. + +## Scenario Description + +This directory supports building a practical training and inference integrated scenario from scratch. The scenario uses two clusters to deploy online inference services and offline training tasks respectively. The online inference service exhibits tidal characteristics: during the day (8:00-18:00 daily), usage is high and requires all GPU resources of the online cluster, while at night (18:00-8:00 daily), usage is low and only requires half of the GPU resources. Offline training tasks are only deployed on the offline cluster and only when the online cluster frees up GPU resources at night. + +The online inference service uses vLLM to deploy the online inference service, while the offline training task uses KubeRay + VeRL to deploy large model reinforcement learning training tasks. The model used is `Qwen2.5-0.5B-Instruct`, and the training dataset is GSM8K. + +**Goal**: Automatically utilize idle GPUs for training tasks at night, and automatically exit training tasks during the day to ensure resources for online inference services. + +**Note**: This project is simplified. Models and datasets are stored in images, and trained checkpoints are stored in local directories. Best practice recommends that models and datasets be loaded through cloud storage systems such as file storage, and trained checkpoints can also be saved through cloud storage systems. + + +## Directory Structure + +``` +. +├── config.env.example # Configuration file example +├── run-demo.sh # One-click deployment script (recommended) +├── clean-demo.sh # One-click cleanup script (recommended) +├── vllm-infer-demo.sh # vLLM inference service deployment script +├── llm-kuberay-verl-demo.sh # VERL training job deployment script +├── is-qwen2-5-05b-vllm.yaml # vLLM inference service YAML configuration +└── verl-raycluster.yaml # RayCluster YAML configuration +``` + +## Prerequisites + +### Resource Requirements + +- At least one TKE standard cluster as the virtual compute cluster and one TKE standard cluster as the worker cluster. +- The worker cluster should have at least 4 GPU nodes, each with at least 2 GPUs (A10 or newer), 24 CPU cores, and 100GB memory. If these requirements are not met, you can manually adjust the resource requirements in the YAML files in this directory. +- Internal network access must be enabled for both compute and worker clusters, which requires deploying internal load balancers at a certain cost. + +### Environment Requirements + +- Compute cluster and worker cluster Pod networks are directly interconnected, and node networks are interconnected +- Access to clusters using `kubectl` through internal network +- Local environment has `helm` installed, version v3 +- Install and configure [Tencent Cloud CLI tool (tccli)](https://cloud.tencent.com/document/product/440/6176) +- Install [jq](https://jqlang.org/download/) JSON processing tool +- Prepare TKE cluster region, cluster ID, and subnet ID +- Other requirements: [Requirements](../../../docs/requirements.md) + +## One-Click Deployment (Recommended) + +### 1. Create Configuration File + +```bash +cat > config.env < [English](README.md) | 中文 + +本目录包含使用 Kubeocean 进行 LLM 训练和推理跨集群部署的完整示例。 + +## 场景描述 + +本目录支持从零构建了一个实际的训推一体化的场景,该场景使用了两个集群,分别部署在线推理业务和离线训练业务,在线推理业务存在潮汐特性,白天(每日8点-18点)用量较高,需要使用在线集群的全部 GPU 资源,而晚上(每日18点-次日8点)用量较低,只需一半的 GPU 资源。离线训练业务只部署在离线集群,且只在每日晚上在线集群腾挪出 GPU 资源时才部署任务。 + +在线推理业务使用 vLLM 部署在线推理服务,而离线训练任务使用了 Kuberay+VeRL 部署大模型强化学习训练任务。使用的模型都是`Qwen2.5-0.5B-Instruct`,训练数据集为 GSM8K。 + +目标:实现夜间训练任务自动利用闲置GPU,白天训练任务自动退出,保障在线推理业务资源。 + +注:本项目做了一定简化,模型和数据集都保存在镜像中,训练完的 checkpoints 也保存在本地目录,最佳实践建议,模型和数据集可通过文件存储等云存储系统加载,训练完的 checkpoints 也可通过文件存储等云存储系统保存。 + + +## 目录结构 + +``` +. +├── config.env.example # 配置文件示例 +├── run-demo.sh # 一键部署脚本(推荐使用) +├── clean-demo.sh # 一键清理脚本(推荐使用) +├── vllm-infer-demo.sh # vLLM 推理服务部署脚本 +├── llm-kuberay-verl-demo.sh # VERL 训练任务部署脚本 +├── is-qwen2-5-05b-vllm.yaml # vLLM 推理服务 YAML 配置 +└── verl-raycluster.yaml # RayCluster YAML 配置 +``` + +## 前置准备 + +### 资源准备 + +- 需要至少一个 TKE 标准集群作为虚拟算力集群,以及一个 TKE 标准集群作为工作集群。 +- 工作集群中至少包含4个 GPU 节点,每个节点至少带有2个 GPU,型号为 A10 或更新的代次型号,CPU 至少为 24核,内存100G。若不满足此要求,可手动调整本目录下 YAML 中的资源要求。 +- 需要为算力集群和工作集群开启内网访问,需要耗费一定成本部署内网负载均衡。 + +### 环境准备 + +- 算力集群和业务集群 Pod 网络直接互通、节点网络互通 +- 可以使用 `kubectl` 并通过内网访问集群 +- 本地环境已安装 `helm`,且版本为 v3 +- 安装并配置 [腾讯云 CLI 工具 (tccli)](https://cloud.tencent.com/document/product/440/6176) +- 安装 [jq](https://jqlang.org/download/) JSON 处理工具 +- 准备 TKE 集群所在的地域、集群 ID 和子网 ID +- 其他要求参考:[要求](../../../docs/requirements_zh.md) + +## 一键部署(推荐) + +### 1. 创建配置文件 + +```bash +cat > config.env < /dev/null; then + missing_tools+=("tccli") + else + log_success "tccli is installed" + fi + + # Check jq + if ! command -v jq &> /dev/null; then + missing_tools+=("jq") + else + log_success "jq is installed" + fi + + # Check kubectl + if ! command -v kubectl &> /dev/null; then + missing_tools+=("kubectl") + else + log_success "kubectl is installed" + fi + + # Check helm (only if KubeRay uninstall is not skipped) + if [ "$SKIP_UNINSTALL_KUBERAY" = "false" ]; then + if ! command -v helm &> /dev/null; then + missing_tools+=("helm") + else + log_success "helm is installed" + fi + fi + + # If any tools are missing, report and exit + if [ ${#missing_tools[@]} -gt 0 ]; then + log_error "Missing required tools: ${missing_tools[*]}" + log_error "Please install missing tools before running this script" + echo "" + echo "Installation guides:" + echo " tccli: https://cloud.tencent.com/document/product/440/6176" + echo " jq: https://jqlang.org/download/" + echo " kubectl: https://kubernetes.io/docs/tasks/tools/" + echo " helm: https://helm.sh/docs/intro/install/" + exit 1 + fi + + # Check tccli configuration + if ! tccli --version &> /dev/null; then + log_error "tccli is not configured properly. Please run 'tccli configure' first" + exit 1 + fi + + log_success "tccli configuration check passed" + + # Check required configuration variables + if [ -z "$MANAGER_REGION" ]; then + log_error "MANAGER_REGION is required. Please set it in config file" + exit 1 + fi + + if [ -z "$MANAGER_CLUSTER_ID" ]; then + log_error "MANAGER_CLUSTER_ID is required. Please set it in config file" + exit 1 + fi + + if [ -z "$WORKER_REGION" ]; then + log_error "WORKER_REGION is required. Please set it in config file" + exit 1 + fi + + if [ -z "$WORKER_CLUSTER_ID" ]; then + log_error "WORKER_CLUSTER_ID is required. Please set it in config file" + exit 1 + fi + + log_success "All prerequisites check passed" +} + +# Check kubectl contexts +check_kubectl_contexts() { + log_info "Checking kubectl contexts..." + + local manager_context="manager-admin-$MANAGER_CLUSTER_ID" + local worker_context="worker-admin-$WORKER_CLUSTER_ID" + + # Check manager context + if ! kubectl config get-contexts "$manager_context" &> /dev/null; then + log_error "Manager cluster context not found: $manager_context" + log_error "Please ensure manager cluster is configured in kubectl" + log_error "You can run install-tke.sh first or manually configure kubectl" + exit 1 + fi + log_success "Manager context found: $manager_context" + + # Check worker context + if ! kubectl config get-contexts "$worker_context" &> /dev/null; then + log_error "Worker cluster context not found: $worker_context" + log_error "Please ensure worker cluster is configured in kubectl" + log_error "You can run install-tke.sh first or manually configure kubectl" + exit 1 + fi + log_success "Worker context found: $worker_context" + + log_success "All required kubectl contexts are available" +} + +# Delete VERL training resources from manager cluster +delete_verl_training() { + log_info "Deleting VERL training resources from manager cluster..." + + local manager_context="manager-admin-$MANAGER_CLUSTER_ID" + + # Switch to manager cluster context + if ! kubectl config use-context "$manager_context" &> /dev/null; then + log_error "Failed to switch to manager cluster context: $manager_context" + exit 1 + fi + log_success "Switched to manager cluster context: $manager_context" + + # Check if RayCluster YAML exists + if [ ! -f "$RAYCLUSTER_YAML" ]; then + log_warning "RayCluster YAML not found: $RAYCLUSTER_YAML" + log_warning "Skipping RayCluster deletion" + else + # Delete RayCluster + log_info "Deleting RayCluster resources..." + if kubectl delete -f "$RAYCLUSTER_YAML" -n "$MANAGER_NAMESPACE" --ignore-not-found=true 2>&1; then + log_success "RayCluster delete command executed" + else + log_warning "Failed to delete RayCluster, but continuing..." + fi + fi + + # Wait for pods to be deleted (40 seconds polling) + log_info "Waiting for Ray pods to be deleted (up to 40 seconds)..." + local max_wait=40 + local wait_time=0 + local interval=2 + + while [ $wait_time -lt $max_wait ]; do + local pod_count + pod_count=$(kubectl get pods -n "$MANAGER_NAMESPACE" -l ray.io/cluster=verl-cluster --no-headers 2>/dev/null | wc -l || echo "0") + + if [ "$pod_count" -eq 0 ]; then + log_success "All Ray pods have been deleted" + return 0 + fi + + if [ $((wait_time % 10)) -eq 0 ]; then + log_info "Still waiting... ($pod_count pods remaining, ${wait_time}s elapsed)" + fi + + sleep $interval + wait_time=$((wait_time + interval)) + done + + # Check final status + local remaining_pods + remaining_pods=$(kubectl get pods -n "$MANAGER_NAMESPACE" -l ray.io/cluster=verl-cluster --no-headers 2>/dev/null | wc -l || echo "0") + + if [ "$remaining_pods" -eq 0 ]; then + log_success "All Ray pods have been deleted" + else + log_warning "Timeout: $remaining_pods Ray pods still exist after ${max_wait}s" + log_warning "You may need to manually check and clean up remaining pods" + fi +} + +# Uninstall KubeRay operator +uninstall_kuberay() { + if [ "$SKIP_UNINSTALL_KUBERAY" = "true" ]; then + log_info "Skipping KubeRay operator uninstall (SKIP_UNINSTALL_KUBERAY=true)" + return 0 + fi + + log_info "Uninstalling KubeRay operator..." + + local manager_context="manager-admin-$MANAGER_CLUSTER_ID" + + # Switch to manager cluster context + kubectl config use-context "$manager_context" &> /dev/null || true + + # Check if KubeRay operator is installed + if ! helm list -n "$MANAGER_NAMESPACE" 2>/dev/null | grep -q kuberay-operator; then + log_warning "KubeRay operator not found, skipping uninstall" + return 0 + fi + + # Uninstall KubeRay operator + log_info "Uninstalling kuberay-operator helm release..." + if helm uninstall kuberay-operator -n "$MANAGER_NAMESPACE" 2>&1; then + log_success "KubeRay operator uninstalled successfully" + else + log_warning "Failed to uninstall KubeRay operator, but continuing..." + fi + + # Wait a moment for cleanup + sleep 3 +} + +# Delete vLLM inference service from worker cluster +delete_vllm_service() { + log_info "Deleting vLLM inference service from worker cluster..." + + local worker_context="worker-admin-$WORKER_CLUSTER_ID" + + # Switch to worker cluster context + if ! kubectl config use-context "$worker_context" &> /dev/null; then + log_error "Failed to switch to worker cluster context: $worker_context" + exit 1 + fi + log_success "Switched to worker cluster context: $worker_context" + + # Check if vLLM YAML exists + if [ ! -f "$VLLM_YAML" ]; then + log_warning "vLLM YAML not found: $VLLM_YAML" + log_warning "Skipping vLLM service deletion" + else + # Delete vLLM resources + log_info "Deleting vLLM inference service resources..." + if kubectl delete -f "$VLLM_YAML" -n "$WORKER_NAMESPACE" --ignore-not-found=true 2>&1; then + log_success "vLLM service delete command executed" + else + log_warning "Failed to delete vLLM service, but continuing..." + fi + fi + + # Wait for pods to be deleted (40 seconds polling) + log_info "Waiting for vLLM pods to be deleted (up to 40 seconds)..." + local max_wait=40 + local wait_time=0 + local interval=2 + + while [ $wait_time -lt $max_wait ]; do + local pod_count + pod_count=$(kubectl get pods -n "$WORKER_NAMESPACE" -l app.kubernetes.io/instance=is-qwen2-5-05b-vllm --no-headers 2>/dev/null | wc -l || echo "0") + + if [ "$pod_count" -eq 0 ]; then + log_success "All vLLM pods have been deleted" + return 0 + fi + + if [ $((wait_time % 10)) -eq 0 ]; then + log_info "Still waiting... ($pod_count pods remaining, ${wait_time}s elapsed)" + fi + + sleep $interval + wait_time=$((wait_time + interval)) + done + + # Check final status + local remaining_pods + remaining_pods=$(kubectl get pods -n "$WORKER_NAMESPACE" -l app.kubernetes.io/instance=is-qwen2-5-05b-vllm --no-headers 2>/dev/null | wc -l || echo "0") + + if [ "$remaining_pods" -eq 0 ]; then + log_success "All vLLM pods have been deleted" + else + log_warning "Timeout: $remaining_pods vLLM pods still exist after ${max_wait}s" + log_warning "You may need to manually check and clean up remaining pods" + fi +} + +# Uninstall HPC controller addon +uninstall_hpc_controller() { + if [ "$SKIP_UNINSTALL_HPC" = "true" ]; then + log_info "Skipping HPC controller uninstall (SKIP_UNINSTALL_HPC=true)" + return 0 + fi + + log_info "Uninstalling HPC controller addon..." + + local addon_name="tke-hpc-controller" + + # Check if addon exists + local addon_info + ret=0 + addon_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeAddon \ + --ClusterId "$WORKER_CLUSTER_ID" \ + --AddonName "$addon_name" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_warning "Failed to query HPC controller addon status, it may not exist" + log_info "Skipping HPC controller uninstall" + return 0 + fi + + # Check addon phase + local phase + phase=$(echo "$addon_info" | jq -r '.Addons[0].Phase // empty') + + if [ -z "$phase" ] || [ "$phase" = "null" ]; then + log_warning "HPC controller addon not found, skipping uninstall" + return 0 + fi + + log_info "Current HPC controller addon phase: $phase" + + # Delete addon + log_info "Deleting HPC controller addon..." + ret=0 + local delete_result + delete_result=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DeleteAddon \ + --ClusterId "$WORKER_CLUSTER_ID" \ + --AddonName "$addon_name" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_warning "Failed to delete HPC controller addon" + log_warning "$delete_result" + log_info "Continuing with cleanup..." + return 0 + fi + + log_success "HPC controller addon deletion initiated" + + # Wait for addon to be deleted (up to 60 seconds) + log_info "Waiting for HPC controller addon to be deleted..." + local max_retries=30 + local retry=0 + + while [ $retry -lt $max_retries ]; do + sleep 2 + + ret=0 + addon_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeAddon \ + --ClusterId "$WORKER_CLUSTER_ID" \ + --AddonName "$addon_name" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_success "HPC controller addon deleted successfully" + return 0 + fi + + phase=$(echo "$addon_info" | jq -r '.Addons[0].Phase // empty') + + if [ -z "$phase" ] || [ "$phase" = "null" ]; then + log_success "HPC controller addon deleted successfully" + return 0 + fi + + if [ $((retry % 10)) -eq 0 ]; then + log_info "Still deleting... (phase: $phase, attempt $((retry + 1))/$max_retries)" + fi + + retry=$((retry + 1)) + done + + log_warning "Timeout waiting for HPC controller addon deletion" + log_warning "Addon may still be in deletion process, but continuing..." +} + +# Uninstall Kubeocean components +uninstall_kubeocean() { + log_info "Uninstalling Kubeocean components..." + + # Check if uninstall script exists + if [ ! -f "$UNINSTALL_TKE_SCRIPT" ]; then + log_error "Uninstall script not found: $UNINSTALL_TKE_SCRIPT" + exit 1 + fi + + # Call uninstall-tke.sh with current config file + log_info "Calling uninstall-tke.sh with config: $CONFIG_FILE" + + if bash "$UNINSTALL_TKE_SCRIPT" -c "$CONFIG_FILE"; then + log_success "Kubeocean uninstallation completed" + else + log_error "Kubeocean uninstallation failed" + exit 1 + fi +} + +# Display final summary +show_final_summary() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + log_success "LLM Training and Inference Demo Cleanup Complete!" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "📋 Cleanup Summary:" + echo " • Configuration: $CONFIG_FILE" + echo " • VERL training resources deleted ✓" + if [ "$SKIP_UNINSTALL_KUBERAY" = "false" ]; then + echo " • KubeRay operator uninstalled ✓" + else + echo " • KubeRay operator retained (SKIP_UNINSTALL_KUBERAY=true)" + fi + echo " • vLLM inference service deleted ✓" + if [ "$SKIP_UNINSTALL_HPC" = "false" ]; then + echo " • HPC controller uninstalled ✓" + else + echo " • HPC controller retained (SKIP_UNINSTALL_HPC=true)" + fi + echo " • Kubeocean components uninstalled ✓" + echo "" + echo "🔍 Verification Commands:" + echo "" + echo " # Check if contexts are cleaned up" + echo " kubectl config get-contexts" + echo "" + echo " # Verify manager cluster (if context still exists)" + echo " kubectl config use-context manager-admin-$MANAGER_CLUSTER_ID" + echo " kubectl get all -n kubeocean-system" + echo " kubectl get raycluster -n $MANAGER_NAMESPACE" + echo "" + echo " # Verify worker cluster (if context still exists)" + echo " kubectl config use-context worker-admin-$WORKER_CLUSTER_ID" + echo " kubectl get all -n kubeocean-worker" + echo " kubectl get deployment is-qwen2-5-05b-vllm -n $WORKER_NAMESPACE" + echo "" + echo "🎉 Cleanup completed successfully!" + echo "" +} + +# Main function +main() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo " 🧹 Kubeocean LLM Training and Inference Demo" + echo " 🧹 Cleanup Script" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + + # Parse arguments + parse_args "$@" + + # Check config file + check_config_file + echo "" + + # Load configuration + load_config + echo "" + + # Step 1: Check prerequisites + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "📋 Step 1/7: Checking Prerequisites" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + check_prerequisites + echo "" + + # Step 2: Check kubectl contexts + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🔍 Step 2/7: Checking Kubectl Contexts" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + check_kubectl_contexts + echo "" + + # Step 3: Delete VERL training resources + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🎓 Step 3/7: Deleting VERL Training Resources" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + delete_verl_training + echo "" + + # Step 4: Uninstall KubeRay operator + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🔌 Step 4/7: KubeRay Operator Cleanup" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + uninstall_kuberay + echo "" + + # Step 5: Delete vLLM inference service + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🤖 Step 5/7: Deleting vLLM Inference Service" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + delete_vllm_service + echo "" + + # Step 6: Uninstall HPC controller + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🔧 Step 6/7: HPC Controller Cleanup" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + uninstall_hpc_controller + echo "" + + # Step 7: Uninstall Kubeocean + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "📦 Step 7/7: Uninstalling Kubeocean" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + uninstall_kubeocean + echo "" + + # Display final summary + show_final_summary +} + +# Execute main function +main "$@" + diff --git a/examples/cookbook/llm-train-and-infer-cross-clusters/config.env.example b/examples/cookbook/llm-train-and-infer-cross-clusters/config.env.example new file mode 100644 index 0000000..d64b40f --- /dev/null +++ b/examples/cookbook/llm-train-and-infer-cross-clusters/config.env.example @@ -0,0 +1,34 @@ +# vLLM Inference Service Deployment Configuration Example + +# Manager Cluster Settings +MANAGER_REGION="ap-guangzhou" +MANAGER_CLUSTER_ID="cls-xxx1" +MANAGER_SUBNET_ID="subnet-xxx1" +SKIP_MANAGER_UNINSTALL="false" + +# Worker Cluster Settings +WORKER_REGION="ap-guangzhou" +WORKER_CLUSTER_ID="cls-xxx1" +WORKER_SUBNET_ID="subnet-xxx2" +WORKER_KUBECONFIG="/tmp/kubeconfig-worker" +WORKER_CLUSTER_NAME="example-cluster" + +# Optional: Skip tke-hpc-controller installation (default: false) +# Set to "true" if tke-hpc-controller is already installed or not needed +SKIP_INSTALL_HPC="false" + +# Training Script Settings +# Optional: Skip time window checking for training job submission (default: false) +# When false, training can only be submitted between 18:00-08:00 +# When true, skips time checking and clears RLP timeWindows in worker cluster +SKIP_CURRENT_TIME_CHECKING="false" + +# Cleanup Script Settings +# Optional: Skip HPC controller uninstall during cleanup (default: true) +# Set to "false" to also uninstall tke-hpc-controller when running clean-demo.sh +SKIP_UNINSTALL_HPC="true" + +# Optional: Skip KubeRay operator uninstall during cleanup (default: true) +# Set to "false" to also uninstall KubeRay operator when running clean-demo.sh +SKIP_UNINSTALL_KUBERAY="true" + diff --git a/examples/cookbook/llm-train-and-infer-cross-clusters/is-qwen2-5-05b-vllm.yaml b/examples/cookbook/llm-train-and-infer-cross-clusters/is-qwen2-5-05b-vllm.yaml new file mode 100644 index 0000000..11c9795 --- /dev/null +++ b/examples/cookbook/llm-train-and-infer-cross-clusters/is-qwen2-5-05b-vllm.yaml @@ -0,0 +1,112 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: is-qwen2-5-05b-vllm +spec: + replicas: 2 + selector: + matchLabels: + app.kubernetes.io/name: vllm-inference-service + app.kubernetes.io/instance: is-qwen2-5-05b-vllm + template: + metadata: + labels: + app.kubernetes.io/name: vllm-inference-service + app.kubernetes.io/instance: is-qwen2-5-05b-vllm + spec: + enableServiceLinks: false + containers: + - name: vllm + image: ccr.ccs.tencentyun.com/tke-ai-playbook/vllm-openai:v0.11.1-Qwen2.5-0.5B + imagePullPolicy: IfNotPresent + command: ["bash", "-c"] + args: + - | + # 启动 vLLM Server (单节点,双卡 GPU) + vllm serve /vllm-workspace/models/Qwen2.5-0.5B-Instruct \ + --served-model-name Qwen2.5-0.5B-Instruct \ + --tensor-parallel-size 2 \ + --max-model-len 1024 \ + --max-num-seqs 32 \ + --host 0.0.0.0 \ + --port 8000 + ports: + - containerPort: 8000 + name: vllm + protocol: TCP + resources: + limits: + cpu: "10" + memory: 40Gi + nvidia.com/gpu: "2" + ephemeral-storage: "30Gi" + requests: + cpu: "10" + memory: 40Gi + nvidia.com/gpu: "2" + ephemeral-storage: "8Gi" + startupProbe: + httpGet: + path: /health + port: vllm + scheme: HTTP + timeoutSeconds: 3 + periodSeconds: 10 + failureThreshold: 360 + livenessProbe: + httpGet: + path: /health + port: vllm + scheme: HTTP + timeoutSeconds: 3 + periodSeconds: 10 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /health + port: vllm + scheme: HTTP + timeoutSeconds: 3 + periodSeconds: 10 + failureThreshold: 3 +--- +apiVersion: v1 +kind: Service +metadata: + name: is-qwen2-5-05b-vllm + labels: + app.kubernetes.io/name: vllm-inference-service + app.kubernetes.io/instance: is-qwen2-5-05b-vllm +spec: + type: ClusterIP + ports: + - name: http + port: 8000 + protocol: TCP + targetPort: vllm + selector: + app.kubernetes.io/name: vllm-inference-service + app.kubernetes.io/instance: is-qwen2-5-05b-vllm +--- +apiVersion: autoscaling.cloud.tencent.com/v1 +kind: HorizontalPodCronscaler +metadata: + finalizers: + - autoscaling.cloud.tencent.com + labels: + qcloud-app: is-qwen + name: is-qwen + namespace: default +spec: + crons: + - name: scale-down + schedule: 0 0 18 * * * + targetSize: 2 + - name: scale-up + schedule: 0 0 8 * * * + targetSize: 4 + scaleTarget: + apiVersion: apps/v1 + kind: Deployment + name: is-qwen2-5-05b-vllm + namespace: default diff --git a/examples/cookbook/llm-train-and-infer-cross-clusters/llm-kuberay-verl-demo.sh b/examples/cookbook/llm-train-and-infer-cross-clusters/llm-kuberay-verl-demo.sh new file mode 100755 index 0000000..8c9843f --- /dev/null +++ b/examples/cookbook/llm-train-and-infer-cross-clusters/llm-kuberay-verl-demo.sh @@ -0,0 +1,854 @@ +#!/bin/bash + +# Copyright 2025 The Kubeocean Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# LLM KubeRay VERL Training Deployment Script for TKE +# Features: +# 1. Pre-check TKE cluster and required tools +# 2. Get manager cluster kubeconfig and set context +# 3. Deploy KubeRay Operator using Helm +# 4. Create RayCluster for VERL training +# 5. Wait for RayCluster pods to be ready +# 6. Submit VERL training job to Ray cluster +# 7. Verify job submission status + +set -e +set -o pipefail + +# Color definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Logging functions +log_info() { + echo -e "${BLUE}ℹ️ $1${NC}" +} + +log_success() { + echo -e "${GREEN}✅ $1${NC}" +} + +log_warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +log_error() { + echo -e "${RED}❌ $1${NC}" +} + +# Error handling +trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR + +# Script root directory +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Manager Cluster Configuration variables +MANAGER_REGION="" +MANAGER_CLUSTER_ID="" +MANAGER_SUBNET_ID="" +MANAGER_NAMESPACE="default" + +# RayCluster configuration +RAYCLUSTER_NAME="verl-cluster" +RAYCLUSTER_YAML="${SCRIPT_DIR}/verl-raycluster.yaml" + +# Time window checking (default: false = check time window) +SKIP_CURRENT_TIME_CHECKING="false" + +# VPC information (will be extracted from cluster info) +VPC_ID="" + +# Ray job ID (will be extracted after job submission) +RAY_JOB_ID="" + +# Display help information +show_help() { + cat << EOF +LLM KubeRay VERL Training Deployment Script for TKE + +This script automates the deployment of VERL training on RayCluster with the following steps: +1. Pre-check tccli, kubectl, helm and cluster connectivity +2. Get manager cluster admin kubeconfig and set kubectl context +3. Deploy KubeRay Operator using Helm +4. Create RayCluster for VERL training +5. Wait for RayCluster pods to be ready +6. Submit VERL training job to Ray cluster +7. Verify job submission status + +Usage: $0 [options] + +Options: + -h, --help Display this help message + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + MANAGER_REGION Manager cluster region + MANAGER_CLUSTER_ID Manager cluster ID + MANAGER_SUBNET_ID Manager cluster subnet ID for internal network access + +Optional Configuration Variables: + MANAGER_NAMESPACE Namespace for deployment (default: default) + SKIP_CURRENT_TIME_CHECKING Skip time window checking (default: false) + When false, script checks if current time is between 18:00-08:00 + When true, skips the time check + +Prerequisites: + tccli Tencent Cloud CLI tool (must be installed and configured) + jq Command-line JSON processor + kubectl Kubernetes command-line tool + helm Helm package manager (version 3.x required) + +Examples: + # Use default configuration file (./config.env) + $0 + + # Use custom configuration file + $0 --config /path/to/my-config.env + +Configuration File Example: + cp config.env.example config.env + # Edit config.env with your values + vim config.env + +EOF +} + +# Load configuration from file +load_config() { + if [[ -f "$CONFIG_FILE" ]]; then + log_info "Loading configuration from: $CONFIG_FILE" + set -a + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + set +a + log_success "Configuration loaded" + fi +} + +# Parse command line arguments +parse_args() { + while [[ $# -gt 0 ]]; do + case $1 in + -h|--help) + show_help + exit 0 + ;; + -c|--config) + CONFIG_FILE="$2" + shift 2 + ;; + *) + log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + log_info "All other configurations should be provided in config file" + show_help + exit 1 + ;; + esac + done +} + +# Check prerequisites +check_prerequisites() { + log_info "Checking prerequisites..." + + # Check required parameters + if [ -z "$MANAGER_REGION" ]; then + log_error "MANAGER_REGION is required. Please set it in config file" + exit 1 + fi + + if [ -z "$MANAGER_CLUSTER_ID" ]; then + log_error "MANAGER_CLUSTER_ID is required. Please set it in config file" + exit 1 + fi + + log_info "Manager Region: $MANAGER_REGION" + log_info "Manager Cluster ID: $MANAGER_CLUSTER_ID" + log_info "Manager Namespace: $MANAGER_NAMESPACE" + + # Check tccli installation + if ! command -v tccli &> /dev/null; then + log_error "tccli is not installed. Please install Tencent Cloud CLI first" + log_error "Installation guide: https://cloud.tencent.com/document/product/440/6176" + exit 1 + fi + + log_success "tccli is installed" + + # Check tccli configuration + if ! tccli --version &> /dev/null; then + log_error "tccli is not configured properly. Please run 'tccli configure' first" + exit 1 + fi + + log_success "tccli configuration check passed" + + # Check jq installation + if ! command -v jq &> /dev/null; then + log_error "jq is not installed. Please install jq first" + log_error "Installation guide: https://jqlang.org/download/" + exit 1 + fi + + log_success "jq is installed" + + # Check kubectl + if ! command -v kubectl &> /dev/null; then + log_error "kubectl is not installed. Please install kubectl first" + exit 1 + fi + + log_success "kubectl is installed" + + # Check helm + if ! command -v helm &> /dev/null; then + log_error "helm is not installed. Please install helm first" + log_error "Installation guide: https://helm.sh/docs/intro/install/" + exit 1 + fi + + log_success "helm is installed" + + # Check if cluster exists and get VPC info + log_info "Checking cluster existence and getting VPC info..." + local cluster_info + ret=0 + cluster_info=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusters --ClusterIds "[\"$MANAGER_CLUSTER_ID\"]" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to query cluster information" + log_error "$cluster_info" + exit 1 + fi + + # Check if cluster exists + local total_count + total_count=$(echo "$cluster_info" | jq -r '.TotalCount // 0') + + if [ "$total_count" -eq 0 ]; then + log_error "Cluster $MANAGER_CLUSTER_ID not found in region $MANAGER_REGION" + exit 1 + fi + + # Extract VPC ID + VPC_ID=$(echo "$cluster_info" | jq -r '.Clusters[0].ClusterNetworkSettings.VpcId // empty') + + if [ -z "$VPC_ID" ]; then + log_error "Failed to get VPC ID from cluster information" + exit 1 + fi + + log_success "Cluster found in VPC: $VPC_ID" + + # Check RayCluster YAML file + if [ ! -f "$RAYCLUSTER_YAML" ]; then + log_error "RayCluster YAML file not found: $RAYCLUSTER_YAML" + exit 1 + fi + + log_success "RayCluster YAML file found: $RAYCLUSTER_YAML" + + # Check time window + check_time_window + + log_success "Prerequisites check passed" +} + +# Check if current time is within allowed time window (18:00-08:00) +check_time_window() { + if [ "$SKIP_CURRENT_TIME_CHECKING" = "true" ]; then + log_warning "Time window checking is skipped (SKIP_CURRENT_TIME_CHECKING=true)" + return 0 + fi + + log_info "Checking current time window..." + + # Get current hour (0-23) + local current_hour + current_hour=$(date +%H) + + # Remove leading zero if present + current_hour=$((10#$current_hour)) + + # Check if time is between 18:00-23:59 or 00:00-08:00 + if [ $current_hour -ge 18 ] || [ $current_hour -lt 8 ]; then + log_success "Current time is within allowed window (18:00-08:00)" + log_info "Current hour: ${current_hour}:00" + else + log_error "Current time is NOT within allowed window!" + log_error "Training jobs can only be submitted between 18:00-08:00" + log_error "Current hour: ${current_hour}:00" + log_error "" + log_error "Options:" + log_error " 1. Wait until 18:00 to run this script" + log_error " 2. Set SKIP_CURRENT_TIME_CHECKING=true in config.env to skip this check" + exit 1 + fi +} + +# Enable cluster internal network access +enable_cluster_internal_access() { + log_info "Checking cluster internal network access status..." + + # Check if internal access is already enabled + local endpoint_status + ret=0 + endpoint_status=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$MANAGER_CLUSTER_ID" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to query cluster endpoint status" + log_error "$endpoint_status" + exit 1 + fi + + local status + status=$(echo "$endpoint_status" | jq -r '.Status // empty') + + if [ "$status" = "Created" ]; then + log_success "Cluster internal network access is already enabled" + return 0 + fi + + # Need to enable internal access + log_info "Enabling cluster internal network access..." + + # Check subnet ID parameter + if [ -z "$MANAGER_SUBNET_ID" ]; then + log_error "MANAGER_SUBNET_ID is required for enabling internal access. Please set it in config file" + exit 1 + fi + + # Enable internal access + local enable_result + ret=0 + enable_result=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke CreateClusterEndpoint \ + --ClusterId "$MANAGER_CLUSTER_ID" \ + --SubnetId "$MANAGER_SUBNET_ID" \ + --IsExtranet false 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to enable cluster internal network access" + log_error "$enable_result" + exit 1 + fi + + log_info "Waiting for internal network access to be ready..." + + # Poll for status (timeout: 120 seconds) + local max_retries=60 + local retry=0 + + while [ $retry -lt $max_retries ]; do + sleep 2 + + ret=0 + endpoint_status=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$MANAGER_CLUSTER_ID" 2>&1) || ret=$? + + if [ $ret -eq 0 ]; then + status=$(echo "$endpoint_status" | jq -r '.Status // empty') + + if [ "$status" = "Created" ]; then + log_success "Cluster internal network access enabled successfully" + return 0 + elif [ "$status" = "Creating" ]; then + log_info "Still creating... (attempt $((retry + 1))/$max_retries)" + else + log_warning "Unexpected status: $status" + fi + fi + + retry=$((retry + 1)) + done + + log_error "Timeout waiting for cluster internal network access to be ready" + exit 1 +} + +# Get cluster kubeconfig and set context +get_cluster_kubeconfig() { + log_info "Getting cluster kubeconfig..." + + # Check if context already exists + local context_name="manager-admin-$MANAGER_CLUSTER_ID" + if kubectl config get-contexts "$context_name" &> /dev/null; then + log_info "Context '$context_name' already exists, reusing it" + kubectl config use-context "$context_name" + log_success "Switched to existing context: $context_name" + return 0 + fi + + log_info "Context not found, fetching kubeconfig from TKE..." + + # Wait for internal load balancer information to update + log_info "Waiting 5 seconds for internal load balancer information to update..." + sleep 5 + + # Get cluster security info (including kubeconfig) + local security_info + ret=0 + security_info=$(TENCENTCLOUD_REGION="$MANAGER_REGION" tccli tke DescribeClusterSecurity --ClusterId "$MANAGER_CLUSTER_ID" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to get cluster security information" + log_error "$security_info" + exit 1 + fi + + # Extract internal kubeconfig + local kubeconfig_content + kubeconfig_content=$(echo "$security_info" | jq -r '.Kubeconfig // empty') + + if [ -z "$kubeconfig_content" ]; then + log_error "Failed to extract kubeconfig from cluster security information" + exit 1 + fi + + # Extract PgwEndpoint (internal network endpoint) + local pgw_endpoint + pgw_endpoint=$(echo "$security_info" | jq -r '.PgwEndpoint // empty') + + if [ -z "$pgw_endpoint" ]; then + log_error "Failed to extract PgwEndpoint from cluster security information" + exit 1 + fi + + log_info "PgwEndpoint: $pgw_endpoint" + + # Save kubeconfig to temporary file + local temp_kubeconfig="/tmp/tke-kubeconfig-$MANAGER_CLUSTER_ID" + echo "$kubeconfig_content" > "$temp_kubeconfig" + + # Replace server address with PgwEndpoint + log_info "Replacing server address with PgwEndpoint..." + local cluster_name + cluster_name=$(kubectl --kubeconfig="$temp_kubeconfig" config view -o jsonpath='{.clusters[0].name}' 2>/dev/null) + + if [ -n "$cluster_name" ]; then + kubectl --kubeconfig="$temp_kubeconfig" config set-cluster "$cluster_name" --server="https://$pgw_endpoint" &> /dev/null + log_success "Server address updated to: $pgw_endpoint" + else + log_error "Failed to get cluster name from kubeconfig" + exit 1 + fi + + # Verify connection before merging + log_info "Verifying cluster connection..." + if ! kubectl --kubeconfig="$temp_kubeconfig" cluster-info &> /dev/null; then + log_error "Failed to connect to cluster using the kubeconfig" + log_error "Please check if the cluster internal network access is properly configured" + rm -f "$temp_kubeconfig" + exit 1 + fi + log_success "Successfully verified cluster connection" + + # Set kubectl context name + local context_name="manager-admin-$MANAGER_CLUSTER_ID" + + log_info "Merging kubeconfig to default kubectl config" + log_info "Context name: $context_name" + + # Get default KUBECONFIG path + local default_kubeconfig="${HOME}/.kube/config" + + # Create .kube directory if it doesn't exist + mkdir -p "${HOME}/.kube" + + # If default kubeconfig doesn't exist, create an empty one + if [ ! -f "$default_kubeconfig" ]; then + touch "$default_kubeconfig" + fi + + # Merge the TKE kubeconfig into default kubeconfig + KUBECONFIG="$default_kubeconfig:$temp_kubeconfig" kubectl config view --flatten > "${temp_kubeconfig}.merged" + mv "${temp_kubeconfig}.merged" "$default_kubeconfig" + + # Get the original context name from temp kubeconfig + local original_context + original_context=$(kubectl --kubeconfig="$temp_kubeconfig" config current-context 2>/dev/null) + + # Rename context to our desired name + if [ -n "$original_context" ]; then + kubectl config rename-context "$original_context" "$context_name" &> /dev/null || true + fi + + # Set as current context + kubectl config use-context "$context_name" &> /dev/null + + # Clean up temporary file + rm -f "$temp_kubeconfig" + + log_success "Kubeconfig merged to: $default_kubeconfig" + log_success "Context name: $context_name" + log_success "Cluster is ready for use" +} + +# Deploy KubeRay Operator using Helm +deploy_kuberay_operator() { + log_info "Checking KubeRay Operator status..." + + # Check if KubeRay Operator is already installed in the namespace + if helm list -n "$MANAGER_NAMESPACE" | grep -q kuberay-operator; then + log_success "KubeRay Operator is already installed in namespace: $MANAGER_NAMESPACE" + return 0 + fi + + log_info "Installing KubeRay Operator using Helm in namespace: $MANAGER_NAMESPACE" + + # Add KubeRay Helm repository + log_info "Adding KubeRay Helm repository..." + helm repo add kuberay https://ray-project.github.io/kuberay-helm/ || log_warning "Helm repo may already exist" + helm repo update + + # Install KubeRay Operator + ret=0 + helm install kuberay-operator kuberay/kuberay-operator \ + --namespace "$MANAGER_NAMESPACE" \ + --version 1.5.0 2>&1 || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to install KubeRay Operator" + exit 1 + fi + + log_success "KubeRay Operator installed successfully" + + # Wait for operator to be ready + log_info "Waiting for KubeRay Operator to be ready..." + kubectl wait --for=condition=available --timeout=180s \ + deployment/kuberay-operator -n "$MANAGER_NAMESPACE" || log_warning "Operator may still be starting" + + log_success "KubeRay Operator is ready" +} + +# Create RayCluster +create_raycluster() { + log_info "Checking if RayCluster '$RAYCLUSTER_NAME' already exists..." + + # Check if RayCluster already exists + if kubectl get raycluster "$RAYCLUSTER_NAME" -n "$MANAGER_NAMESPACE" &> /dev/null; then + log_success "RayCluster '$RAYCLUSTER_NAME' already exists, skipping creation" + return 0 + fi + + log_info "Creating RayCluster '$RAYCLUSTER_NAME'..." + + # Apply RayCluster YAML + ret=0 + kubectl apply -f "$RAYCLUSTER_YAML" -n "$MANAGER_NAMESPACE" 2>&1 || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to create RayCluster" + exit 1 + fi + + log_success "RayCluster '$RAYCLUSTER_NAME' created successfully" +} + +# Wait for RayCluster pods to be ready +wait_for_raycluster_ready() { + log_info "Waiting for RayCluster pods to be ready..." + + local max_retries=180 # 6 minutes (180 * 2 seconds) + local retry=0 + + while [ $retry -lt $max_retries ]; do + sleep 2 + + # Get all pods for this RayCluster + local pods_status + pods_status=$(kubectl get pods -n "$MANAGER_NAMESPACE" \ + -l ray.io/cluster="$RAYCLUSTER_NAME" \ + -o jsonpath='{range .items[*]}{.metadata.name}:{.status.phase}{"\n"}{end}' 2>&1) || true + + if [ -z "$pods_status" ]; then + if [ $((retry % 10)) -eq 0 ]; then + log_info "Waiting for pods to be created... (attempt $((retry + 1))/$max_retries)" + fi + retry=$((retry + 1)) + continue + fi + + # Check if all pods are Running + local all_running=true + local pod_count=0 + local running_count=0 + + while IFS=: read -r pod_name pod_phase; do + if [ -n "$pod_name" ]; then + pod_count=$((pod_count + 1)) + if [ "$pod_phase" = "Running" ]; then + running_count=$((running_count + 1)) + else + all_running=false + fi + fi + done <<< "$pods_status" + + if [ "$pod_count" -eq 0 ]; then + if [ $((retry % 10)) -eq 0 ]; then + log_info "Waiting for pods to be created... (attempt $((retry + 1))/$max_retries)" + fi + elif $all_running && [ "$running_count" -gt 0 ]; then + log_success "All RayCluster pods are running ($running_count/$pod_count)" + + # Display pod status + log_info "RayCluster pods status:" + kubectl get pods -n "$MANAGER_NAMESPACE" -l ray.io/cluster="$RAYCLUSTER_NAME" + + return 0 + else + if [ $((retry % 10)) -eq 0 ]; then + log_info "Pods status: $running_count/$pod_count running (attempt $((retry + 1))/$max_retries)" + fi + fi + + retry=$((retry + 1)) + done + + log_error "Timeout waiting for RayCluster pods to be ready" + log_info "Current pod status:" + kubectl get pods -n "$MANAGER_NAMESPACE" -l ray.io/cluster="$RAYCLUSTER_NAME" + exit 1 +} + +# Submit VERL training job +submit_verl_training_job() { + log_info "Submitting VERL training job to RayCluster..." + + # Find the Ray head pod + local head_pod + head_pod=$(kubectl get pods -n "$MANAGER_NAMESPACE" \ + -l ray.io/cluster="$RAYCLUSTER_NAME",ray.io/node-type=head \ + -o jsonpath='{.items[0].metadata.name}' 2>&1) + + if [ -z "$head_pod" ]; then + log_error "Failed to find Ray head pod" + exit 1 + fi + + log_info "Ray head pod: $head_pod" + + # Prepare working directory + log_info "Preparing working directory in Ray head pod..." + kubectl exec -n "$MANAGER_NAMESPACE" "$head_pod" -- bash -c "mkdir -p /workspace/work" || true + + # Submit Ray job + log_info "Submitting VERL training job..." + + local job_submit_output + job_submit_output=$(kubectl exec -n "$MANAGER_NAMESPACE" "$head_pod" -- bash -c " +cd /workspace/work && \ +ray job submit --no-wait --working-dir /workspace/work -- \ +PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \ +data.train_files=/root/data/gsm8k/train.parquet \ +data.val_files=/root/data/gsm8k/test.parquet \ +data.train_batch_size=256 \ +data.max_prompt_length=512 \ +data.max_response_length=256 \ +actor_rollout_ref.model.path=/root/models/Qwen2.5-0.5B-Instruct \ +actor_rollout_ref.actor.optim.lr=1e-6 \ +actor_rollout_ref.actor.ppo_mini_batch_size=64 \ +actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \ +actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \ +actor_rollout_ref.rollout.tensor_model_parallel_size=1 \ +actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \ +actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \ +critic.optim.lr=1e-5 \ +critic.model.path=/root/models/Qwen2.5-0.5B-Instruct \ +critic.ppo_micro_batch_size_per_gpu=4 \ +algorithm.kl_ctrl.kl_coef=0.001 \ +trainer.logger=['console'] \ +trainer.val_before_train=False \ +trainer.default_hdfs_dir=null \ +trainer.n_gpus_per_node=2 \ +trainer.nnodes=2 \ +trainer.save_freq=150 \ +trainer.test_freq=10 \ +trainer.default_local_dir=/data/checkpoints/verl_Qwen2.5_gsm8k_2 \ +trainer.total_epochs=10 2>&1 +" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to submit VERL training job" + log_error "$job_submit_output" + exit 1 + fi + + log_success "VERL training job submitted" + echo "$job_submit_output" + + # Extract job ID from output + local job_id + job_id=$(echo "$job_submit_output" | grep -oP "Job '.*?' submitted" | grep -oP "'.*?'" | tr -d "'") + + if [ -z "$job_id" ]; then + # Try alternative format + job_id=$(echo "$job_submit_output" | grep -oP "raysubmit_[a-zA-Z0-9]+" | head -1) + fi + + if [ -n "$job_id" ]; then + # Save job ID to global variable + RAY_JOB_ID="$job_id" + log_info "Ray Job ID: $RAY_JOB_ID" + + # Wait a moment for job to initialize + log_info "Waiting 5 seconds for job to initialize..." + sleep 5 + + # Check job status + log_info "Checking job status..." + local job_status + job_status=$(kubectl exec -n "$MANAGER_NAMESPACE" "$head_pod" -- bash -c "ray job status $RAY_JOB_ID" 2>&1) || true + + log_info "Job status output:" + echo "$job_status" + + if echo "$job_status" | grep -qE "(RUNNING|SUCCEEDED)"; then + log_success "VERL training job is running successfully" + elif echo "$job_status" | grep -q "PENDING"; then + log_info "VERL training job is pending, it will start soon" + else + log_warning "Job status check returned unexpected result, but job was submitted" + fi + else + log_warning "Could not extract job ID, but job submission command completed" + fi +} + +# Display summary information +show_summary() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + log_success "VERL Training Deployment Complete!" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "📋 Deployment Information:" + echo " • Region: $MANAGER_REGION" + echo " • Cluster ID: $MANAGER_CLUSTER_ID" + echo " • Namespace: $MANAGER_NAMESPACE" + echo " • RayCluster Name: $RAYCLUSTER_NAME" + echo " • Context: manager-admin-$MANAGER_CLUSTER_ID" + if [ -n "$RAY_JOB_ID" ]; then + echo " • Ray Job ID: $RAY_JOB_ID" + fi + echo "" + echo "🔍 Next Steps:" + echo " 1. Check RayCluster status:" + echo " kubectl get raycluster $RAYCLUSTER_NAME -n $MANAGER_NAMESPACE" + echo " kubectl describe raycluster $RAYCLUSTER_NAME -n $MANAGER_NAMESPACE" + echo "" + echo " 2. Check Ray pods:" + echo " kubectl get pods -n $MANAGER_NAMESPACE -l ray.io/cluster=$RAYCLUSTER_NAME" + echo "" + echo " 3. View Ray head pod logs:" + echo " HEAD_POD=\$(kubectl get pods -n $MANAGER_NAMESPACE -l ray.io/cluster=$RAYCLUSTER_NAME,ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')" + echo " kubectl logs -n $MANAGER_NAMESPACE \$HEAD_POD -f" + echo "" + + if [ -n "$RAY_JOB_ID" ]; then + echo " 4. Check Ray job status:" + echo " kubectl exec -n $MANAGER_NAMESPACE \$HEAD_POD -- ray job status $RAY_JOB_ID" + echo "" + echo " 5. Monitor training progress:" + echo " kubectl exec -n $MANAGER_NAMESPACE \$HEAD_POD -- ray job logs $RAY_JOB_ID --follow" + echo "" + echo " 6. List all Ray jobs:" + echo " kubectl exec -n $MANAGER_NAMESPACE \$HEAD_POD -- ray job list" + echo "" + else + echo " 4. Check Ray job status (replace JOB_ID with actual job ID):" + echo " kubectl exec -n $MANAGER_NAMESPACE \$HEAD_POD -- ray job status JOB_ID" + echo "" + echo " 5. List all Ray jobs:" + echo " kubectl exec -n $MANAGER_NAMESPACE \$HEAD_POD -- ray job list" + echo "" + fi + + echo " 7. Access Ray Dashboard (port-forward):" + echo " kubectl port-forward -n $MANAGER_NAMESPACE \$HEAD_POD 8265:8265" + echo " # Then open http://localhost:8265 in your browser" + echo "" +} + +# Main function +main() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo " 🚀 LLM KubeRay VERL Training Deployment Script" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + + # Step 0: Parse config file path first + for arg in "$@"; do + if [[ "$arg" == "-c" || "$arg" == "--config" ]]; then + shift + CONFIG_FILE="$1" + break + fi + shift + done + + # Load configuration file + load_config + + # Parse arguments (command line overrides config file) + parse_args "$@" + + # Step 1: Check prerequisites + echo "📋 Step 1/7: Checking Prerequisites" + check_prerequisites + echo "" + + # Step 2: Enable cluster internal network access and get kubeconfig + echo "🔐 Step 2/7: Setting up Manager Cluster Access" + enable_cluster_internal_access + get_cluster_kubeconfig + echo "" + + # Step 3: Deploy KubeRay Operator + echo "🔌 Step 3/7: Deploying KubeRay Operator" + deploy_kuberay_operator + echo "" + + # Step 4: Create RayCluster + echo "📦 Step 4/7: Creating RayCluster" + create_raycluster + echo "" + + # Step 5: Wait for RayCluster pods to be ready + echo "⏳ Step 5/7: Waiting for RayCluster Pods" + wait_for_raycluster_ready + echo "" + + # Step 6: Submit VERL training job + echo "🚀 Step 6/7: Submitting VERL Training Job" + submit_verl_training_job + echo "" + + # Step 7: Display summary + echo "📊 Step 7/7: Summary" + show_summary +} + +# Execute main function +main "$@" + diff --git a/examples/cookbook/llm-train-and-infer-cross-clusters/run-demo.sh b/examples/cookbook/llm-train-and-infer-cross-clusters/run-demo.sh new file mode 100755 index 0000000..08fe3aa --- /dev/null +++ b/examples/cookbook/llm-train-and-infer-cross-clusters/run-demo.sh @@ -0,0 +1,569 @@ +#!/bin/bash + +# Copyright 2025 The Kubeocean Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Kubeocean LLM Training and Inference Demo - One-Click Deployment Script +# Features: +# 1. Pre-check required tools +# 2. Load configuration from config.env +# 3. Install Kubeocean using install-tke.sh +# 4. Deploy vLLM inference service +# 5. Patch ResourceLeasingPolicy timeWindows (if SKIP_CURRENT_TIME_CHECKING=true) +# 6. Deploy VERL training job on manager cluster + +set -e +set -o pipefail + +# Color definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Logging functions +log_info() { + echo -e "${BLUE}ℹ️ $1${NC}" +} + +log_success() { + echo -e "${GREEN}✅ $1${NC}" +} + +log_warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +log_error() { + echo -e "${RED}❌ $1${NC}" +} + +# Error handling +trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR + +# Script root directory +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Installation script paths +INSTALL_TKE_SCRIPT="${SCRIPT_DIR}/../installation-tke/install-tke.sh" +VLLM_INFER_SCRIPT="${SCRIPT_DIR}/vllm-infer-demo.sh" +VERL_TRAINING_SCRIPT="${SCRIPT_DIR}/llm-kuberay-verl-demo.sh" + +# Configuration variables (will be loaded from config.env) +MANAGER_REGION="" +MANAGER_CLUSTER_ID="" +MANAGER_SUBNET_ID="" +WORKER_REGION="" +WORKER_CLUSTER_ID="" +WORKER_SUBNET_ID="" +WORKER_KUBECONFIG="/tmp/kubeconfig-worker" +WORKER_CLUSTER_NAME="example-cluster" +SKIP_MANAGER_UNINSTALL="false" +SKIP_INSTALL_HPC="false" +SKIP_CURRENT_TIME_CHECKING="false" + +# Ray job ID (will be extracted after training deployment) +RAY_JOB_ID="" +RAYCLUSTER_NAME="verl-cluster" +MANAGER_NAMESPACE="default" + +# Display help information +show_help() { + cat << EOF +Kubeocean LLM Training and Inference Demo - One-Click Deployment Script + +This script automates the complete deployment process by sequentially executing: +1. Pre-check required tools (tccli, jq, kubectl, helm) +2. Load configuration from config.env +3. Install Kubeocean using install-tke.sh +4. Deploy vLLM inference service using vllm-infer-demo.sh +5. Deploy VERL training job using llm-kuberay-verl-demo.sh + +Usage: $0 [options] + +Options: + -h, --help Display this help message + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + MANAGER_REGION Manager cluster region + MANAGER_CLUSTER_ID Manager cluster ID + MANAGER_SUBNET_ID Manager cluster subnet ID + WORKER_REGION Worker cluster region + WORKER_CLUSTER_ID Worker cluster ID + WORKER_SUBNET_ID Worker cluster subnet ID + +Optional Configuration Variables: + WORKER_KUBECONFIG Worker kubeconfig path (default: /tmp/kubeconfig-worker) + WORKER_CLUSTER_NAME Worker cluster name for binding (default: example-cluster) + SKIP_MANAGER_UNINSTALL Skip manager uninstall (default: false) + SKIP_INSTALL_HPC Skip tke-hpc-controller installation (default: false) + SKIP_CURRENT_TIME_CHECKING Skip time window checking (default: false) + When false, training can only be submitted between 18:00-08:00 + When true, skips time check and clears RLP timeWindows + +Prerequisites: + tccli Tencent Cloud CLI tool (must be installed and configured) + jq Command-line JSON processor + kubectl Kubernetes command-line tool + helm v3.x Helm package manager (version 3.x required) + +Examples: + # Use default configuration file (./config.env) + $0 + + # Use custom configuration file + $0 -c /path/to/my-config.env + + # Quick setup + cp config.env.example config.env + vim config.env + $0 + +EOF +} + +# Parse command line arguments +parse_args() { + while [[ $# -gt 0 ]]; do + case $1 in + -h|--help) + show_help + exit 0 + ;; + -c|--config) + CONFIG_FILE="$2" + shift 2 + ;; + *) + log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + show_help + exit 1 + ;; + esac + done +} + +# Check if config file exists +check_config_file() { + if [ ! -f "$CONFIG_FILE" ]; then + log_error "Configuration file not found: $CONFIG_FILE" + log_info "Please create config file first:" + log_info " cp config.env.example config.env" + log_info " vim config.env" + exit 1 + fi + + log_success "Configuration file found: $CONFIG_FILE" +} + +# Load configuration from config file +load_config() { + log_info "Loading configuration from: $CONFIG_FILE" + + # Source the config file + set -a + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + set +a + + # Log loaded configuration + log_info "Manager Region: ${MANAGER_REGION}" + log_info "Manager Cluster ID: ${MANAGER_CLUSTER_ID}" + log_info "Worker Region: ${WORKER_REGION}" + log_info "Worker Cluster ID: ${WORKER_CLUSTER_ID}" + log_info "Worker Cluster Name: ${WORKER_CLUSTER_NAME}" + + log_success "Configuration loaded" +} + +# Check prerequisites +check_prerequisites() { + log_info "Checking required tools..." + + local missing_tools=() + + # Check tccli + if ! command -v tccli &> /dev/null; then + missing_tools+=("tccli") + else + log_success "tccli is installed" + fi + + # Check jq + if ! command -v jq &> /dev/null; then + missing_tools+=("jq") + else + log_success "jq is installed" + fi + + # Check kubectl + if ! command -v kubectl &> /dev/null; then + missing_tools+=("kubectl") + else + log_success "kubectl is installed" + fi + + # Check helm + if ! command -v helm &> /dev/null; then + missing_tools+=("helm") + else + log_success "helm is installed" + fi + + # If any tools are missing, report and exit + if [ ${#missing_tools[@]} -gt 0 ]; then + log_error "Missing required tools: ${missing_tools[*]}" + log_error "Please install missing tools before running this script" + echo "" + echo "Installation guides:" + echo " tccli: https://cloud.tencent.com/document/product/440/6176" + echo " jq: https://jqlang.org/download/" + echo " kubectl: https://kubernetes.io/docs/tasks/tools/" + echo " helm: https://helm.sh/docs/intro/install/" + exit 1 + fi + + # Check tccli configuration + if ! tccli --version &> /dev/null; then + log_error "tccli is not configured properly. Please run 'tccli configure' first" + exit 1 + fi + + log_success "tccli configuration check passed" + + # Check required configuration variables + if [ -z "$MANAGER_REGION" ]; then + log_error "MANAGER_REGION is required. Please set it in config file" + exit 1 + fi + + if [ -z "$MANAGER_CLUSTER_ID" ]; then + log_error "MANAGER_CLUSTER_ID is required. Please set it in config file" + exit 1 + fi + + if [ -z "$MANAGER_SUBNET_ID" ]; then + log_error "MANAGER_SUBNET_ID is required. Please set it in config file" + exit 1 + fi + + if [ -z "$WORKER_REGION" ]; then + log_error "WORKER_REGION is required. Please set it in config file" + exit 1 + fi + + if [ -z "$WORKER_CLUSTER_ID" ]; then + log_error "WORKER_CLUSTER_ID is required. Please set it in config file" + exit 1 + fi + + if [ -z "$WORKER_SUBNET_ID" ]; then + log_error "WORKER_SUBNET_ID is required. Please set it in config file" + exit 1 + fi + + log_success "All prerequisites check passed" +} + +# Check required scripts exist +check_scripts() { + log_info "Checking required scripts..." + + if [ ! -f "$INSTALL_TKE_SCRIPT" ]; then + log_error "install-tke.sh not found: $INSTALL_TKE_SCRIPT" + exit 1 + fi + log_success "install-tke.sh found" + + if [ ! -f "$VLLM_INFER_SCRIPT" ]; then + log_error "vllm-infer-demo.sh not found: $VLLM_INFER_SCRIPT" + exit 1 + fi + log_success "vllm-infer-demo.sh found" + + if [ ! -f "$VERL_TRAINING_SCRIPT" ]; then + log_error "llm-kuberay-verl-demo.sh not found: $VERL_TRAINING_SCRIPT" + exit 1 + fi + log_success "llm-kuberay-verl-demo.sh found" + + log_success "All required scripts exist" +} + +# Install Kubeocean using install-tke.sh +install_kubeocean() { + log_info "Installing Kubeocean using install-tke.sh..." + log_info "This will install both worker and manager clusters and create cluster binding" + + # Create a temporary config file for install-tke.sh in the installation-tke directory + local tke_config_file="${SCRIPT_DIR}/../installation-tke/config.env" + + # Copy our config to the installation-tke directory if it doesn't exist + if [ ! -f "$tke_config_file" ]; then + log_info "Creating config file for install-tke.sh at: $tke_config_file" + cp "$CONFIG_FILE" "$tke_config_file" + else + log_warning "Config file already exists at $tke_config_file, using existing one" + fi + + if bash "$INSTALL_TKE_SCRIPT" -c "$tke_config_file"; then + log_success "Kubeocean installation completed" + else + log_error "Kubeocean installation failed" + exit 1 + fi +} + +# Deploy vLLM inference service +deploy_vllm_inference() { + log_info "Deploying vLLM inference service..." + + if bash "$VLLM_INFER_SCRIPT"; then + log_success "vLLM inference service deployment completed" + else + log_error "vLLM inference service deployment failed" + exit 1 + fi +} + +# Patch ResourceLeasingPolicy timeWindows if SKIP_CURRENT_TIME_CHECKING is true +patch_rlp_time_windows() { + if [ "$SKIP_CURRENT_TIME_CHECKING" != "true" ]; then + log_info "Time checking is enabled, skipping RLP timeWindows patch" + return 0 + fi + + log_info "Patching ResourceLeasingPolicy timeWindows (SKIP_CURRENT_TIME_CHECKING=true)..." + + # Switch to worker cluster context + local worker_context="worker-admin-$WORKER_CLUSTER_ID" + + if ! kubectl config use-context "$worker_context" &> /dev/null; then + log_error "Failed to switch to worker cluster context: $worker_context" + log_error "Please ensure the worker cluster is properly configured" + exit 1 + fi + + log_success "Switched to worker cluster context: $worker_context" + + # Check if the RLP exists + if ! kubectl get resourceleasingpolicy example-policy &> /dev/null; then + log_warning "ResourceLeasingPolicy 'example-policy' not found, skipping patch" + return 0 + fi + + # Patch the RLP to set timeWindows to empty + log_info "Removing timeWindows from example-policy..." + + if kubectl patch resourceleasingpolicy example-policy --type='json' \ + -p='[{"op": "replace", "path": "/spec/timeWindows", "value": []}]'; then + log_success "ResourceLeasingPolicy 'example-policy' timeWindows cleared" + else + log_warning "Failed to patch ResourceLeasingPolicy, but continuing..." + fi +} + +# Deploy VERL training job +deploy_verl_training() { + log_info "Deploying VERL training job on manager cluster..." + + if bash "$VERL_TRAINING_SCRIPT"; then + log_success "VERL training job deployment completed" + + # Try to extract Ray job ID + extract_ray_job_id + else + log_error "VERL training job deployment failed" + exit 1 + fi +} + +# Extract Ray job ID from manager cluster +extract_ray_job_id() { + log_info "Attempting to extract Ray job ID..." + + # Switch to manager cluster context + local manager_context="manager-admin-$MANAGER_CLUSTER_ID" + + if ! kubectl config use-context "$manager_context" &> /dev/null; then + log_warning "Failed to switch to manager cluster context, skipping job ID extraction" + return 0 + fi + + # Find the Ray head pod + local head_pod + head_pod=$(kubectl get pods -n "$MANAGER_NAMESPACE" \ + -l ray.io/cluster="$RAYCLUSTER_NAME",ray.io/node-type=head \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) + + if [ -z "$head_pod" ]; then + log_warning "Ray head pod not found, skipping job ID extraction" + return 0 + fi + + # Get the latest job ID using ray job list + local job_list_output + job_list_output=$(kubectl exec -n "$MANAGER_NAMESPACE" "$head_pod" -- bash -c "ray job list 2>&1" 2>&1) || true + + if [ -z "$job_list_output" ]; then + log_warning "Failed to get Ray job list" + return 0 + fi + + # Extract the most recent job ID (first non-header line with raysubmit_) + local job_id + job_id=$(echo "$job_list_output" | grep -oP "raysubmit_[a-zA-Z0-9]+" | head -1) + + if [ -n "$job_id" ]; then + RAY_JOB_ID="$job_id" + log_success "Extracted Ray Job ID: $RAY_JOB_ID" + else + log_warning "Could not extract Ray job ID from job list output" + fi +} + +# Display final summary +show_final_summary() { + local manager_context="manager-admin-$MANAGER_CLUSTER_ID" + local worker_context="worker-admin-$WORKER_CLUSTER_ID" + + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + log_success "LLM Training and Inference Demo Deployment Complete!" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "📋 Deployment Summary:" + echo " • Configuration: $CONFIG_FILE" + echo " • Kubeocean installed ✓" + echo " • vLLM inference service deployed ✓" + echo " • VERL training job submitted ✓" + if [ -n "$RAY_JOB_ID" ]; then + echo " • Ray Job ID: $RAY_JOB_ID" + fi + echo "" + echo "📊 Cluster Information:" + echo " Manager Cluster:" + echo " • Region: $MANAGER_REGION" + echo " • Cluster ID: $MANAGER_CLUSTER_ID" + echo " • Context: $manager_context" + echo "" + echo " Worker Cluster:" + echo " • Region: $WORKER_REGION" + echo " • Cluster ID: $WORKER_CLUSTER_ID" + echo " • Context: $worker_context" + echo "" + echo "🔍 Quick Verification Commands:" + echo "" + echo " # Check Kubeocean components" + echo " kubectl config use-context $manager_context" + echo " kubectl get all -n kubeocean-system" + echo " kubectl get clusterbindings" + echo "" + echo " # Check vLLM inference service on worker cluster" + echo " kubectl config use-context $worker_context" + echo " kubectl get deployment is-qwen2-5-05b-vllm -n default" + echo " kubectl get pods -l app.kubernetes.io/instance=is-qwen2-5-05b-vllm -n default" + echo " kubectl get service is-qwen2-5-05b-vllm -n default" + echo "" + echo " # Check VERL training on manager cluster" + echo " kubectl config use-context $manager_context" + echo " kubectl get raycluster verl-cluster -n default" + echo " kubectl get pods -l ray.io/cluster=verl-cluster -n default" + echo "" + echo " # Monitor training job" + echo " HEAD_POD=\$(kubectl get pods -n default -l ray.io/cluster=verl-cluster,ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')" + + if [ -n "$RAY_JOB_ID" ]; then + echo " kubectl exec -n default \$HEAD_POD -- ray job status $RAY_JOB_ID" + echo " kubectl exec -n default \$HEAD_POD -- ray job logs $RAY_JOB_ID --follow" + else + echo " kubectl exec -n default \$HEAD_POD -- ray job list" + fi + + echo " kubectl logs -n default \$HEAD_POD -f" + echo "" + echo "🎉 Demo environment is ready to use!" + echo "" +} + +# Main function +main() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo " 🚀 Kubeocean LLM Training and Inference Demo" + echo " 🚀 One-Click Deployment Script" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + + # Parse arguments + parse_args "$@" + + # Check config file + check_config_file + echo "" + + # Load configuration + load_config + echo "" + + # Step 1: Check prerequisites + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "📋 Step 1/5: Checking Prerequisites" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + check_prerequisites + check_scripts + echo "" + + # Step 2: Install Kubeocean + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "📦 Step 2/5: Installing Kubeocean" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + install_kubeocean + echo "" + + # Step 3: Deploy vLLM inference service + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🤖 Step 3/5: Deploying vLLM Inference Service" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + deploy_vllm_inference + echo "" + + # Step 4: Deploy VERL training job + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🎓 Step 4/5: Deploying VERL Training Job" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + patch_rlp_time_windows + deploy_verl_training + echo "" + + # Step 5: Display final summary + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "📊 Step 5/5: Summary" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + show_final_summary +} + +# Execute main function +main "$@" + diff --git a/examples/cookbook/llm-train-and-infer-cross-clusters/verl-raycluster.yaml b/examples/cookbook/llm-train-and-infer-cross-clusters/verl-raycluster.yaml new file mode 100644 index 0000000..06196e7 --- /dev/null +++ b/examples/cookbook/llm-train-and-infer-cross-clusters/verl-raycluster.yaml @@ -0,0 +1,62 @@ +apiVersion: ray.io/v1 +kind: RayCluster +metadata: + name: verl-cluster +spec: + rayVersion: '2.43.0' + headGroupSpec: + rayStartParams: {} + template: + spec: + containers: + - name: ray-head + image: ccr.ccs.tencentyun.com/firmlyzhu/verl:ngc-th2.6.0-cu126-vllm0.8.4-flashinfer0.2.2-gsm8k-Qwen2.5-2 + resources: + limits: + cpu: 36 + memory: 140G + nvidia.com/gpu: "2" + requests: + cpu: 24 + memory: 100G + nvidia.com/gpu: "2" + ports: + - containerPort: 6379 + name: gcs-server + - containerPort: 8265 + name: dashboard + - containerPort: 10001 + name: client + tolerations: + - operator: Exists + key: kubeocean.io/vnode + workerGroupSpecs: + - groupName: worker-group + maxReplicas: 1 + minReplicas: 1 + replicas: 1 + numOfHosts: 1 + template: + spec: + containers: + - name: ray-worker + image: ccr.ccs.tencentyun.com/firmlyzhu/verl:ngc-th2.6.0-cu126-vllm0.8.4-flashinfer0.2.2-gsm8k-Qwen2.5-2 + resources: + limits: + cpu: 36 + memory: 150G + nvidia.com/gpu: "2" + requests: + cpu: 24 + memory: 100G + nvidia.com/gpu: "2" + ports: + - containerPort: 6379 + name: gcs-server + - containerPort: 8265 + name: dashboard + - containerPort: 10001 + name: client + tolerations: + - operator: Exists + key: kubeocean.io/vnode \ No newline at end of file diff --git a/examples/cookbook/llm-train-and-infer-cross-clusters/vllm-infer-demo.sh b/examples/cookbook/llm-train-and-infer-cross-clusters/vllm-infer-demo.sh new file mode 100755 index 0000000..4bd80c0 --- /dev/null +++ b/examples/cookbook/llm-train-and-infer-cross-clusters/vllm-infer-demo.sh @@ -0,0 +1,668 @@ +#!/bin/bash + +# Copyright 2025 The Kubeocean Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# vLLM Inference Service Deployment Script for TKE +# Features: +# 1. Pre-check TKE cluster and tccli tool +# 2. Get cluster kubeconfig and set context +# 3. Check and install tke-hpc-controller addon if needed +# 4. Deploy vLLM inference service with HorizontalPodCronscaler + +set -e +set -o pipefail + +# Color definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Logging functions +log_info() { + echo -e "${BLUE}ℹ️ $1${NC}" +} + +log_success() { + echo -e "${GREEN}✅ $1${NC}" +} + +log_warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +log_error() { + echo -e "${RED}❌ $1${NC}" +} + +# Error handling +trap 'log_error "Script execution failed with exit code: $?, at line: $LINENO"' ERR + +# Script root directory +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Configuration file path +CONFIG_FILE="${SCRIPT_DIR}/config.env" + +# Worker Cluster Configuration variables +WORKER_REGION="" +WORKER_CLUSTER_ID="" +WORKER_SUBNET_ID="" +WORKER_NAMESPACE="default" +SKIP_INSTALL_HPC="false" + +# VPC information (will be extracted from cluster info) +VPC_ID="" + +# Display help information +show_help() { + cat << EOF +vLLM Inference Service Deployment Script for TKE + +This script automates the deployment of vLLM inference service with the following steps: +1. Pre-check tccli and cluster connectivity +2. Get cluster admin kubeconfig and set kubectl context +3. Check and install tke-hpc-controller addon if needed +4. Deploy vLLM inference service with HorizontalPodCronscaler + +Usage: $0 [options] + +Options: + -h, --help Display this help message + -c, --config FILE Configuration file path (default: ./config.env) + +Required Configuration Variables (in config file): + WORKER_REGION Worker cluster region + WORKER_CLUSTER_ID Worker cluster ID + WORKER_SUBNET_ID Worker cluster subnet ID for internal network access + +Optional Configuration Variables: + WORKER_NAMESPACE Namespace for deployment (default: default) + SKIP_INSTALL_HPC Skip tke-hpc-controller installation (default: false) + +Prerequisites: + tccli Tencent Cloud CLI tool (must be installed and configured) + jq Command-line JSON processor + kubectl Kubernetes command-line tool + +Examples: + # Use default configuration file (./config.env) + $0 + + # Use custom configuration file + $0 --config /path/to/my-config.env + +Configuration File Example: + cp config.env.example config.env + # Edit config.env with your values + vim config.env + +EOF +} + +# Load configuration from file +load_config() { + if [[ -f "$CONFIG_FILE" ]]; then + log_info "Loading configuration from: $CONFIG_FILE" + set -a + # shellcheck disable=SC1090 + source "$CONFIG_FILE" + set +a + log_success "Configuration loaded" + fi +} + +# Parse command line arguments +parse_args() { + while [[ $# -gt 0 ]]; do + case $1 in + -h|--help) + show_help + exit 0 + ;; + -c|--config) + CONFIG_FILE="$2" + shift 2 + ;; + *) + log_error "Unknown argument: $1" + log_info "Only -h/--help and -c/--config options are supported" + log_info "All other configurations should be provided in config file" + show_help + exit 1 + ;; + esac + done +} + +# Check prerequisites +check_prerequisites() { + log_info "Checking prerequisites..." + + # Check required parameters + if [ -z "$WORKER_REGION" ]; then + log_error "WORKER_REGION is required. Please set it in config file" + exit 1 + fi + + if [ -z "$WORKER_CLUSTER_ID" ]; then + log_error "WORKER_CLUSTER_ID is required. Please set it in config file" + exit 1 + fi + + log_info "Worker Region: $WORKER_REGION" + log_info "Worker Cluster ID: $WORKER_CLUSTER_ID" + log_info "Worker Namespace: $WORKER_NAMESPACE" + + # Check tccli installation + if ! command -v tccli &> /dev/null; then + log_error "tccli is not installed. Please install Tencent Cloud CLI first" + log_error "Installation guide: https://cloud.tencent.com/document/product/440/6176" + exit 1 + fi + + log_success "tccli is installed" + + # Check tccli configuration + if ! tccli --version &> /dev/null; then + log_error "tccli is not configured properly. Please run 'tccli configure' first" + exit 1 + fi + + log_success "tccli configuration check passed" + + # Check jq installation + if ! command -v jq &> /dev/null; then + log_error "jq is not installed. Please install jq first" + log_error "Installation guide: https://jqlang.org/download/" + exit 1 + fi + + log_success "jq is installed" + + # Check if cluster exists and get VPC info + log_info "Checking cluster existence and getting VPC info..." + local cluster_info + ret=0 + cluster_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusters --ClusterIds "[\"$WORKER_CLUSTER_ID\"]" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to query cluster information" + log_error "$cluster_info" + exit 1 + fi + + # Check if cluster exists + local total_count + total_count=$(echo "$cluster_info" | jq -r '.TotalCount // 0') + + if [ "$total_count" -eq 0 ]; then + log_error "Cluster $WORKER_CLUSTER_ID not found in region $WORKER_REGION" + exit 1 + fi + + # Extract VPC ID + VPC_ID=$(echo "$cluster_info" | jq -r '.Clusters[0].ClusterNetworkSettings.VpcId // empty') + + if [ -z "$VPC_ID" ]; then + log_error "Failed to get VPC ID from cluster information" + exit 1 + fi + + log_success "Cluster found in VPC: $VPC_ID" + + # Check kubectl + if ! command -v kubectl &> /dev/null; then + log_error "kubectl is not installed. Please install kubectl first" + exit 1 + fi + + log_success "Prerequisites check passed" +} + +# Enable cluster internal network access +enable_cluster_internal_access() { + log_info "Checking cluster internal network access status..." + + # Check if internal access is already enabled + local endpoint_status + ret=0 + endpoint_status=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$WORKER_CLUSTER_ID" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to query cluster endpoint status" + log_error "$endpoint_status" + exit 1 + fi + + local status + status=$(echo "$endpoint_status" | jq -r '.Status // empty') + + if [ "$status" = "Created" ]; then + log_success "Cluster internal network access is already enabled" + return 0 + fi + + # Need to enable internal access + log_info "Enabling cluster internal network access..." + + # Check subnet ID parameter + if [ -z "$WORKER_SUBNET_ID" ]; then + log_error "WORKER_SUBNET_ID is required for enabling internal access. Please set it in config file" + exit 1 + fi + + # Enable internal access + local enable_result + ret=0 + enable_result=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke CreateClusterEndpoint \ + --ClusterId "$WORKER_CLUSTER_ID" \ + --SubnetId "$WORKER_SUBNET_ID" \ + --IsExtranet false 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to enable cluster internal network access" + log_error "$enable_result" + exit 1 + fi + + log_info "Waiting for internal network access to be ready..." + + # Poll for status (timeout: 120 seconds) + local max_retries=60 + local retry=0 + + while [ $retry -lt $max_retries ]; do + sleep 2 + + ret=0 + endpoint_status=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusterEndpointStatus --ClusterId "$WORKER_CLUSTER_ID" 2>&1) || ret=$? + + if [ $ret -eq 0 ]; then + status=$(echo "$endpoint_status" | jq -r '.Status // empty') + + if [ "$status" = "Created" ]; then + log_success "Cluster internal network access enabled successfully" + return 0 + elif [ "$status" = "Creating" ]; then + log_info "Still creating... (attempt $((retry + 1))/$max_retries)" + else + log_warning "Unexpected status: $status" + fi + fi + + retry=$((retry + 1)) + done + + log_error "Timeout waiting for cluster internal network access to be ready" + exit 1 +} + +# Get cluster kubeconfig and set context +get_cluster_kubeconfig() { + log_info "Getting cluster kubeconfig..." + + # Check if context already exists + local context_name="worker-admin-$WORKER_CLUSTER_ID" + if kubectl config get-contexts "$context_name" &> /dev/null; then + log_info "Context '$context_name' already exists, reusing it" + kubectl config use-context "$context_name" + log_success "Switched to existing context: $context_name" + return 0 + fi + + log_info "Context not found, fetching kubeconfig from TKE..." + + # Wait for internal load balancer information to update + log_info "Waiting 5 seconds for internal load balancer information to update..." + sleep 5 + + # Get cluster security info (including kubeconfig) + local security_info + ret=0 + security_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeClusterSecurity --ClusterId "$WORKER_CLUSTER_ID" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to get cluster security information" + log_error "$security_info" + exit 1 + fi + + # Extract internal kubeconfig + local kubeconfig_content + kubeconfig_content=$(echo "$security_info" | jq -r '.Kubeconfig // empty') + + if [ -z "$kubeconfig_content" ]; then + log_error "Failed to extract kubeconfig from cluster security information" + exit 1 + fi + + # Extract PgwEndpoint (internal network endpoint) + local pgw_endpoint + pgw_endpoint=$(echo "$security_info" | jq -r '.PgwEndpoint // empty') + + if [ -z "$pgw_endpoint" ]; then + log_error "Failed to extract PgwEndpoint from cluster security information" + exit 1 + fi + + log_info "PgwEndpoint: $pgw_endpoint" + + # Save kubeconfig to temporary file + local temp_kubeconfig="/tmp/tke-kubeconfig-$WORKER_CLUSTER_ID" + echo "$kubeconfig_content" > "$temp_kubeconfig" + + # Replace server address with PgwEndpoint + log_info "Replacing server address with PgwEndpoint..." + local cluster_name + cluster_name=$(kubectl --kubeconfig="$temp_kubeconfig" config view -o jsonpath='{.clusters[0].name}' 2>/dev/null) + + if [ -n "$cluster_name" ]; then + kubectl --kubeconfig="$temp_kubeconfig" config set-cluster "$cluster_name" --server="https://$pgw_endpoint" &> /dev/null + log_success "Server address updated to: $pgw_endpoint" + else + log_error "Failed to get cluster name from kubeconfig" + exit 1 + fi + + # Verify connection before merging + log_info "Verifying cluster connection..." + if ! kubectl --kubeconfig="$temp_kubeconfig" cluster-info &> /dev/null; then + log_error "Failed to connect to cluster using the kubeconfig" + log_error "Please check if the cluster internal network access is properly configured" + rm -f "$temp_kubeconfig" + exit 1 + fi + log_success "Successfully verified cluster connection" + + # Set kubectl context name + local context_name="worker-admin-$WORKER_CLUSTER_ID" + + log_info "Merging kubeconfig to default kubectl config" + log_info "Context name: $context_name" + + # Get default KUBECONFIG path + local default_kubeconfig="${HOME}/.kube/config" + + # Create .kube directory if it doesn't exist + mkdir -p "${HOME}/.kube" + + # If default kubeconfig doesn't exist, create an empty one + if [ ! -f "$default_kubeconfig" ]; then + touch "$default_kubeconfig" + fi + + # Merge the TKE kubeconfig into default kubeconfig + KUBECONFIG="$default_kubeconfig:$temp_kubeconfig" kubectl config view --flatten > "${temp_kubeconfig}.merged" + mv "${temp_kubeconfig}.merged" "$default_kubeconfig" + + # Get the original context name from temp kubeconfig + local original_context + original_context=$(kubectl --kubeconfig="$temp_kubeconfig" config current-context 2>/dev/null) + + # Rename context to our desired name + if [ -n "$original_context" ]; then + kubectl config rename-context "$original_context" "$context_name" &> /dev/null || true + fi + + # Set as current context + kubectl config use-context "$context_name" &> /dev/null + + # Clean up temporary file + rm -f "$temp_kubeconfig" + + log_success "Kubeconfig merged to: $default_kubeconfig" + log_success "Context name: $context_name" + log_success "Cluster is ready for use" +} + +# Check and install tke-hpc-controller addon +check_and_install_hpc_controller() { + # Check if we should skip HPC installation + if [ "$SKIP_INSTALL_HPC" = "true" ]; then + log_warning "SKIP_INSTALL_HPC is set to true, skipping tke-hpc-controller installation" + return 0 + fi + + log_info "Checking tke-hpc-controller addon status..." + + local addon_name="tke-hpc-controller" + + # Check if addon is already installed + local addon_info + ret=0 + addon_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeAddon \ + --ClusterId "$WORKER_CLUSTER_ID" \ + --AddonName "$addon_name" 2>&1) || ret=$? + + if [ $ret -eq 0 ]; then + # Addon exists, check its phase + local phase + phase=$(echo "$addon_info" | jq -r '.Addons[0].Phase // empty') + + if [ "$phase" = "Succeeded" ]; then + log_success "tke-hpc-controller addon is already installed and running" + return 0 + elif [ "$phase" = "Failed" ]; then + log_warning "tke-hpc-controller addon exists but in Failed state" + log_info "Will attempt to reinstall..." + else + log_info "tke-hpc-controller addon is in phase: $phase" + log_info "Waiting for it to become Succeeded..." + # Continue to polling below + fi + else + # Addon doesn't exist, need to install + log_info "tke-hpc-controller addon not found, installing..." + + local install_result + ret=0 + install_result=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke InstallAddon \ + --ClusterId "$WORKER_CLUSTER_ID" \ + --AddonName "$addon_name" 2>&1) || ret=$? + + if [ $ret -ne 0 ]; then + log_error "Failed to install tke-hpc-controller addon" + log_error "$install_result" + exit 1 + fi + + log_success "tke-hpc-controller addon installation initiated" + fi + + # Poll for addon status (timeout: 300 seconds / 5 minutes) + log_info "Waiting for tke-hpc-controller addon to be ready..." + local max_retries=150 + local retry=0 + + while [ $retry -lt $max_retries ]; do + sleep 2 + + ret=0 + addon_info=$(TENCENTCLOUD_REGION="$WORKER_REGION" tccli tke DescribeAddon \ + --ClusterId "$WORKER_CLUSTER_ID" \ + --AddonName "$addon_name" 2>&1) || ret=$? + + if [ $ret -eq 0 ]; then + local phase + phase=$(echo "$addon_info" | jq -r '.Addons[0].Phase // empty') + + if [ "$phase" = "Succeeded" ]; then + log_success "tke-hpc-controller addon is ready" + return 0 + elif [ "$phase" = "Failed" ]; then + log_error "tke-hpc-controller addon installation failed" + local reason + reason=$(echo "$addon_info" | jq -r '.Addons[0].Reason // "Unknown reason"') + log_error "Reason: $reason" + exit 1 + elif [ "$phase" = "Installing" ] || [ "$phase" = "Upgrading" ]; then + if [ $((retry % 10)) -eq 0 ]; then + log_info "Still installing... (attempt $((retry + 1))/$max_retries)" + fi + else + log_info "Current phase: $phase (attempt $((retry + 1))/$max_retries)" + fi + else + log_warning "Failed to query addon status, retrying..." + fi + + retry=$((retry + 1)) + done + + log_error "Timeout waiting for tke-hpc-controller addon to be ready" + exit 1 +} + +# Deploy vLLM inference service +deploy_vllm_service() { + log_info "Deploying vLLM inference service..." + + local yaml_file="${SCRIPT_DIR}/is-qwen2-5-05b-vllm.yaml" + + if [ ! -f "$yaml_file" ]; then + log_error "YAML file not found: $yaml_file" + exit 1 + fi + + log_info "Using YAML file: $yaml_file" + + # Apply the YAML file + ret=0 + if kubectl apply -f "$yaml_file" -n "$WORKER_NAMESPACE" 2>&1; then + log_success "vLLM inference service deployed successfully" + else + log_error "Failed to deploy vLLM inference service" + exit 1 + fi + + log_info "Checking deployment status..." + + # Wait a moment for resources to be created + sleep 2 + + # Display deployment status + log_info "Deployment resources:" + kubectl get deployment is-qwen2-5-05b-vllm -n "$WORKER_NAMESPACE" || log_warning "Deployment not found yet" + + log_info "Service resources:" + kubectl get service is-qwen2-5-05b-vllm -n "$WORKER_NAMESPACE" || log_warning "Service not found yet" + + log_info "HorizontalPodCronscaler resources:" + kubectl get horizontalpodcronscaler is-qwen -n "$WORKER_NAMESPACE" || log_warning "HorizontalPodCronscaler not found yet" + + log_success "Deployment initiated. Use 'kubectl get pods -n $WORKER_NAMESPACE' to check pod status" +} + +# Display summary information +show_summary() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + log_success "vLLM Inference Service Deployment Complete!" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "📋 Deployment Information:" + echo " • Region: $WORKER_REGION" + echo " • Cluster ID: $WORKER_CLUSTER_ID" + echo " • Namespace: $WORKER_NAMESPACE" + echo " • Context: worker-admin-$WORKER_CLUSTER_ID" + echo "" + echo "🔍 Next Steps:" + echo " 1. Check deployment status:" + echo " kubectl get deployment is-qwen2-5-05b-vllm -n $WORKER_NAMESPACE" + echo "" + echo " 2. Check pod status:" + echo " kubectl get pods -l app.kubernetes.io/instance=is-qwen2-5-05b-vllm -n $WORKER_NAMESPACE" + echo "" + echo " 3. Check service:" + echo " kubectl get service is-qwen2-5-05b-vllm -n $WORKER_NAMESPACE" + echo "" + echo " 4. Check HorizontalPodCronscaler:" + echo " kubectl get horizontalpodcronscaler is-qwen -n $WORKER_NAMESPACE" + echo " kubectl describe horizontalpodcronscaler is-qwen -n $WORKER_NAMESPACE" + echo "" + echo " 5. View pod logs:" + echo " kubectl logs -l app.kubernetes.io/instance=is-qwen2-5-05b-vllm -n $WORKER_NAMESPACE" + echo "" + echo " 6. Test the inference service (after pods are ready):" + echo " kubectl port-forward svc/is-qwen2-5-05b-vllm 8000:8000 -n $WORKER_NAMESPACE" + echo " curl http://localhost:8000/health" + echo "" + echo "📝 Scheduled Scaling:" + echo " • Scale up to 4 replicas: Every day at 08:00" + echo " • Scale down to 2 replicas: Every day at 18:00" + echo "" +} + +# Main function +main() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo " 🚀 vLLM Inference Service Deployment Script" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + + # Step 0: Parse config file path first + for arg in "$@"; do + if [[ "$arg" == "-c" || "$arg" == "--config" ]]; then + shift + CONFIG_FILE="$1" + break + fi + shift + done + + # Load configuration file + load_config + + # Parse arguments (command line overrides config file) + parse_args "$@" + + # Step 1: Check prerequisites + echo "📋 Step 1/5: Checking Prerequisites" + check_prerequisites + echo "" + + # Step 2: Enable cluster internal network access and get kubeconfig + echo "🔐 Step 2/5: Setting up Cluster Access" + enable_cluster_internal_access + get_cluster_kubeconfig + echo "" + + # Step 3: Check and install tke-hpc-controller addon + if [ "$SKIP_INSTALL_HPC" != "true" ]; then + echo "🔌 Step 3/5: Checking tke-hpc-controller Addon" + check_and_install_hpc_controller + echo "" + else + echo "🔌 Step 3/5: Checking tke-hpc-controller Addon" + log_warning "Skipping tke-hpc-controller installation (SKIP_INSTALL_HPC=true)" + echo "" + fi + + # Step 4: Deploy vLLM inference service + echo "📦 Step 4/5: Deploying vLLM Inference Service" + deploy_vllm_service + echo "" + + # Step 5: Display summary + echo "📊 Step 5/5: Summary" + show_summary +} + +# Execute main function +main "$@" + diff --git a/examples/playbook/README.md b/examples/playbook/README.md deleted file mode 100644 index 3a0dcd3..0000000 --- a/examples/playbook/README.md +++ /dev/null @@ -1,208 +0,0 @@ -# Kubeocean One-Click Deployment Playbook - -> English | [中文](README_zh.md) - -This directory provides one-click deployment scripts for Kubeocean components and cluster binding, supporting a one-stop experience of Kubeocean-related features on TKE clusters. - -## Features - -- Deploy Kubeocean components on compute clusters and worker clusters -- Bind worker clusters to compute clusters and configure simple resource leasing policies - -## Prerequisites (Environment Requirements) - -- At least one Kubernetes cluster is required as a virtual compute cluster, and another Kubernetes cluster as a worker cluster -- Pod network and node network are directly interconnected between compute cluster and worker cluster -- Can access clusters using `kubectl` -- Local environment has `helm` installed with version v3 -- Both clusters have APIServer internal network access enabled, i.e., there is a service named `kubernetes-intranet` with type `LoadBalancer` in the `default` namespace. For TKE standard clusters, refer to the following image to enable it in the cluster console: -![k8s-svc](../../docs/images/k8s-svc.png) -- Other requirements refer to: [Requirements](../../docs/requirements.md) - -## Basic Usage - -### Install and Bind -```bash -# Switch kubectl to worker cluster -bash install-worker.sh -# Add label to nodes that need resource extraction -kubectl label node kubeocean.io/role=worker - -# Switch kubectl to compute cluster, and copy the generated /tmp/kubeconfig-worker to local -bash install-manager.sh -``` -*Note: Script execution order cannot be changed* - -### Uninstall -```bash -# Switch kubectl to compute cluster -bash uninstall-manager.sh - -# Switch kubectl to worker cluster -bash uninstall-worker.sh -``` -*Note: Script execution order cannot be changed* - -## Advanced Usage - -### Install and Bind -```bash -# Specify output path for worker kubeconfig -bash install-worker.sh -o /tmp/my-kubeconfig -bash install-worker.sh --output /tmp/my-kubeconfig - -# Skip ResourceLeasingPolicy deployment -bash install-worker.sh --skip-rlp - -# Install manager with specified worker cluster ID and name -bash install-manager.sh -i cls-prod -n prod-cluster -bash install-manager.sh --cluster-id cls-prod --cluster-name prod-cluster - -# Install manager with specified worker kubeconfig input path -bash install-manager.sh -w /tmp/kubeconfig-worker1 -bash install-manager.sh --worker-kubeconfig /tmp/kubeconfig-worker1 - -# Only bind cluster, skip manager installation, and specify cluster ID and name -bash install-manager.sh -i cls-prod -n prod-cluster --skip-manager -``` - -### Uninstall -```bash -# Unbind specific worker cluster by name -bash uninstall-manager.sh -n worker1 -bash uninstall-manager.sh --cluster-name worker1 - -# Only unbind worker cluster without uninstalling manager components -bash uninstall-manager.sh -n worker1 --skip-manager - -# Uninstall and clean up specific RLP object by name -bash uninstall-worker.sh -r my-policy -bash uninstall-worker.sh --rlp-name my-policy -``` - -## TKE Cluster One-Click Deployment - -For Tencent Cloud TKE clusters, dedicated one-click deployment scripts are provided to automatically complete cluster configuration (enable internal network access, create kube-dns-intranet, etc.) and component installation. - -### Prerequisites - -In addition to the basic environment requirements above, you also need: -- Install and configure [Tencent Cloud CLI (tccli)](https://cloud.tencent.com/document/product/440/6176) -- Install [jq](https://jqlang.org/download/) JSON processor -- Prepare TKE cluster region, cluster ID, and subnet ID - -### Deploy Worker Cluster - -```bash -# Basic usage (uses default kubeconfig output path /tmp/kubeconfig-worker) -bash install-worker-tke.sh \ - --region ap-guangzhou \ - --cluster-id cls-xxxxxxxx \ - --subnet-id subnet-xxxxxxxx - -# Specify custom kubeconfig output path -bash install-worker-tke.sh \ - --region ap-guangzhou \ - --cluster-id cls-xxxxxxxx \ - --subnet-id subnet-xxxxxxxx \ - --output /tmp/my-kubeconfig - -# Short parameter form -bash install-worker-tke.sh \ - -r ap-guangzhou \ - -c cls-xxxxxxxx \ - -s subnet-xxxxxxxx -``` - -The script will automatically: -1. Check prerequisites (tccli, jq, kubectl, helm, etc.) -2. Enable cluster internal network access (if not enabled) -3. Get cluster kubeconfig and configure context (name: `worker-admin-`) -4. Install kubeocean-worker components -5. Generate worker kubeconfig for manager cluster binding - -### Deploy Manager Cluster - -```bash -# Basic usage (uses default worker kubeconfig path /tmp/kubeconfig-worker) -bash install-manager-tke.sh \ - --region ap-guangzhou \ - --cluster-id cls-xxxxxxxx \ - --subnet-id subnet-xxxxxxxx \ - --worker-cluster-id cls-worker-xxx - -# Specify custom worker kubeconfig path -bash install-manager-tke.sh \ - --region ap-guangzhou \ - --cluster-id cls-xxxxxxxx \ - --subnet-id subnet-xxxxxxxx \ - --worker-kubeconfig /tmp/my-kubeconfig \ - --worker-cluster-id cls-worker-xxx - -# Short parameter form -bash install-manager-tke.sh \ - -r ap-guangzhou \ - -c cls-xxxxxxxx \ - -s subnet-xxxxxxxx \ - -i cls-worker-xxx -``` - -The script will automatically: -1. Check prerequisites (tccli, jq, kubectl, helm, worker kubeconfig, etc.) -2. Enable cluster internal network access (if not enabled) -3. Get cluster kubeconfig and configure context (name: `manager-admin-`) -4. Create kube-dns-intranet service (if not exists) -5. Install kubeocean-manager components -6. Create ClusterBinding to bind worker cluster - -### Complete Example - -```bash -# Step 1: Deploy worker cluster -bash install-worker-tke.sh \ - -r ap-guangzhou \ - -c cls-xxxxxxxx \ - -s subnet-xxxxxxxx - -# Add label to nodes for resource extraction -kubectl label node kubeocean.io/role=worker - -# Step 2: Deploy manager cluster (will automatically use /tmp/kubeconfig-worker) -bash install-manager-tke.sh \ - -r ap-guangzhou \ - -c cls-xxxxxxxx \ - -s subnet-xxxxxxxx \ - -i cls-xxxxxxxx - -# Step 3: Verify deployment -# Switch to manager cluster context -kubectl config use-context manager-admin-cls-manager-xxx -kubectl get clusterbindings -kubectl get all -n kubeocean-system - -# Switch to worker cluster context -kubectl config use-context worker-admin-cls-worker-xxx -kubectl get all -n kubeocean-worker -``` - -### Parameter Description - -#### install-worker-tke.sh - -| Parameter | Short | Description | Required | Default | -|-----------|-------|-------------|----------|---------| -| `--region` | `-r` | TKE cluster region | Yes | - | -| `--cluster-id` | `-c` | TKE cluster ID | Yes | - | -| `--subnet-id` | `-s` | Subnet ID (for internal network access) | Yes (when enabling internal access) | - | -| `--output` | `-o` | Worker kubeconfig output path | No | `/tmp/kubeconfig-worker` | - -#### install-manager-tke.sh - -| Parameter | Short | Description | Required | Default | -|-----------|-------|-------------|----------|---------| -| `--region` | `-r` | TKE cluster region | Yes | - | -| `--cluster-id` | `-c` | TKE cluster ID | Yes | - | -| `--subnet-id` | `-s` | Subnet ID (for internal access and DNS service) | Yes (when enabling internal access or creating DNS service) | - | -| `--worker-kubeconfig` | `-w` | Worker cluster kubeconfig path | No | `/tmp/kubeconfig-worker` | -| `--worker-cluster-id` | `-i` | Worker cluster ID | Yes | - | -``` diff --git a/examples/playbook/README_zh.md b/examples/playbook/README_zh.md deleted file mode 100644 index 4fbc059..0000000 --- a/examples/playbook/README_zh.md +++ /dev/null @@ -1,208 +0,0 @@ -# Kubeocean 一键部署脚本箱 - -> [English](README.md) | 中文 - -本目录提供了 Kubeocean 组件和集群绑定的一键部署脚本箱,支持在 TKE 集群上一站式体验 Kubeocean 相关功能的能力。 - -## 功能 - -- 在算力集群和工作集群部署 Kubeocean 组件 -- 绑定工作集群到算力集群,并配置简单资源抽取策略 - -## 前置准备(环境要求) - -- 需要至少一个 kubernetes 集群作为虚拟算力集群,以及一个 kubernetes 集群作为工作集群。 -- 算力集群和业务集群 Pod 网络直接互通、节点网络互通 -- 可以使用 `kubectl` 访问集群 -- 本地环境已安装 `helm`,且版本为 v3 -- 集群都已开启 APIServer 内网访问,即在 `default` namespace 下存在名为 `kubernetes-intranet` 且类型为 `LoadBalancer` 的服务。若集群为 TKE 标准集群,可参考下图在集群控制台开启: -![k8s-svc](../../docs/images/k8s-svc.png) -- 其他要求参考:[要求](../../docs/requirements_zh.md) - -## 基础使用 - -### 安装并绑定 -``` -# kubectl 切换到工作集群 -bash install-worker.sh -# 给期望抽取资源的节点添加 label -kubectl label node kubeocean.io/role=worker - -# 将 kubectl 切换到算力集群,并将上一步生成的 /tmp/kubeconfig-worker 拷贝到本地 -bash install-manager.sh -``` -*注意:脚本执行顺序不能修改* - -### 卸载 -``` -# kubectl 切换到算力集群 -bash uninstall-manager.sh - -# kubectl 切换到工作集群 -bash uninstall-worker.sh -``` -*注意:脚本执行顺序不能修改* - -## 进阶使用 - -### 安装并绑定 -``` -# 指定 worker kubeconfig 的输出路径 -bash install-worker.sh -o /tmp/my-kubeconfig -bash install-worker.sh --output /tmp/my-kubeconfig - -# 跳过 ResourceLeasingPolicy 的部署 -bash install-worker.sh --skip-rlp - -# 安装 manager 指定工作集群的 ID 和名称 -bash install-manager.sh -i cls-prod -n prod-cluster -bash install-manager.sh --cluster-id cls-prod --cluster-name prod-cluster - -# 安装 manager 指定 worker kubeconfig 的输入路径 -bash install-manager.sh -w /tmp/kubeconfig-worker1 -bash install-manager.sh --worker-kubeconfig /tmp/kubeconfig-worker1 - -# 只绑定集群,跳过安装 manager 步骤,并指定集群ID和名称 -bash install-manager.sh -i cls-prod -n prod-cluster --skip-manager -``` - -### 卸载 -``` -# 解绑特定名称的工作集群 -bash uninstall-manager.sh -n worker1 -bash uninstall-manager.sh --cluster-name worker1 - -# 只解绑工作集群,不卸载 manager 组件 -bash uninstall-manager.sh -n worker1 --skip-manager - -# 卸载并清理特定名称的 rlp 对象 -bash uninstall-worker.sh -r my-policy -bash uninstall-worker.sh --rlp-name my-policy -``` - -## TKE 集群一键部署 - -针对腾讯云 TKE 集群,提供了专门的一键部署脚本,可以自动完成集群配置(开启内网访问、创建 kube-dns-intranet 等)和组件安装。 - -### 前置准备 - -除了上述基础环境要求外,还需要: -- 安装并配置 [腾讯云 CLI 工具 (tccli)](https://cloud.tencent.com/document/product/440/6176) -- 安装 [jq](https://jqlang.org/download/) JSON 处理工具 -- 准备 TKE 集群所在的地域、集群 ID 和子网 ID - -### 部署 Worker 集群 - -```bash -# 基本使用(使用默认 kubeconfig 输出路径 /tmp/kubeconfig-worker) -bash install-worker-tke.sh \ - --region ap-guangzhou \ - --cluster-id cls-xxxxxxxx \ - --subnet-id subnet-xxxxxxxx - -# 指定自定义 kubeconfig 输出路径 -bash install-worker-tke.sh \ - --region ap-guangzhou \ - --cluster-id cls-xxxxxxxx \ - --subnet-id subnet-xxxxxxxx \ - --output /tmp/my-kubeconfig - -# 短参数形式 -bash install-worker-tke.sh \ - -r ap-guangzhou \ - -c cls-xxxxxxxx \ - -s subnet-xxxxxxxx -``` - -脚本会自动完成以下操作: -1. 检查前置条件(tccli、jq、kubectl、helm 等) -2. 开启集群内网访问(如未开启) -3. 获取集群 kubeconfig 并配置 context(名称:`worker-admin-<集群ID>`) -4. 安装 kubeocean-worker 组件 -5. 生成 worker kubeconfig 用于 manager 集群绑定 - -### 部署 Manager 集群 - -```bash -# 基本使用(使用默认 worker kubeconfig 路径 /tmp/kubeconfig-worker) -bash install-manager-tke.sh \ - --region ap-guangzhou \ - --cluster-id cls-xxxxxxxx \ - --subnet-id subnet-xxxxxxxx \ - --worker-cluster-id cls-worker-xxx - -# 指定自定义 worker kubeconfig 路径 -bash install-manager-tke.sh \ - --region ap-guangzhou \ - --cluster-id cls-xxxxxxxx \ - --subnet-id subnet-xxxxxxxx \ - --worker-kubeconfig /tmp/my-kubeconfig \ - --worker-cluster-id cls-worker-xxx - -# 短参数形式 -bash install-manager-tke.sh \ - -r ap-guangzhou \ - -c cls-xxxxxxxx \ - -s subnet-xxxxxxxx \ - -i cls-worker-xxx -``` - -脚本会自动完成以下操作: -1. 检查前置条件(tccli、jq、kubectl、helm、worker kubeconfig 等) -2. 开启集群内网访问(如未开启) -3. 获取集群 kubeconfig 并配置 context(名称:`manager-admin-<集群ID>`) -4. 创建 kube-dns-intranet 服务(如不存在) -5. 安装 kubeocean-manager 组件 -6. 创建 ClusterBinding 绑定 worker 集群 - -### 完整示例 - -```bash -# Step 1: 部署 worker 集群 -bash install-worker-tke.sh \ - -r ap-guangzhou \ - -c cls-xxxxxxxx \ - -s subnet-xxxxxxxx - -# 给节点添加 label 以标记资源抽取节点 -kubectl label node kubeocean.io/role=worker - -# Step 2: 部署 manager 集群(会自动使用 /tmp/kubeconfig-worker) -bash install-manager-tke.sh \ - -r ap-guangzhou \ - -c cls-xxxxxxxx \ - -s subnet-xxxxxxxx \ - -i cls-xxxxxxxx - -# Step 3: 验证部署 -# 切换到 manager 集群 context -kubectl config use-context manager-admin-cls-manager-xxx -kubectl get clusterbindings -kubectl get all -n kubeocean-system - -# 切换到 worker 集群 context -kubectl config use-context worker-admin-cls-worker-xxx -kubectl get all -n kubeocean-worker -``` - -### 参数说明 - -#### install-worker-tke.sh - -| 参数 | 短参数 | 说明 | 是否必需 | 默认值 | -|------|--------|------|----------|--------| -| `--region` | `-r` | TKE 集群所在地域 | 是 | - | -| `--cluster-id` | `-c` | TKE 集群 ID | 是 | - | -| `--subnet-id` | `-s` | 子网 ID(用于内网访问) | 是(开启内网访问时) | - | -| `--output` | `-o` | Worker kubeconfig 输出路径 | 否 | `/tmp/kubeconfig-worker` | - -#### install-manager-tke.sh - -| 参数 | 短参数 | 说明 | 是否必需 | 默认值 | -|------|--------|------|----------|--------| -| `--region` | `-r` | TKE 集群所在地域 | 是 | - | -| `--cluster-id` | `-c` | TKE 集群 ID | 是 | - | -| `--subnet-id` | `-s` | 子网 ID(用于内网访问和 DNS 服务) | 是(开启内网访问或创建 DNS 服务时) | - | -| `--worker-kubeconfig` | `-w` | Worker 集群 kubeconfig 路径 | 否 | `/tmp/kubeconfig-worker` | -| `--worker-cluster-id` | `-i` | Worker 集群 ID | 是 | - | -``` \ No newline at end of file diff --git a/examples/resourceleasingpolicy_sample.yaml b/examples/resourceleasingpolicy_sample.yaml index 1e316f6..ab88783 100644 --- a/examples/resourceleasingpolicy_sample.yaml +++ b/examples/resourceleasingpolicy_sample.yaml @@ -20,7 +20,7 @@ spec: days: ["Saturday", "Sunday"] resourceLimits: - resource: cpu - quantity: "4" + quantity: "96" percent: 90 # Take the smaller of 4 CPUs or 90% of available CPUs - resource: memory percent: 95 # Take 95% of available memory diff --git a/examples/tke-dns-svc.yaml b/examples/tke-dns-svc.yaml new file mode 100644 index 0000000..6c67a08 --- /dev/null +++ b/examples/tke-dns-svc.yaml @@ -0,0 +1,29 @@ +apiVersion: v1 +kind: Service +metadata: + annotations: + service.cloud.tencent.com/direct-access: "true" + service.cloud.tencent.com/pass-to-target: "true" + service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: + name: kube-dns-intranet + namespace: kube-system +spec: + allocateLoadBalancerNodePorts: true + externalTrafficPolicy: Cluster + internalTrafficPolicy: Cluster + ipFamilies: + - IPv4 + ipFamilyPolicy: SingleStack + ports: + - name: dns + port: 53 + protocol: UDP + targetPort: 53 + - name: dns-tcp + port: 53 + protocol: TCP + targetPort: 53 + selector: + k8s-app: kube-dns + sessionAffinity: None + type: LoadBalancer \ No newline at end of file diff --git a/hack/makelib/kind.mk b/hack/makelib/kind.mk index 6722cce..7a9ab3b 100644 --- a/hack/makelib/kind.mk +++ b/hack/makelib/kind.mk @@ -137,7 +137,7 @@ kind-create-all: kind-create-manager kind-create-worker1 kind-create-worker2 ## @echo "" @echo "🎉 All kubeocean KIND clusters created successfully!" @echo "" - @echo "🔍 Check status with: make kind-status" + @echo "🔍 Check status with: make kind-status-all" .PHONY: kind-delete-manager kind-delete-manager: kind ## Delete KIND manager cluster. diff --git a/hack/rules/documentation-rules.md b/hack/rules/documentation-rules.md new file mode 100644 index 0000000..aae4437 --- /dev/null +++ b/hack/rules/documentation-rules.md @@ -0,0 +1,236 @@ +# Kubeocean 文档编写规则 + +## 文档格式规范 + +### 1. 代码块格式 +- **强制要求**:所有代码块必须明确指定语言标记 + - Shell 命令使用 `sh` 标记 + - YAML 配置使用 `yaml` 标记 + - 其他语言使用对应标记 +- **空行要求**:代码块前后必须有空行,提高可读性 +- **示例**: + ```markdown + 执行以下命令: + + ```sh + kubectl get pods + ``` + + 查看结果... + ``` + +### 2. YAML Front Matter +- **执行类文档**:需要添加 front matter 指定工作目录 + ```yaml + --- + cwd: ../ + --- + ``` +- **适用文档**:快速开始、安装指南等需要执行命令的教程文档 + +### 3. 段落结构 +- 标题后添加空行 +- 步骤间添加空行分隔 +- 说明性文字与代码块之间添加空行 +- 代码块与代码块之间添加空行 + +## 命令编写规范 + +### 1. 优先使用环境变量 +- **原则**:将重复使用的值提取为环境变量 +- **位置**:在开始步骤或需要自定义的地方设置环境变量 +- **好的示例**: + ```sh + export CLUSTER_NAME=kubeocean-worker1 + export CLUSTERID=cls-worker1 + kubectl config use-context kind-$CLUSTER_NAME + ``` +- **避免**:硬编码重复值 + +### 2. 使用 Heredoc 创建配置文件 +- **原则**:优先使用 heredoc 直接创建 YAML/配置文件,而非引用外部文件或使用占位符 +- **优势**: + - 用户可以直接复制粘贴执行 + - 环境变量自动展开,无需手动编辑 + - 避免 `sed -i` 原地修改文件 +- **标准格式**: + ```sh + # 设置必要的环境变量 + export SUBNET_ID=subnet-abcdefgh + + # 使用 heredoc 创建配置文件 + cat > config.yaml <|subnet-xxx|" config.yaml + + # 不推荐:管道方式虽可行但不如 heredoc 直观 + cat examples/config.yaml | sed "s//$SUBNET_ID/g" | kubectl create -f - + ``` + +### 3. 简化多集群操作 +- **模式**:使用环境变量 + 通用步骤 +- **说明**:在步骤开始提供环境变量设置,告知用户如何复用步骤 +- **示例**: + ```markdown + 0. 设置环境变量 + + ```sh + export CLUSTER_NAME=kubeocean-worker1 + export CLUSTERID=cls-worker1 + # 将 CLUSTER_NAME 设为 kubeocean-worker2,CLUSTERID 设为 cls-worker2, + # 再重新执行即可完成第二个业务集群注册 + ``` + ``` + +### 4. 命令与结果分离 +- **原则**:执行命令和查看结果分开展示 +- **格式**: + ```markdown + 部署完成后,可使用 `kubectl` 查看结果 + + ```sh + kubectl get pods -owide -w + ``` + + 可以观察到以下输出: + + ```sh + NAME READY STATUS RESTARTS AGE + test-pod 1/1 Running 0 10s + ``` + ``` +- **避免**:将命令、说明和输出混在一起 + +## 示例和注释规范 + +### 1. 占位符格式 +- **环境变量替换**:使用 `$VARIABLE_NAME` 或 `${VARIABLE_NAME}` +- **示例值**:提供具体示例,如 `subnet-abcdefgh`、`cls-worker1` +- **注释说明**:必要时添加注释说明如何修改 + +## 文档类型特定规则 + +### 快速开始文档(Quick Start) +1. 必须包含 YAML front matter +2. 步骤编号从 0 开始(当需要预先设置环境变量时) +3. 提供完整的端到端流程 +4. 每个步骤都要验证结果 +5. 使用环境变量支持多集群场景 +6. 命令可直接复制粘贴执行 + +### 安装文档(Installation) +1. 必须包含 YAML front matter +2. 按照依赖关系组织步骤 +3. 提供多种安装方式(图形界面 + CLI) +4. 使用 heredoc 创建所有配置文件 +5. 命令可直接复制粘贴执行 + +## 多语言文档同步规则 + +### 1. 内容一致性 +- 中英文档内容必须保持同步 +- 修改一个语言版本后立即同步到另一个版本 +- 仅翻译文字说明,命令和配置保持一致 + +### 2. 命名约定 +- 中文文档:`*_zh.md` +- 英文文档:`*.md` +- 同一主题的文档放在相同目录 + +### 3. 引用路径 +- 图片、文件引用使用相对路径 +- 跨语言文档互相引用时使用正确的文件名后缀 + +## 检查清单 + +在提交文档前,确保: + +- [ ] 所有代码块都有语言标记 +- [ ] 代码块前后有空行 +- [ ] 可执行教程包含 front matter +- [ ] 重复值已提取为环境变量 +- [ ] YAML 配置使用 heredoc 创建 +- [ ] 命令可以直接复制粘贴执行 +- [ ] 提供了预期输出示例 +- [ ] 中英文文档已同步 +- [ ] 环境变量命名清晰(大写+下划线) +- [ ] 包含必要的验证步骤 + +## 反例总结 + +### 不推荐的做法 + +1. **代码块无语言标记** + ```markdown + # 错误 + ``` + kubectl get pods + ``` + + # 正确 + ```sh + kubectl get pods + ``` + ``` + +2. **硬编码重复值** + ```sh + # 错误 + kubectl config use-context kind-kubeocean-worker1 + kubectl get pods -n kubeocean-worker1 + + # 正确 + export CLUSTER_NAME=kubeocean-worker1 + kubectl config use-context kind-$CLUSTER_NAME + kubectl get pods -n $CLUSTER_NAME + ``` + +3. **需要手动编辑文件** + ```sh + # 错误:用户需要手动编辑 config.yaml + kubectl apply -f config.yaml # 文件中有 + + # 正确:直接生成可用配置 + export VALUE=real-value + cat > config.yaml <