From 4f55d49821597a94a1912945a3954666cdfff5fd Mon Sep 17 00:00:00 2001 From: Yancey1989 Date: Mon, 23 Mar 2020 20:09:04 +0800 Subject: [PATCH 1/3] add pipline design --- doc/design.md | 134 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) diff --git a/doc/design.md b/doc/design.md index e7191a5..701ba7b 100644 --- a/doc/design.md +++ b/doc/design.md @@ -186,3 +186,137 @@ We hope Fluid users could represent it by the following Python/Fluid code. ```python def build_docker_image_from_git_source( ``` + +### Pipeline + +A Pipeline object is like function decleration. + +A Pipeline in Tekton defined an ordered series of Tasks. A valid Pipeline declearation +must include a reference to at last one `Task`, for example: + +``` yaml +tasks: + - name: build-the-image + taskRef: + name: build-push +``` + +The `Pipeline Tasks` in a Pipeline can be connected and run as a Directed Acyclic Graph(DAG), each of the Pipeline Tasks is a node, which can be connected with: + +- `runAfter` clauses on the `Pipeline Tasks`. +- `from` clauses on the `PipelineResources` needed by a `Task`. + +For an example `Pipeline` spec with `runAfter`: + +``` yaml +- name: lint-repo + taskRef: + name: pylint + resources: + inputs: + - name: workspace + resource: my-repo +- name: test-app + taskRef: + name: make-test + resources: + inputs: + - name: workspace + resource: my-repo +- name: build-app + taskRef: + name: kaniko-build-app + runAfter: + - test-app + resources: + inputs: + - name: workspace + resource: my-repo + outputs: + - name: image + resource: my-app-image +- name: build-frontend + taskRef: + name: kaniko-build-frontend + runAfter: + - test-app + resources: + inputs: + - name: workspace + resource: my-repo + outputs: + - name: image + resource: my-frontend-image +- name: deploy-all + taskRef: + name: deploy-kubectl + resources: + inputs: + - name: my-app-image + resource: my-app-image + from: + - build-app + - name: my-frontend-image + resource: my-frontend-image + from: + - build-frontend +``` + +This will result the following execution graph: + +``` text + | | + v v + test-app lint-repo + / \ + v v +build-app build-frontend + \ / + v v + deploy-all +``` + +In Python, a function is like a node of the DAG, which connected by the input +and of output of the functions. + +We hope Fluid users can write the following program to express a DAG with `fluid.pipeline`: + +``` python +@fluid.task +def pylint(): + fluid.step(...) + +@fluid.task +def make_test(): + fluid.step(...) + +@fluid.task +def kaniko_build_app(): + fluid.step(...) + +@fluid.task +def kaniko_build_frontend(): + fluid.step(...) + +@fluid.task +def deploy_kubectl(): + fluid.step(...) + +@fluid.pipeline +def dag_demo(): + lint_repo = pylint() + test_app = make_test() + build_app = kaniko_build_app().run_after(test_app) + build_frontend = kaniko_build_frontend().run_after(test_app) + deploy_all = deploy_kubectl() + deploy_all.inputs.my_frontend_image.from(build_app) + deploy_all.inputs.my_app_image.from(build_frontend) +``` + +### PipelineRun + +A PipelineRun object is like a function invocation: + +``` python +build_pipeline() +``` From e52bd0abf010df2b08dd4d461c536ed026d2774d Mon Sep 17 00:00:00 2001 From: Yancey1989 Date: Mon, 23 Mar 2020 22:50:00 +0800 Subject: [PATCH 2/3] add pipeline design --- doc/design.md | 208 ++++++++++++++++++++++---------------------------- 1 file changed, 93 insertions(+), 115 deletions(-) diff --git a/doc/design.md b/doc/design.md index 701ba7b..c94df80 100644 --- a/doc/design.md +++ b/doc/design.md @@ -101,7 +101,7 @@ spec: We hope Fluid users could represent it by the following line. ```python -skaffold_git = fluid.Git( +skaffold_git = fluid.git_resource( revision="master", url="https://github.com/GoogleContainerTools/skaffold) ``` @@ -125,7 +125,7 @@ spec: We hope Fluid users could represent the above YAML file by the following line. ```python -skaffold_image_leeroy_web = fluid.Image( +skaffold_image_leeroy_web = fluid.image_resource( url="gcr.io/wangkuiyi/leeroy-web") ``` @@ -142,7 +142,7 @@ According to the [document](https://github.com/tektoncd/pipeline/blob/master/doc The following example from the [Tekton tutorial](https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#task-inputs-and-outputs) takes an input resource, an output resource, and two input parameters. ```yaml -goapiVersion: tekton.dev/v1alpha1 +apiVersion: tekton.dev/v1alpha1 kind: Task metadata: name: build-docker-image-from-git-source @@ -185,138 +185,116 @@ We hope Fluid users could represent it by the following Python/Fluid code. ```python def build_docker_image_from_git_source( + docker_source: "input,git", + built_image: "output,image", + path_to_dockerfile="/workspace/docker-source/Dockerfile", + path_to_context="/workspace/docker-source"): + '''Define a Tekton Task that builds a Docker image from a Git repo''' + couler.step(image="gcr.io/kaniko-project/executor:v0.14.0", + cmd=["/kaniko/executor"], + args=[f"--dockerfile={path_to_dockerfile}", + f"--destination={built_image.url}", + f"--context={path_to_context}"], + env={"DOCKER_CONFIG": "/tekton/home/.docker/"}) ``` ### Pipeline -A Pipeline object is like function decleration. - -A Pipeline in Tekton defined an ordered series of Tasks. A valid Pipeline declearation -must include a reference to at last one `Task`, for example: - -``` yaml -tasks: - - name: build-the-image - taskRef: - name: build-push -``` - -The `Pipeline Tasks` in a Pipeline can be connected and run as a Directed Acyclic Graph(DAG), each of the Pipeline Tasks is a node, which can be connected with: +A Pipeline object is like function declaration. -- `runAfter` clauses on the `Pipeline Tasks`. -- `from` clauses on the `PipelineResources` needed by a `Task`. +A Pipeline in Tekton defined an ordered series of Tasks. Users can specify whether +the output of a `Task` is used as an input for the next `Task` using `from` property on `PipelineResources` -For an example `Pipeline` spec with `runAfter`: +As the following example comes from [Tekton's tutorial](https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#creating-and-running-a-pipeline) ``` yaml -- name: lint-repo - taskRef: - name: pylint - resources: - inputs: - - name: workspace - resource: my-repo -- name: test-app - taskRef: - name: make-test - resources: - inputs: - - name: workspace - resource: my-repo -- name: build-app - taskRef: - name: kaniko-build-app - runAfter: - - test-app - resources: - inputs: - - name: workspace - resource: my-repo - outputs: - - name: image - resource: my-app-image -- name: build-frontend - taskRef: - name: kaniko-build-frontend - runAfter: - - test-app - resources: - inputs: - - name: workspace - resource: my-repo - outputs: - - name: image - resource: my-frontend-image -- name: deploy-all - taskRef: - name: deploy-kubectl +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: tutorial-pipeline +spec: resources: - inputs: - - name: my-app-image - resource: my-app-image - from: - - build-app - - name: my-frontend-image - resource: my-frontend-image - from: - - build-frontend -``` - -This will result the following execution graph: - -``` text - | | - v v - test-app lint-repo - / \ - v v -build-app build-frontend - \ / - v v - deploy-all + - name: source-repo + type: git + - name: web-image + type: image + tasks: + - name: build-skaffold-web + taskRef: + name: build-docker-image-from-git-source + params: + - name: pathToDockerFile + value: Dockerfile + - name: pathToContext + value: /workspace/docker-source/examples/microservices/leeroy-web #configure: may change according to your source + resources: + inputs: + - name: docker-source + resource: source-repo + outputs: + - name: builtImage + resource: web-image + - name: deploy-web + taskRef: + name: deploy-using-kubectl + resources: + inputs: + - name: source + resource: source-repo + - name: image + resource: web-image + from: + - build-skaffold-web + params: + - name: path + value: /workspace/source/examples/microservices/leeroy-web/kubernetes/deployment.yaml #configure: may change according to your source + - name: yamlPathToImage + value: "spec.template.spec.containers[0].image" ``` -In Python, a function is like a node of the DAG, which connected by the input -and of output of the functions. - -We hope Fluid users can write the following program to express a DAG with `fluid.pipeline`: +We hope Fluid users can write the following program to express the above YAML file: ``` python -@fluid.task -def pylint(): - fluid.step(...) +@fluid.pipeline +def tutorial(source_repo:"resource,git", web_image="resource,image"): + build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image) -@fluid.task -def make_test(): - fluid.step(...) + deploy_web = deploy_using_kubectl(source_repo, web_image) + deploy_web.web_image.from(build_skaffold_web) +``` -@fluid.task -def kaniko_build_app(): - fluid.step(...) +### PipelineRun -@fluid.task -def kaniko_build_frontend(): - fluid.step(...) +A PipelineRun object is like a function invocation. -@fluid.task -def deploy_kubectl(): - fluid.step(...) +A PipelineRun object defines a call to a Pipeline. The following is a PipelineRun example from [Tekton's tutorial](https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#creating-and-running-a-pipeline): -@fluid.pipeline -def dag_demo(): - lint_repo = pylint() - test_app = make_test() - build_app = kaniko_build_app().run_after(test_app) - build_frontend = kaniko_build_frontend().run_after(test_app) - deploy_all = deploy_kubectl() - deploy_all.inputs.my_frontend_image.from(build_app) - deploy_all.inputs.my_app_image.from(build_frontend) +``` yaml +apiVersion: tekton.dev/v1beta1 +kind: PipelineRun +metadata: + name: tutorial-pipeline-run-1 +spec: + serviceAccountName: tutorial-service + pipelineRef: + name: tutorial-pipeline + resources: + - name: source-repo + resourceRef: + name: skaffold-git + - name: web-image + resourceRef: + name: skaffold-image-leeroy-web ``` -### PipelineRun - -A PipelineRun object is like a function invocation: +We hope Fluid users write the following program: ``` python -build_pipeline() +skaffold_git = fluid.git_resource( + revision="master", + url="https://github.com/GoogleContainerTools/skaffold") +skaffold_image_leeroy_web = fluid.image_resource( + url="gcr.io/wangkuiyi/leeroy-web") + +tutorial(skaffold_git, skaffold_image_leeroy_web) ``` From bcd2f8e1a0281528d641164daf5a45be9e66993d Mon Sep 17 00:00:00 2001 From: Yancey1989 Date: Fri, 27 Mar 2020 19:58:45 +0800 Subject: [PATCH 3/3] update --- doc/design.md | 162 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 159 insertions(+), 3 deletions(-) diff --git a/doc/design.md b/doc/design.md index c94df80..3eda4ea 100644 --- a/doc/design.md +++ b/doc/design.md @@ -184,6 +184,7 @@ spec: We hope Fluid users could represent it by the following Python/Fluid code. ```python +@fluid.task def build_docker_image_from_git_source( docker_source: "input,git", built_image: "output,image", @@ -200,9 +201,9 @@ def build_docker_image_from_git_source( ### Pipeline -A Pipeline object is like function declaration. +A `Pipeline` object is like function declaration, according to the [definition](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md). -A Pipeline in Tekton defined an ordered series of Tasks. Users can specify whether +A `Pipeline` in Tekton defines an ordered series of Tasks. Users can specify whether the output of a `Task` is used as an input for the next `Task` using `from` property on `PipelineResources` As the following example comes from [Tekton's tutorial](https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#creating-and-running-a-pipeline) @@ -252,17 +253,172 @@ spec: value: "spec.template.spec.containers[0].image" ``` +The above `Pipeline` is referencing a `Task` called `deploy-using-kubectl` defined as follows: + +``` yaml +apiVersion: tekton.dev/v1beta1 +kind: Task +metadata: + name: deploy-using-kubectl +spec: + params: + - name: path + type: string + description: Path to the manifest to apply + - name: yamlPathToImage + type: string + description: | + The path to the image to replace in the yaml manifest (arg to yq) + resources: + inputs: + - name: source + type: git + - name: image + type: image + steps: + - name: replace-image + image: mikefarah/yq + command: ["yq"] + args: + - "w" + - "-i" + - "$(params.path)" + - "$(params.yamlPathToImage)" + - "$(resources.inputs.image.url)" + - name: run-kubectl + image: lachlanevenson/k8s-kubectl + command: ["kubectl"] + args: + - "apply" + - "-f" + - "$(params.path)" +``` + We hope Fluid users can write the following program to express the above YAML file: ``` python +@fluid.task +def deploy_using_kubectl( + source_repo: "input, git", + web_image: "input,image", + path="/workspace/source/examples/microservices/leeroy-web/kubernetes/deployment.yaml", + yaml_path_to_image="spec.template.spec.containers[0].image"): + fluid.step(image="mikefarah/yq", + command=["yq"], + args=["w", + "-i", + f"{path}", + f"{yaml_path_to_image}", + f"{image.url}"]) + fluid.step(image="lachlanevenson/k8s-kubectl", + command=["kubectl"], + args=["apply", "-f", f"{path}"]) + @fluid.pipeline -def tutorial(source_repo:"resource,git", web_image="resource,image"): +def tutorial(source_repo: "resource,git", web_image: "resource,image"): build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image) deploy_web = deploy_using_kubectl(source_repo, web_image) deploy_web.web_image.from(build_skaffold_web) ``` +### Pipeline with DAG + +The `Pipeline Tasks` in a `Pipeline` can be connected as a Directed Acyclic Graph (DAG). Each of the Tasks is a node, which can be +connected with an edge by: + +- `from`: clauses on the PipelineResources needed by a Task. +- `runAfter`: clauses on the Pipeline Tasks. + +As the following example of `Pipeline spec` comes from Tekton Pipeline tutorials: + +``` yaml +- name: lint-repo + taskRef: + name: pylint + resources: + inputs: + - name: workspace + resource: my-repo +- name: test-app + taskRef: + name: make-test + resources: + inputs: + - name: workspace + resource: my-repo +- name: build-app + taskRef: + name: kaniko-build-app + runAfter: + - test-app + resources: + inputs: + - name: workspace + resource: my-repo + outputs: + - name: image + resource: my-app-image +- name: build-frontend + taskRef: + name: kaniko-build-frontend + runAfter: + - test-app + resources: + inputs: + - name: workspace + resource: my-repo + outputs: + - name: image + resource: my-frontend-image +- name: deploy-all + taskRef: + name: deploy-kubectl + resources: + inputs: + - name: my-app-image + resource: my-app-image + from: + - build-app + - name: my-frontend-image + resource: my-frontend-image + from: + - build-frontend +``` + +This will result in the following execution graph: + +``` text + | | + v v + test-app lint-repo + / \ + v v +build-app build-frontend + \ / + v v + deploy-all +``` + +We hope users can write the following Fluid program to construct the above DAG: + +``` python +@fluid.pipeline +def dag(my_repo: "resource,git", my_app_image: "resource,image", my_frontend_image: "resource,image"): + lint_repo = pylint() + test_app = make_test() + + build_app = kaniko-build-app(my_repo, my_app_image) + build_app.run_after(test_app) + + build_frontend = kaniko-build-frontend(my_repo, my_frontend_image) + build_frontend.run_after(test_app) + + deploy_all = deploy_kubectl(my_app_image, my_frontend_image) + deploy_all.inputs.my_app_image.from(build_app) + deploy_all.inputs.my_frontend_image.from(build_frontend) +``` + ### PipelineRun A PipelineRun object is like a function invocation.