Skip to content

Commit e52bd0a

Browse files
committed
add pipeline design
1 parent 4f55d49 commit e52bd0a

File tree

1 file changed

+93
-115
lines changed

1 file changed

+93
-115
lines changed

doc/design.md

+93-115
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ spec:
101101
We hope Fluid users could represent it by the following line.
102102

103103
```python
104-
skaffold_git = fluid.Git(
104+
skaffold_git = fluid.git_resource(
105105
revision="master",
106106
url="https://github.com/GoogleContainerTools/skaffold)
107107
```
@@ -125,7 +125,7 @@ spec:
125125
We hope Fluid users could represent the above YAML file by the following line.
126126

127127
```python
128-
skaffold_image_leeroy_web = fluid.Image(
128+
skaffold_image_leeroy_web = fluid.image_resource(
129129
url="gcr.io/wangkuiyi/leeroy-web")
130130
```
131131

@@ -142,7 +142,7 @@ According to the [document](https://github.com/tektoncd/pipeline/blob/master/doc
142142
The following example from the [Tekton tutorial](https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#task-inputs-and-outputs) takes an input resource, an output resource, and two input parameters.
143143

144144
```yaml
145-
goapiVersion: tekton.dev/v1alpha1
145+
apiVersion: tekton.dev/v1alpha1
146146
kind: Task
147147
metadata:
148148
name: build-docker-image-from-git-source
@@ -185,138 +185,116 @@ We hope Fluid users could represent it by the following Python/Fluid code.
185185

186186
```python
187187
def build_docker_image_from_git_source(
188+
docker_source: "input,git",
189+
built_image: "output,image",
190+
path_to_dockerfile="/workspace/docker-source/Dockerfile",
191+
path_to_context="/workspace/docker-source"):
192+
'''Define a Tekton Task that builds a Docker image from a Git repo'''
193+
couler.step(image="gcr.io/kaniko-project/executor:v0.14.0",
194+
cmd=["/kaniko/executor"],
195+
args=[f"--dockerfile={path_to_dockerfile}",
196+
f"--destination={built_image.url}",
197+
f"--context={path_to_context}"],
198+
env={"DOCKER_CONFIG": "/tekton/home/.docker/"})
188199
```
189200

190201
### Pipeline
191202

192-
A Pipeline object is like function decleration.
193-
194-
A Pipeline in Tekton defined an ordered series of Tasks. A valid Pipeline declearation
195-
must include a reference to at last one `Task`, for example:
196-
197-
``` yaml
198-
tasks:
199-
- name: build-the-image
200-
taskRef:
201-
name: build-push
202-
```
203-
204-
The `Pipeline Tasks` in a Pipeline can be connected and run as a Directed Acyclic Graph(DAG), each of the Pipeline Tasks is a node, which can be connected with:
203+
A Pipeline object is like function declaration.
205204

206-
- `runAfter` clauses on the `Pipeline Tasks`.
207-
- `from` clauses on the `PipelineResources` needed by a `Task`.
205+
A Pipeline in Tekton defined an ordered series of Tasks. Users can specify whether
206+
the output of a `Task` is used as an input for the next `Task` using `from` property on `PipelineResources`
208207

209-
For an example `Pipeline` spec with `runAfter`:
208+
As the following example comes from [Tekton's tutorial](https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#creating-and-running-a-pipeline)
210209

211210
``` yaml
212-
- name: lint-repo
213-
taskRef:
214-
name: pylint
215-
resources:
216-
inputs:
217-
- name: workspace
218-
resource: my-repo
219-
- name: test-app
220-
taskRef:
221-
name: make-test
222-
resources:
223-
inputs:
224-
- name: workspace
225-
resource: my-repo
226-
- name: build-app
227-
taskRef:
228-
name: kaniko-build-app
229-
runAfter:
230-
- test-app
231-
resources:
232-
inputs:
233-
- name: workspace
234-
resource: my-repo
235-
outputs:
236-
- name: image
237-
resource: my-app-image
238-
- name: build-frontend
239-
taskRef:
240-
name: kaniko-build-frontend
241-
runAfter:
242-
- test-app
243-
resources:
244-
inputs:
245-
- name: workspace
246-
resource: my-repo
247-
outputs:
248-
- name: image
249-
resource: my-frontend-image
250-
- name: deploy-all
251-
taskRef:
252-
name: deploy-kubectl
211+
apiVersion: tekton.dev/v1beta1
212+
kind: Pipeline
213+
metadata:
214+
name: tutorial-pipeline
215+
spec:
253216
resources:
254-
inputs:
255-
- name: my-app-image
256-
resource: my-app-image
257-
from:
258-
- build-app
259-
- name: my-frontend-image
260-
resource: my-frontend-image
261-
from:
262-
- build-frontend
263-
```
264-
265-
This will result the following execution graph:
266-
267-
``` text
268-
| |
269-
v v
270-
test-app lint-repo
271-
/ \
272-
v v
273-
build-app build-frontend
274-
\ /
275-
v v
276-
deploy-all
217+
- name: source-repo
218+
type: git
219+
- name: web-image
220+
type: image
221+
tasks:
222+
- name: build-skaffold-web
223+
taskRef:
224+
name: build-docker-image-from-git-source
225+
params:
226+
- name: pathToDockerFile
227+
value: Dockerfile
228+
- name: pathToContext
229+
value: /workspace/docker-source/examples/microservices/leeroy-web #configure: may change according to your source
230+
resources:
231+
inputs:
232+
- name: docker-source
233+
resource: source-repo
234+
outputs:
235+
- name: builtImage
236+
resource: web-image
237+
- name: deploy-web
238+
taskRef:
239+
name: deploy-using-kubectl
240+
resources:
241+
inputs:
242+
- name: source
243+
resource: source-repo
244+
- name: image
245+
resource: web-image
246+
from:
247+
- build-skaffold-web
248+
params:
249+
- name: path
250+
value: /workspace/source/examples/microservices/leeroy-web/kubernetes/deployment.yaml #configure: may change according to your source
251+
- name: yamlPathToImage
252+
value: "spec.template.spec.containers[0].image"
277253
```
278254

279-
In Python, a function is like a node of the DAG, which connected by the input
280-
and of output of the functions.
281-
282-
We hope Fluid users can write the following program to express a DAG with `fluid.pipeline`:
255+
We hope Fluid users can write the following program to express the above YAML file:
283256

284257
``` python
285-
@fluid.task
286-
def pylint():
287-
fluid.step(...)
258+
@fluid.pipeline
259+
def tutorial(source_repo:"resource,git", web_image="resource,image"):
260+
build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image)
288261
289-
@fluid.task
290-
def make_test():
291-
fluid.step(...)
262+
deploy_web = deploy_using_kubectl(source_repo, web_image)
263+
deploy_web.web_image.from(build_skaffold_web)
264+
```
292265

293-
@fluid.task
294-
def kaniko_build_app():
295-
fluid.step(...)
266+
### PipelineRun
296267

297-
@fluid.task
298-
def kaniko_build_frontend():
299-
fluid.step(...)
268+
A PipelineRun object is like a function invocation.
300269

301-
@fluid.task
302-
def deploy_kubectl():
303-
fluid.step(...)
270+
A PipelineRun object defines a call to a Pipeline. The following is a PipelineRun example from [Tekton's tutorial](https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#creating-and-running-a-pipeline):
304271

305-
@fluid.pipeline
306-
def dag_demo():
307-
lint_repo = pylint()
308-
test_app = make_test()
309-
build_app = kaniko_build_app().run_after(test_app)
310-
build_frontend = kaniko_build_frontend().run_after(test_app)
311-
deploy_all = deploy_kubectl()
312-
deploy_all.inputs.my_frontend_image.from(build_app)
313-
deploy_all.inputs.my_app_image.from(build_frontend)
272+
``` yaml
273+
apiVersion: tekton.dev/v1beta1
274+
kind: PipelineRun
275+
metadata:
276+
name: tutorial-pipeline-run-1
277+
spec:
278+
serviceAccountName: tutorial-service
279+
pipelineRef:
280+
name: tutorial-pipeline
281+
resources:
282+
- name: source-repo
283+
resourceRef:
284+
name: skaffold-git
285+
- name: web-image
286+
resourceRef:
287+
name: skaffold-image-leeroy-web
314288
```
315289

316-
### PipelineRun
317-
318-
A PipelineRun object is like a function invocation:
290+
We hope Fluid users write the following program:
319291

320292
``` python
321-
build_pipeline()
293+
skaffold_git = fluid.git_resource(
294+
revision="master",
295+
url="https://github.com/GoogleContainerTools/skaffold")
296+
skaffold_image_leeroy_web = fluid.image_resource(
297+
url="gcr.io/wangkuiyi/leeroy-web")
298+
299+
tutorial(skaffold_git, skaffold_image_leeroy_web)
322300
```

0 commit comments

Comments
 (0)