-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathprefect.yaml
More file actions
370 lines (365 loc) · 19.2 KB
/
prefect.yaml
File metadata and controls
370 lines (365 loc) · 19.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.
# Generic metadata about this project
name: qa-wolf-flows
# build section allows you to manage and build docker images
build:
# push section allows you to manage if and how this project is uploaded to remote locations
push:
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
repository: https://github.com/PrefectHQ/qa-wolf-flows.git
branch: main
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: oneTask
version:
tags: []
description:
"### Basic hello world flow\n```python\nfrom prefect import flow, task\n
\n@task(log_prints=True)\ndef long_long_long_long_long_name(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@flow(log_prints=True)\ndef
basic_flow(name:str=\"world\"):\n long_long_long_long_long_name()\n print(\"\
hello\", name)\n return 'hello'\n```\n "
entrypoint: simpleFlowWithTasks.py:basic_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: artifacts
version:
tags: []
description:
"Flow that creates markdown, link and table artifacts\n \n## Flow
code\n```python\nfrom prefect import flow, task\nfrom prefect.artifacts import
create_markdown_artifact, create_link_artifact, create_table_artifact\n\n@task\n
def my_first_link_task():\n create_link_artifact(\n key=\"variable-data-link\"\
,\n link=\"https://nyc3.digitaloceanspaces.com/my-bucket-name/highly_variable_data_.csv\"\
,\n description=\"## Highly variable data\",\n link_text=\"\
Highly variable data\",\n )\n\n@task\ndef my_second_link_task():\n \
\ create_link_artifact(\n key=\"variable-data-link\",\n \
\ link=\"https://nyc3.digitaloceanspaces.com/my-bucket-name/low_pred_data_.csv\"\
,\n description=\"# Low prediction accuracy data\",\n )\n@task\n
def my_table_task():\n highest_churn_possibility = [\n {'customer_id':'12345',
'name': 'John Smith', 'churn_probability': 0.85 }, \n {'customer_id':'56789',
'name': 'Jane Jones', 'churn_probability': 0.65 } \n ]\n\n create_table_artifact(\n\
\ key=\"personalized-reachout\",\n table=highest_churn_possibility,\n\
\ description= \"# Marvin, please reach out to these customers today!\"\
\n )\n \n@task\ndef markdown_task():\n na_revenue = 500000\n markdown_report
= f\"\"\"# Sales Report\n\n## Summary\n\nIn the past quarter, our company saw
a significant increase in sales, with a total revenue of $1,000,000. This represents
a 20% increase over the same period last year.\n\n## Sales by Region\n\n| Region\
\ | Revenue |\n|:--------------|-------:|\n| North America | ${na_revenue:,}
|\n| Europe | $250,000 |\n| Asia | $150,000 |\n| South America
| $75,000 |\n| Africa | $25,000 |\n\n## Top Products\n\n1. Product A -
$300,000 in revenue\n2. Product B - $200,000 in revenue\n3. Product C - $150,000
in revenue\n\n## Conclusion\n\nOverall, these results are very encouraging and
demonstrate the success of our sales team in increasing revenue across all regions.
However, we still have room for improvement and should focus on further increasing
sales in the coming quarter.\n\"\"\"\n create_markdown_artifact(\n key=\"\
gtm-markdown-report\",\n markdown=markdown_report,\n description=\"\
Quarterly Sales Report\",\n )\n\n@flow()\ndef artifacts_flow():\n markdown_task()\n\
\ my_first_link_task()\n my_second_link_task()\n my_table_task()\n \
\ \n```\n "
entrypoint: artifacts.py:artifacts_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: parameter-input
version:
tags: []
description:
"Flow to test display of Pydantic fields with default values\n \n
#Flow code\n\n```python\nimport datetime\nfrom typing import Dict, List, Optional,
Set, Tuple\n\nfrom prefect import flow, get_run_logger\nfrom pydantic import UUID4,
BaseModel, Field, Json\n\n\nclass PydanticFieldsDefault(BaseModel): \n\n bool_field_default:
Optional[bool] = Field(\n title=\"Title bool_field_default\", description=\"\
Description bool_field_default\", default=True\n )\n int_field_default:
Optional[int] = Field(\n title=\"Title int_field_default\", description=\"\
Description int_field_default\", default=42\n )\n float_field_default: Optional[float]
= Field(\n title=\"Title float_field_default\", description=\"Description
float_field_default\", default=1.23\n )\n str_field_default: Optional[str]
= Field(\n title=\"Title str_field_default\", description=\"Description
str_field_default\", default=\"default\"\n )\n list_field_default: Optional[List[str]]
= Field(\n title=\"Title list_field_default\",\n description=\"
Description list_field_default\",\n default=[\"default\", \"default\",
\"default\"],\n )\n #tuple inputs currently not supported - can be done
via json\n # tuple_field_default: Optional[Tuple[str, str]] = Field(\n #\
\ title=\"Title tuple_field_default\",\n # description=\"Description
tuple_field_default\",\n # default=(\"default\", \"default\"),\n # )\n\
\ dict_field_default: Optional[Dict[str, str]] = Field(\n title=\"Title
dict_field_default\", description=\"Description dict_field_default\", default={\"\
name\": \"default\"}\n )\n set_field_default: Optional[Set[str]] = Field(\n\
\ title=\"Title set_field_default\",\n description=\"Description
set_field_default\",\n default=(\"default_0\", \"default_1\"),\n )\n\
\ date_field_default: Optional[datetime.date] = Field(\n title=\"Title
date_field_default\",\n description=\"Description date_field_default\"\
,\n default=datetime.date(2022, 1, 1),\n )\n time_field_default:
Optional[datetime.time] = Field(\n title=\"Title time_field_default\",
description=\"Description time_field_default\", default=datetime.time(12, 12)\n\
\ )\n datetime_field_default: Optional[datetime.datetime] = Field(\n \
\ title=\"Title datetime_field_default\",\n description=\"Description
datetime_field_default\",\n default=datetime.datetime(2022, 1, 1, 12, 12),\n\
\ )\n #UUID objects are immutable\n # uuid_field_default: Optional[UUID4]
= Field(\n # title=\"Title uuid_field_default\",\n # description=\"\
Description uuid_field_default\",\n # default=UUID4(\"aa763817-0ba2-4771-bfc7-1550d1646874\"\
),\n # )\n json_field_default: Optional[Json] = Field(\n title=\"\
Title json_field_default\",\n description=\"Description json_field_default\"\
,\n default='{\"name\":\"John\", \"age\":30, \"car\":null}',\n )\n \
\ enum_field_default: Optional[FruitEnum] = Field(\n title=\"Title enum_field_default\"\
,\n description=\"Description enum_field_default\",\n default=FruitEnum.pear,\n\
\ )\n\n\n@flow(name=\"Pydantic Fields Default\")\ndef mainPydantic(\n pydantic_fields_default_defaults:
PydanticFieldsDefault = PydanticFieldsDefault(),\n) -> None:\n logger = get_run_logger()\n\
\ logger.info(f\"{pydantic_fields_default_defaults=}\")\n\n```\n "
entrypoint: parameterInputs.py:parameterDisplays
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-root
version:
tags:
- automation-test
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$1
version:
tags:
- automation-test
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$2
version:
tags:
- automation-test
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$3
version:
tags:
- automation-test
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$4
version:
tags:
- automation-test
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$5
version:
tags: [automation-test]
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$6
version:
tags: [automation-test]
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$7
version:
tags: [automation-test]
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$8
version:
tags: [automation-test]
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$9
version:
tags: [automation-test]
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: automation-test-$10
version:
tags: [automation-test]
description:
"### Sleep task flow\n```python\nfrom prefect import flow, task\nfrom
prefect import time\n\n@task(log_prints=True)\ndef name_task(name):\n print(\"\
hello again\", name)\n return \"hello again\"\n\n@task(log_prints=True)\ndef
sleep_task(sleepLength:int=10):\n time.sleep(sleepLength)\n print(\"slept\"\
)\n return \"slept\"\n\n@flow(log_prints=True)\ndef sleep_task_flow(name:str=\"\
world\"sleepLength:int=10):\n name_task()\n print(\"hello\", name)\n \
\ return 'hello'\n```\n "
entrypoint: sleepTaskFlow.py:sleep_task_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: subFlows
version:
tags: []
description:
entrypoint: parentChildFlow.py:main_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: interactive
version:
tags: []
description:
"Interactive flow\n```python \nclass UserName(RunInput):\n name:
str\n\n@flow(log_prints=True)\ndef interactive_flow():\n user_input = pause_flow_run(\n\
\ wait_for_input=UserName\n )\n print(f\"Hello, {user_input.name}!\"\
)```"
entrypoint: interactive.py:interactive_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}
- name: timber-logger
version:
tags: []
description:
"Flow that demonstrates comprehensive logging capabilities and custom
event emission\n\n## Purpose\nTests all Prefect logging levels (DEBUG, INFO,
WARNING, ERROR, CRITICAL) and custom event emission with a dystopian forestry
theme.\n\n## Features\n- Task per logging level\n- Custom Prefect events from
each task\n- ANSI color codes in select logs\n- Subflow with same configuration\n\n```python\n
from prefect import flow, task, get_run_logger\nfrom prefect.events import emit_event\n\n@task(log_prints=True)\n
def debug_task():\n logger = get_run_logger()\n logger.debug(\"Chainsaw
blade sharpness: optimal\")\n emit_event(\n event=\"equipment.chainsaw.initialized\"\
,\n resource={\"prefect.resource.id\": \"logging-system-debug\"}\n )\n\n@flow(log_prints=True)\n
def logger_flow():\n logger = get_run_logger()\n logger.debug(\"System
initialization...\")\n logger.info(\"Morning report generated\")\n logger.warning(\"\
Warning: Spotted owl nest detected\")\n logger.error(\"Error: Tree protestor
encountered\")\n logger.critical(\"CRITICAL: Last old growth stand identified\"\
)\n \n debug_task()\n info_task()\n warning_task()\n error_task()\n\
\ critical_task()\n subflow_logger_flow()\n```"
entrypoint: logger.py:logger_flow
parameters: {}
work_pool:
name: "kubernetes-prd-internal-tools"
work_queue_name:
job_variables: {}