Research if pods such as notifier can be a long-lived container that is also part of the workflow

[olaals]

Describe the improvement you would like to see
Currently we spin up a series of containers on-demand in workflows.
This makes sense for some heavier workloads, but for lightweight pods such as cloe / notifier it might be a better trade-off to have long-lived pod(s) that multiple workloads can share, resulting in faster workflow execution times.

For example the workflow notifier processing work time is very short, resulting in that just spinning up the image dominates the the total time it takes for the workflow step. Since the processing time is very low, it is suitable to be shared among several workflows since it wont be overloaded as easily.

Global agent pod instead of one per workflow issue: argoproj/argo-workflows#7891
Discussion about DEFAULT_REQUEUE_TIME which is relevant for quick workflow steps: argoproj/argo-workflows#12381 (comment)

Questions:

• Can global agents be a deployment with multiple pods?
• Can we create some common functionality in such pods that covers multiple use-cases that only does api calls (such as CLOE)? This would make Argo a better fit for both the normal job processing pattern and cases where the processing is done be

How will this change existing functionality?
Make certain pods shared long-lived pods across workflows instead of spinning up per workflow

How will this improvement affect the current Threat Model?
NA