Feature request: support loading multiple workflow YAMLs per Zigflow worker

[ShlomitSibony]

Hi Zigflow folks 👋

We’re rolling Zigflow out for a large set of partner workflows. Today, the recommended approach seems to be one Zigflow worker process per workflow YAML (e.g. zigflow run -f workflow.yaml).

That works well for isolation, but at our scale it becomes operationally heavy: we expect hundreds of workflow YAMLs over time, which translates into hundreds of Kubernetes deployments (monitoring, scaling, resource tuning, gitops changes, etc.). Many of these workflows are low-volume and don’t need independent scaling.

What we’d like

An option for a single Zigflow worker to load multiple workflows from a directory or list, for example:

• zigflow run --workflow-dir /workflows/
• zigflow run --workflow-glob "/workflows/*.yaml"
• or zigflow run -f a.yaml -f b.yaml -f c.yaml

Expected behavior

• Zigflow loads all workflow definitions at startup and registers them for the configured Temporal namespace/task queue(s).
• If there are conflicts (same document.name / same workflow type), fail fast with a clear error.
• Ideally clarify handling of task queues:
  • If YAMLs use different document.namespace / taskQueue: can one Zigflow process register multiple task queues, or should multi-YAML mode require a shared task queue?

Why this helps

• Reduces deployment count dramatically (especially for low-scale workflows)
• Simplifies GitOps + operational overhead
• Still compatible with “one worker per workflow” for high-volume workflows that need isolation

Open questions

1. Is there a design constraint in Zigflow/Temporal that makes multi-YAML workers undesirable or tricky?
2. Is “multiple workflows in a single YAML file” intended to be the official approach?
3. Would maintainers be open to this as an optional mode, keeping the current 1 yaml per worker as the default?

[mrsimonemms]

Thanks @ShlomitSibony. This is a great discussion point and something I'd not considered. When I created Zigflow, I was definitely in the "1 YAML, 1 runner" mindset. The more I think about this, the more I think this is a good idea.

I don't think there are any technical limitations to avoiding multi-YAML workers. After all, Zigflow does handle multiple workflows already.

There were two reasons why I wanted "1 YAML, 1 runner":

1. avoid having workers requiring too many resources
2. avoid workflows requiring secrets and those being shared to workflows that didn't require it (eg, a DB password set as an envvar).

Now I've thought about it, I think those are both a little spurious - workers can be replicated and, if you want to isolate secrets, you can still run it in a separate process/container.

The next is a question of timing - I'm mainly focused on the Studio at the moment, so I don't know when I'll be able to look at this. I'm happy to receive PRs on this, but I'd like to have a design session on this before we do any work to think about the implementation.

[mrsimonemms]

One other observation - the namespace is a Temporal architectural boundary. Multiple YAML files must maintain that boundary - ie, you cannot run multiple YAML files across different namespaces.

On the subject of the document.namespace in the YAML, this is the Temporal task queue - there are no limitations about having multiple task queues (or workflow types in document.name)