Endpoint to expose registered engines and their configurations

[JaeAeich]

Summary

Currently, one can discover all engines configured in a WES server via the /service-info endpoint. However, this information is not sufficient for real-world production use.
To safely and dynamically use WES in a production setting, additional visibility into engine configuration and defaults is required.

Problem

In the current WES specification, while /service-info provides generic details, it does not expose engine-level configuration or metadata.

For example:

• Different WES deployments may register multiple workflow engines (e.g., Cromwell, Nextflow, Snakemake, Toil).
• Each engine may support custom configurations like maximum CPU, memory, or runtime limits.
• Currently, these configurations are opaque to the WES user or UI.

This lack of visibility makes it difficult to:

• Build UIs or services that adapt dynamically to backend capabilities.
• Enforce safe guardrails on user-submitted resource requests.
• Communicate which engine is the default and what parameters are allowed or overridden.

Proposed Solution

Introduce a new endpoint, for example:

GET /ga4gh/wes/v1/engines

and

GET /ga4gh/wes/v1/engines/{id}

This endpoint should return all registered engines in the WES implementation, along with key metadata such as:

• Engine name and version
• Whether it’s the default engine
• Supported workflow languages
• Default or maximum CPU/memory configurations
• Any additional parameters exposed by the engine

Example Response

{
  "engines": [
    {
      "name": "cromwell",
      "version": "85",
      "default": true,
      "languages": ["WDL", "CWL"],
      "config": {
        "max_cpu": {
          "value": 16,
          "cli_flag": "--max-concurrent-jobs",
          "is_default": true,
          "user_overridable": false,
          "description": "Maximum number of concurrent jobs Cromwell can execute."
        },
        "max_memory_gb": {
          "value": 64,
          "cli_flag": "--max-memory",
          "is_default": true,
          "user_overridable": false,
          "description": "Maximum total memory allocation allowed for Cromwell jobs."
        },
        "default_cpu": {
          "value": 4,
          "cli_flag": "--default-cpu",
          "is_default": true,
          "user_overridable": true,
          "description": "Default CPU cores assigned per task if not specified in the workflow."
        },
        "default_memory_gb": {
          "value": 8,
          "cli_flag": "--default-memory",
          "is_default": true,
          "user_overridable": true,
          "description": "Default memory per task if not explicitly defined in workflow inputs."
        },
        "queue": {
          "value": "production",
          "is_default": true,
          "user_overridable": false,
          "description": "Execution queue used for submitted workflows."
        }
      }
    },
    {
      "name": "nextflow",
      "version": "24.04.1",
      "default": false,
      "languages": ["Nextflow"],
      "config": {
        "max_cpu": {
          "value": 8,
          "cli_flag": "-process.maxForks",
          "is_default": true,
          "user_overridable": false,
          "description": "Maximum parallel processes allowed in the executor."
        },
        "max_memory_gb": {
          "value": 32,
          "cli_flag": "-process.maxMemory",
          "is_default": true,
          "user_overridable": false,
          "description": "Upper limit for memory requested per process."
        },
        "default_cpu": {
          "value": 2,
          "cli_flag": "-process.cpus",
          "is_default": true,
          "user_overridable": true,
          "description": "Default number of CPU cores assigned per process."
        },
        "runtime": {
          "value": {
            "container_engine": "docker",
            "default_image": "nextflow/nextflow:24.04.1"
          },
          "is_default": true,
          "user_overridable": false,
          "description": "Runtime environment for workflow execution."
        }
      }
    }
  ]
}

**The above is just an example

Benefits

• Enables frontends and clients to display engine information dynamically.
• Helps enforce and communicate backend resource limits to users.
• Improves transparency for multi-engine WES deployments.
• Makes it easier to integrate WES with custom orchestration or authentication systems that set user-specific quotas.

[patmagee]

Do you envision any of these values as things the user can manipulate? If so would they be passed as engine_parameters and could they be capture in the default_engine_params block in the service-info endpoint?

Also, there is already the workflow_engine_versions block which was added for a similar use case (IE users discovering the engines available). You mention that this is not enough for production use, I would be interested in understanding what is lacking

I reread your issue and see you explained it.

It feels like the expansion of the engine metadata embraces WES as a more leaky abstraction by allowing more of the engine specifics to show through. I think that is understandable (i have made the argument that WES itself relies on the engine directly), but I really am not sure how generalizable your ask is. I would not be surprised if every implementation of WES had a different set of keys for the config. For example I typically use Cloud for my backends and hook into various other servers and none of those would be applicable in that use case... there likely is a different set of configs that it would return. It also leads to the question of how much config gets put there? Should it only be the things that are overridable?

So then the quesion I think is what is the intended use case here? Are you planning on building a UI around these config properties and expect some level of consistency across implementations? I think that would be reasonable for your own APIs or close collaborators and would highly suggest just going ahead and doing it and testing it out. I am not sure how useful it would be to hook into a "WES in the Wild" though if you plan on building a UI kit around it. In my experience You typically need custom handling for each engine / WES API and this would likely be no different

[vinjana]

I would not be surprised if every implementation of WES had a different set of keys for the config

I think this is a valid criticism, but not of this specific proposal. There have been discussions to standardize the vocabulary for engine parameters that are independent of the present proposal.

It also leads to the question of how much config gets put there? Should it only be the things that are overridable?

I think that would be a reasonable approach. I don't think there is much added value for the client, to expose engine parameters that are not overridable. Eventually, all such problems need to be solved with the WES-operators and consequently will pass the operator who does have the internal knowledge. So why burden the client?

The alternative is that we don't regulate which engine parameters are displayed but leave that to the WES implementer. In that case, however, we need to be able to discriminate in the response to the client, which parameters are overridable. That is covered by @JaeAeich's example already ("user_overridable": false).

So then the question I think is what is the intended use case here? Are you planning on building a UI around these config properties and expect some level of consistency across implementations?

I think this is really the core question. I think the use case for inclusion into the standard could also be some kind of external systems that has to interoperate with multiple WES implementations, e.g., to select the right WES service based on its capabilities, e.g., a WES-proxy. AFAIR mostly @uniqueg is working in that direction and might want to say something about this use case.