Using the specification

[tloubrieu-jpl]

Hello,

For the EDRN project (https://edrn.nci.nih.gov/) we are interested in using the Workflow Execution Service API to trigger our worflows running in Apache Airflow and Nextflow. It looks like a very good and solid basis for our development.

Regarding the specification, I have 2 questions:

1. I don't see a /workflows and /workflows/{id} end-point to enable the search/discovery of available workflows. I used the file as reference. Is there an end-point that I missed ? Is this /workflows end point coming in a future version of the specification ?
2. for an Airflow implementation of the WES, the list of runs is given per workflow (DAGs in the Airflow vocabulary), that would make the /runs end-points easier to implement as /workflows/{id}/runs assuming a /workflows end-point is something we can have in the WES specification. Do you have thoughts on that ?

I am also having a question on the activity on this development. I am not seeing a lot of activity lately in the repository and especially on the Starter Kit implementation. Was that a one-off experience proposed for a workshop ? Do you advise using the WES standard and if yes, are there maintained implementations for this standard server side ?

Any advice are welcome since we don't want to re-invent the wheel and if we need to do developments, your experience is very valuable to us anyhow.

Thanks

[uniqueg]

Welcome to the GA4GH Cloud community, @tloubrieu-jpl - and thanks for your feedback!

Allowing your users to trigger Nextflow and Apache runs sounds like an exciting project, and using WES for triggering Nextflow and Airflow workflow runs seems like a very good idea to me! 😀

To answer your questions as best as I can:

1. Indeed, /workflows endpoints are currently not included in the specs. As far as I know, they have also not yet been suggested suggested by anyone. What resource did you have in mind exactly that would be provided at /workflows? Or is this suggestion only/primarily for filtering /runs? In that case, something like that has been (fairly) recently proposed here, and there is also an open PR on run filtering. Neither currently covers workflow filtering, but I guess that could be easily added. The good thing about filtering vs dedicated endpoints is that it's much easier to maintain and implement, both on the server and client side. Also, filtering is already supported by the GA4GH Tool Registry Service (TRS) and Task Execution Service (TES) API specifications, so we have some experience here. In my personal opinion, filtering runs by workflows would be a very useful addition to the specs. However, perhaps you had other use cases in mind than just run filtering? In that case, maybe you could describe these in more detail and, ideally, also provide a dummy response data model for /workflows//workflows/{id}?
2. Oh, so perhaps your main use case is that you would like users to be able to see only Apache or Nextflow runs, but not both together? In that case, a filter on workflow_type (instead of workflow_url) would do the trick, I guess. Probably an even better idea (or perhaps we could just add both).

More generally on how standard/product development works in GA4GH:

• Additions to and maintenance of the standard are provided by the community. In particular, issues and PRs are expected to originate from the community of implementers. Whether a proposed change is accepted, is then decided by the entire GA4GH Cloud Work Stream commmunity and, ultimately, the broader GA4GH community, consisting of Driver Projects](https://www.ga4gh.org/our-community/driver-projects/#) and other major stakeholders.
• That being said, each standard/product has a dedicated leadership team. For WES, these are currently (as far as I know): @patmagee, @vschnei and @inutano. So perhaps they can give you an idea if a new release is currently on the horizon?
• WES (and GA4GH products in general) are meant to support a common denominator of core features, ensuring a wide variety of use cases and keeping the implementation barrier low - while still providing a basic level of interoperability. So for every proposed change, the community and leadership would ask the question: Does (almost) every implementation need to support this? If the answer is yes, the changes are typically accepted (perhaps after modification) - if it is not, then the suggestion is for implementers to simply extend the specification, e.g., with their own custom properties and endpoints (if the product is an OpenAPI specification).
• Sparsity of community contributions (we are almost all doing this voluntarily) and speed of development can be an issue. On the other hand, it's probably not a bad thing that standard development is not too fast, as this ensures that development remains supported by a broad community, making (eventual) widespread adoption (and the benefits thereof in terms of interoperability) more likely.

On that last note: Are you planning to work on two separate implementations, or one implementation, with plugins for each workflow type? The former would have the advantage of making it easier for the open source software community to support one good WES implementation per workflow type (ideally led by the workflow type/engine community itself), rather than fragmenting the already relatively sparse contributions across multiple different WES implementations. On the other hand, maintaining two repos is of course more initial workload on you for realizing your use cases, which is (I guess) why WESkit (Snakemake, Nextflow) and Sapporo WES (CWL, WDL, Snakemake, Nextflow) chose the latter approach. As a potential compromise, perhaps you could check whether WESkit or Sapporo WES are working well for you - and in that case perhaps focus on providing an Apache Airflow implementation only (I am not aware of any others, currently - although it has been mentioned a few times before). Along these lines, the ELIXIR Cloud & AAI Driver Project, which I am co-leading, is adopting an approach of using already existing WES implementations. We are developing a WES gateway with middleware to distribute workflows to an appropriate WES instance. Something like that:

Last note: I am not really familiar with the starter kit, its intention and future plans. However, I wouldn't count on it to be further developed. There are certainly a bunch of other WES implementations being developed, both in academia and industry. However, I currently don't have an exhaustive/extensive list. WESkit and Sapporo are certainly two, with core developers of both being involved in WES leadership.