Race condition when set semaphore at both workflow and template level

[quantm0806]

Hi, I'm meeting race condition when using semaphore.
What I have setup

• A semaphore named total-pods=2
• A cron workflow acquiring semaphore at both workflow and template level. It looks like the image below. The format will be like:

"spec": {
    "schedule": "* * * * *",
    "concurrencyPolicy": "Replace",
    "workflowSpec": {
      "synchronization": {
        "semaphore": {
          "configMapKeyRef": {
            "name": "semaphore",
            "key": "total-pods"
          }
        }
      },
      "entrypoint": "main",
      "templates": [
        {
          "name": "main",
          "synchronization": {
            "semaphore": {
              "configMapKeyRef": {
                "name": "semaphore",
                "key": "total-pods"
              }
            }
          }

• A normal workflow also acquiring semaphore at both workflow and template level.

The cron workflow and normal workflow are attached here:
test_parallelism_cwf.json
test_semaphore_templates_with_priority_wf.json

How to reproduce

1. Submit the cron workflow and wait until the first cron workflow appears
2. When the first cron workflow runs, submit the normal workflow. The normal workflow doesn't run because the semaphore total-pods has been all used by the cron workflows (its workflow takes 1 and its template takes 1)
3. After one minute, when new cron workflow appears -> old cron workflow is replaced and releases semaphore -> new cron wf take 1 and the waiting normal workflow also take 1. Now, we have race condition. Two workflows acquire the semaphore but no template within them are running.

Note: I set priority: 1 in normal workflow at both workflow + template level but somehow it isn't prioritized to get 2 total-pods

Question

1. Why does my priority not work?
2. The docs say: You can set a [priority](https://argo-workflows.readthedocs.io/en/latest/parallelism/#priority) on Workflows. So does it mean that I cannot set a priority on template but workflow?

Thanks in advance.