Race condition while looping over hosts in template #191
Description
There is a race condition triggered in template loop
{% for host in ansible_play_hosts %}
{% if host != inventory_hostname %}
[Peer]
# {{ host }}
PublicKey = {{hostvars[host].wireguard__fact_public_key}}
...
What happens here is that in the loop, wireguard__fact_public_key - a fact that is dynamically generated on other hosts, is needed in the loop body, however, there is no wait statement to make sure that at the point of execution, this fact really is defined. One will normally notice this when inventory is geographically spread, like in my case where I have some hosts in local network and some in the cloud. Hosts in local network are typically faster and will reach loop in template before cloud hosts had a chance to set the fact.
Simple workaround is to put something like this in main.yml just before template task:
- name: Sleep for 10 seconds and continue with play
ansible.builtin.wait_for:
timeout: 10
Of course, 10 seconds wait is arbitrary and this is no real solution. Solution may involve more flexible waiting in template loop by periodically checking for fact existence before using them or, even better, removing race condition altogether so that all wireguard keys and configs are delegated to ansible controller and distributed to the clients afterwards. The second solution may raise some security concerns regarding private keys, however, one should bear in mind that if ansible controller somehow gets compromised, one probably will have much bigger problems than wireguard keys anyway.
Notice also that serializing the play won't cut it either, because if one does that, play will always fail: none of the dynamically generated facts from other hosts will be available in the template loop for obvious reason. Similarly, when executing the play with big number of hosts, say 1000, and having serial set to say 20, loop will fail too, even if all hosts are in local network, i.e. equally fast.
Explicit looping over ansible inventory is generally discouraged anyway, but I can see that in this particular case this lends itself nicely to create wireguard config in pretty elegant way. Unfortunately, elegance comes with price.