Home
It has been quite common recently for the pipeline to get stuck during the rsync process. This can be detected because the fetcher task is still running, while the hist_canning workflow is marked as failed.
See: https://github.com/ooni/sysadmin/issues/403
In order to fix it you should:
- Connect to datacollector and
pkill rsync:
ssh datacollector.infra.ooni.io
datacollector:~$ sudo pkill rsync
-
Once rsync has been killed check that the task is marked as up-for-retry from the airflow UI and wait for it to conclude. Tip: sometimes you need to kill rsync a couple of times before it gets a socket which has decent throughput
-
Once it has concluded you can restart the hist_canning DAG. See screenshots below:
