Failed file transfers are never retried

[Xarthisius]

Occasionally fetching remote resources fails with reasons unrelated to WT, e.g.:

Traceback (most recent call last):
  File "/girder/plugins/wt_data_manager/server/lib/tm_utils.py", line 39, in run
    self.transfer()
  File "/girder/plugins/wt_data_manager/server/lib/handlers/common.py", line 30, in transfer
    with open(self.psPath, 'wb') as outf, self.openInputStream() as inf:
  File "/girder/plugins/wt_data_manager/server/lib/handlers/http.py", line 25, in openInputStream
    return HttpStream(urlopen(self.url))
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 503: Service Unavailable

This results in an empty file being created in DMS cache (/tmp/ps). On the receiving end (girderfs), the call that triggered the fetch polls /dm/fs/<itemId>/raw indefinitely (which is a separate issue). Subsequent calls from the client also hang, even if remote issue is resolved. File is never downloaded to cache and stays as an empty file. The only workaround is to manually drop dm attribute from the item and try accessing the file via the client.