(probably) truncated response from pulp will cause ubipop crash

[rbikar]

Description:
On rare occasions, pulp return probably truncated json in response so e.g. resp.json() will fail on DecodeError.
See:
Traceback (most recent call last): File "/home/fedora/jenkins/workspace/ubipop-stage/ubipop_env/bin/ubipop", line 10, in <module> sys.exit(entry_point()) File "/home/fedora/jenkins/workspace/ubipop-stage/ubipop_env/lib/python3.7/site-packages/ubipop/cli.py", line 69, in entry_point main(sys.argv[1:]) File "/home/fedora/jenkins/workspace/ubipop-stage/ubipop_env/lib/python3.7/site-packages/ubipop/cli.py", line 64, in main opts.output_repos)\ File "/home/fedora/jenkins/workspace/ubipop-stage/ubipop_env/lib/python3.7/site-packages/ubipop/__init__.py", line 115, in populate_ubi_repos self._executor).run_ubi_population() File "/home/fedora/jenkins/workspace/ubipop-stage/ubipop_env/lib/python3.7/site-packages/ubipop/__init__.py", line 469, in run_ubi_population current_debug_rpms_ft) File "/home/fedora/jenkins/workspace/ubipop-stage/ubipop_env/lib/python3.7/site-packages/ubipop/__init__.py", line 396, in _get_pulp_actions current_rpms_ft.result()) File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib64/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/fedora/jenkins/workspace/ubipop-stage/ubipop_env/lib/python3.7/site-packages/ubipop/_pulp_client.py", line 116, in search_rpms for item in ret.json(): File "/usr/lib/python3.7/site-packages/requests/models.py", line 896, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1752239 (char 1752238) Traceback (most recent call last): File "scripts/ubipop/ubi-population-cron.py", line 35, in <module> subprocess.check_call(cmd) File "/usr/lib64/python3.7/subprocess.py", line 328, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['ubipop', '--pulp-hostname', 'pulp.dist.stage.ext.phx2.redhat.com', '--user', '****', '--password', '****', '--conf-src', 'https://gitlab.cee.redhat.com/ubi/data', '--output-repos', 'artifacts/repo-list.txt']' returned non-zero exit status 1.

This task is to investigate and confirm that pulp response was really truncated and implement retry for this case.

Acceptance criteria:

• Confirm root cause of the issue
• Implement retry for truncated response

[rohanpm]

confirm that pulp response was really truncated

I don't think it needs more confirmation actually - our other tools used for same Pulp servers have already hit this problem and added workarounds years ago. I wouldn't say the root cause has been deeply analyzed, but it rather seems to be introduced by the load balancer applications in front of Pulp and not by Pulp itself.