Retry flaky tests

[ofek]

Motivation: #817 (comment)

[ofek]

CI failure due to: ipython/ipython#13554

[uranusjr]

Do we need this many? I’d start with 2.

[ofek]

Done!

[cs01]

Before we go to retries, we should look to understand the root cause of the problem. I would rather hold off on landing this before we try other methods, since this is adding more tech debt. I appreciate the PR and if you can’t root cause it, that’s fine, but I’d prefer creating an issue instead and closing this PR, especially because CI tests still failed.

[ofek]

Basically, iPython needs to release the fix ipython/ipython#13554 (comment)

I think retries are still a useful thing to add, though it's up to you!

[uranusjr]

From experience, subprocess (also os.fork and multiprocess which all share a same vein under the hood) sometimes just fail like this due to OS internal states, and I’ve yet to see anyone come up with a good solution to make this always work. Retry is the usual solution for this kind of things.

[cs01]

From experience, subprocess (also os.fork and multiprocess which all share a same vein under the hood) sometimes just fail like this due to OS internal states, and I’ve yet to see anyone come up with a good solution to make this always work. Retry is the usual solution for this kind of things.

We have had zero of such issues with pipx so far. I just looked into the failures we're seeing a little, and the root cause is a repeatable python3.8/ipython issue. ipython/ipython#12677

I appreciate the PR, but I am going to close it since I believe it is trying to solve a problem that does not exist, and is adding tech debt.