Describe the bug
Many test cases in GatorGrader check the newline count of generated output. It turns out that this count can change depending on an external variable: the number of columns in the terminal used to run pytest. With a $COLUMNS value fewer than 80, certain tests will fail, with more failures the smaller the terminal becomes. The same is true in reverse, with a $COLUMNS value of greater than 82.
To Reproduce
Steps to reproduce the behavior:
- Resize a terminal such that
echo $COLUMNS prints a number less than 80 or greater than 82
- Execute
pipenv run cover to see how many tests fail at that given size.
Expected behavior
Tests should not depend on terminal size for correctness.
Environment
GitHub Actions terminal, Elementary OS 5 (based on Ubuntu 18.04) gnome-terminal.
Python 3.9.7 (also reproducible in 3.8.12).
I can confirm that this bug does not appear in Python 3.7.12.
Proposed Solution
This bug is caused by asserting an exact amount of new lines in expected produced output; instead of asserting an exact amount, the minimum amount possible should be "at least" (with >=) asserted. Alternatively, new line count should be dropped as a measure of correctness for produced output -- this may be the desired solution, since new line count can be influenced by many factors, and often does not indicate regressions or errors.
Describe the bug
Many test cases in GatorGrader check the newline count of generated output. It turns out that this count can change depending on an external variable: the number of columns in the terminal used to run
pytest. With a$COLUMNSvalue fewer than 80, certain tests will fail, with more failures the smaller the terminal becomes. The same is true in reverse, with a$COLUMNSvalue of greater than 82.To Reproduce
Steps to reproduce the behavior:
echo $COLUMNSprints a number less than 80 or greater than 82pipenv run coverto see how many tests fail at that given size.Expected behavior
Tests should not depend on terminal size for correctness.
Environment
GitHub Actions terminal, Elementary OS 5 (based on Ubuntu 18.04) gnome-terminal.
Python 3.9.7 (also reproducible in 3.8.12).
I can confirm that this bug does not appear in Python 3.7.12.
Proposed Solution
This bug is caused by asserting an exact amount of new lines in expected produced output; instead of asserting an exact amount, the minimum amount possible should be "at least" (with
>=) asserted. Alternatively, new line count should be dropped as a measure of correctness for produced output -- this may be the desired solution, since new line count can be influenced by many factors, and often does not indicate regressions or errors.