Open
Description
I reviewed all the configuration options which are used in the code and found quite a few inconsistencies. I have listed proposed fixes for them, separated into breaking and non-breaking changes. We can add warning for the breaking changes now, and then change in v1.0.0
.
Let's discuss these lists. Are there any items which should be added or removed? For the breaking changes, how long of a warning period should we have? For any of them, should we only start warning in v1.0.0
and then change in v1.1.0
?
Review of configuration options
Option | kwarg | CLI | ini | default |
---|---|---|---|---|
Enable testing | N/A | --mpl |
N/A | False |
Enable baseline image generation, at specified directory path (relative to where pytest was run) | N/A | --mpl-generate-path |
N/A | None |
Enable baseline hash generation, at specified file path (relative to where pytest was run) | N/A | --mpl-generate-hash-library |
N/A | None |
Directory containing baseline images (relative to the test file) |
baseline_dir |
--mpl-baseline-path (relative to where pytest was run) |
baseline/ |
|
Whether --mpl-baseline-path should also be relative to the test file |
N/A | --mpl-baseline-relative |
N/A | False |
Filename of the baseline image | filename |
N/A | N/A | test name |
Whether to include the module name in the baseline image filename | ? | mpl-use-full-test-name |
False |
|
File containing baseline hashes (relative to the test file) |
hash_library |
--mpl-hash-library |
no hash comparison | |
RMS tolerance | tolerance |
2 |
||
Whether to standardise metadata | deterministic |
True (PNG: False ) |
||
kwargs to pass to savefig | savefig_kwargs |
N/A | N/A | {} |
Matplotlib style | style |
classic |
||
Whether to remove axis tick labels | remove_text |
False |
||
Matplotlib backend | backend |
agg |
||
Directory to write testing artifacts to (relative to where pytest was run) |
N/A | --mpl-results-path |
mpl-results-path |
temp dir |
Whether to save result images for passing tests | N/A | --mpl-results-always |
mpl-results-always |
False (True if generating a HTML summary) |
Which test summaries to generate, if any | N/A | --mpl-generate-summary={html,json,basic-html} |
None |
Notes
- N/A are options which we don't think should exist.
- Empty boxes are options which we do think should exist.
- ? are options which could exist but would be of limited use.
- Due to how paths are computed by Python, the baseline directory and hash library paths can be absolute. In this case
--mpl-baseline-relative
and what directory the paths are interpreted to be relative to does not have any effect. - Path specified in kwargs should be relative to the test file, and paths specified in CLI and ini options should be relative to where pytest was run.
Non-breaking changes
- Document that backend can be specified. Migrate docs from
README.rst
to website #199 - Document the option to use the full test name. Migrate docs from
README.rst
to website #199 - Document that
mpl-use-full-test-name
ini overrides thefilename
kwarg. Migrate docs fromREADME.rst
to website #199 - Add
mpl-baseline-path
ini option (relative to where pytest was run). Add missing CLI and INI options for a consistent API #181 - Expand
mpl-use-full-test-name
ini option to CLI (and maybe kwarg). Add missing CLI and INI options for a consistent API #181 - Add
mpl-hash-library
ini option (relative to where pytest was run). Add missing CLI and INI options for a consistent API #181 - Add
--mpl-default-tolerance
CLI andmpl-default-tolerance
ini options. Add missing CLI and INI options for a consistent API #181 - Add CLI and ini options for
deterministic
. Set deterministic to True for vector graphics and warn about change to True in future for PNG #197 - Add
--mpl-default-style
CLI andmpl-default-style
ini options. Add missing CLI and INI options for a consistent API #181 - Add CLI and ini options for
remove_text
. - Add CLI and ini options for
backend
. Add missing CLI and INI options for a consistent API #181 - Add
generate-summary
ini option for generating test summaries. Add missing CLI and INI options for a consistent API #181
Breaking changes
-
--mpl-hash-library
should be relative to where pytest was run (to match--mpl-baseline-path
). - Local
hash_library
kwarg should take precedence over global--mpl-hash-library
CLI option. decorator hash_library kwarg option precedence #154 - Deprecate the use of multiple hash libraries in the same pytest run (not necessary because files are small and include the full module path anyway). decorator hash_library kwarg option precedence #154
- Deprecate the use of multiple testing modes in the same pytest run (e.g. warn if one test only has hash comparison while the others only have image comparison)
- Use Matplotlib's default style instead of
'classic'
. - Change the default RMS tolerance to
0
. - Enable deterministic PNG files by default.
- Deprecate
remove_text
kwarg and replace with better namedremove_ticks_and_titles
- (???) Add a
remove_text
kwarg that actually removes all text. - (???) Rename
baseline_dir
/--mpl-baseline-path
to have consistent name.