Add a script to decode and compare PSNR of the entire test suite. #148
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Encoder bug-fixes improve this file's PSNR test performance. - Previously demixing matrices were applied incorrectly and led to bad scores in some channels. - Based on AOMediaCodec/iamf-tools@cdc930082
- `psnr_calc`: Same usage, except use new flag for verbose per-channel logging. - `calc_average_channel_psnr_wav` validates the wav files are consistent, and calls the low-level function. - `calc_average_channel_psnr_pcm` performs low-level signal analysis. - Easier for other utilities to share code, when they are not bundled with the main program.
- These are only used by tests, and not the main repo.
- `user_metadata_parsing_utils`
- Introduce utilities to get statistics about each test vector.
- Introduce a function to get `iamfdec` arguments associated with a test file.
- `run_decode_and_psnr_test`
- Decodes a set of files, runs PSNR tests, and produces a summary of all tests.
- By default, this runs on all test vectors.
- Uses same comparison as `psnr_calc`, but automatically decodes and matches combos.
- Document in `README.md`.
|
Thanks for this script. It works really well. I tested it on v1.0.1 with the updated 70 and 210 test files and most of the tests pass. I would like to list the few things which fail when tested against the v1.0.1 tests folder:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow up of #142. Merge after #145, because while developing this script, I found some bugs in the reference files.
This PR leaves the old
psnr_calcworking, but it aims to be easier to use.psnr_calcwas difficult to use because it required knowing all of the arguments to decode the correct {mix id, sample rate, bit-depth, layout, limiter settings} etc. to create comparison wav files.This script actually parses the test suite, determines the command line arguments to use, runs decode, and runs the comparisons. It prints a summary, and optionally writes a CSV with test details. It also filters out certain tests that should not be compared, such as tests that exceed profile limits, decode to an extension layout, are tagged as "should-fail", etc.
This PR aims to establish a baseline of the actual verification that was performed under v1.0.0. To establish this baseline it must reuse the metric of averaging PSNR across channels. Although this has been flagged before, by establishing this baseline it will be easier to investigate other metrics.