AV2 ctc test script update sept 2026 - #5371
Open
RyanLei-Meta wants to merge 10 commits into
Open
Conversation
RyanLei-Meta
commented
Sep 4, 2026
RyanLei-Meta
commented
Contributor
- document clean up and update
- add perceptual quality metrics, including lpips, dists, and cvvdp for evaluating AI extension. disabled by default
- other code cleanup
1. use a relative root path in config.yaml so the framework works from any checkout; bin/ and work/ are commented out and derive from root 2. fix USER_GUIDE.md inconsistencies with the code: - vmaf flag is --aom_ctc, not --avm_ctc - ConvexHullTest.py has no summary function and spells it "concatnate" - -t/--ScaleMethod has no default and must be passed explicitly - CheckEncoding.py requires -c RA|AS and reads hardcoded paths - ECF tiling rules use "and", not "or"; the LD 720p tier is an exact match - local-run examples need -CmdOnly false, else nothing is executed - AV2CTCProgress.py uses get_anchor_tag(), not a single anchor variable 3. document paths.bin/paths.work, the full executables and feature flag lists, the per-version Excel template mapping, and the cluster path caveat 4. correct stale references: avmenc-v14.0.0, releases through v15.0.0
1. ignore venv/ anywhere in the tree 2. ignore *.pdf and *.xlsm, which AV2CTCProgress.py generates into ctc_result/ 3. keep tracking the CTC Excel templates in convexhull_framework/bin/, which are required inputs selected by Config.py per ctc_version
Adds three optional perceptual quality metrics for evaluating AI / learned
coding tools, where pixel-fidelity metrics correlate poorly with perceived
quality. Off by default: with perceptual_metrics.enabled false the framework
behaves exactly as before and never imports torch.
1. new PerceptualMetricsRunner.py computes the metrics in a subprocess
- LPIPS and DISTS via pyiqa, ColorVideoVDP via the pycvvdp API
- runs as a subprocess, not in-process, so the command is captured into
the per-job shell scripts and reaches the compute cluster
- one ffmpeg y4m->RGB conversion feeds all three metrics; the matrix and
range are stated explicitly because CTC y4m carries no colour tags
- accurate_rnd+full_chroma_int is required: without it swscale's
limited->full expansion lands 2/255 short (Y=235 -> 253, not 255)
2. new CalcPerceptualMetrics.py mirrors CalcQtyWithVmafTool.py and plugs
into CalculateQualityMetrics.py, the existing single fan-out point
3. results are appended AFTER the MD5 columns rather than added to
QualityList: AV2CTCProgress.WriteSheet copies the RD CSV into the .xlsm
templates by absolute column index, so inserting columns mid-row would
shift EncT/DecT/instr/cycles/MD5 and corrupt every workbook
4. BD-rate support in AV2CTCProgress.py
- LPIPS and DISTS are lower-is-better, so they are negated before
BD_RATE, which assumes the opposite; negative BD-rate keeps meaning
"bitrate saving" for every metric
- encodes where a metric was not computed are excluded, not counted as 0
- the three duplicated .agg() dicts and the positional fields_name list
are now derived from qtys so they cannot drift
5. ParseCSVFile reads the new columns with .get() and Record defaults them
to NaN, so the existing result CSVs in ctc_result/ still parse
6. LPIPS/DISTS are skipped on the HDR classes (G1, G2, ECF-3, ECF-4), where
they are out of distribution; ColorVideoVDP runs there with a PQ display
7. optional deps live in requirements-perceptual.{in,txt}, not hash-pinned,
installed via setup_env.sh --perceptual; the core requirements.txt and
its --require-hashes install are untouched
Verified: pre-existing BD-rate values bit-identical across all 4242 rows of
Bdrate-Summary; generated CSV header byte-identical to the historical format
when disabled; EncT[s] still lands in .xlsm column 26; colour conversion
exact on a grey ramp and within 1/255 on BT.709 primaries.
Place LPIPS/DISTS/CVVDP directly after CAMBI in the RD CSV instead of after DecMD5, so they are grouped with the metrics they belong with. 1. AV2CTCTest.py and ConvexHullTest.py write the perceptual columns before the timing block, in both the header and the value rows 2. AV2CTCProgress.WriteSheet now drops those columns before filling a CTC .xlsm template. The templates are laid out by absolute column index, so without this the shift would put LPIPS where EncT[s] is expected and corrupt every generated workbook 3. Utils.GatherPerframeStat always emits one field per enabled perceptual metric, even when the perceptual log is short or missing, so the per-frame CSV cannot become ragged. Its ordering was already correct -- the per-frame CSV has no timing or MD5 columns, so the perceptual fields already followed the quality metrics directly Verified: WriteSheet output byte-identical on all 10 historical RDResults CSVs; EncT[s] still resolves to .xlsm column 26 once the new columns are stripped; both writers now produce the same header; per-frame rows stay rectangular with complete, short and empty perceptual logs.
… Utils
Match the ffmpeg invocation from "Evaluation Report for AI-based Image
Codecs" so both frameworks convert the same way, and consolidate the
conversion helpers so there is a single y4m -> RGB path.
1. scaler flags now match the study exactly:
lanczos+accurate_rnd+full_chroma_int, sws_dither=none, param0=5,
explicit out_color_matrix, and format=<pixfmt> in the filter chain.
Given the study's matrix and range the generated filter string is
byte-identical to the one in the doc.
2. matrix and range moved to config (perceptual_metrics.color_conversion):
- sdr_matrix defaults to bt709, correct for CTC video. The study uses
bt601 for its PNG-derived still images; set that to reproduce it.
- default_range defaults to tv, the norm for CTC video sequences.
- Range is now DETECTED from the y4m XCOLORRANGE tag when present and
only falls back to default_range when untagged, so full-range files
such as the study's PNG-derived y4m are handled correctly either way.
The value used and the reason are recorded in the output JSON.
3. moved into Utils.py, replacing the runner's private copies:
GetY4MColorRange, GetY4MInfo, BuildY4MToRGBCmd, DecodeY4MToRGB,
ToUnitFloat, plus the Y4M_TO_RGB_SWS_* flag constants. The runner's
probe_y4m is now a thin wrapper that adds its error reporting, and
GetY4MInfo builds on the existing parseY4MHeader rather than
duplicating the parse.
Verified: filter string identical to the doc's when given bt601/pc;
conversion still exact on a grey ramp and within 1/255 on BT.709
primaries; XCOLORRANGE detection correct for tagged full, tagged limited
and untagged files; runner works from any working directory; feature-off
path unchanged and still torch-free.
The AV2 CTC y4m sequences are full range, and the AI image codec evaluation study reads them with in_range=pc. Default to that instead of tv so untagged CTC content is converted correctly out of the box. Changed in all four places the default was declared, so they cannot drift: config.yaml, the Config.py fallback, the runner's --in-range argparse default, and USER_GUIDE.md. A y4m carrying an XCOLORRANGE tag still overrides the default, so limited-range content is handled correctly when it says so. Verified on a Y ramp: untagged now passes through unscaled (0->0, 235->235, 255->255) rather than being expanded, and a file tagged XCOLORRANGE=LIMITED still gets the limited->full expansion (16->0, 235->255).
No functional change: ParseCSVFile reads columns by name via DictReader, so the CSV's physical column order never affected it. But after the perceptual metrics moved next to the other quality metrics, the argument list no longer read in CSV order, which invites someone to "fix" it and silently shift every field -- Record takes ~30 positional arguments. Move lpips/dists/cvvdp ahead of the timing arguments in both Record.__init__ and the ParseCSVFile call so the two, and the CSV, all read in the same order. Record is constructed in exactly one place, so this is a private contract between those two functions. Dropping their "" defaults costs nothing: backward compatibility with CSVs that predate these columns comes from data.get(), not from the parameter defaults. Added a docstring on ParseCSVFile recording that column order is irrelevant but argument order is not. Verified: historical CSVs parse unchanged (enc_time 770612.08, not an lpips value) and a CSV carrying the perceptual columns maps every field correctly with nothing shifted by three.
Handle metric polarity where the problem actually is -- the monotonicity check -- rather than transforming the data before the call. BD_RATE and check_monotonicity take a lower_is_better flag, defaulting to False so every existing caller is unchanged. When set, the check expects the bit rate to be non-increasing after the sort by quality ascending, which is what a well behaved LPIPS or DISTS curve looks like: more bits means a lower score. Previously that curve was rejected outright and the BD-rate silently recorded as 0.0 for every clip. Nothing else in BD_RATE needs to know the direction. The result is a difference of log-bit-rate integrals taken at matched quality, so reversing the quality axis reorders the samples without changing the sign; negative still means a bit rate saving for every metric. AV2CTCProgress.CalcOneBDRate now passes lower_is_better=(qty in LOWER_IS_BETTER_QTYS) and no longer negates. Verified: - flag=False is bit-identical to the previous check across 2000 random RD point sets - flag=True gives exactly the same numbers as the negation it replaces across randomised non-uniform curves (max disagreement 0) - sign convention holds: -30% for a 30% better codec, +30% for a worse one - genuinely non-monotonic data is still rejected under the flag - full BD-rate recomputed over the real ctc_result data: all 4242 rows and every pre-existing metric column bit-identical to the baseline
The VMAF saturation filter was gated on a hand-written list of four
literals that included "vmaf-neg" -- a spelling nothing in the framework
produces. The name AV2CTCProgress actually passes is "vmaf_neg", so
VMAF-NEG silently skipped the filter on that path while plain VMAF got it.
Replace the inline literals with a single VMAF_METRIC_NAMES set and an
is_vmaf_metric() helper, and drop the phantom spelling. Verified complete
against the real sources: every VMAF name in Config.QualityList and
AV2CTCProgress.qtys matches, and no non-VMAF metric does.
The three remaining spellings cannot be collapsed into one, so the set
documents why each exists:
VMAF_Y / VMAF_Y-NEG RD CSV columns. libvmaf's own naming, and baked
into the 20 committed result directories under
ctc_result/ that AV2CTCProgress still reads.
vmaf / vmaf_neg BD-rate keys, which become Bdrate-Summary columns.
vmaf_y_neg Utils.Record attribute; "-" is not a valid
Python identifier.
No published number moves: BD-rate recomputed over the real ctc_result
data is bit-identical across all 4242 rows. VMAF-NEG never saturates to
>=99.5 in the current result set, so the filter would have been a no-op
there anyway -- the bug was latent and this closes it for future content.
Audited every third-party import in src/ against the requirements files: nothing is missing. numpy, pandas, scipy, matplotlib, openpyxl, PyYAML and tabulate are all in requirements.in; torch, torchvision, pyiqa and cvvdp are in requirements-perceptual.in. 1. USER_GUIDE.md dependency table now says which module uses each package, rather than just listing names. 2. New "External Tools" section: requirements.txt covers Python packages only, so the binaries the framework shells out to are now documented in one place -- vmaf, avmenc/avmdec, ffmpeg, HDRConvert, lanczos_resample_y4m, SvtAv1EncApp, TAppEncoderStatic, and the per-OS timing utility (perf / gtime / ptime). 3. Recorded that xlrd and xlsxwriter are pinned but unused: nothing in src/ imports them, and pandas is used only for CSV so they are not needed as its Excel engines either. Left in place rather than churning the hash-pinned lock, which cannot be validated for the Linux cluster from a macOS machine. Flagged in requirements.in and the guide for the next regeneration. 4. New CLAUDE.md for the framework, recording the invariants that are easy to break: commands must go through ExecuteCmd to reach the cluster, LogCmdOnly defaults to true, WriteSheet maps to the .xlsm by absolute column index, QualityList is positional, Record takes ~30 positional arguments, BD_RATE assumes higher-is-better, the three VMAF naming layers, y4m carries no colorimetry, and the perceptual metrics must stay torch-free when disabled. Scoped to tools/convexhull_framework/ rather than the repo root, which is the AVM codec itself.
Contributor
Author
|
@urvangjoshi @leolzhao please review this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.