It looks like VideoDetails always assumes YUV, which is not the case. From what I can tell, this makes it impossible to determine the exact video format being used. E.g. the vapoursynth decoder supplies more information via the VSFormat and ColorFamily types, but most of it is never exposed through the av API.
This makes things like rust-av/ssimulacra2_bin#29 impossible.