Consider providing a facility for integer-fraction timescales

[rcombs]

It's pretty well-established that Matroska's poor timebase support is one of the format's worst properties. While it support very precise timestamps (down to the nanosecond), it's very inefficient to do so (and the resulting values still aren't exact for most input rates), so muxers tend to default to 1ms timestamps, which can lead to a variety of subtle issues, especially with high-packet-rate streams (e.g. audio) and VFR video content. Muxers can choose rates that are closer to the time base of their inputs (or the packet rate of the content), but exactly how best to do so has always been unclear, and some of the possible options would lead to either worse player behavior, or timestamp drift. I'm proposing a format addition to remedy this.

The only actual normative change I propose is this: in addition to the classic nanosecond-denominator time scale, muxers could provide 2 additional integers, serving as a numerator and denominator time base value, which is required to round to the existing nanosecond-scaled value.

This should be paired with some advice for muxer implementations on how to make use of this feature. This depends on the properties of the input. For reference, here are some examples of the error produced by rounding a variety of common time bases to the nearest nanosecond, scaled by 3 hours (a reasonable target for the duration of a film):

nearest_ns(x) = round(x * 1,000,000,000) / 1,000,000,000
ceil_ns(x) = ceil(x * 1,000,000,000) / 1,000,000,000
floor_ns(x) = floor(x * 1,000,000,000) / 1,000,000,000
nearest_error(x) = 1 - (x / nearest_ns(x))
ceil_error(x) = 1 - (x / ceil_ns(x))
floor_error(x) = 1 - (x / floor_ns(x))
nearest_error_3h(x) = nearest_error(x) * 60 * 60 * 3
ceil_error_3h(x) = ceil_error(x) * 60 * 60 * 3
floor_error_3h(x) = floor_error(x) * 60 * 60 * 3
e(x) = nearest_error_3h(1 / x)
ce(x) = ceil_error_3h(1 / x)
fe(x) = floor_error_3h(1 / x)

# Integer video frame rates
e(24)        => 8.64e-5
e(25)        => 0
e(30)        => -0.0001
e(48)        => -0.0002
e(50)        => 0
e(60)        => 0.0002
e(120)       => -0.0004

# NTSC video frame rates
e(24/1.001)  => -8.6314e-5
e(30/1.001)  => 0.0001
e(48/1.001)  => 0.0002
e(60/1.001)  => -0.0002
e(120/1.001) => 0.0004

# TrueHD frame rates
e(44100/40)   => -0.0057
e(48000/40)   => -0.0043
e(88200/40)   => 0.0062
e(96000/40)   => 0.0086

# AAC frame rates
e(44100/960)  => -0.0002
e(48000/960)  => 0
e(88200/960)  => 0.0003
e(96000/960)  => 0
e(44100/1024) => 0.0002
e(48000/1024) => -0.0002
e(88200/1024) => -0.0003
e(96000/1024) => 0.0003

# MP3 frame rates
e(44100/1152)  => 8.4375e-6
e(48000/1152)  => 0
e(88200/1152)  => -0.0004
e(96000/1152)  => 0

# Other audio frame rates
e(44100/128)   => -0.0012
e(48000/128)   => 0.0013
e(88200/128)   => -0.0012
e(96000/128)   => -0.0027
e(44100/2880)  => -7.425e-5
e(48000/2880)  => 2.3981e-12
e(88200/2880)  => -7.425e-5
e(96000/2880)  => 2.3981e-12

# GCF of common short-first audio frame sizes
e(44100/64)   => -0.0012
e(48000/64)   => -0.0027
e(88200/64)   => 0.0062
e(96000/64)   => 0.0054

# Raw audio sample rates
e(44100)     => 0.1253
e(48000)     => -0.1728
e(88200)     => 0.1253
e(96000)     => 0.3456

fe(44100)    => -0.351
ce(48000)    => 0.3456
fe(88200)    => -0.8273
fe(96000)    => -0.6912

# MPEGTS time base
e(90000)     => -0.108

ce(90000)    => 0.8639

# Common multiples
e(30000)     => -0.108
e(60000)     => 0.216
e(120000)    => -0.432
e(240000)    => 0.8639
e(480000)    => -1.7283

ce(30000)    => 0.216
fe(60000)    => -0.432
ce(120000)   => 0.8639
fe(240000)   => -1.7283
ce(480000)   => 3.4549

As we can see, rounding common video and audio frame rates (including e.g. the least common multiple of 24 and 60 for that VFR case) produces a negligible amount of error over a reasonable duration. This means that for content where all timestamps can reasonably be expressed in integer values of those rates, there would be no significant error over common file durations, even if different streams were muxed with different time bases.

There are a few real-world time bases that would produce significant rounding error (upwards of 100ms) over the course of 3 hours when used in existing players: MPEGTS's 90000Hz, all common raw audio sample rates, and least-common-multiples between integer and NTSC video frame rates. This essentially means that mixing these rates with others would produce significant desync over a reasonable duration for static on-disk content; the same issue could occur when muxing very lengthy content (e.g. streaming).

All of these issues can be addressed in one of the following ways:

• Using a lower rate (e.g. 90,000Hz isn't usually the real content rate but instead an artifact of its previous container; expressing timestamps in samples rather than frames is usually unnecessary)
• Choosing the highest of the input rates for all streams (e.g. 48000 is a multiple of many common frame rates, including 24/1.001)
• Choosing a more precise common-multiple rate that may create a larger total drift, but does so equally for all streams (see the "Common multiples" section; 1/30000 is suitable for mixing 24fps and 30/1.001fps content alongside most common framed audio rates, while the later listed bases are suitable for increasingly large sets).
• Round some tracks' nanosecond timescales in the opposite direction, creating a larger drift, but potentially one with the same sign (and thus a closer value) as the drift in other tracks (this is probably too complex and niche to have substantial use)
• Fall back to classic rounded nanosecond-based timestamps (and don't write an integer-fraction time base at all)
• Use the extension, resulting in significant sync drift in older players that haven't implemented the change

This last option is usually unacceptable, but may be fine for files that use codecs that become available after the change is made (and thus are unavoidably non-backwards-compatible anyway).

If combined with clear advice in the spec on how muxers SHOULD (or MAY) decide on time bases for various possible input cases, I think this extension could get actual adoption in muxers and solve one of the format's longest-standing problems.

[rcombs]

…I just realized I'd misremembered where timescales are specified when writing this (they're on the segment, not the track). Still, the same concept applies, just requiring common-multiple rates (though the TrackTimestampScale element could be used to account for this to some extent; it's deprecated, but all existing players other than MPlayer-derived ones seem to support it).

[dericed]

Hi, I see this was brought up as an issue in the GitHub repository and am cross-posting to the cellar working group.

On Sep 7, 2020, at 12:02 AM, rcombs ***@***.***> wrote: It's pretty well-established that Matroska's poor timebase support is one of the format's worst properties. While it support very precise timestamps (down to the nanosecond), it's very inefficient to do so (and the resulting values still aren't exact for most input rates), so muxers tend to default to 1ms timestamps, which can lead to a variety of subtle issues, especially with high-packet-rate streams (e.g. audio) and VFR video content. Muxers can choose rates that are closer to the time base of their inputs (or the packet rate of the content), but exactly how best to do so has always been unclear, and some of the possible options would lead to either worse player behavior, or timestamp drift. I'm proposing a format addition to remedy this. The only actual normative change I propose is this: in addition to the classic nanosecond-denominator time scale, muxers could provide 2 additional integers, serving as a numerator and denominator time base value, which is required to round to the existing nanosecond-scaled value. This should be paired with some advice for muxer implementations on how to make use of this feature. This depends on the properties of the input. For reference, here are some examples of the error produced by rounding a variety of common time bases to the nearest nanosecond, scaled by 3 hours (a reasonable target for the duration of a film): nearest_ns(x) = round(x * 1,000,000,000) / 1,000,000,000 ceil_ns(x) = ceil(x * 1,000,000,000) / 1,000,000,000 floor_ns(x) = floor(x * 1,000,000,000) / 1,000,000,000 nearest_error(x) = 1 - (x / nearest_ns(x)) ceil_error(x) = 1 - (x / ceil_ns(x)) floor_error(x) = 1 - (x / floor_ns(x)) nearest_error_3h(x) = nearest_error(x) * 60 * 60 * 3 ceil_error_3h(x) = ceil_error(x) * 60 * 60 * 3 floor_error_3h(x) = floor_error(x) * 60 * 60 * 3 e(x) = nearest_error_3h(1 / x) ce(x) = ceil_error_3h(1 / x) fe(x) = floor_error_3h(1 / x) # Integer video frame rates e(24) => 8.64e-5 e(25) => 0 e(30) => -0.0001 e(48) => -0.0002 e(50) => 0 e(60) => 0.0002 e(120) => -0.0004 # NTSC video frame rates e(24/1.001) => -8.6314e-5 e(30/1.001) => 0.0001 e(48/1.001) => 0.0002 e(60/1.001) => -0.0002 e(120/1.001) => 0.0004 # TrueHD frame rates e(44100/40) => -0.0057 e(48000/40) => -0.0043 e(88200/40) => 0.0062 e(96000/40) => 0.0086 # AAC frame rates e(44100/960) => -0.0002 e(48000/960) => 0 e(88200/960) => 0.0003 e(96000/960) => 0 e(44100/1024) => 0.0002 e(48000/1024) => -0.0002 e(88200/1024) => -0.0003 e(96000/1024) => 0.0003 # MP3 frame rates e(44100/1152) => 8.4375e-6 e(48000/1152) => 0 e(88200/1152) => -0.0004 e(96000/1152) => 0 # Other audio frame rates e(44100/128) => -0.0012 e(48000/128) => 0.0013 e(88200/128) => -0.0012 e(96000/128) => -0.0027 e(44100/2880) => -7.425e-5 e(48000/2880) => 2.3981e-12 e(88200/2880) => -7.425e-5 e(96000/2880) => 2.3981e-12 # GCF of common short-first audio frame sizes e(44100/64) => -0.0012 e(48000/64) => -0.0027 e(88200/64) => 0.0062 e(96000/64) => 0.0054 # Raw audio sample rates e(44100) => 0.1253 e(48000) => -0.1728 e(88200) => 0.1253 e(96000) => 0.3456 fe(44100) => -0.351 ce(48000) => 0.3456 fe(88200) => -0.8273 fe(96000) => -0.6912 # MPEGTS time base e(90000) => -0.108 ce(90000) => 0.8639 # Common multiples e(30000) => -0.108 e(60000) => 0.216 e(120000) => -0.432 e(240000) => 0.8639 e(480000) => -1.7283 ce(30000) => 0.216 fe(60000) => -0.432 ce(120000) => 0.8639 fe(240000) => -1.7283 ce(480000) => 3.4549 As we can see, rounding common video and audio frame rates (including e.g. the least common multiple of 24 and 60 for that VFR case) produces a negligible amount of error over a reasonable duration. This means that for content where all timestamps can reasonably be expressed in integer values of those rates, there would be no significant error over common file durations, even if different streams were muxed with different time bases. There are a few real-world time bases that would produce significant rounding error (upwards of 100ms) over the course of 3 hours when used in existing players: MPEGTS's 90000Hz, all common raw audio sample rates, and least-common-multiples between integer and NTSC video frame rates. This essentially means that mixing these rates with others would produce significant desync over a reasonable duration for static on-disk content; the same issue could occur when muxing very lengthy content (e.g. streaming). All of these issues can be addressed in one of the following ways: Using a lower rate (e.g. 90,000Hz isn't usually the real content rate but instead an artifact of its previous container; expressing timestamps in samples rather than frames is usually unnecessary) Choosing the highest of the input rates for all streams (e.g. 48000 is a multiple of many common frame rates, including 24/1.001) Choosing a more precise common-multiple rate that may create a larger total drift, but does so equally for all streams (see the "Common multiples" section; 1/30000 is suitable for mixing 24fps and 30/1.001fps content alongside most common framed audio rates, while the later listed bases are suitable for increasingly large sets). Round some tracks' nanosecond timescales in the opposite direction, creating a larger drift, but potentially one with the same sign (and thus a closer value) as the drift in other tracks (this is probably too complex and niche to have substantial use) Fall back to classic rounded nanosecond-based timestamps (and don't write an integer-fraction time base at all) Use the extension, resulting in significant sync drift in older players that haven't implemented the change This last option is usually unacceptable, but may be fine for files that use codecs that become available after the change is made (and thus are unavoidably non-backwards-compatible anyway). If combined with clear advice in the spec on how muxers SHOULD (or MAY) decide on time bases for various possible input cases, I think this extension could get actual adoption in muxers and solve one of the format's longest-standing problems.

This has been discussed on the list before though I don’t remember clear consensus on how to address this. Steve even compiled a list of discussions on this at https://mailarchive.ietf.org/arch/msg/cellar/ZpZxhG1gML9xVx_ir1Jf6_gcI8U/ <https://mailarchive.ietf.org/arch/msg/cellar/ZpZxhG1gML9xVx_ir1Jf6_gcI8U/>. I proposed an option in this https://mailarchive.ietf.org/arch/msg/cellar/mTprgjNqVbe20e6hyYxns8ZnVwY/ <https://mailarchive.ietf.org/arch/msg/cellar/mTprgjNqVbe20e6hyYxns8ZnVwY/> where one of the existing reserved bits of the Block Header (in the byte that contains the keyframe, invisible, and lacing flags) be used as a flag for Timescale Alignment. With this approach, new elements could be added to the track header with a numerator and denominator of a rationale time scale and if Timescale Alignment were set to true, then the nearest increment of the rationale time scale would be used. Example:

Thus if the frame rate of the track header is 120000/1001, then If Matroska timecode is 4 and Enable TimeScale Alignment is 0, than it is at 4 / (1000000000 / TimecodeScale ). If Matroska timecode is 4 and Enable TimeScale Alignment is 1, than it is at 0 / 1200000 (nearest increment of the rationale frame rate). If Matroska timecode is 17 and Enable TimeScale Alignment is 0, than it is at 17 / (1000000000 / TimecodeScale ). If Matroska timecode is 17 and Enable TimeScale Alignment is 1, than it is at 2002 / 1200000 (nearest increment of the rationale frame rate).

In a Matroska demuxer doesn’t understand the new nom/denom elements or the Alignment flag then it would simply use the existing nanosecond timestamp system. In that thread there were other proposals, for example Steve discussed using a float to depict a point in time. Dave

Did anyone suggest storing the rounding error as a fraction? (With denominator stored in the track header, this is only 3 bytes per packet in the best case.)

Of course all of these ideas are terrible hacks compared to just storing it in the correct way.

[dericed]

On Sep 8, 2020, at 11:52 AM, wm4 ***@***.***> wrote: Did anyone suggest storing the rounding error as a fraction? (With denominator stored in the track header, this is only 3 bytes per packet in the best case.)

That sounds interesting: to have the rounding error numerator in each block and the rounding error denominator in the track header. Perhaps a rounding error denominator could also be in the block but defaults to the one within the track header.

Of course all of these ideas are terrible hacks compared to just storing it in the correct way.

Yes, it is a challenge to fix this and maintain reverse compatibility. Dave

[mbunkus]

Did anyone suggest storing the rounding error as a fraction? (With denominator stored in the track header, this is only 3 bytes per packet in the best case.)

I don't like this as storing rounding errors ist imprecise as well (unless the global timestamp scaling factor is a multiple of the rounding error's denominator). I'm also quite unsure which denominator a multiplexer should chose. In order to express a rounding error precisely it must have a much higher resolution that the usual 1ms resolution of Matroska timestamps. For example, with 1001/30000 FPS content the rounding error will always be below one frame duration, therefore you'll have to make the denominator much larger.

Something else that came to mind when reading our previous discussion that Dave linked to: please keep in mind that any solution that sets values for the whole track in the track header will inevitably fail with mixed frame rate content or content with different interlacing, e.g. when multiplexing from an MPEG transport stream recorded from a DVB broadcast. Those bloody streams change frame rates all the time when the program changes, e.g. when transitioning to and from commercials (or just from an announcement to the movie). With our new and shiny precise timestamp calculation we'll either have to forbid such changes (unrealistic) or provide facilities to signal such changes in the form of some type of global index similar to cues. Unlike cues, though, such an index would have to be mandatory (a file without cues can be played just fine, even seeking works similar to seeking in Ogg files — meaning some kind of binary search).

File types whose timestamps are based solely on a stream's regular sampling frequency (MP4 usually is, but doesn't have to; Ogg does, too) all share those issues. MPEG TS on the other hand uses a 90 KHz-based clock which is fine for most video stuff but doesn't have enough resolution for sample-precision timing of audio tracks with high sampling frequency.

… in the correct way.

Due to what I've written above I'm pretty sure that there is no one correct way to store timestamps for a general purpose container that allows its content to change its time base in the middle.

In theory Matroska's timestamps can have sample-precision already (just make global timestamp scale small enough to match all of the tracks' time bases). The problem is with the waste of space that follows due to the bloody 16-bit integer offset in Block & SimpleBlock.

So if we're thinking about breaking compatibility anyway, why not think about a whole new SimpleBlock V2 that allows for much larger relative timestamps? Would make all existing players incompatible, though.

Another idea that only wastes space but doesn't destroy existing players' ability to play the file: adding a new child to Block called PreciseRelativeTimestamp or whatever that contains the difference between the timestamp-scaling-based timestamp & the actual, precise one, in nanoseconds. Cannot be used with SimpleBlocks, of course. Will take several bytes per BlockGroup.

I don't like this as storing rounding errors ist imprecise as well (unless the global timestamp scaling factor is a multiple of the rounding error's denominator).

It can be 100% exact. It's the rounding error after all - the number that needs to be added to the "classic" ms timestamp to get the fractional timestamp

I'm also quite unsure which denominator a multiplexer should chose. In order to express a rounding error precisely it must have a much higher resolution that the usual 1ms resolution of Matroska timestamps. For example, with 1001/30000 FPS content the rounding error will always be below one frame duration, therefore you'll have to make the denominator much larger.

It seems the denominator of the rounding error is simply the denominator of the original timestamp. E.g. in this case, the rounding error would have denominator 30000 and nominator (n*1001/30000 - int(n*1001/30000*1000)/1000) * 30000) for frame n or something like this. This is probably wrong, just typing this out casually. Actually probably also needs a constant nominator part (to be stored in the track header) of 1001.

Something else that came to mind when reading our previous discussion that Dave linked to: please keep in mind that any solution that sets values for the whole track in the track header will inevitably fail with mixed frame rate content or content with different interlacing, e.g. when multiplexing from an MPEG transport stream recorded from a DVB broadcast. Those bloody streams change frame rates all the time when the program changes, e.g. when transitioning to and from commercials (or just from an announcement to the movie). With our new and shiny precise timestamp calculation we'll either have to forbid such changes (unrealistic) or provide facilities to signal such changes in the form of some type of global index similar to cues. Unlike cues, though, such an index would have to be mandatory (a file without cues can be played just fine, even seeking works similar to seeking in Ogg files — meaning some kind of binary search).

What does Matroska do if the codec changes? Transport streams can do that, Matroska can't do that. I feel like bringing up such cases just complicates the whole discussion. You can't fix everything at the same time. But you can stall any progress by wanting to consider every possible future feature and requirement.

Besides, as was suggested in a previous post, the denominator part could be overridden per packet. This would cause some bytes of overhead in such obscure cases as mixing multiple framerates that are not known in advance.

In theory Matroska's timestamps can have sample-precision already (just make global timestamp scale small enough to match all of the tracks' time bases). The problem is with the waste of space that follows due to the bloody 16-bit integer offset in Block & SimpleBlock.

I guess you mean the fact that every packet will need its own cluster. But AFAIK that still doesn't give a way to get fractional timestamps? So, not an option.

So if we're thinking about breaking compatibility anyway, why not think about a whole new SimpleBlock V2 that allows for much larger relative timestamps? Would make all existing players incompatible, though.

Obviously not an option. If it were specified, it's likely everyone would disable this by default, except people who use Matroska in special setups where they control producer and consumer.

Another idea that only wastes space but doesn't destroy existing players' ability to play the file: adding a new child to Block called PreciseRelativeTimestamp or whatever that contains the difference between the timestamp-scaling-based timestamp & the actual, precise one, in nanoseconds. Cannot be used with SimpleBlocks, of course. Will take several bytes per BlockGroup.

I thought that was what I proposed here (except I wanted to use fractional numbers).

PS: I think obsessing about a few bytes per packet isn't useful. Having precise timestamps, even if it introduces overhead, is much more important. Nobody will discard Matroska as an option because it doesn't go to the edge of the theoretically possible for saving overhead.

[mbunkus]

What does Matroska do if the codec changes? Transport streams can do that, Matroska can't do that.

True. The difference is that having multiple time bases in the same track is something that exists & works today.

I'm really not trying not prevent progress here, and I'm not talking about each and every possible situation. I am talking about one specific situation that is in wide-spread use today.

What I am trying to prevent is implementing a scheme that's supposed to improve one aspect that simultaneously makes another aspect worse. Hence me talking about ways to signal a change in time base mid-stream. We'd also have to signal a precise timestamp at the point of change in time base so that the player can reconstruct the whole timeline properly without having to read blocks at each change in time base.

[gbooker]

It seems like there are a few ways discussed to correct this:

1. Express the time-base in the track so the demuxer can adjust the timestamps in the file to the closest increment of the time-base
2. Express a fractional error value using a denominator in the track and numerator in the packet so the demuxer can give more precise timestamps
   1. Potentially allow overriding the denominator on a per-packet basis
3. Express a second timestamp using a fractional time-base stored in the track
   1. Potentially allow just expressing the timestamp in a numerator/denominator so as to ignore/override the track's time-base

All of these would still require the current timestamp to still exist and thus would be compatible with current demuxers but newer demuxers would be able to read/derive more precise timestamps.

It seems the denominator of the rounding error is simply the denominator of the original timestamp.

Close. When I saw this first suggested, I did some math and figured out what it would be for the case of 44.1kHz AAC audio (this is what really sparked this conversation; see below). In this case the samples are 1024/44100 seconds long with the MKA using 1ms precision on the timestamps and so the error can be expressed as m/1000 - n*1024/44100 where m is the timestamp in the MKA and n is the packet number. To express the error exactly in integers, the denominator is lcm(1000, 44100) (your basic fractions with common denominator) which in this case is 441000. Using some quick examples:

Packet number	MKA timestamp	Error (using 441,000 as the denominator)
0	0	0
1	23	97
2	46	194
3	70	-150
354	8220	-60
355	8243	37

Also worth noting: the duration would likely need the same treatment.

Aside: I sparked this conversation in an internal discussion with @rcombs about AAC 44.1kHz audio in an MKA format. I was remuxing this to MPEG-TS and the MKA had only 1ms precision timestamps. Well, a simple remux would be multiplying these timestamps by 90 to match MPEG-TS. This simple remux resulted in packets at times 0, 2070, 4140, 6300, 8370 … which gave them effective durations of 0, 2070, 2070, 2160, 2070 and this inconsistency would cause stuttering audio in Apple's HLS demuxer. So this meant that remuxing MKA -> MPEG-TS required opening a codec in lavc to get more precise durations and thus derive timestamps without error.

P.S. These imprecise timestamps were one of the more annoying things we had to deal in Perian's MKV demuxer and that was over a decade ago.

[robUx4]

The reason not to introduce a SimpleBlock v2 is that any hardware/software player that doesn't know it won't be able to play the files. It can be done in Matroska v5. Such files being unreabable by v5 parsers will also be marked as such. We might as well call it Matroska2 or something, just like WebM shares a lot with Matroska.

The practical question is whether there is a convenient way to have precise timestamps in v4 and make it work in existing players (and WebM, I know that's something they want as well).

The question about VFR (Variable Frame Rate) is not really an issue IMO. In the end you only have 1 or 2 frame rates mixed, and maybe with the same denominator. All you need is a fraction that handle both. Facebook even created a timebase that covers most common timebases for video. As long as you know the timebases you'll have to deal with before muxing you should be fine.

An important thing to note is that floating point should not be used at all (we want precision). All we have is the Matroska timebase x/1.000.000.000 s (x=TimestampScale) and the source material timebase(s) (a/44.100, b/48.000, c/24, d*10001/30000, etc). They are all fractions. So we should be able to find something that works with just fractions, using common denominators, fraction reduction, etc. It can get to large numbers very quickly as there are multiple tracks with different timebase (or odd fractions when a track uses VFR, see above).

What we have now is a timestamp for each Block as a fraction of TimestampScale/1.000.000.000.

What we want is a timestamp for each Block as a fraction of the source material. The difference between the two values is still a fraction. We can store this difference as a fraction. And we must also store the source material fraction.

Now we just have to do the math the find this "difference as a fraction". In particular to minimize the storage needed to do so if possible (if not, mandating BlockGroup for precise tracks is always an option). If we can fit in inside the (3) reserved bits of the SimpleBlock it would be perfect.

[t11s]

ISO/IEC 14496-12 "ISO base media file format" uses a "timescale" (counts per second) and "media sample durations". If timescale=30000 and media sample duration is 1001, you get NTSC fractional frame rate.

Similarly, ISO/IEC 14496-10 "Advanced Video Coding" has a clock tick defined as num_units_in_tick divided by the time_scale (see equation C-1). The presence of these in VUI is indicated by the timing_info_present_flag. For NTSC, time_scale may be 30000 and num_units_in_tick may be 1001.

[robUx4]

Following my "pure rational numbers" approach we can say the following, for a Track sampled at the original frequency, stored in a Matroska Segment with TimestampScale:

The real timestamp for each sample S is: real(S) = S / frequency

The Matroska timestamp for the same sample is matroska(S) = S * TimestampScale / 1,000,000,000

The Cluster timestamp is just a value to add to S to get the proper value, so we can skip it for now.
As we just check the rational values, the rounding introduced by divisions is not taken in account.

The difference between the real timestamp and the one we get from Matroska is:

real(S) - matroska(S)
= S / frequency - S * TimestampScale / 1,000,000,000
= (S * 1,000,000,000) / (frequency * 1,000,000,000) - (S * TimestampScale * frequency) / (frequency * 1,000,000,000)
= S * (1,000,000,000 - TimestampScale * frequency) / (frequency * 1,000,000,000)

We can already deduce a few things from this:

• The error grows linearly with the value of S.
• if TimestampScale is exactly 1,000,000,000 / frequency, there is no error.
• the bigger the rounding error of 1,000,000,000 / frequency, the bigger the Matroska and real timestamps will diverge.

That gives some sampling frequencies where it's possible to achieve 0 error per sample:

• Audio: 8000 Hz, 16000 Hz, 32000 Hz, 50000 Hz, 64000 Hz,
• Video: 25 fps, 50 fps, 100 fps

That leaves out a lot of common ones:

• Audio: 11025, 22050, 37800, 44056, 44100, 48000, 50400, 88200, 96000, 176400, 192000, 352800
• Video: 16, 24000/1001, 24, 30000/1001, 30, 60000/1001, 48, 60, 90, 100, 120000/1001, 120

The other way to reduce the error, is to reduce the value of S. We already effectively reduce the value we store to a 16 bits integer, so the value is always between -32,768 and 32,767. If we were to store the error in the remaining 3 bits of a SimpleBlock that's still 13 bits too many.

By limiting the possible values of S in a Cluster to [-4,3] (3 bits), in other words 8 frames, it is possible to store each frame with the Matroska timestamp and the error based on TimestampScale * frequency. This is also feasible because audio is usually not found in 1 samples, but by chunks of samples in one Frame. Sometimes all chunks have the same amount of samples, sometimes not, but each amount of samples is based on the same multiple (worst case scenario is many chunk sizes unrelated).
For video that means at most 8 frames per Cluster, for a 29,97 fps file that's 267ms. This is very small.

A Block has one extra free bit, so we could double these values. That's still very small IMO. And that's the case where the TimestampScale is precisely adjusted for one track. When you have 2 or more, finding a value of TimestampScale that works well with all frequency becomes even harder.

I think the scope where it works, even with the proper muxing guidelines, is too narrow to be worth using all the reserved bits. In particular because common frequencies like 44100 Hz or 30000/1001 fps will introduce errors no matter what and will need to use this system.

There could be other clever ways to do this. We could use a bit in the Block that says the timestamp "shift" is stored after/before the Block data, but that would be incompatible with all existing readers. That would be equivalent to using a new lacing format.

Another way would be to force using a BlockGroup to have precise timing and store the "shift" in a new element. It might only need 16 bits of storage, so that would translate in 3 extra octets per BlockGroup

[robUx4]

It seems one of the aspect of this not discussed is how the rounding of the current system works and how it could be adapted. We assume that we start with the current system and try to fit the correct fraction in there. We may do the other way around, ie have the fraction and use that to set the Block/Cluster timestamp value. The rounding error is then on older parsers assuming a timestamp value when in fact it's another value. But the old system is already known to be imprecise/inaccurate. It's not assumed to be sample precise. So a little more, a little less rounding error should not be a big deal.

What we cannot really do is add some information per-track to modify how the Block/SimpleBlock values are interpreted. That would break backward compatibily. For that we would need BlockV2 and SimpleBlockV2.

So we could store the TimestampScale and a fraction that is the actual fraction it's based one.

Let's see what happens for 29.97fps video, or 30000/1001 Hz. The most accurate TimestampScale is 33,366,667 (nanosecond per frame/lace, rounded). We also store the Segment timestamp fraction as {30000, 1001}:

Frame Number	New Block Value	Old Parser timestamp	Real timestamp	Difference

       0  |               0 |                 0 ns |             0 ns |      0 ns
       1  |               1 |          33366667 ns |      33366666 ns |      1 ns
       2  |               2 |          66733334 ns |      66733333 ns |      1 ns
       3  |               3 |         100100001 ns |     100100000 ns |      1 ns
       4  |               4 |         133466668 ns |     133466666 ns |      2 ns
       5  |               5 |         166833335 ns |     166833333 ns |      2 ns
       6  |               6 |         200200002 ns |     200200000 ns |      2 ns
       7  |               7 |         233566669 ns |     233566666 ns |      3 ns
       8  |               8 |         266933336 ns |     266933333 ns |      3 ns
       9  |               9 |         300300003 ns |     300300000 ns |      3 ns
      10  |              10 |         333666670 ns |     333666666 ns |      4 ns
       .. |              .. |                  ..  |               .. |       ..
   65532  |           65532 |     2186584421844 ns | 2186584400000 ns |  21844 ns
   65533  |           65533 |     2186617788511 ns | 2186617766666 ns |  21845 ns
   65534  |           65534 |     2186651155178 ns | 2186651133333 ns |  21845 ns
   65535  |           65535 |     2186684521845 ns | 2186684500000 ns |  21845 ns

The Old Parser timestamp is the timestamp older parsers would see: Block Value * TimestampScale. The Real timestamp is the one using the fraction: Block Value * 1001 / 30000.

For 44100 Hz audio we get the following, with a TimestampScale of 22,676 (nanosecond per frame/lace, rounded).

Frame Number	New Block Value	Old Parser timestamp	Real timestamp	Difference

       0  |               0 |                 0 ns |             0 ns |      0 ns
       1  |               1 |             22676 ns |         22675 ns |      1 ns
       2  |               2 |             45352 ns |         45351 ns |      1 ns
       3  |               3 |             68028 ns |         68027 ns |      1 ns
       4  |               4 |             90704 ns |         90702 ns |      2 ns
       5  |               5 |            113380 ns |        113378 ns |      2 ns
       6  |               6 |            136056 ns |        136054 ns |      2 ns
       7  |               7 |            158732 ns |        158730 ns |      2 ns
       8  |               8 |            181408 ns |        181405 ns |      3 ns
       9  |               9 |            204084 ns |        204081 ns |      3 ns
      10  |              10 |            226760 ns |        226757 ns |      3 ns
       .. |              .. |                  ..  |               .. |       ..
    1636  |            1636 |          37097936 ns |      37097505 ns |    431 ns
    1637  |            1637 |          37120612 ns |      37120181 ns |    431 ns
    1638  |            1638 |          37143288 ns |      37142857 ns |    431 ns
    1639  |            1639 |          37165964 ns |      37165532 ns |    432 ns
    1640  |            1640 |          37188640 ns |      37188208 ns |    432 ns
       .. |              .. |                  ..  |               .. |       ..
   65532  |           65532 |        1486003632 ns |    1485986394 ns |  17238 ns
   65533  |           65533 |        1486026308 ns |    1486009070 ns |  17238 ns
   65534  |           65534 |        1486048984 ns |    1486031746 ns |  17238 ns
   65535  |           65535 |        1486071660 ns |    1486054421 ns |  17239 ns

The difference is less than one sample.
When packed at 40 samples per frames (shortest packing in @rcombs example). We would then use a fraction of {40, 44100} and a TimestampScale of 907029 :

Frame Number	New Block Value	Old Parser timestamp	Real timestamp	Difference

       0  |               0 |                 0 ns |             0 ns |      0 ns
       1  |               1 |            907029 ns |        907029 ns |      0 ns
       2  |               2 |           1814058 ns |       1814058 ns |      0 ns
       3  |               3 |           2721087 ns |       2721088 ns |     -1 ns
       4  |               4 |           3628116 ns |       3628117 ns |     -1 ns
       5  |               5 |           4535145 ns |       4535147 ns |     -2 ns
       .. |              .. |                  ..  |               .. |       ..
   47392  |           47392 |       42985918368 ns |   42985941043 ns | -22675 ns
   47393  |           47393 |       42986825397 ns |   42986848072 ns | -22675 ns
   47394  |           47394 |       42987732426 ns |   42987755102 ns | -22676 ns
   47395  |           47395 |       42988639455 ns |   42988662131 ns | -22676 ns
   47396  |           47396 |       42989546484 ns |   42989569160 ns | -22676 ns
   47397  |           47397 |       42990453513 ns |   42990476190 ns | -22677 ns
       .. |              .. |                  ..  |               .. |       ..
   65533  |           65533 |       59440331457 ns |   59440362811 ns | -31354 ns
   65534  |           65534 |       59441238486 ns |   59441269841 ns | -31355 ns
   65535  |           65535 |       59442145515 ns |   59442176870 ns | -31355 ns

We get less than 1 sample error with 47393 frames stored, or 42s worth of samples in a Cluster.

The worst case scenario is the highest, not easily divisible, frequency 352800. It gives:

Frame Number	New Block Value	Old Parser timestamp	Real timestamp	Difference

       0  |               0 |                 0 ns |             0 ns |      0 ns
       1  |               1 |              2834 ns |          2834 ns |      0 ns
       2  |               2 |              5668 ns |          5668 ns |      0 ns
       3  |               3 |              8502 ns |          8503 ns |     -1 ns
       4  |               4 |             11336 ns |         11337 ns |     -1 ns
       5  |               5 |             14170 ns |         14172 ns |     -2 ns
       .. |              .. |                  ..  |               .. |       ..
    6066  |            6066 |          17191044 ns |      17193877 ns |  -2833 ns
    6067  |            6067 |          17193878 ns |      17196712 ns |  -2834 ns
    6068  |            6068 |          17196712 ns |      17199546 ns |  -2834 ns
    6069  |            6069 |          17199546 ns |      17202380 ns |  -2834 ns
       .. |              .. |                  ..  |               .. |       ..
   65533  |           65533 |         185720522 ns |     185751133 ns | -30611 ns
   65534  |           65534 |         185723356 ns |     185753968 ns | -30612 ns
   65535  |           65535 |         185726190 ns |     185756802 ns | -30612 ns

Here we achieve less than on sample error when there is less than 6067 samples in a Cluster. This can be doubled by using signed values for the Block timestamp value. The range to get less than one sample error becomes [-6067,6066]. And by packing samples by at least 11 samples, we always get less than 1 sample error. With 22 samples we get less than half a sample duration error, which should be enough with rounding.

So with single track files we can probably achieve sample precision easily.

With mixed frequencies it becomes more complicated. For example the 29.97 fps video with the 44100 Hz audio. We have 1001/30000 and 1/44100 so the fraction to use would be 1001/reduced(30000, 44100) where reduced(A, B) is each number multiplied and divided by their Greatest Common Denominator. In this case (30000*44100)/100 = 13230000. That gives a round TimestampScale of 75661 ns/tick.

That gives these Blocks:

Block Value	Old Parser timestamp	Real timestamp	Difference

      0 |                 0 ns |             0 ns |      0 ns
      1 |             75661 ns |         75661 ns |      0 ns
      2 |            151322 ns |        151322 ns |      0 ns
      3 |            226983 ns |        226984 ns |     -1 ns
      4 |            302644 ns |        302645 ns |     -1 ns
      5 |            378305 ns |        378306 ns |     -1 ns
      6 |            453966 ns |        453968 ns |     -2 ns
      7 |            529627 ns |        529629 ns |     -2 ns
      8 |            605288 ns |        605291 ns |     -3 ns
      9 |            680949 ns |        680952 ns |     -3 ns
     10 |            756610 ns |        756613 ns |     -3 ns
     11 |            832271 ns |        832275 ns |     -4 ns
     12 |            907932 ns |        907936 ns |     -4 ns
     .. |                   .. |               .. |        ..
    441 |          33366501 ns |      33366666 ns |   -165 ns
     .. |                   .. |               .. |        ..
    882 |          66733002 ns |      66733333 ns |   -331 ns
     .. |                   .. |               .. |        ..
   1323 |         100099503 ns |     100100000 ns |   -497 ns
     .. |                   .. |               .. |        ..
  32634 |        2469121074 ns |    2469133333 ns | -12259 ns

For the video track we would get something like this:

Frame Number	New Block Value	Old Parser timestamp	Real timestamp	Difference

       0  |               0 |                 0 ns |             0 ns |      0 ns
       1  |             441 |          33366501 ns |      33366666 ns |   -165 ns
       2  |             882 |          66733002 ns |      66733333 ns |   -331 ns
       3  |            1323 |         100099503 ns |     100100000 ns |   -497 ns
       .. |              .. |                  ..  |               .. |       ..
      74  |           32634 |        2469121074 ns |    2469133333 ns | -12259 ns
       .. |              .. |                  ..  |               .. |       ..
     148  |           65268 |        4938242148 ns |    4938266666 ns | -24518 ns

We can store almost 5s In a Cluster.

For the audio track, on the other hand, we cannot recover each sample easily

Sample Number	Real timestamp	Block Value

        0  |            0 ns |          0
        1  |        22675 ns |         ~0
        2  |        45351 ns |         ~1
        3  |        68027 ns |         ~1
        4  |        90702 ns |         ~1
        5  |       113378 ns |         ~1
        6  |       136054 ns |         ~2
        7  |       158730 ns |         ~2
        .. |              .. |         ..
      300  |      6802721 ns |        ~90

The Block Value doesn't map to an exact sample timestamp (and vice versa).

It seems that if we apply a factor of 3 we may get better results. So we could have a Segment fraction of 1001/3*13230000, with a rounded TimestampScale of 25220 ns/tick.

Sample Number	Block Value	Old Parser timestamp	Real timestamp	Difference

        0  |           0 |                 0 ns |             0 ns |        0 ns
        1  |           1 |             25220 ns |         22675 ns |     2545 ns
        2  |           2 |             50440 ns |         45351 ns |     5089 ns
        3  |           3 |             75660 ns |         68027 ns |     7633 ns
        4  |           4 |            100880 ns |         90702 ns |    10178 ns
        5  |           5 |            126100 ns |        113378 ns |    12722 ns
        6  |           6 |            151320 ns |        136054 ns |    15266 ns
        7  |           7 |            176540 ns |        158730 ns |    17810 ns
        8  |           8 |            201760 ns |        181405 ns |    20355 ns
        9  |           9 |            226980 ns |        204081 ns |    22899 ns
       10  |          10 |            252200 ns |        226757 ns |    25443 ns
       ..  |          .. |                   .. |               .. |          ..
    65534  |       65534 |        1652767480 ns |    1486031746 ns | 166735734 ns
    65535  |       65535 |        1652792700 ns |    1486054421 ns | 166738279 ns

We lose about 1 sample precision every 10 samples, or 10%. For a full Block that's about 166ms shift (or rather half when using signed 16 bits). That's a lot. Even packed at 40 samples per frame that's still about 20ms, when such a frame is 1 ms.

If we use the full fraction {1001, 30000*44100} we cannot store more than one video frame per Cluster.

There doesn't seem to be a system where it works by storing the Block value as a real fraction value. At least when mixing "heterogeneous" frequencies. It works with single tracks or frequency that are easily divisible. And not if we want to keep backward compatibility (Block/SimpleBlock).

[robUx4]

A little background on this, for adaptive streaming it's important when you switch to one "quality" (representation) to another one to switch exactly the frame and audio you want. I don't know if they are sample exact for audio, especially as each codec (or different encoding parameters) may pack a different amount of samples per frame. So the boundaries don't totally overlap. Maybe there's an offset that tells on which sample to start. Or an exact clock gives the exact timestamp for each sample in each representation anyway.

Give that, the important phrase here is

So with single track files we can probably achieve sample precision easily.

In adaptive streaming you don't (usually) use muxed tracks. So you can pick each channel independently with the best possible choice at any given time. So in these conditions we can be sample precise. All we need is to tell the original clock (numerator/denominator) of the Track. A new parser would use that value with the Block timestamp value. Older parsers would not see it and would use the Block timestamp value with the global TimestampScale. As described above, the difference is minimal, as long as the TimestampScale is matching the fraction.

I'll send a proposal for new elements to store this fraction and the necessary changes on how to interpret the timestamps.