# Bug: Critical LEB128 Encoding Error in AV1 Codec Causes Browser Playback Issues

[neil-mao]

🐛 Bug Description

The LEB128 encoding implementation in rtc-rtp/src/codec/av1/leb128.rs contains a critical algorithmic error that causes incorrect encoding for all values greater than 127. This results in malformed AV1 RTP packets that cause severe playback issues (stuttering, frame drops) in web browsers.

🔍 Root Cause Analysis

The Problematic Code

File: rtc-master/rtc-rtp/src/codec/av1/leb128.rs

pub fn encode_leb128(mut val: u32) -> u32 {
    let mut b = 0;
    loop {
        b |= val & 0b_0111_1111;
        val >>= 7;
        if val != 0 {
            b |= 0b_1000_0000;
            b <<= 8;  // ❌ CRITICAL BUG: This destroys byte boundaries
        } else {
            return b;
        }
    }
}

impl BytesMutExt for BytesMut {
    fn put_leb128(&mut self, n: u32) {
        let mut encoded = encode_leb128(n);
        while encoded >= 0b_1000_0000 {
            self.put_u8(0b_1000_0000 | (encoded & 0b_0111_1111) as u8);
            encoded >>= 7;
        }
        self.put_u8(encoded as u8);
    }
}

Why It's Wrong

The encode_leb128 function attempts to pack multiple bytes into a single u32 by using left shift operations (b <<= 8). This approach fundamentally breaks the LEB128 encoding rules:

1. Destroys byte boundaries: Shifting by 8 bits corrupts the data structure
2. Generates incorrect length: Produces more bytes than necessary
3. Wrong content: The actual byte values are completely wrong

Detailed Example: Encoding Value 128

Correct LEB128 encoding should be: [0x80, 0x01] (2 bytes)

What the buggy code produces: [129, 128, 2] (3 bytes) ❌

Step-by-step breakdown of the bug:

First iteration:
  b = 0 | (128 & 0x7F) = 0 | 0 = 0
  val = 128 >> 7 = 1
  val != 0, so b = 0 | 0x80 = 0x80
  b <<= 8 → b = 0x8000  ← ERROR HERE!

Second iteration:
  b = 0x8000 | (1 & 0x7F) = 0x8001
  val = 0
  Returns 0x8001 (32769)

Then in put_leb128:
  Iteration 1: Output 0x80 | (32769 & 0x7F) = 129
  Iteration 2: Output 0x80 | (256 & 0x7F) = 128  
  Final: Output 2
  Result: [129, 128, 2] ❌ WRONG!

📊 Test Results

OBU Size	Official (Buggy)	Correct Encoding	Difference
128	[129, 128, 2] (3 bytes)	[128, 1] (2 bytes)	❌ +1 byte, wrong content
256	[130, 128, 2] (3 bytes)	[128, 2] (2 bytes)	❌ +1 byte, wrong content
1000	[135, 208, 3] (3 bytes)	[232, 7] (2 bytes)	❌ +1 byte, wrong content
2048	[144, 128, 2] (3 bytes)	[128, 16] (2 bytes)	❌ +1 byte, wrong content

Critical finding: ALL values greater than 127 produce incorrect encoding!

💥 Impact on Users

When browsers receive RTP packets with incorrectly encoded LEB128 OBU sizes:

1. OBU boundary parsing fails: Browsers cannot correctly determine OBU sizes
2. Cascading errors: One misparsed OBU causes all subsequent OBUs to be misaligned
3. Frame decoding failures: Decoders encounter invalid bitstreams and drop frames
4. Poor user experience: Video playback exhibits stuttering, frame drops, and visual artifacts

Real-world symptom: AV1 video streams play with noticeable stuttering and frame skipping in web browsers when using the official Av1Payloader.

✅ Proposed Fix

Replace the buggy implementation with a correct one that follows the LEB128 specification:

/// Encode value to LEB128 format (correct implementation)
pub trait BytesMutExt {
    fn put_leb128(&mut self, n: u32);
}

impl BytesMutExt for BytesMut {
    fn put_leb128(&mut self, mut value: u32) {
        loop {
            let mut byte = (value & 0x7f) as u8;
            value >>= 7;
            if value != 0 {
                byte |= 0x80;  // Set continuation bit
            }
            self.put_u8(byte);
            if value == 0 {
                break;
            }
        }
    }
}

This implementation:

• ✅ Outputs bytes one at a time, following LEB128 spec
• ✅ Correctly handles continuation bits
• ✅ Generates compact and correct encoding
• ✅ Matches the standard LEB128 encoding used by all other implementations

🧪 Verification

We created test tools to verify the fix works correctly:

# Test various values
for value in 0 1 127 128 255 256 1000 2048 16383 16384; do
    echo "Value $value:"
    echo "  Correct: $(python3 -c "
val = $value
result = []
while True:
    byte = val & 0x7f
    val >>= 7
    if val != 0:
        byte |= 0x80
    result.append(byte)
    if val == 0:
        break
print(result)
")"
done

📝 Additional Context

LEB128 Specification Reference

LEB128 (Little-Endian Base 128) is a variable-length integer encoding:

• Each byte uses 7 bits for data (bits 0-6)
• Bit 7 is the continuation flag:
  • 1: More bytes follow
  • 0: This is the last byte

Example encoding of value 128:

Binary: 10000000
Step 1: Low 7 bits = 0000000, needs continuation → 10000000 (0x80)
Step 2: Remaining = 10000000 >> 7 = 1, no continuation → 00000001 (0x01)
Result: [0x80, 0x01] = [128, 1] ✅

🎯 Severity

High - This bug affects all users of rtc-rtp who use AV1 codec for RTP streaming. Any OBU larger than 127 bytes will be incorrectly encoded, which is extremely common in real-world video streams.

📋 Environment

• Library: rtc-rtp
• Version: Latest (as of April 2026)
• Affected Component: src/codec/av1/leb128.rs
• Platform: All platforms

🔗 Related Files

• Buggy implementation: rtc-master/rtc-rtp/src/codec/av1/leb128.rs
• Usage in payloader: rtc-master/rtc-rtp/src/codec/av1/mod.rs

🙏 Request

Please review and merge this critical fix. The current implementation is fundamentally broken and causes real-world playback issues for all AV1 users.

---