IndexError: bytearray index out of range intermittently

[sitkevij]

To reproduce please run the following, we are seeing bytearray index out of range intermittently.

python hls-analyzer.py "https://cp541868-vh.akamaihd.net/i/mtvnorigin/gsp.alias/mediabus/mtv.com/2017/01/31/10/32/54/299d114ee3854f93a52389da76944592/1556899/1ftyUd0NW7_HDTNW610A4_1556899_7122114_,384x216_278,512x288_498,640x360_1028,768x432_1528,960x540_2128,1280x720_3128,1920x1080_5128,.mp4.csmil/master.m3u8?hdnea=st%3D1486567247%7Eexp%3D1487172047%7Eacl%3D%2Fi%2Fmtvnorigin%2Fgsp.alias%2Fmediabus%2Fmtv.com%2F2017%2F01%2F31%2F10%2F32%2F54%2F299d114ee3854f93a52389da76944592%2F1556899%2F1ftyUd0NW7_HDTNW610A4_1556899_7122114_%2C384x216_278%2C512x288_498%2C640x360_1028%2C768x432_1528%2C960x540_2128%2C1280x720_3128%2C1920x1080_5128%2C.mp4.csmil%2F*%7Ehmac%3Df8d18581e1bfd5b46a86b3c8f42182939fad51876dc7486d223c893f1620bb4b&__a__=off&__b__=450&__viacc__=NONE" >out.txt
Traceback (most recent call last):
  File "hls-analyzer.py", line 210, in <module>
    analyze_variant(m3u8.load(playlist.absolute_uri), playlist.stream_info.bandwidth)
  File "hls-analyzer.py", line 63, in analyze_variant
    analyze_segment(variant.segments[i], bw, variant.media_sequence + i)
  File "hls-analyzer.py", line 167, in analyze_segment
    ts_parser.prepare()
  File "/home/ubuntu/workspace/ts_segment.py", line 33, in prepare
    self._readHeader()
  File "/home/ubuntu/workspace/ts_segment.py", line 94, in _readHeader
    self._processTSPacket(packet)
  File "/home/ubuntu/workspace/ts_segment.py", line 120, in _processTSPacket
    self._parseProgramId(payload_unit_start_indicator, packetParser)
  File "/home/ubuntu/workspace/ts_segment.py", line 137, in _parseProgramId
    packetParser.readBits(12)
  File "/home/ubuntu/workspace/bitreader.py", line 43, in readBits
    return self.readBitsLong(n)
  File "/home/ubuntu/workspace/bitreader.py", line 53, in readBitsLong
    retVal |= (self.readUnsignedByte() << n)
  File "/home/ubuntu/workspace/bitreader.py", line 75, in readUnsignedByte
    value = ((self.data[self.byteOffset]) << self.bitOffset) | ((self.data[self.byteOffset + 1]) >> (8 - self.bitOffset))
IndexError: bytearray index out of range

[jeoliva]

@sitkevij, unfortunately I don't have access to that stream. I get an HTTP 403 response because the security token has expired (Akamai token, the one provided in the query string parameter hdnea). If you provide a stream url with an enough long duration token I will take a look.

[sitkevij]

TTL temporarily extended

https://cp541868-vh.akamaihd.net/i/mtvnorigin/gsp.alias/mediabus/mtv.com/2017/01/31/10/32/54/299d114ee3854f93a52389da76944592/1556899/1ftyUd0NW7_HDTNW610A4_1556899_7122114_,384x216_278,512x288_498,640x360_1028,768x432_1528,960x540_2128,1280x720_3128,1920x1080_5128,.mp4.csmil/master.m3u8?hdnea=st%3D1487173811%7Eexp%3D1488383411%7Eacl%3D%2Fi%2Fmtvnorigin%2Fgsp.alias%2Fmediabus%2Fmtv.com%2F2017%2F01%2F31%2F10%2F32%2F54%2F299d114ee3854f93a52389da76944592%2F1556899%2F1ftyUd0NW7_HDTNW610A4_1556899_7122114_%2C384x216_278%2C512x288_498%2C640x360_1028%2C768x432_1528%2C960x540_2128%2C1280x720_3128%2C1920x1080_5128%2C.mp4.csmil%2F*%7Ehmac%3D69ebdac7b97a4411805d4946b0eff9f275d1a7f5649e4043c43bb53dea367c3f&__a__=off&__b__=450&__viacc__=NONE`

[jeoliva]

@sitkevij , I don't get the issue once I used a valid token. For how long are you testing this stream? I am thinking on 2 possible reasons for this error:

• Using a short duration token that has already expired (hdnea query string parameter). I need to manage this situation in the app instead of returning a horrible exception message.

• If you test the stream for a long time (many segments) the long duration token (generated by Akamai after the first request) could also expire and then an error will be raised.

I will take a look to both situations.

Many thanks for your help

[sitkevij]

Hi @jeoliva - I can indeed replicate the issue, here's the latest run. Hope that helps.

$ python --version
Python 2.7.6

***** Analyzing variant (1139000) *****

        ** Generic information **
        Version: 3
        Start Media sequence: 1
        Is Live: False
        Encrypted: True
        Number of segments: 58
        Playlist duration: 342.008

        ** Downloading https://cp541868-vh.akamaihd.net/i/mtvnorigin/gsp.alias/mediabus/mtv.com/2017/01/31/10/32/54/299d114ee3854f93a52389da76944592/1556899/1ftyUd0NW7_HDTNW610A4_1556899_7122114_,384x216_278,512x288_498,640x360_1028,768x432_1528,960x540_2128,1280x720_3128,1920x1080_5128,.mp4.csmil/segment1_4_av.ts?null=0&id=AgBoxiOLF9eCZzI4plgmcCbbgHEtR1bpgZW%2fVLnHr671TiXfdEMul2coWphAF92DHJO0bgGA6RyKjg%3d%3d&hdntl=exp=1487374770~acl=/i/mtvnorigin/gsp.alias/mediabus/mtv.com/2017/01/31/10/32/54/299d114ee3854f93a52389da76944592/1556899/1ftyUd0NW7_HDTNW610A4_1556899_7122114_,384x216_278,512x288_498,640x360_1028,768x432_1528,960x540_2128,1280x720_3128,1920x1080_5128,.mp4.csmil/*~data=hdntl~hmac=dc26447e81a3d8df2e6d351895940db67940d8acb72be74877f2548c5a781174, Range: None **
Traceback (most recent call last):
  File "hls-analyzer.py", line 210, in <module>
    analyze_variant(m3u8.load(playlist.absolute_uri), playlist.stream_info.bandwidth)
  File "hls-analyzer.py", line 63, in analyze_variant
    analyze_segment(variant.segments[i], bw, variant.media_sequence + i)
  File "hls-analyzer.py", line 167, in analyze_segment
    ts_parser.prepare()
  File "/home/ubuntu/workspace/ts_segment.py", line 33, in prepare
    self._readHeader()
  File "/home/ubuntu/workspace/ts_segment.py", line 94, in _readHeader
    self._processTSPacket(packet)
  File "/home/ubuntu/workspace/ts_segment.py", line 120, in _processTSPacket
    self._parseProgramId(payload_unit_start_indicator, packetParser)
  File "/home/ubuntu/workspace/ts_segment.py", line 137, in _parseProgramId
    packetParser.readBits(12)
  File "/home/ubuntu/workspace/bitreader.py", line 43, in readBits
    return self.readBitsLong(n)
  File "/home/ubuntu/workspace/bitreader.py", line 53, in readBitsLong
    retVal |= (self.readUnsignedByte() << n)
  File "/home/ubuntu/workspace/bitreader.py", line 75, in readUnsignedByte
    value = ((self.data[self.byteOffset]) << self.bitOffset) | ((self.data[self.byteOffset + 1]) >> (8 - self.bitOffset))
IndexError: bytearray index out of range

[Taffjones]

Bump

Same issue here - is the project still being maintained?