panic: runtime error: index out of range error when using random partition keys

[anappi-vc]

Occasionally, I will get the following error:

panic: runtime error: index out of range [-2]

goroutine 18 [running, locked to thread]:
math/rand.(*rngSource).Uint64(...)
        /home/.gimme/versions/go1.19.5.linux.amd64/src/math/rand/rng.go:249
math/rand.(*rngSource).Int63(0x2c000d19150?)
        /home/.gimme/versions/go1.19.5.linux.amd64/src/math/rand/rng.go:234 +0x9f
math/rand.(*Rand).Int63(...)
        /home/.gimme/versions/go1.19.5.linux.amd64/src/math/rand/rand.go:84
math/rand.(*Rand).Int31(...)
        /home/.gimme/versions/go1.19.5.linux.amd64/src/math/rand/rand.go:98
math/rand.(*Rand).Int31n(0x2c0004952c0, 0x3e)
        /home/.gimme/versions/go1.19.5.linux.amd64/src/math/rand/rand.go:133 +0x59
math/rand.(*Rand).Intn(0x2c000407c70?, 0x0?)
        /home/.gimme/versions/go1.19.5.linux.amd64/src/math/rand/rand.go:171 +0x2e
github.com/aws/amazon-kinesis-streams-for-fluent-bit/util.(*RandomStringGenerator).RandomString(...)
        /kinesis-streams/util/random.go:31
github.com/aws/amazon-kinesis-streams-for-fluent-bit/kinesis.(*OutputPlugin).AddRecord(0x2c000407c70, 0x2c0004b3ce8, 0x2c000a4a040?, 0x7?)
        /kinesis-streams/kinesis/kinesis.go:269 +0x785
main.unpackRecords(0x2c000407c70, 0x2c0004117d0?, 0xe6100)
        /kinesis-streams/fluent-bit-kinesis.go:251 +0x17c
main.FLBPluginFlushCtx(0x7fa166131398?, 0x2c00008a000?, 0x0?, 0x2c000182301?)
        /kinesis-streams/fluent-bit-kinesis.go:206 +0xba

It occurs randomly, so it has been very difficult to reproduce, but here is my conf file:

[INPUT]
    Name              tail
    Tag               var-log
    Path              /var/log/router/*.log
    Path_Key          source_file
    DB                /var/log/flb_service.db
    DB.locking        true
    Refresh_Interval  10
    Rotate_Wait       30
    Buffer_Max_Size   64k
    Skip_Long_Lines   On

[OUTPUT]
    Name                kinesis
    Match               *
    region              local
    stream              my-stream
    compression         gzip
    workers             2
    endpoint            http://localstack:4566
    Retry_Limit       no_limits

This example is using localstack but I have seen it happen in production as well.

I have fluent bit running in a container using the image amazon/aws-for-fluent-bit:latest.

I found this issue on the go lang issue tracker which suggests that this line would be causing the problem. It seems to be a concurrency issue, perhaps from using multiple workers(?)

The work around I have been using for now is to manually set the partition key.

If you need any more information, I will be happy to provide it.