The output of a decoder produced from Encoding.NewDecoder differs depending on how you chunk the input to it. I noticed these differences:
- The decoder may ignore "internal" padding (
= characters not at the end of the stream). For example, decoding ["QQ==Qg=="] (correctly) results in an error, but ["QQ==", "Qg=="] (incorrectly) decodes to "AB".
- The byte offset in error messages may get reset to 0 instead of indicating the absolute offset in the stream. For example, decoding
["AAAA####"] says the error occurs at offset 4, but decoding ["AAAA" "####"] says the error occurs at offset 0.
I think that the output of a decoder should always be the same as if the entire Reader were serialized to a string and then passed to DecodeString.
Item 1 is more important IMO. Item 2 was unexpected but I can live with inconsistent byte offsets in error messages. However seeing as CorruptInputError is already an int64, it would be nice to have if it doesn't complicate the internals too much.
This bug is somewhat similar to #25296 for encoding/base32.
What version of Go are you using (go version)?
$ go version
go version go1.11.5 linux/amd64
Does this issue reproduce with the latest release?
Yes, using go1.12 on play.golang.org
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
What did you do?
https://play.golang.org/p/6rcDYtro36S
package main
import (
"encoding/base64"
"fmt"
"io"
"io/ioutil"
)
func test(chunks []string) {
fmt.Printf("\n")
fmt.Printf("%+q\n", chunks)
pr, pw := io.Pipe()
go func() {
for _, chunk := range chunks {
pw.Write([]byte(chunk))
}
pw.Close()
}()
dec := base64.NewDecoder(base64.StdEncoding, pr)
output, err := ioutil.ReadAll(dec)
fmt.Printf("%+q %v\n", output, err)
}
func main() {
fmt.Printf("DecodeString(%+q)\n", "QQ==Qg==")
output, err := base64.StdEncoding.DecodeString("QQ==Qg==")
fmt.Printf("%+q %v\n", output, err)
fmt.Printf("DecodeString(%+q)\n", "AAAA####")
output, err = base64.StdEncoding.DecodeString("AAAA####")
fmt.Printf("%+q %v\n", output, err)
for _, chunks := range [][]string{
{"QQ==Qg=="},
{"Q", "Q==Qg=="},
{"QQ==", "Qg=="},
{"QQ==Qg=", "="},
{"Q", "Q", "=", "=", "Q", "g", "=", "="},
{"AAAA####"},
{"AAAA", "####"},
} {
test(chunks)
}
}
What did you expect to see?
DecodeString("QQ==Qg==")
"A" illegal base64 data at input byte 4
DecodeString("AAAA####")
"\x00\x00\x00" illegal base64 data at input byte 4
["QQ==Qg=="]
"A" illegal base64 data at input byte 4
["Q" "Q==Qg=="]
"A" illegal base64 data at input byte 4
["QQ==" "Qg=="]
"A" illegal base64 data at input byte 4
["QQ==Qg=" "="]
"A" illegal base64 data at input byte 4
["Q" "Q" "=" "=" "Q" "g" "=" "="]
"A" illegal base64 data at input byte 4
["AAAA####"]
"\x00\x00\x00" illegal base64 data at input byte 4
["AAAA" "####"]
"\x00\x00\x00" illegal base64 data at input byte 4
What did you see instead?
DecodeString("QQ==Qg==")
"A" illegal base64 data at input byte 4
DecodeString("AAAA####")
"\x00\x00\x00" illegal base64 data at input byte 4
["QQ==Qg=="]
"A" illegal base64 data at input byte 4
["Q" "Q==Qg=="]
"A" illegal base64 data at input byte 4
["QQ==" "Qg=="]
"AB" <nil>
["QQ==Qg=" "="]
"AB" <nil>
["Q" "Q" "=" "=" "Q" "g" "=" "="]
"AB" <nil>
["AAAA####"]
"\x00\x00\x00" illegal base64 data at input byte 4
["AAAA" "####"]
"\x00\x00\x00" illegal base64 data at input byte 0
The output of a decoder produced from Encoding.NewDecoder differs depending on how you chunk the input to it. I noticed these differences:
=characters not at the end of the stream). For example, decoding["QQ==Qg=="](correctly) results in an error, but["QQ==", "Qg=="](incorrectly) decodes to"AB".["AAAA####"]says the error occurs at offset 4, but decoding["AAAA" "####"]says the error occurs at offset 0.I think that the output of a decoder should always be the same as if the entire Reader were serialized to a string and then passed to DecodeString.
Item 1 is more important IMO. Item 2 was unexpected but I can live with inconsistent byte offsets in error messages. However seeing as CorruptInputError is already an int64, it would be nice to have if it doesn't complicate the internals too much.
This bug is somewhat similar to #25296 for encoding/base32.
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes, using go1.12 on play.golang.org
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
https://play.golang.org/p/6rcDYtro36S
What did you expect to see?
What did you see instead?