AsciiJSON silently corrupts non-BMP characters (emoji) by emitting malformed \u escapes

[xd-sarthak]

Description

render.AsciiJSON corrupts any Unicode code point above U+FFFF (non-BMP characters such as emoji). It escapes every non-ASCII rune with fmt.Appendf(buf, "\\u%04x", r), but %04x is a minimum width, not a fixed width. A non-BMP rune needs 5+ hex digits, so a single 5-digit \uXXXXX token is written. JSON \u escapes are exactly 4 hex digits, so a decoder reads the first 4 as one character and the trailing digit(s) as literal text.

The output is still syntactically valid JSON (so this is not a parse error and is easy to miss), but the value is silently wrong — it does not round-trip.

Per RFC 8259 §7, code points outside the BMP must be encoded as a UTF-16 surrogate pair (e.g. U+1F600 → \ud83d\ude00).

Example — 😀 (U+1F600):

	value
Input	😀
AsciiJSON output	{"msg":"\u1f600"}
After json.Unmarshal	ὠ0 (U+1F60 ὠ + literal 0)
Expected after round-trip	😀

Offending code — render/json.go (AsciiJSON.Render):

for _, r := range bytesconv.BytesToString(ret) {
    if r > unicode.MaxASCII {
        escapeBuf = fmt.Appendf(escapeBuf[:0], "\\u%04x", r) // 5+ digits for non-BMP → invalid escape
        buffer.Write(escapeBuf)
    } else {
        buffer.WriteByte(byte(r))
    }
}

Related (but distinct — that one is about naming/content-type/deprecation, not this corruption): #2546.

Gin Version

v1.12.0 (master, commit d83f866)

Can you reproduce the bug?

Yes.

Steps:

1. Render any payload containing a non-BMP character via c.AsciiJSON(...) (or render.AsciiJSON).
2. Observe the response body contains a malformed \uXXXXX escape.
3. json.Unmarshal the body — the decoded value differs from the original.

Source Code

Minimal reproducible example:

package main

import (
	"encoding/json"
	"fmt"
	"net/http/httptest"

	"github.com/gin-gonic/gin/render"
)

func main() {
	w := httptest.NewRecorder()
	_ = render.AsciiJSON{Data: map[string]string{"msg": "😀"}}.Render(w)

	out := w.Body.String()
	fmt.Println("rendered:", out) // {"msg":"\u1f600"}

	var decoded map[string]string
	_ = json.Unmarshal([]byte(out), &decoded)
	fmt.Printf("decoded: %q\n", decoded["msg"]) // "ὠ0", expected "😀"
}

Go Version

go1.25.0

Operating System

Linux 6.17.0 x86_64 (codec-independent — the defect is in Gin's ASCII-escaping loop, not the JSON marshaler).

[Bury-Lee]

Thank you for discovering and analyzing this issue in detail!

I ran your test code and added more non-BMP characters (including emojis and CJK Extension B characters). The results fully confirm your conclusion.

Test Results

package main

import (
	"encoding/json"
	"fmt"
	"net/http/httptest"

	"github.com/gin-gonic/gin/render"
)

func main() {
	testCases := map[string]string{
		"grinning_face":     "😀",  // U+1F600
		"crying_laugh":      "😂",  // U+1F602
		"rocket":            "🚀",  // U+1F680
		"heart":             "❤️", // U+2764 + U+FE0F (within BMP)
		"dragon":            "🐉",  // U+1F409
		"alien":             "👽",  // U+1F47D
		"ancient_character": "𠀀",  // U+20000 (CJK Extension B)
	}

	for name, original := range testCases {
		data := map[string]string{"msg": original}
		w := httptest.NewRecorder()

		_ = render.AsciiJSON{Data: data}.Render(w)
		output := w.Body.String()

		var decoded map[string]string
		_ = json.Unmarshal([]byte(output), &decoded)

		fmt.Printf("Test: %s\n", name)
		fmt.Printf("  Original: %q (code points: %U)\n", original, []rune(original))
		fmt.Printf("  Rendered: %s\n", output)
		fmt.Printf("  Decoded:  %q\n", decoded["msg"])
		fmt.Printf("  Round-trip: %v\n\n", original == decoded["msg"])
	}
}

Output:

=== AsciiJSON Non-BMP Character Corruption Test ===

Test: ancient_character
  Original: "𠀀" (code points: [U+20000])
  Rendered: {"msg":"\u20000"}
  Decoded:  "\u20000"
  Round-trip: false

Test: grinning_face
  Original: "😀" (code points: [U+1F600])
  Rendered: {"msg":"\u1f600"}
  Decoded:  "ὠ0"
  Round-trip: false

Test: crying_laugh
  Original: "😂" (code points: [U+1F602])
  Rendered: {"msg":"\u1f602"}
  Decoded:  "ὠ2"
  Round-trip: false

Test: rocket
  Original: "🚀" (code points: [U+1F680])
  Rendered: {"msg":"\u1f680"}
  Decoded:  "Ὠ0"
  Round-trip: false

Test: heart
  Original: "❤️" (code points: [U+2764 U+FE0F])
  Rendered: {"msg":"\u2764\ufe0f"}
  Decoded:  "❤️"
  Round-trip: true

Test: dragon
  Original: "🐉" (code points: [U+1F409])
  Rendered: {"msg":"\u1f409"}
  Decoded:  "ὀ9"
  Round-trip: false

Test: alien
  Original: "👽" (code points: [U+1F47D])
  Rendered: {"msg":"\u1f47d"}
  Decoded:  "\u1f47d"
  Round-trip: false

Issue Confirmed

Scenario	Result
BMP characters (U+0000 ~ U+FFFF)	✅ Working (e.g., ❤️)
Non-BMP characters (U+10000 and above)	❌ Corrupted, cannot round-trip

I will try to fix this and submit a PR.

[xd-sarthak]

Thank you for discovering and analyzing this issue in detail!

I ran your test code and added more non-BMP characters (including emojis and CJK Extension B characters). The results fully confirm your conclusion.

Test Results

package main

import (
"encoding/json"
"fmt"
"net/http/httptest"

"github.com/gin-gonic/gin/render"
)

func main() {
testCases := map[string]string{
"grinning_face": "😀", // U+1F600
"crying_laugh": "😂", // U+1F602
"rocket": "🚀", // U+1F680
"heart": "❤️", // U+2764 + U+FE0F (within BMP)
"dragon": "🐉", // U+1F409
"alien": "👽", // U+1F47D
"ancient_character": "𠀀", // U+20000 (CJK Extension B)
}

for name, original := range testCases {
data := map[string]string{"msg": original}
w := httptest.NewRecorder()

  _ = render.AsciiJSON{Data: data}.Render(w)
  output := w.Body.String()

  var decoded map[string]string
  _ = json.Unmarshal([]byte(output), &decoded)

  fmt.Printf("Test: %s\n", name)
  fmt.Printf("  Original: %q (code points: %U)\n", original, []rune(original))
  fmt.Printf("  Rendered: %s\n", output)
  fmt.Printf("  Decoded:  %q\n", decoded["msg"])
  fmt.Printf("  Round-trip: %v\n\n", original == decoded["msg"])

}
}
Output:

=== AsciiJSON Non-BMP Character Corruption Test ===

Test: ancient_character
  Original: "𠀀" (code points: [U+20000])
  Rendered: {"msg":"\u20000"}
  Decoded:  "\u20000"
  Round-trip: false

Test: grinning_face
  Original: "😀" (code points: [U+1F600])
  Rendered: {"msg":"\u1f600"}
  Decoded:  "ὠ0"
  Round-trip: false

Test: crying_laugh
  Original: "😂" (code points: [U+1F602])
  Rendered: {"msg":"\u1f602"}
  Decoded:  "ὠ2"
  Round-trip: false

Test: rocket
  Original: "🚀" (code points: [U+1F680])
  Rendered: {"msg":"\u1f680"}
  Decoded:  "Ὠ0"
  Round-trip: false

Test: heart
  Original: "❤️" (code points: [U+2764 U+FE0F])
  Rendered: {"msg":"\u2764\ufe0f"}
  Decoded:  "❤️"
  Round-trip: true

Test: dragon
  Original: "🐉" (code points: [U+1F409])
  Rendered: {"msg":"\u1f409"}
  Decoded:  "ὀ9"
  Round-trip: false

Test: alien
  Original: "👽" (code points: [U+1F47D])
  Rendered: {"msg":"\u1f47d"}
  Decoded:  "\u1f47d"
  Round-trip: false

Issue Confirmed

Scenario Result
BMP characters (U+0000 ~ U+FFFF) ✅ Working (e.g., ❤️)
Non-BMP characters (U+10000 and above) ❌ Corrupted, cannot round-trip
I will try to fix this and submit a PR.

Thanks for verifying and expanding the test cases.

The BMP vs non-BMP split is a nice confirmation that the issue is specifically in the escape generation path for code points above U+FFFF. The CJK Extension B example is especially helpful because it shows the problem isn't emoji-specific.

Feel free to submit a PR if you'd like. I've also been investigating the issue locally and writing a regression test, so I'm happy to review and test any proposed fix as well.