Merge pull request #46 from lpgauth/bench-glazer-validate-utf8

Benchmark glazer with UTF-8 validation enabled

Author: lpgauth

README.md

@@ -202,62 +202,67 @@ Apple M2 Pro, OTP 29, Elixir 1.20. Both libraries are profile-guided
 optimised (PGO) builds: **Torque PGO** (via `scripts/pgo-build.sh`) and
 **Glazer PGO** (via `OPTIMIZE=1`).
 
+glazer is benchmarked with UTF-8 validation enabled (`validate_utf8` on
+decode, `force_utf8` on encode — both off by default in glazer) so every
+library provides the same guarantee Torque always does: JSON strings are
+valid UTF-8.
+
 ### Decode (1.2 KB OpenRTB)
 
 | Library | ips | mean | median | p99 | memory |
 |---|---|---|---|---|---|
-| **torque** | **413.3K** | **2.42 μs** | **2.29 μs** | **4.04 μs** | 1.56 KB |
-| **glazer** | 401.8K | 2.49 μs | 2.38 μs | 4.58 μs | 1.56 KB |
-| **jiffy** | 202.5K | 4.94 μs | 4.50 μs | 9.75 μs | **1.55 KB** |
-| **otp json** | 147.4K | 6.78 μs | 6.58 μs | 10.58 μs | 7.73 KB |
-| **jason** | 113.2K | 8.83 μs | 8.46 μs | 13.96 μs | 9.54 KB |
+| **torque** | **412.8K** | **2.42 μs** | **2.29 μs** | **4.67 μs** | 1.56 KB |
+| **glazer** | 355.5K | 2.81 μs | 2.67 μs | 5.00 μs | 1.56 KB |
+| **jiffy** | 200.0K | 5.00 μs | 4.50 μs | 11.00 μs | **1.55 KB** |
+| **otp json** | 143.8K | 6.95 μs | 6.71 μs | 12.54 μs | 7.73 KB |
+| **jason** | 109.0K | 9.17 μs | 8.54 μs | 20.58 μs | 9.54 KB |
 
 ### Decode (750 KB Twitter)
 
 | Library | ips | mean | median | p99 | memory |
 |---|---|---|---|---|---|
-| **torque** | **660.7** | **1.51 ms** | **1.33 ms** | 2.21 ms | **1.57 KB** |
-| **glazer** | 659.6 | 1.52 ms | 1.43 ms | **1.94 ms** | 1.58 KB |
-| **jiffy** | 301.4 | 3.32 ms | 3.39 ms | 3.96 ms | 2.30 MB |
-| **otp json** | 205.3 | 4.87 ms | 4.80 ms | 7.52 ms | 2.48 MB |
-| **jason** | 151.7 | 6.59 ms | 6.58 ms | 6.97 ms | 3.52 MB |
+| **torque** | **659.4** | **1.52 ms** | **1.32 ms** | **2.14 ms** | **1.57 KB** |
+| **glazer** | 597.0 | 1.68 ms | 1.59 ms | 2.26 ms | 1.58 KB |
+| **jiffy** | 298.8 | 3.35 ms | 3.37 ms | 4.52 ms | 2.30 MB |
+| **otp json** | 211.3 | 4.73 ms | 4.77 ms | 5.57 ms | 2.48 MB |
+| **jason** | 150.2 | 6.66 ms | 6.59 ms | 8.21 ms | 3.54 MB |
 
 ### Encode (1.2 KB OpenRTB)
 
 | Library | ips | mean | median | p99 | memory |
 |---|---|---|---|---|---|
-| **otp json** [map() :: iodata()] | **1180K** | **0.85 μs** | **0.79 μs** | **1.08 μs** | 3928 B |
-| **torque** [proplist() :: binary()] | 1110K | 0.90 μs | 0.83 μs | **1.08 μs** | 88 B |
-| **torque** [proplist() :: iodata()] | 1080K | 0.92 μs | 0.83 μs | 1.83 μs | **64 B** |
-| **torque** [map() :: iodata()] | 1000K | 1.00 μs | 0.96 μs | 1.13 μs | **64 B** |
-| **glazer** [map() :: binary()] | 990K | 1.01 μs | 0.92 μs | 1.29 μs | **64 B** |
-| **torque** [map() :: binary()] | 980K | 1.02 μs | 0.96 μs | 1.17 μs | 88 B |
-| **jiffy** [proplist() :: iodata()] | 640K | 1.57 μs | 1.33 μs | 1.83 μs | 120 B |
-| **jason** [map() :: iodata()] | 620K | 1.62 μs | 1.54 μs | 2.63 μs | 3848 B |
-| **jiffy** [map() :: iodata()] | 520K | 1.91 μs | 1.75 μs | 2.25 μs | 824 B |
-| **jason** [map() :: binary()] | 380K | 2.63 μs | 2.38 μs | 6.42 μs | 3912 B |
+| **otp json** [map() :: iodata()] | **1174K** | **0.85 μs** | **0.79 μs** | 1.21 μs | 3928 B |
+| **torque** [proplist() :: iodata()] | 1084K | 0.92 μs | 0.88 μs | **1.08 μs** | **64 B** |
+| **torque** [proplist() :: binary()] | 1048K | 0.95 μs | 0.88 μs | 1.21 μs | 88 B |
+| **torque** [map() :: iodata()] | 957K | 1.04 μs | 1.00 μs | 1.25 μs | **64 B** |
+| **torque** [map() :: binary()] | 953K | 1.05 μs | 1.00 μs | 1.21 μs | 88 B |
+| **glazer** [map() :: binary()] | 932K | 1.07 μs | 1.00 μs | 1.21 μs | **64 B** |
+| **jiffy** [proplist() :: iodata()] | 654K | 1.53 μs | 1.33 μs | 1.88 μs | 120 B |
+| **jason** [map() :: iodata()] | 598K | 1.67 μs | 1.54 μs | 3.21 μs | 3848 B |
+| **jiffy** [map() :: iodata()] | 526K | 1.90 μs | 1.75 μs | 2.17 μs | 824 B |
+| **jason** [map() :: binary()] | 401K | 2.49 μs | 2.33 μs | 4.63 μs | 3912 B |
 
 ### Encode (750 KB Twitter)
 
 | Library | ips | mean | median | p99 | memory |
 |---|---|---|---|---|---|
-| **torque** [proplist() :: iodata()] | **1156.0** | **0.87 ms** | **0.85 ms** | **1.06 ms** | **64 B** |
-| **torque** [proplist() :: binary()] | 1154.1 | **0.87 ms** | **0.85 ms** | **1.06 ms** | 88 B |
-| **torque** [map() :: binary()] | 1045.4 | 0.96 ms | 0.95 ms | 1.13 ms | 88 B |
-| **torque** [map() :: iodata()] | 1040.4 | 0.96 ms | 0.96 ms | 1.13 ms | **64 B** |
-| **glazer** [map() :: binary()] | 1033.3 | 0.97 ms | 0.96 ms | 1.08 ms | **64 B** |
-| **jiffy** [proplist() :: iodata()] | 469.5 | 2.13 ms | 2.10 ms | 2.51 ms | 37.7 KB |
-| **jiffy** [map() :: iodata()] | 347.6 | 2.88 ms | 2.96 ms | 3.90 ms | 1.06 MB |
-| **otp json** [map() :: iodata()] | 251.8 | 3.97 ms | 4.13 ms | 6.44 ms | 5.40 MB |
-| **jason** [map() :: iodata()] | 233.5 | 4.28 ms | 4.03 ms | 6.50 ms | 4.96 MB |
-| **jason** [map() :: binary()] | 127.9 | 7.82 ms | 7.84 ms | 8.43 ms | 4.96 MB |
+| **torque** [proplist() :: iodata()] | **1163.7** | **0.86 ms** | **0.84 ms** | **1.08 ms** | **64 B** |
+| **torque** [proplist() :: binary()] | 1143.3 | 0.87 ms | **0.84 ms** | 1.44 ms | 88 B |
+| **torque** [map() :: binary()] | 1056.0 | 0.95 ms | 0.93 ms | 1.15 ms | 88 B |
+| **torque** [map() :: iodata()] | 1019.0 | 0.98 ms | 0.96 ms | 1.21 ms | **64 B** |
+| **glazer** [map() :: binary()] | 843.9 | 1.19 ms | 1.17 ms | 1.38 ms | **64 B** |
+| **jiffy** [proplist() :: iodata()] | 473.8 | 2.11 ms | 2.09 ms | 2.31 ms | 37.7 KB |
+| **jiffy** [map() :: iodata()] | 357.1 | 2.80 ms | 2.95 ms | 3.26 ms | 1.06 MB |
+| **otp json** [map() :: iodata()] | 270.5 | 3.70 ms | 3.93 ms | 4.77 ms | 5.40 MB |
+| **jason** [map() :: iodata()] | 261.0 | 3.83 ms | 3.54 ms | 5.86 ms | 4.96 MB |
+| **jason** [map() :: binary()] | 138.5 | 7.22 ms | 7.18 ms | 8.09 ms | 4.96 MB |
 
 ### Parse (1.2 KB OpenRTB)
 
 | Library | ips | mean | median | p99 |
 |---|---|---|---|---|
-| **torque** parse(unique_keys) | **542.0K** | **1.84 μs** | 1.46 μs | **5.63 μs** |
-| **torque** parse | 522.8K | 1.91 μs | 1.46 μs | 6.00 μs |
+| **torque** parse(unique_keys) | **556.8K** | **1.80 μs** | 1.46 μs | **5.21 μs** |
+| **torque** parse | 555.4K | **1.80 μs** | **1.42 μs** | **5.21 μs** |
 
 ### Extract 5 fields from raw JSON (1.2 KB OpenRTB)
 
@@ -268,10 +273,10 @@ selective extraction skips materializing the whole document.
 
 | Library | ips | mean | median | p99 |
 |---|---|---|---|---|
-| **torque** parse + get_many | **436.6K** | **2.29 μs** | **1.79 μs** | **6.21 μs** |
-| **torque** parse(unique_keys) + get_many | 429.5K | 2.33 μs | **1.79 μs** | 6.67 μs |
-| **torque** parse + get x5 | 409.8K | 2.44 μs | 2.00 μs | 6.67 μs |
-| **glazer** decode + find x5 | 344.9K | 2.90 μs | 2.75 μs | 6.71 μs |
+| **torque** parse(unique_keys) + get_many | **467.3K** | **2.14 μs** | **1.79 μs** | **4.79 μs** |
+| **torque** parse + get_many | 455.1K | 2.20 μs | **1.79 μs** | 5.46 μs |
+| **torque** parse + get x5 | 420.1K | 2.38 μs | 1.96 μs | 6.08 μs |
+| **glazer** decode + find x5 | 315.1K | 3.17 μs | 3.04 μs | 4.88 μs |
 
 Run benchmarks locally:
 

bench/glazer_pgo_workload.exs

@@ -6,6 +6,10 @@
 # benchmark uses — keeping the glazer-vs-torque comparison apples-to-apples
 # (both libraries profiled against equivalent decode/encode/find workloads).
 #
+# Uses the same UTF-8 validation options as bench/torque_bench.exs
+# (`validate_utf8` decode, `force_utf8` encode) so the profile matches the
+# benchmarked configuration — keep them in sync.
+#
 # Run with an instrumented glazer.so loaded: MIX_ENV=bench mix run this file.
 
 small_json =
@@ -40,17 +44,17 @@ paths =
 IO.puts("glazer PGO workload: small=#{byte_size(small_json)}B large=#{byte_size(large_json)}B")
 
 decode = fn ->
-  :glazer_json.decode(small_json)
-  :glazer_json.decode(large_json)
+  :glazer_json.decode(small_json, [:validate_utf8])
+  :glazer_json.decode(large_json, [:validate_utf8])
 end
 
 encode = fn ->
-  :glazer_json.encode(small_term)
-  :glazer_json.encode(large_term)
+  :glazer_json.encode(small_term, [:force_utf8])
+  :glazer_json.encode(large_term, [:force_utf8])
 end
 
 find = fn ->
-  d = :glazer_json.decode(small_json)
+  d = :glazer_json.decode(small_json, [:validate_utf8])
   Enum.each(paths, &:glazer.find(d, &1))
 end
 

bench/index.html

@@ -108,7 +108,10 @@ <h1>Torque Benchmarks</h1>
     </p>
 
     <h2>Library Comparison</h2>
-    <p class="section-desc">Latest CI results — all libraries, sorted by throughput (higher is better).</p>
+    <p class="section-desc">Latest CI results — all libraries, sorted by throughput (higher is better).
+      glazer runs with UTF-8 validation enabled (<code>validate_utf8</code> on decode,
+      <code>force_utf8</code> on encode; both default off in glazer) so all libraries
+      provide the same guarantee Torque always does: JSON strings are valid UTF-8.</p>
     <div id="comparison"><div class="empty">Loading...</div></div>
 
     <h2>Performance Over Time</h2>

bench/torque_bench.exs

@@ -2,6 +2,13 @@
 #
 # Run with: MIX_ENV=bench mix run bench/torque_bench.exs
 # CI JSON output: BENCH_OUTPUT=json MIX_ENV=bench mix run bench/torque_bench.exs
+#
+# glazer is benchmarked with UTF-8 validation enabled — `validate_utf8` on
+# decode, `force_utf8` on encode (both default OFF in glazer, needs >= 0.5.15)
+# — so every library in the comparison provides the guarantee Torque always
+# does: JSON strings are valid UTF-8. Keep these options in sync with
+# bench/glazer_pgo_workload.exs so glazer's PGO profile matches the
+# benchmarked configuration.
 
 # CI formatter for github-action-benchmark (customBiggerIsBetter)
 if System.get_env("BENCH_OUTPUT") == "json" do
@@ -306,7 +313,7 @@ IO.puts("=== DECODE BENCHMARK ===\n")
 
 Benchee.run(
   %{
-    "glazer decode" => fn -> :glazer_json.decode(sample_json) end,
+    "glazer decode" => fn -> :glazer_json.decode(sample_json, [:validate_utf8]) end,
     "jason decode" => fn -> Jason.decode!(sample_json) end,
     "jiffy decode" => fn -> :jiffy.decode(sample_json, [:return_maps]) end,
     "otp json decode" => fn -> :json.decode(sample_json) end,
@@ -327,7 +334,7 @@ IO.puts("\n=== LARGE JSON DECODE BENCHMARK ===\n")
 
 Benchee.run(
   %{
-    "glazer decode" => fn -> :glazer_json.decode(large_json) end,
+    "glazer decode" => fn -> :glazer_json.decode(large_json, [:validate_utf8]) end,
     "jason decode" => fn -> Jason.decode!(large_json) end,
     "jiffy decode" => fn -> :jiffy.decode(large_json, [:return_maps]) end,
     "otp json decode" => fn -> :json.decode(large_json) end,
@@ -355,7 +362,7 @@ Benchee.run(
       :jiffy.encode(bid_response_proplist, [:force_utf8])
     end,
     "otp json [map() :: iodata()]" => fn -> :json.encode(bid_response) end,
-    "glazer [map() :: binary()]" => fn -> :glazer_json.encode(bid_response) end,
+    "glazer [map() :: binary()]" => fn -> :glazer_json.encode(bid_response, [:force_utf8]) end,
     "torque [map() :: binary()]" => fn -> Torque.encode!(bid_response) end,
     "torque [map() :: iodata()]" => fn -> Torque.encode_to_iodata(bid_response) end,
     "torque [proplist() :: binary()]" => fn -> Torque.encode!(bid_response_proplist) end,
@@ -381,7 +388,9 @@ Benchee.run(
     "jiffy [map() :: iodata()]" => fn -> :jiffy.encode(large_decoded_json) end,
     "jiffy [proplist() :: iodata()]" => fn -> :jiffy.encode(large_decoded_proplist) end,
     "otp json [map() :: iodata()]" => fn -> :json.encode(large_decoded_json) end,
-    "glazer [map() :: binary()]" => fn -> :glazer_json.encode(large_decoded_json) end,
+    "glazer [map() :: binary()]" => fn ->
+      :glazer_json.encode(large_decoded_json, [:force_utf8])
+    end,
     "torque [map() :: binary()]" => fn -> Torque.encode!(large_decoded_json) end,
     "torque [map() :: iodata()]" => fn -> Torque.encode_to_iodata(large_decoded_json) end,
     "torque [proplist() :: binary()]" => fn -> Torque.encode!(large_decoded_proplist) end,
@@ -432,7 +441,7 @@ IO.puts("\n=== EXTRACT 5 FIELDS BENCHMARK ===\n")
 Benchee.run(
   %{
     "glazer decode + find x5" => fn ->
-      d = :glazer_json.decode(sample_json)
+      d = :glazer_json.decode(sample_json, [:validate_utf8])
       for p <- glazer_paths, do: :glazer.find(d, p)
     end,
     "torque parse + get x5" => fn ->

mix.lock

@@ -6,7 +6,7 @@
   "earmark_parser": {:hex, :earmark_parser, "1.4.44", "f20830dd6b5c77afe2b063777ddbbff09f9759396500cdbe7523efd58d7a339c", [:mix], [], "hexpm", "4778ac752b4701a5599215f7030989c989ffdc4f6df457c5f36938cc2d2a2750"},
   "erlex": {:hex, :erlex, "0.2.9", "7debbbaa9f4f368b8cd648983e0f1d7963028508e9c59e9d4ed504e94ef52a55", [:mix], [], "hexpm", "8cfffc0ec7159e6d73de2ab28a588064de80f88b2798d5cbe4482cbbc200178b"},
   "ex_doc": {:hex, :ex_doc, "0.40.3", "4a972ffe64bc07dc605af487e98fc19b72a4185f55ca031b94c0552d6071c1d9", [:mix], [{:earmark_parser, "~> 1.4.44", [hex: :earmark_parser, repo: "hexpm", optional: false]}, {:makeup_c, ">= 0.1.0", [hex: :makeup_c, repo: "hexpm", optional: true]}, {:makeup_elixir, "~> 0.14 or ~> 1.0", [hex: :makeup_elixir, repo: "hexpm", optional: false]}, {:makeup_erlang, "~> 0.1 or ~> 1.0", [hex: :makeup_erlang, repo: "hexpm", optional: false]}, {:makeup_html, ">= 0.1.0", [hex: :makeup_html, repo: "hexpm", optional: true]}], "hexpm", "2756e357742fecd9749b489b85d67c9ce99c465f2e75728d9e6dc8d704b973de"},
-  "glazer": {:hex, :glazer, "0.5.11", "c6f4bf32c3da539351d4e3811484ffe0ba341a871dc2ce47ddb5284c78f5166f", [:rebar3], [], "hexpm", "f5abec34de2f4cec226632f0db25567d4c6ffc0fd2e7ed5d285df81afb5582f8"},
+  "glazer": {:hex, :glazer, "0.5.15", "cdbe06d3f82332575e3c10123989977e0d136241799d7292b657d2e01db89f1b", [:rebar3], [], "hexpm", "bc2cc19b63e459985029f995c96b29883e1b8d61383594b58dc8edb23718d314"},
   "jason": {:hex, :jason, "1.4.5", "2e3a008590b0b8d7388c20293e9dcc9cf3e5d642fd2a114e4cbbb52e595d940a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0 or ~> 3.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "b0c823996102bcd0239b3c2444eb00409b72f6a140c1950bc8b457d836b30684"},
   "jiffy": {:hex, :jiffy, "2.0.0", "1d7605ee31c4fd9b454725d744716b59e2abd88297862c8931b605c83b076480", [:rebar3], [], "hexpm", "73f58877ccaae14e14601c95928e2913fd1c1ef28e6ef4f274a8d62819f9e04f"},
   "makeup": {:hex, :makeup, "1.2.1", "e90ac1c65589ef354378def3ba19d401e739ee7ee06fb47f94c687016e3713d1", [:mix], [{:nimble_parsec, "~> 1.4", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "d36484867b0bae0fea568d10131197a4c2e47056a6fbe84922bf6ba71c8d17ce"},