Skip to content

Commit f999cab

Browse files
lpgauthlpgauth
authored
Merge pull request #19 from lpgauth/feat/unique-keys-option
Add unique_keys option to parse/2 for faster lookups
2 parents 09efa1e + 08e6b15 commit f999cab

10 files changed

Lines changed: 452 additions & 201 deletions

File tree

‎README.md‎

Lines changed: 58 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,13 @@ results = Torque.get_many(doc, ["/id", "/site/domain", "/device/ip"])
7171
# [{:ok, "req-1"}, {:ok, "example.com"}, {:ok, "1.2.3.4"}]
7272
```
7373

74+
When your JSON is known to have no duplicate object keys, pass `unique_keys: true`
75+
for faster field lookups (uses sonic-rs internal indexing instead of linear scan):
76+
77+
```elixir
78+
{:ok, doc} = Torque.parse(json, unique_keys: true)
79+
```
80+
7481
### Encoding
7582

7683
```elixir
@@ -94,7 +101,7 @@ json = Torque.encode_to_iodata(%{id: "abc"})
94101
|----------|-------------|
95102
| `Torque.decode(binary)` | Decode JSON to Elixir terms |
96103
| `Torque.decode!(binary)` | Decode JSON, raising on error |
97-
| `Torque.parse(binary)` | Parse JSON into opaque document reference |
104+
| `Torque.parse(binary, opts)` | Parse JSON into opaque document reference |
98105
| `Torque.get(doc, path)` | Extract field by JSON Pointer path |
99106
| `Torque.get(doc, path, default)` | Extract field with default for missing paths |
100107
| `Torque.get_many(doc, paths)` | Extract multiple fields in one NIF call |
@@ -118,7 +125,7 @@ json = Torque.encode_to_iodata(%{id: "abc"})
118125
| `true`, `false` | `true`, `false` |
119126
| `null` | `nil` |
120127

121-
For objects with duplicate keys, the last value wins.
128+
For objects with duplicate keys, the last value wins (unless `unique_keys: true` is passed to `parse/2`).
122129

123130
### Elixir to JSON
124131

@@ -165,62 +172,73 @@ Apple M2 Pro, OTP 28, Elixir 1.19:
165172

166173
| Library | ips | mean | median | p99 | memory |
167174
|---|---|---|---|---|---|
168-
| **torque** | **257.2K** | **3.89 μs** | **3.67 μs** | **8.17 μs** | **1.56 KB** |
169-
| **simdjsone** | 170.0K | 5.88 μs | 5.13 μs | 14.75 μs | 1.59 KB |
170-
| **jiffy** | 146.1K | 6.85 μs | 6.00 μs | 17.08 μs | **1.56 KB** |
171-
| **otp json** | 127.2K | 7.86 μs | 7.29 μs | 18.00 μs | 7.73 KB |
172-
| **jason** | 107.7K | 9.29 μs | 8.71 μs | 18.08 μs | 9.54 KB |
175+
| **torque** | **262.5K** | **3.81 μs** | **3.63 μs** | **7.83 μs** | **1.56 KB** |
176+
| **simdjsone** | 182.7K | 5.47 μs | 5.13 μs | 11.88 μs | 1.59 KB |
177+
| **jiffy** | 144.6K | 6.92 μs | 6.21 μs | 17.17 μs | **1.56 KB** |
178+
| **otp json** | 129.6K | 7.72 μs | 7.21 μs | 16.50 μs | 7.73 KB |
179+
| **jason** | 103.6K | 9.65 μs | 8.71 μs | 22.75 μs | 9.54 KB |
173180

174181
### Decode (750 KB Twitter)
175182

176183
| Library | ips | mean | median | p99 | memory |
177184
|---|---|---|---|---|---|
178-
| **torque** | **505.0** | **1.98 ms** | **1.82 ms** | **2.58 ms** | **1.56 KB** |
179-
| **simdjsone** | 415.3 | 2.41 ms | 1.90 ms | 3.82 ms | **1.56 KB** |
180-
| **otp json** | 182.5 | 5.48 ms | 5.45 ms | 6.58 ms | 2.49 MB |
181-
| **jason** | 136.8 | 7.31 ms | 7.13 ms | 12.29 ms | 3.55 MB |
182-
| **jiffy** | 100.7 | 9.93 ms | 10.01 ms | 11.91 ms | 5.53 MB |
185+
| **torque** | **476.0** | **2.10 ms** | **1.87 ms** | **4.73 ms** | **1.56 KB** |
186+
| **simdjsone** | 459.4 | 2.18 ms | 1.85 ms | 3.20 ms | **1.56 KB** |
187+
| **otp json** | 195.1 | 5.13 ms | 5.12 ms | 6.16 ms | 2.49 MB |
188+
| **jason** | 142.0 | 7.04 ms | 6.91 ms | 11.47 ms | 3.55 MB |
189+
| **jiffy** | 115.9 | 8.63 ms | 8.72 ms | 9.94 ms | 5.53 MB |
183190

184191
### Encode (1.2 KB OpenRTB)
185192

186193
| Library | ips | mean | median | p99 | memory |
187194
|---|---|---|---|---|---|
188-
| **torque** [proplist() :: binary()] | **1274.3K** | **0.78 μs** | **0.71 μs** | **0.92 μs** | 88 B |
189-
| **torque** [proplist() :: iodata()] | 1261.4K | 0.79 μs | **0.71 μs** | 0.96 μs | **64 B** |
190-
| **otp json** [map() :: iodata()] | 1078.0K | 0.93 μs | 0.88 μs | 1.38 μs | 3928 B |
191-
| **torque** [map() :: iodata()] | 1064.8K | 0.94 μs | 0.88 μs | 1.13 μs | **64 B** |
192-
| **torque** [map() :: binary()] | 1053.1K | 0.95 μs | 0.88 μs | 1.17 μs | 88 B |
193-
| **jason** [map() :: iodata()] | 591.3K | 1.69 μs | 1.50 μs | 3.54 μs | 3848 B |
194-
| **jiffy** [proplist() :: iodata()] | 579.2K | 1.73 μs | 1.50 μs | 2.13 μs | 120 B |
195-
| **jiffy** [map() :: iodata()] | 498.1K | 2.01 μs | 1.83 μs | 2.50 μs | 824 B |
196-
| **simdjsone** [proplist() :: iodata()] | 441.6K | 2.26 μs | 2.00 μs | 3.71 μs | 184 B |
197-
| **jason** [map() :: binary()] | 399.7K | 2.50 μs | 2.33 μs | 4.21 μs | 3912 B |
198-
| **simdjsone** [map() :: iodata()] | 386.7K | 2.59 μs | 2.38 μs | 4.54 μs | 888 B |
195+
| **otp json** [map() :: iodata()] | **1091.6K** | **0.92 μs** | **0.83 μs** | 1.46 μs | 3928 B |
196+
| **torque** [proplist() :: binary()] | 1073.6K | 0.93 μs | 0.88 μs | **1.13 μs** | 88 B |
197+
| **torque** [proplist() :: iodata()] | 1069.3K | 0.94 μs | 0.88 μs | 1.17 μs | **64 B** |
198+
| **torque** [map() :: binary()] | 917.5K | 1.09 μs | 1.00 μs | 1.33 μs | 88 B |
199+
| **torque** [map() :: iodata()] | 914.6K | 1.09 μs | 1.00 μs | 1.42 μs | **64 B** |
200+
| **jason** [map() :: iodata()] | 571.8K | 1.75 μs | 1.54 μs | 3.75 μs | 3848 B |
201+
| **jiffy** [proplist() :: iodata()] | 518.4K | 1.93 μs | 1.67 μs | 2.75 μs | 120 B |
202+
| **jiffy** [map() :: iodata()] | 427.6K | 2.34 μs | 2.08 μs | 4.33 μs | 824 B |
203+
| **simdjsone** [proplist() :: iodata()] | 415.4K | 2.41 μs | 2.21 μs | 3.96 μs | 184 B |
204+
| **jason** [map() :: binary()] | 385.1K | 2.60 μs | 2.38 μs | 5.00 μs | 3912 B |
205+
| **simdjsone** [map() :: iodata()] | 346.8K | 2.88 μs | 2.67 μs | 4.33 μs | 888 B |
199206

200207
### Encode (750 KB Twitter)
201208

202209
| Library | ips | mean | median | p99 | memory |
203210
|---|---|---|---|---|---|
204-
| **torque** [proplist() :: iodata()] | **1272.5** | **0.79 ms** | **0.76 ms** | **0.99 ms** | **64 B** |
205-
| **torque** [proplist() :: binary()] | 1252.1 | 0.80 ms | 0.77 ms | 1.04 ms | 88 B |
206-
| **torque** [map() :: iodata()] | 1102.9 | 0.91 ms | 0.89 ms | 1.09 ms | **64 B** |
207-
| **torque** [map() :: binary()] | 1084.0 | 0.92 ms | 0.89 ms | 1.20 ms | 88 B |
208-
| **jiffy** [proplist() :: iodata()] | 342.0 | 2.92 ms | 2.82 ms | 4.75 ms | 37.7 KB |
209-
| **jiffy** [map() :: iodata()] | 287.1 | 3.48 ms | 3.32 ms | 4.29 ms | 1.06 MB |
210-
| **simdjsone** [proplist() :: iodata()] | 259.7 | 3.85 ms | 3.78 ms | 5.79 ms | 37.7 KB |
211-
| **jason** [map() :: iodata()] | 241.3 | 4.14 ms | 3.94 ms | 6.99 ms | 4.96 MB |
212-
| **simdjsone** [map() :: iodata()] | 216.1 | 4.63 ms | 4.66 ms | 6.51 ms | 1.06 MB |
213-
| **otp json** [map() :: iodata()] | 200.2 | 4.99 ms | 5.10 ms | 6.97 ms | 5.40 MB |
214-
| **jason** [map() :: binary()] | 130.9 | 7.64 ms | 7.53 ms | 9.09 ms | 4.96 MB |
215-
216-
### Parse + Get (5 fields) (1.2 KB OpenRTB)
211+
| **torque** [proplist() :: iodata()] | **1026.4** | **0.97 ms** | **0.96 ms** | **1.18 ms** | **64 B** |
212+
| **torque** [proplist() :: binary()] | 983.5 | 1.02 ms | 0.98 ms | 1.69 ms | 88 B |
213+
| **torque** [map() :: binary()] | 918.5 | 1.09 ms | 1.08 ms | 1.31 ms | 88 B |
214+
| **torque** [map() :: iodata()] | 905.4 | 1.10 ms | 1.09 ms | 1.35 ms | **64 B** |
215+
| **jiffy** [proplist() :: iodata()] | 342.6 | 2.92 ms | 2.86 ms | 4.35 ms | 37.7 KB |
216+
| **jiffy** [map() :: iodata()] | 270.8 | 3.69 ms | 3.53 ms | 5.94 ms | 1.06 MB |
217+
| **jason** [map() :: iodata()] | 254.9 | 3.92 ms | 3.70 ms | 6.50 ms | 4.96 MB |
218+
| **simdjsone** [proplist() :: iodata()] | 247.4 | 4.04 ms | 3.98 ms | 5.63 ms | 37.7 KB |
219+
| **otp json** [map() :: iodata()] | 246.9 | 4.05 ms | 4.13 ms | 5.64 ms | 5.40 MB |
220+
| **simdjsone** [map() :: iodata()] | 210.5 | 4.75 ms | 4.78 ms | 5.41 ms | 1.06 MB |
221+
| **jason** [map() :: binary()] | 141.1 | 7.09 ms | 7.02 ms | 8.40 ms | 4.96 MB |
222+
223+
### Parse (1.2 KB OpenRTB)
224+
225+
| Library | ips | mean | median | p99 |
226+
|---|---|---|---|---|
227+
| **torque** parse(unique_keys) | **596.6K** | **1.68 μs** | **1.33 μs** | **3.13 μs** |
228+
| **torque** parse | 579.2K | 1.73 μs | **1.33 μs** | 3.88 μs |
229+
| **simdjsone** parse | 364.9K | 2.74 μs | 1.17 μs | 4.92 μs |
230+
231+
### Get (5 fields) (1.2 KB OpenRTB)
217232

218233
| Library | ips | mean | median | p99 | memory |
219234
|---|---|---|---|---|---|
220-
| **torque** parse+get_many_nil | **455.8K** | **2.19 μs** | **1.75 μs** | **6.21 μs** | **288 B** |
221-
| **torque** parse+get_many | 431.6K | 2.32 μs | 1.75 μs | 6.33 μs | 408 B |
222-
| **torque** parse+get | 415.5K | 2.41 μs | 1.96 μs | 7.13 μs | 432 B |
223-
| **simdjsone** parse+get | 353.8K | 2.83 μs | 1.71 μs | 7.25 μs | 408 B |
235+
| **torque** get_many_nil (unique_keys) | **2.49M** | **402 ns** | **375 ns** | **500 ns** | **240 B** |
236+
| **torque** get_many (unique_keys) | 2.37M | 422 ns | **375 ns** | **500 ns** | 360 B |
237+
| **torque** get_many_nil | 2.16M | 463 ns | 458 ns | 583 ns | **240 B** |
238+
| **torque** get_many | 2.07M | 483 ns | 458 ns | 584 ns | 360 B |
239+
| **simdjsone** get | 1.77M | 564 ns | 458 ns | 1083 ns | 384 B |
240+
| **torque** get (unique_keys) | 1.67M | 601 ns | 583 ns | 709 ns | 384 B |
241+
| **torque** get | 1.50M | 669 ns | 625 ns | 792 ns | 384 B |
224242

225243
Run benchmarks locally:
226244

‎bench/torque_bench.exs‎

Lines changed: 34 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -399,27 +399,43 @@ Benchee.run(
399399
] ++ ci_formatters
400400
)
401401

402-
BenchGroup.set("Parse+Get (5 fields) — 1.2 KB OpenRTB")
403-
IO.puts("\n=== PARSE + GET BENCHMARK ===\n")
402+
{:ok, pre_doc} = Torque.parse(sample_json)
403+
{:ok, pre_doc_uk} = Torque.parse(sample_json, unique_keys: true)
404+
pre_ref = :simdjson.parse(sample_json)
404405

406+
BenchGroup.set("Parse — 1.2 KB OpenRTB")
407+
IO.puts("\n=== PARSE BENCHMARK ===\n")
408+
409+
# NOTE: simdjsone segfaults when Benchee measures memory (GC triggers a
410+
# use-after-free in the NIF resource destructor), so memory_time is 0 here.
405411
Benchee.run(
406412
%{
407-
"simdjsone parse+get" => fn ->
408-
ref = :simdjson.parse(sample_json)
409-
for f <- fields, do: :simdjson.get(ref, f)
410-
end,
411-
"torque parse+get" => fn ->
412-
{:ok, doc} = Torque.parse(sample_json)
413-
for f <- fields, do: Torque.get(doc, f)
414-
end,
415-
"torque parse+get_many" => fn ->
416-
{:ok, doc} = Torque.parse(sample_json)
417-
Torque.get_many(doc, fields)
418-
end,
419-
"torque parse+get_many_nil" => fn ->
420-
{:ok, doc} = Torque.parse(sample_json)
421-
Torque.get_many_nil(doc, fields)
422-
end
413+
"simdjsone parse" => fn -> :simdjson.parse(sample_json) end,
414+
"torque parse" => fn -> Torque.parse(sample_json) end,
415+
"torque parse(unique_keys)" => fn -> Torque.parse(sample_json, unique_keys: true) end
416+
},
417+
warmup: 2,
418+
time: 5,
419+
memory_time: 0,
420+
percentiles: [50, 95, 99],
421+
formatters:
422+
[
423+
{Benchee.Formatters.Console, percentiles: [50, 95, 99]}
424+
] ++ ci_formatters
425+
)
426+
427+
BenchGroup.set("Get (5 fields) — 1.2 KB OpenRTB")
428+
IO.puts("\n=== GET BENCHMARK ===\n")
429+
430+
Benchee.run(
431+
%{
432+
"simdjsone get" => fn -> for f <- fields, do: :simdjson.get(pre_ref, f) end,
433+
"torque get" => fn -> for f <- fields, do: Torque.get(pre_doc, f) end,
434+
"torque get_many" => fn -> Torque.get_many(pre_doc, fields) end,
435+
"torque get_many_nil" => fn -> Torque.get_many_nil(pre_doc, fields) end,
436+
"torque get (unique_keys)" => fn -> for f <- fields, do: Torque.get(pre_doc_uk, f) end,
437+
"torque get_many (unique_keys)" => fn -> Torque.get_many(pre_doc_uk, fields) end,
438+
"torque get_many_nil (unique_keys)" => fn -> Torque.get_many_nil(pre_doc_uk, fields) end
423439
},
424440
warmup: 2,
425441
time: 5,

‎bench/torque_only_bench.exs‎

Lines changed: 26 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -339,22 +339,35 @@ Benchee.run(
339339
]
340340
)
341341

342-
IO.puts("\n=== PARSE + GET BENCHMARK ===\n")
342+
IO.puts("\n=== PARSE BENCHMARK ===\n")
343343

344344
Benchee.run(
345345
%{
346-
"torque parse+get" => fn ->
347-
{:ok, doc} = Torque.parse(sample_json)
348-
for f <- fields, do: Torque.get(doc, f)
349-
end,
350-
"torque parse+get_many" => fn ->
351-
{:ok, doc} = Torque.parse(sample_json)
352-
Torque.get_many(doc, fields)
353-
end,
354-
"torque parse+get_many_nil" => fn ->
355-
{:ok, doc} = Torque.parse(sample_json)
356-
Torque.get_many_nil(doc, fields)
357-
end
346+
"torque parse" => fn -> Torque.parse(sample_json) end,
347+
"torque parse(unique_keys)" => fn -> Torque.parse(sample_json, unique_keys: true) end
348+
},
349+
warmup: 2,
350+
time: 5,
351+
memory_time: 2,
352+
percentiles: [50, 95, 99],
353+
formatters: [
354+
{Benchee.Formatters.Console, percentiles: [50, 95, 99]}
355+
]
356+
)
357+
358+
{:ok, pre_doc} = Torque.parse(sample_json)
359+
{:ok, pre_doc_uk} = Torque.parse(sample_json, unique_keys: true)
360+
361+
IO.puts("\n=== GET BENCHMARK ===\n")
362+
363+
Benchee.run(
364+
%{
365+
"torque get" => fn -> for f <- fields, do: Torque.get(pre_doc, f) end,
366+
"torque get_many" => fn -> Torque.get_many(pre_doc, fields) end,
367+
"torque get_many_nil" => fn -> Torque.get_many_nil(pre_doc, fields) end,
368+
"torque get (unique_keys)" => fn -> for f <- fields, do: Torque.get(pre_doc_uk, f) end,
369+
"torque get_many (unique_keys)" => fn -> Torque.get_many(pre_doc_uk, fields) end,
370+
"torque get_many_nil (unique_keys)" => fn -> Torque.get_many_nil(pre_doc_uk, fields) end
358371
},
359372
warmup: 2,
360373
time: 5,

0 commit comments

Comments
 (0)