Skip to content

Commit 3803520

Browse files
lpgauthlpgauth
committed
Initial commit
 Torque is a high-performance JSON library for Elixir, backed by sonic-rs (SIMD-accelerated) via Rustler NIFs. Decoding: - Full decode: parse entire JSON binary into Elixir terms in one pass - Parse + Get: parse into opaque document reference, then extract fields by JSON Pointer (RFC 6901) path — ideal for selective field access (e.g. OpenRTB bid request processing) - get_many/2: extract multiple fields in a single NIF call to minimize NIF boundary crossings Encoding: - Direct term-walking encoder (no intermediate representation) - Supports maps (atom/binary keys), lists, binaries, numbers, booleans, nil, atoms, and jiffy-style {proplist} tuples - encode_to_iodata/1 fast path returning raw binary for direct I/O use Performance: - sonic-rs SIMD-accelerated parsing and serialization - Inputs >10 KB auto-dispatch to dirty CPU schedulers - get/2 always runs on normal schedulers (sub-microsecond pointer traversal) - Stack-allocated array for get_many up to 64 paths, heap fallback beyond - Zero-allocation atom name reads via stack buffer in encoder - itoa/ryu for fast integer/float formatting Targets: aarch64-apple-darwin, aarch64-unknown-linux-gnu, x86_64-apple-darwin, x86_64-unknown-linux-gnu (precompiled via RustlerPrecompiled, TORQUE_BUILD=true for local source builds).
0 parents  commit 3803520

29 files changed

Lines changed: 3057 additions & 0 deletions

‎.formatter.exs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[
2+
inputs: ["{mix,.formatter}.exs", "{config,lib,test,bench}/**/*.{ex,exs}"]
3+
]

‎.github/workflows/ci.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
env:
10+
MIX_ENV: test
11+
TORQUE_BUILD: true
12+
13+
jobs:
14+
format:
15+
name: Format
16+
runs-on: ubuntu-latest
17+
steps:
18+
- uses: actions/checkout@v6
19+
20+
- uses: erlef/setup-beam@v1
21+
with:
22+
otp-version: "27"
23+
elixir-version: "1.18"
24+
25+
- uses: dtolnay/rust-toolchain@stable
26+
with:
27+
components: rustfmt, clippy
28+
29+
- run: mix deps.get
30+
- run: mix format --check-formatted
31+
- run: cargo fmt --check
32+
- run: cargo clippy -- -D warnings
33+
34+
test:
35+
name: Test (OTP ${{ matrix.otp }} / Elixir ${{ matrix.elixir }})
36+
runs-on: ubuntu-latest
37+
strategy:
38+
matrix:
39+
include:
40+
- otp: "27"
41+
elixir: "1.18"
42+
- otp: "26"
43+
elixir: "1.17"
44+
steps:
45+
- uses: actions/checkout@v6
46+
47+
- uses: erlef/setup-beam@v1
48+
with:
49+
otp-version: ${{ matrix.otp }}
50+
elixir-version: ${{ matrix.elixir }}
51+
52+
- uses: dtolnay/rust-toolchain@stable
53+
54+
- run: mix deps.get
55+
- run: mix compile --warnings-as-errors
56+
- run: mix test

‎.github/workflows/release.yml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
name: Build precompiled NIFs
2+
3+
on:
4+
push:
5+
tags:
6+
- "v*"
7+
8+
jobs:
9+
create_release:
10+
name: Create GitHub release
11+
runs-on: ubuntu-22.04
12+
permissions:
13+
contents: write
14+
steps:
15+
- name: Create release
16+
env:
17+
GH_TOKEN: ${{ github.token }}
18+
run: gh release create "${{ github.ref_name }}" --repo "${{ github.repository }}" --title "${{ github.ref_name }}" --draft
19+
20+
build_release:
21+
name: NIF ${{ matrix.nif }} - ${{ matrix.job.target }} (${{ matrix.job.os }})
22+
needs: create_release
23+
runs-on: ${{ matrix.job.os }}
24+
25+
permissions:
26+
contents: write
27+
28+
strategy:
29+
fail-fast: false
30+
matrix:
31+
nif: ["2.15"]
32+
job:
33+
- { target: aarch64-apple-darwin, os: macos-14 }
34+
- { target: aarch64-unknown-linux-gnu, os: ubuntu-22.04, use-cross: true }
35+
- { target: x86_64-apple-darwin, os: macos-14 }
36+
- { target: x86_64-unknown-linux-gnu, os: ubuntu-22.04 }
37+
38+
steps:
39+
- name: Checkout source code
40+
uses: actions/checkout@v6
41+
42+
- name: Extract project version
43+
shell: bash
44+
run: |
45+
echo "PROJECT_VERSION=$(sed -n 's/^ @version "\(.*\)"/\1/p' mix.exs | head -n1)" >> $GITHUB_ENV
46+
47+
- name: Install Rust toolchain
48+
uses: dtolnay/rust-toolchain@stable
49+
with:
50+
target: ${{ matrix.job.target }}
51+
52+
- uses: Swatinem/rust-cache@v2
53+
with:
54+
prefix-key: v0-precomp
55+
shared-key: ${{ matrix.job.target }}-${{ matrix.nif }}
56+
57+
- name: Install cross
58+
if: matrix.job.use-cross
59+
shell: bash
60+
run: cargo install cross --git https://github.com/cross-rs/cross
61+
62+
- name: Build native library
63+
shell: bash
64+
run: |
65+
if [ "${{ matrix.job.use-cross }}" = "true" ]; then
66+
RUSTLER_NIF_VERSION=${{ matrix.nif }} cross build --release --target ${{ matrix.job.target }} --package torque_nif
67+
else
68+
RUSTLER_NIF_VERSION=${{ matrix.nif }} cargo build --release --target ${{ matrix.job.target }} --package torque_nif
69+
fi
70+
71+
- name: Package artifact
72+
shell: bash
73+
run: |
74+
TARGET=${{ matrix.job.target }}
75+
VERSION=${{ env.PROJECT_VERSION }}
76+
NIF=${{ matrix.nif }}
77+
78+
if [[ "$TARGET" == *"darwin"* ]]; then
79+
LIB_FILE="target/${TARGET}/release/libtorque_nif.dylib"
80+
else
81+
LIB_FILE="target/${TARGET}/release/libtorque_nif.so"
82+
fi
83+
84+
# RustlerPrecompiled expects the file inside the archive to match this name
85+
NIF_NAME="libtorque_nif-v${VERSION}-nif-${NIF}-${TARGET}.so"
86+
cp "${LIB_FILE}" "${NIF_NAME}"
87+
88+
ARCHIVE_NAME="${NIF_NAME}.tar.gz"
89+
tar -czf "${ARCHIVE_NAME}" "${NIF_NAME}"
90+
echo "ARCHIVE_NAME=${ARCHIVE_NAME}" >> $GITHUB_ENV
91+
92+
- name: Upload to GitHub release
93+
if: startsWith(github.ref, 'refs/tags/')
94+
shell: bash
95+
env:
96+
GH_TOKEN: ${{ github.token }}
97+
run: gh release upload "${{ github.ref_name }}" "${{ env.ARCHIVE_NAME }}" --clobber

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
*.ez
2+
/_build/
3+
/deps/
4+
/doc/
5+
/priv/native/
6+
**/target/
7+
erl_crash.dump

‎CLAUDE.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Build & Test Commands
6+
7+
```bash
8+
TORQUE_BUILD=true mix deps.get # fetch deps + force local Rust build
9+
TORQUE_BUILD=true mix compile # build (includes Rust NIF compilation)
10+
TORQUE_BUILD=true mix test # run all tests
11+
mix test test/torque_test.exs:42 # run single test by line number
12+
mix compile --warnings-as-errors # build with strict warnings
13+
mix format # format Elixir code
14+
mix format --check-formatted # check Elixir formatting
15+
mix dialyzer # static type analysis
16+
cargo fmt # format Rust code (run from repo root)
17+
cargo fmt --check # check Rust formatting
18+
cargo clippy -- -D warnings # Rust linter
19+
mix run bench/torque_bench.exs # run benchmarks (requires simdjsone + jiffy)
20+
```
21+
22+
`TORQUE_BUILD=true` is required for local development to force compilation from Rust source instead of downloading precompiled binaries. Without it, `RustlerPrecompiled` will try to fetch binaries from GitHub releases.
23+
24+
## Releasing
25+
26+
```bash
27+
./scripts/release.sh # tags, pushes, waits for CI, generates checksums
28+
```
29+
30+
The script reads the version from `mix.exs`, creates a git tag, waits for the release workflow to build precompiled NIFs for all targets, then generates checksums. After it completes, commit the checksum file and run `mix hex.publish`.
31+
32+
## Architecture
33+
34+
Torque is a high-performance JSON library for Elixir using Rustler NIFs backed by sonic-rs (SIMD-accelerated JSON).
35+
36+
### Decoding Strategies
37+
38+
1. **Parse + Get** — `parse/1` returns an opaque reference to a parsed document. `get/2,3` extracts fields by JSON Pointer (RFC 6901) path. `get_many/2` extracts multiple fields in a single NIF call. Ideal when only a subset of fields is needed (e.g. OpenRTB bid request processing).
39+
40+
2. **Full decode** — `decode/1` converts an entire JSON binary into Elixir terms in one pass.
41+
42+
### Encoding
43+
44+
`encode/1` walks Elixir terms directly (no intermediate representation) and writes JSON bytes to a buffer. Supports maps (atom/binary keys), lists, numbers, booleans, nil, and jiffy-style `{proplist}` tuples.
45+
46+
### Scheduler Awareness
47+
48+
Inputs larger than 10 KB are automatically dispatched to dirty CPU schedulers to avoid blocking normal BEAM schedulers. The `get/2` NIF always runs on a normal scheduler (sub-microsecond pointer traversal).
49+
50+
### Type Conversion
51+
52+
| JSON | Elixir |
53+
|------|--------|
54+
| object | map with binary keys |
55+
| array | list |
56+
| string | binary |
57+
| integer | integer (i64/u64) |
58+
| float | float |
59+
| true/false | true/false |
60+
| null | nil |
61+
62+
### Key Files
63+
64+
- `lib/torque.ex` — public API with `@doc`, typespecs, dirty scheduler dispatch
65+
- `lib/torque/native.ex` — RustlerPrecompiled NIF stubs (set `TORQUE_BUILD=true` to compile from source)
66+
- `native/torque_nif/src/lib.rs` — NIF registration, `ParsedDocument` resource
67+
- `native/torque_nif/src/decoder.rs` — parse, get, get_many, decode NIFs
68+
- `native/torque_nif/src/encoder.rs` — direct term-walking JSON encoder
69+
- `native/torque_nif/src/types.rs` — sonic_rs Value → Erlang term conversion
70+
- `native/torque_nif/src/atoms.rs` — cached atoms (ok, error, no_such_field, nil)

0 commit comments

Comments
 (0)