Skip to content

HexDecode is scalar: Buffer.from(hex) ~5x slower than base64 decode #19

Open

Description

@hood

HexDecode decodes one byte per iteration through unhex_table early
exiting on the first invalid digit.
The early exit keeps compilers from vectorizing the loop.

For this very reason in node, Buffer.from(str, 'hex') and buf.write(str, 'hex') run ~5x slower than base64 decoding (which uses simdutf),
even though hex is the simpler format 😄.

Measuring this on Node.js v26.13.0 (official build), Apple M1 Pro, median ns/op:

Input bytes Buffer.from(hex) Buffer.from(base64)
256 266 132
1024 782 212
16384 11347 2068

The cost is all in HexDecode: the out-of-line function alone takes 10.4 µs
for 16 KiB for -O3 (~0.65 ns/byte).

Just sanity checking thex existence of a "famous" consumer: Postgres bytea
(node-postgres decodes it with Buffer.from(hex)), keys and digests decoded
from hex, and ObjectId/UUID-style ids.

Proposal: decode whole 16-byte output blocks with NEON (on arm64) and SSE2
(on x86-64 baseline, no SSSE3 needed), and fall back to the existing scalar
loop at the first block that contains an invalid character. The scalar loop
then finds the exact stopping point, so results stay byte-identical to today's,
including truncation at the first invalid pair. To clarify: the change only
affects one-byte input, as two-byte strings keep the scalar path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions