Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(encoding/unstable): add format option to encodeHex and decodeHex #6480

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
d4ba5bf
perf(encoding): base64
BlackAsLight Feb 22, 2025
c1eccd4
perf(encoding): base64url
BlackAsLight Feb 22, 2025
df36993
perf(encoding): base32
BlackAsLight Feb 23, 2025
ea879eb
perf(encoding): base32hex
BlackAsLight Feb 23, 2025
2103c65
chore(encoding): add missing copyright comment
BlackAsLight Feb 23, 2025
ba88700
adjust(encoding): base64 to use `switch` instead of `if`
BlackAsLight Feb 23, 2025
8f671ae
chore(encoding): update exports
BlackAsLight Feb 23, 2025
56b6f99
chore(encoding): fmt
BlackAsLight Feb 23, 2025
71e0246
perf(encoding): hex
BlackAsLight Feb 23, 2025
a0ac03e
chore(encoding): remove file committed on accident
BlackAsLight Feb 23, 2025
927c2e7
perf(encoding): update streams to be more memory efficient
BlackAsLight Feb 23, 2025
39a3026
perf(encoding): base32crockford
BlackAsLight Feb 23, 2025
6bb70e5
Merge branch 'denoland:main' into encoding_refactor
BlackAsLight Feb 23, 2025
994124b
chore(encoding): fix docs
BlackAsLight Feb 23, 2025
01477c2
revert(encoding): perf brought to stable exports
BlackAsLight Feb 23, 2025
5acf1fc
docs(encoding): fix mismatch type
BlackAsLight Feb 23, 2025
6112d8a
chore(encoding): add missing export
BlackAsLight Feb 23, 2025
4bb9d91
chore(encoding): remove file committed on accident
BlackAsLight Feb 23, 2025
3b91390
perf(encoding): base32crockford stream
BlackAsLight Feb 23, 2025
75778c2
fix(encoding): types for deno v1.x
BlackAsLight Feb 23, 2025
d4fcd64
fix(encoding): types for deno v1.x
BlackAsLight Feb 23, 2025
ec1d78e
refactor(encoding): to bring perf to stable packages
BlackAsLight Feb 26, 2025
78e3c9d
refactor(encoding): merging same base functions & adding tests & docs
BlackAsLight Feb 26, 2025
0bec310
chore(tools): update check_docs.ts entrypoint
BlackAsLight Feb 26, 2025
8edd5e2
chore(encoding): fix types of deno v1.x
BlackAsLight Feb 26, 2025
8cffee7
Merge branch 'main' into encoding_refactor
BlackAsLight Feb 26, 2025
af39bc0
revert(encoding): all changes except unstable hex
BlackAsLight Feb 27, 2025
2b9e86d
revert(tools): change to check_docs.ts
BlackAsLight Feb 27, 2025
92ff5e0
revert(encoding): missing changes
BlackAsLight Feb 27, 2025
bb131a3
Merge branch 'main' into encoding_hex_sync
BlackAsLight Mar 13, 2025
5ed44d5
revert(encoding): changes to hex stream
BlackAsLight Mar 13, 2025
fa8130e
perf(encoding): simplify maths being performed
BlackAsLight Mar 13, 2025
96533b4
add(encoding): default value for format argument
BlackAsLight Mar 13, 2025
c26dd57
test(encoding): added to validate function throws on invalid byte
BlackAsLight Mar 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _tools/check_docs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ const ENTRY_POINTS = [
"../encoding/mod.ts",
"../encoding/unstable_base32.ts",
"../encoding/unstable_base64.ts",
"../encoding/unstable_hex.ts",
"../encoding/unstable_base64_stream.ts",
"../encoding/unstable_base32hex_stream.ts",
"../encoding/unstable_base32_stream.ts",
"../encoding/unstable_base32hex_stream.ts",
"../encoding/unstable_hex_stream.ts",
Expand Down
65 changes: 65 additions & 0 deletions encoding/_common16.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
// Copyright 2018-2025 the Deno authors. MIT license.

import type { Uint8Array_ } from "./_types.ts";
export type { Uint8Array_ };

/**
* Calculate the output size needed to encode a given input size for
* {@linkcode encodeRawHex}.
*
* @param originalSize The size of the input buffer.
* @returns The size of the output buffer.
*
* @example Basic Usage
* ```ts
* import { assertEquals } from "@std/assert";
* import { calcMax } from "@std/encoding/unstable-hex";
*
* assertEquals(calcMax(1), 2);
* ```
*/
export function calcMax(originalSize: number): number {
return originalSize * 2;
}

export function encode(
buffer: Uint8Array_,
i: number,
o: number,
alphabet: Uint8Array,
): number {
for (; i < buffer.length; ++i) {
const x = buffer[i]!;
buffer[o++] = alphabet[x >> 4]!;
buffer[o++] = alphabet[x & 0xF]!;
}
return o;
}

export function decode(
buffer: Uint8Array_,
i: number,
o: number,
alphabet: Uint8Array,
): number {
if ((buffer.length - o) % 2 === 1) {
throw new TypeError(
`Invalid Character (${String.fromCharCode(buffer[buffer.length - 1]!)})`,
);
}

i += 1;
for (; i < buffer.length; i += 2) {
buffer[o++] = (getByte(buffer[i - 1]!, alphabet) << 4) |
getByte(buffer[i]!, alphabet);
}
return o;
}

function getByte(char: number, alphabet: Uint8Array): number {
const byte = alphabet[char] ?? 16;
if (byte === 16) { // alphabet.Hex.length
throw new TypeError(`Invalid Character (${String.fromCharCode(char)})`);
}
return byte;
}
1 change: 1 addition & 0 deletions encoding/deno.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
"./unstable-base64-stream": "./unstable_base64_stream.ts",
"./base64url": "./base64url.ts",
"./hex": "./hex.ts",
"./unstable-hex": "./unstable_hex.ts",
"./unstable-hex-stream": "./unstable_hex_stream.ts",
"./varint": "./varint.ts"
}
Expand Down
216 changes: 216 additions & 0 deletions encoding/unstable_hex.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
// Copyright 2018-2025 the Deno authors. MIT license.
// This module is browser compatible.

/**
* Functions to encode and decode to and from hexadecimal strings.
*
* ```ts
* import { assertEquals } from "@std/assert";
* import { encodeHex, type Uint8Array_ } from "@std/encoding/unstable-hex";
*
* assertEquals(encodeHex("Hello World", "Hex"), "48656c6c6f20576f726c64");
* assertEquals(
* encodeHex(new TextEncoder().encode("Hello World") as Uint8Array_, "Hex"),
* "48656c6c6f20576f726c64",
* );
* ```
*
* @see {@link https://www.rfc-editor.org/rfc/rfc4648.html#section-8}
* @experimental **UNSTABLE**: New API, yet to be vetted.
*
* @module
*/

import type { Uint8Array_ } from "./_types.ts";
export type { Uint8Array_ };
import { calcMax, decode, encode } from "./_common16.ts";
export { calcMax };
import { detach } from "./_common_detach.ts";

const alphabet: Record<HexFormat, Uint8Array> = {
Hex: new TextEncoder().encode("0123456789abcdef"),
};
const rAlphabet: Record<HexFormat, Uint8Array> = {
Hex: new Uint8Array(128).fill(16), // alphabet.Hex.length
};
alphabet.Hex.forEach((byte, i) => rAlphabet.Hex[byte] = i);
new TextEncoder()
.encode("ABCDEF")
.forEach((byte, i) => rAlphabet.Hex[byte] = i + 10);

/**
* The hex encoding formats.
*/
export type HexFormat = "Hex";

/**
* `encodeHex` takes an input source and encodes it into a hexadecimal string.
* If a {@linkcode Uint8Array<ArrayBuffer>} or {@linkcode ArrayBuffer} is
* provided, the underlying source will be detached and reused for the encoding.
* If you need the input source after providing it to this function, call
* `.slice()` to pass in a copy.
*
* @experimental **UNSTABLE**: New API, yet to be vetted.
*
* @param input The input source to encode.
* @param format The format to use for encoding.
* @returns The hexadecimal string representation of the input.
*
* @example Basic Usage
* ```ts
* import { assertEquals } from "@std/assert";
* import { encodeHex, type Uint8Array_ } from "@std/encoding/unstable-hex";
*
* assertEquals(encodeHex("Hello World", "Hex"), "48656c6c6f20576f726c64");
* assertEquals(
* encodeHex(new TextEncoder().encode("Hello World") as Uint8Array_, "Hex"),
* "48656c6c6f20576f726c64",
* );
* ```
*/
export function encodeHex(
input: string | Uint8Array_ | ArrayBuffer,
format: HexFormat = "Hex",
): string {
if (typeof input === "string") {
input = new TextEncoder().encode(input) as Uint8Array_;
} else if (input instanceof ArrayBuffer) {
input = new Uint8Array(input);
}
const [output, i] = detach(
input as Uint8Array_,
calcMax((input as Uint8Array_).length),
);
encode(output, i, 0, alphabet[format]);
return new TextDecoder().decode(output);
}

/**
* `encodeRawHex` is a low-level function that encodes a
* {@linkcode Uint8Array<ArrayBuffer>} to hexadecimal in place. The function
* assumes that the raw data starts at param {@linkcode i} and ends at the end
* of the buffer, and that the entire buffer provided is large enough to hold
* the encoded data.
*
* @experimental **UNSTABLE**: New API, yet to be vetted.
*
* @param buffer The buffer to encode in place.
* @param i The index of where the raw data starts reading from.
* @param o The index of where the encoded data starts writing to.
* @param format The format to use for encoding.
* @returns The index of where the encoded data finished writing to.
*
* @example Basic Usage
* ```ts
* import { assertEquals } from "@std/assert";
* import { calcMax, encodeHex, encodeRawHex } from "@std/encoding/unstable-hex";
*
* const prefix = new TextEncoder().encode("data:url/fake,");
* const input = await Deno.readFile("./deno.lock");
*
* const originalSize = input.length;
* const newSize = prefix.length + calcMax(originalSize);
* const i = newSize - originalSize;
* const o = prefix.length;
*
* // deno-lint-ignore no-explicit-any
* const output = new Uint8Array((input.buffer as any).transfer(newSize));
* output.set(output.subarray(0, originalSize), i);
* output.set(prefix);
*
* encodeRawHex(output, i, o, "Hex");
* assertEquals(
* new TextDecoder().decode(output),
* "data:url/fake," + encodeHex(await Deno.readFile("./deno.lock"), "Hex"),
* );
* ```
*/
export function encodeRawHex(
buffer: Uint8Array_,
i: number,
o: number,
format: HexFormat = "Hex",
): number {
const max = calcMax(buffer.length - i);
if (max > buffer.length - o) throw new RangeError("Buffer too small");
return encode(buffer, i, o, alphabet[format]);
}

/**
* `decodeHex` takes an input source and decodes it into a
* {@linkcode Uint8Array<ArrayBuffer>} using the specified format.
*
* @experimental **UNSTABLE**: New API, yet to be vetted.
*
* @param input The input source to decode.
* @param format The format to use for decoding.
* @returns The decoded {@linkcode Uint8Array<ArrayBuffer>}.
*
* @example Basic Usage
* ```ts
* import { assertEquals } from "@std/assert";
* import { decodeHex } from "@std/encoding/unstable-hex";
*
* assertEquals(
* decodeHex("48656c6c6f20576f726c64", "Hex"),
* new TextEncoder().encode("Hello World"),
* );
* ```
*/
export function decodeHex(
input: string,
format: HexFormat = "Hex",
): Uint8Array_ {
const output = new TextEncoder().encode(input) as Uint8Array_;
return output
.subarray(0, decode(output, 0, 0, rAlphabet[format]));
}

/**
* `decodeRawHex` is a low-level function that decodes a
* {@linkcode Uint8Array<ArrayBuffer>} from hexadecimal in place. Param
* {@linkcode i} must be greater than or equal to param {@linkcode o}. The
* function assumes that the encoded data starts at param {@linkcode i} and ends
* at the end of the buffer.
*
* @experimental **UNSTABLE**: New API, yet to be vetted.
*
* @param buffer The buffer to decode in place.
* @param i The index of where the encoded data starts reading from.
* @param o The index of where the decoded data starts writing to.
* @param format The format to use for decoding.
* @returns The index of where the decoded data finished writing to.
*
* @example Basic Usage
* ```ts
* import { assertEquals } from "@std/assert";
* import {
* decodeRawHex,
* encodeHex,
* type Uint8Array_,
* } from "@std/encoding/unstable-hex";
*
* let buffer = new TextEncoder().encode(
* "data:url/fake," + encodeHex(await Deno.readFile("./deno.lock"), "Hex"),
* ) as Uint8Array_;
*
* const i = buffer.indexOf(",".charCodeAt(0)) + 1;
* const o = decodeRawHex(buffer, i, i, "Hex");
*
* buffer = buffer.subarray(i, o);
* assertEquals(buffer, await Deno.readFile("./deno.lock"));
* ```
*/
export function decodeRawHex(
buffer: Uint8Array_,
i: number,
o: number,
format: HexFormat = "Hex",
): number {
if (i < o) {
throw new RangeError(
"Input (i) must be greater than or equal to output (o)",
);
}
return decode(buffer, i, o, rAlphabet[format]);
}
Loading
Loading