bug(`@std/encoding`): Cannot perform Construct on a detached ArrayBuffer

[leighmcculloch]

Describe the bug

The following errors occurs when using @std/encoding v1.0.8 but not v1.0.7. It seems a bug has been introduced in v1.0.8. The issue still occurs with the latest version v1.0.10.

error: Uncaught (in promise) TypeError: Cannot perform Construct on a detached ArrayBuffer
    at new Uint8Array (<anonymous>)
    at fromArrayLike (ext:deno_node/internal/buffer.mjs:341:15)
    at fromObject (ext:deno_node/internal/buffer.mjs:353:12)
    at _from (ext:deno_node/internal/buffer.mjs:209:15)
    at Buffer.from (ext:deno_node/internal/buffer.mjs:234:10)
    at Object.actualMethods.sign (file:///Users/leighmcculloch/Library/Caches/deno/npm/registry.npmjs.org/@stellar/stellar-base/13.1.0/lib/signing.js:82:19)
    at sign (file:///Users/leighmcculloch/Library/Caches/deno/npm/registry.npmjs.org/@stellar/stellar-base/13.1.0/lib/signing.js:25:24)
    at Keypair.sign (file:///Users/leighmcculloch/Library/Caches/deno/npm/registry.npmjs.org/@stellar/stellar-base/13.1.0/lib/keypair.js:178:32)
    at file:///Users/leighmcculloch/Code/soroban-examples/main/multisig_1_of_n_account/stellar-cli-sign-auth-ed25519-js/main.ts:82:29
    at eventLoopTick (ext:core/01_core.js:178:7)

Steps to Reproduce

I'm triggering the error in a rather large workflow that I have not yet narrowed down to a small example, but I wanted to share this sooner rather than delay because the lib is still only a few patch versions away from when the erroring behaviour was introduced.

Checkout the code:

git clone https://github.com/stellar/soroban-examples
cd soroban-examples
git checkout add-multisig-1-of-n-account
cd multisig_1_of_n_account/stellar-cli-sign-auth-ed25519-js

Change the version to the version with the bug:

sed -i '' 's/@std\/encoding@1.0.7/@std\/encoding@1.0.8/g' deno.json

Run the command:

echo 'AAAAAgAAAABC6aNIWicaOZvbWjNaoGvec9eoaZtxiJSK40S9WE3jBAAMWRMACv+rAAAALQAAAAAAAAAAAAAAAQAAAAAAAAAYAAAAAAAAAAEApfy0ir3P25JeOT8QRjPyxAnQ8OLu2giwz56oChmeUwAAAARtaW50AAAAAgAAABIAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKAAAAAAAAAAAAAAAAAAAAewAAAAEAAAABAAAAAZeQKv/IzrOq8h1qTA8hvoVbXzIop3CGFC44ybxJu3UkEvzbsZmvfowAAAAAAAAAAQAAAAAAAAABAKX8tIq9z9uSXjk/EEYz8sQJ0PDi7toIsM+eqAoZnlMAAAAEbWludAAAAAIAAAASAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgAAAAAAAAAAAAAAAAAAAHsAAAAAAAAAAQAAAAAAAAADAAAABgAAAAEApfy0ir3P25JeOT8QRjPyxAnQ8OLu2giwz56oChmeUwAAABQAAAABAAAABgAAAAGXkCr/yM6zqvIdakwPIb6FW18yKKdwhhQuOMm8Sbt1JAAAABQAAAABAAAAB6J2Feel6+9l/OSx1PnVqbfuw2tQCicshc/wJpjForp9AAAAAgAAAAYAAAABAKX8tIq9z9uSXjk/EEYz8sQJ0PDi7toIsM+eqAoZnlMAAAAQAAAAAQAAAAIAAAAPAAAAB0JhbGFuY2UAAAAAEgAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAGAAAAAZeQKv/IzrOq8h1qTA8hvoVbXzIop3CGFC44ybxJu3UkAAAAFRL827GZr36MAAAAAAAGdVkAAAqYAAABKAAAAAAADFivAAAAAA==' \
    | deno run --allow-read --no-prompt main.ts \
        --secret-key 0000000000000000000000000000000000000000000000000000000000000001 \
        --signature-expiration-ledger 2296800

If the command is run with @std/encoding on v1.0.7, then it works fine.

If the command is run with @std/encoding on v1.0.8, the error above is output.

Expected behavior

No error about a detached ArrayBuffer.

Environment

• OS: macOS 15
• deno version: 2.3.1
• std version: v1.0.8

[BlackAsLight]

I can't seem to find this main.ts file in the code to run, but if you're using the unstable imports from encoding then they detach the input buffers to reuse them for the output buffer. If you still need the input buffer then you need to pass in a copy.

[leighmcculloch]

The main.ts file is located here:
https://github.com/stellar/soroban-examples/tree/add-multisig-1-of-n-account/multisig_1_of_n_account/stellar-cli-sign-auth-ed25519-js

[BlackAsLight]

I believe I figured out the problem after falling down the rabbit hole that is npm dependencies.

At line 33 here you are passing the result of keypair.rawPublicKey() to encodeHex.

encodeHex can accept a string, Uint8Array, or ArrayBuffer, but looking at the source code of @stellar/stellar-base here (a dependency of @stellar/stellar-sdk, which in turn is your dependency), it is returning a nodejs Buffer.

Since Buffer isn't one of the expected types, the JavaScript code falls down to the else statement here and calls .slice() on it, expecting it to be a Uint8Array.

But according to nodejs here, the Buffer.slice() is not the same as a Uint8Array.slice() and is equivalent to the Uint8Array.subarray(). Because of this, a copy of the buffer wasn't made and the underlying buffer of Buffer is then detached, causing you to have an error later in the code when you try and use that same buffer.

A simple fix for this would be to convert your Buffer to a Uint8Array or ArrayBuffer before passing it in. Line 79 might be a similar situation here, but I don't want to figure out it's type

[leighmcculloch]

Wow, very interesting.

So I think it might be the other call to encodeHex that's the problem, because the failure is on line 82 where the payload_hash is reused after it was given to encodeHex on line 79. So it seems like the problem that's arising from the wrong Buffer type use on line 79 is causing the failure on line 82.

Looking at encodeHex across those two versions, see below, it does appear it's been refactored significantly internally and makes new assumptions about what the type is. Those assumptions didn't exist prior in 1.0.7, and those assumptions were stated in the type list, but seemingly still worked regardless even if you passed in other types.

Looks like the change was made in fa656d0 (#6499) as a performance optimisation.

I think the performance optimisation has come at a cost of encodeHex being more brittle, if passing the wrong type to it can seemingly work and then fail later on when the value is reused.

Thoughts?

@std/encoding@1.0.7:

export function encodeHex(src: string | Uint8Array | ArrayBuffer): string {
  const u8 = validateBinaryLike(src);

  const dst = new Uint8Array(u8.length * 2);
  for (let i = 0; i < u8.length; i++) {
    const v = u8[i]!;
    dst[i * 2] = hexTable[v >> 4]!;
    dst[i * 2 + 1] = hexTable[v & 0x0f]!;
  }
  return textDecoder.decode(dst);
}

@std/encoding@1.0.8:

export function encodeHex(src: string | Uint8Array | ArrayBuffer): string {
  if (typeof src === "string") {
    src = new TextEncoder().encode(src) as Uint8Array_;
  } else if (src instanceof ArrayBuffer) src = new Uint8Array(src).slice();
  else src = src.slice();
  const [output, i] = detach(
    src as Uint8Array_,
    calcMax((src as Uint8Array_).length),
  );
  encode(output, i, 0, alphabet);
  return new TextDecoder().decode(output);
}

[leighmcculloch]

Ok yeah, I just confirmed. If the following change is made, removing the use of encodeHex(payload_hash) (line 79) causes the failure with keypair.sign(payload_hash) (line 82) to not longer occur on @encoding/std@1.0.8.

diff --git i/multisig_1_of_n_account/stellar-cli-sign-auth-ed25519-js/main.ts w/multisig_1_of_n_account/stellar-cli-sign-auth-ed25519-js/main.ts
index 5fd8559..ed65e0d 100644
--- i/multisig_1_of_n_account/stellar-cli-sign-auth-ed25519-js/main.ts
+++ w/multisig_1_of_n_account/stellar-cli-sign-auth-ed25519-js/main.ts
@@ -76,7 +76,7 @@ for (const [invocation, creds] of auths) {
     }),
   ).toXDR();
   const payload_hash = hash(payload);
-  stderr("Payload:", encodeHex(payload_hash));
+  //stderr("Payload:", encodeHex(payload_hash));
 
   // Sign the payload hash.
   const signature = keypair.sign(payload_hash);

[BlackAsLight]

I wouldn't characterise it as becoming more brittle. It by chance worked with a type it wasn't expecting previously. One can't guarantee that behaviour will stay the same with unexpected input.

To pass it the correct type, all you need to do is call .buffer on keypair.rawPublicKey() to pass in an ArrayBuffer instead. And same on line 79.