Add Esbuild support

[bretambrose]

Describe the feature

Using esbuild to build/package a JS project that uses the CRT is an exercise in frustration. We should smooth the sharp corners and provide README and/or sample-based guidance for people who wish to use esbuild.

Ideally we should provide solutions that

1. use the browser distribution
2. use the node distribution and support flexible lookups for the NAPI module, with detailed instructions

We may want to special case Lambda as well with a targeted sample.

We've also had a request for flexible NAPI module path lookup here: #214

I believe proper esbuild support would have to include flexible NAPI module resolution, and so I'm consolidating that request into this feature request as well.

Use Case

N/A

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[Lewenhaupt]

@bretambrose Any update on this? I was facing an issue with aws-iot-device-sdk and tried upgrading to v2 but I fail to figure out how I can make aws-crt avialble in the lambda. Both when bundle with esbuild but also when deploying it in a layer.
Any guidance would be greatly appreciated.

[bretambrose]

Don't have an update. It's in the backlog, but hasn't been prioritized/scheduled.

[mitgol]

The problem is in the code for the aws-crt package. In the node_modules/aws-crt/dist/native/binding.js file, there is this:

const binary_name = 'aws-crt-nodejs';
const platformDir = `${os_1.platform}-${os_1.arch}-${cRuntime}`;
let source_root = path.resolve(__dirname, '..', '..');
const dist = path.join(source_root, 'dist');
if ((0, fs_1.existsSync)(dist)) {
    source_root = dist;
}
const bin_path = path.resolve(source_root, 'bin');
const search_paths = [
    path.join(bin_path, platformDir, binary_name),
];
let binding;
for (const path of search_paths) {
    if ((0, fs_1.existsSync)(path + '.node')) {
        binding = require(path);
        break;
    }
}
if (binding == undefined) {
    throw new Error("AWS CRT binary not present in any of the following locations:\n\t" + search_paths.join('\n\t'));
}

As you can see, this is trying to load some binary modules. It is making the assumption that the code that is running is two directories down from the project root, and that there will be a binary in the dist/bin directory. That works fine when running on an EC2, but not when it’s packaged into a single file. When it runs, it yields this error:

{
  "errorType": "Error",
  "errorMessage": "AWS CRT binary not present in any of the following locations:\n\t/bin/linux-x64-glibc/aws-crt-nodejs",
  "trace": [
    "Error: AWS CRT binary not present in any of the following locations:",
    "\t/bin/linux-x64-glibc/aws-crt-nodejs",
    "    at node_modules/aws-crt/dist/native/binding.js (/var/task/index.js:157153:13)",
…
  ]
}

Clearly, it’s not going to find the binaries in /bin ! In fact, as far as I can tell, the binaries haven’t been copied anywhere.
To make it work, there are two steps. (1) add the binaries to the zip, and (2) Make sure there is a dot in front of the path.
When packaged by esbuild, it looks like this:

    var binding;
    for (const path2 of search_paths) {
      if ((0, fs_1.existsSync)(path2 + ".node")) {
        binding = require(path2);
        break;
      }
    }

It’s possible to fix it by making a change to one line:

    var binding;
    for (const path2x of search_paths) { const path2 = '.' + path2x;
      if ((0, fs_1.existsSync)(path2 + ".node")) {
        binding = require(path2);
        break;
      }
    }

I have the below in my package.json file to make sure this happens:

"clean": "rm -rf build/dist build/index.zip node_modules",

"build": "esbuild src/index.ts --bundle --sourcemap --platform=node --target=es2022 --outfile=build/dist/index.js && cp -r configuration/aws_lambda build",

"postbuild": "cp -r node_modules/aws-crt/dist/bin build/dist && cd build/dist && chmod -R ugo+rx bin && sed -i.bak \"s/^    for (const path2 of search_paths) {$/    for (const path2x of search_paths) { const path2 = '.' + path2x;/\" index.js && rm -f ../index.zip index.js.bak && zip -r ../index.zip .",

I should point out that I have not gotten it to work with webpack, because webpack is more intrusive in what it does to the code. Instead of using require, it has something called __webpack_require__ which is more complicated, and the simple hack above still doesn’t enable finding the binaries. Someone who understands it better (or at least has the time and inclination) should probably be able to figure out how to get that to work.

Here’s a link talking about webpack_require which may be of interest: https://devtools.tech/blog/understanding-webpacks-require---rid---7VvMusDzMPVh17YyHdyL

[alonw-cf]

@bretambrose we encounter the same issue here, where we use TypeScipt+esbuild for lambda functions. Any change to increase priority?

[justin-masse]

Having the same exact issue here, unable to load the binaries in lambda when using esbuild and compiling it all into one single file.

This is a major blocker for supporting multi-region services behind a cloudfront distribution. CloudFront cannot guarantee that you will stay within the region that the client request originated from. It is possible that CF directs someone in Virginia to Salt Lake City which would fall into us-west buckets instead of us-east. This means someone would be signing a sigv4 request expecting us-east-1 and then hits us-west-2 (thanks cloudfront) and the signature is not valid. Thus, we need sigv4a to work in lambda to support this * region.

[mitgol]

Justin - I am pretty sure that eslint just doesn't include the binaries in the giant zip file it makes. If you follow the changes I outlined on Dec 12 you should be able to put it together. Note the "cp -r node_modules/aws-crt/dist/bin build/dist" in the postbuild step which adds the binaries to the zip.

Having the same exact issue here, unable to load the binaries in lambda when using esbuild and compiling it all into one single file.

[justin-masse]

Justin - I am pretty sure that eslint just doesn't include the binaries in the giant zip file it makes. If you follow the changes I outlined on Dec 12 you should be able to put it together. Note the "cp -r node_modules/aws-crt/dist/bin build/dist" in the postbuild step which adds the binaries to the zip.

Having the same exact issue here, unable to load the binaries in lambda when using esbuild and compiling it all into one single file.

So I included aws-crt as a layer instead so it would have the files necessary but it still doesn't seem to work even when referencing the files from the layer. I'm not sure why it won't work as a layer unless the compiled index.js references are still broken even when using them?

[anisg]

Thank you @mitgol, for your assistance.

I encountered the same issue with my Node.js Lambda (ARM64) when trying to use a package that use aws-crt.

Fortunately, I managed to resolve it with following bash commands:

# generate folder "build" with index.js inside
esbuild --bundle --platform=node --target=node16 --minify --loader:.html=text --outfile=build/index.js lambda/lambda.ts 

# add binary "aws-crt-nodejs.node" to the folder "build", inside subfolder "build/bin/linux-arm64-glibc"
mkdir -p build/bin/linux-arm64-glibc && cp ./node_modules/aws-crt/dist/bin/linux-arm64/aws-crt-nodejs.node build/bin/linux-arm64-glibc/.

# fix the index.js output so it looks for the binary locally
node ./scripts/fix-aws-crt-not-found.js ./build/index.js 

# zipping the build folder and uploading it to lambda
rm -f ./build.zip && zip -j ./build.zip ./build/index.js && cd ./build && zip -r ../build.zip ./bin
aws lambda update-function-code --function-name YOUR_LAMBDA_NAME --zip-file fileb://build.zip --region YOUR_REGION --profile YOUR_PROFILE

with scripts/fix-aws-crt-not-found.js looking like this: (It adds a prefix "." to the search path for the binary, just as @mitgol suggested.)

// scripts/fix-aws-crt-not-found.js

const fget = (filepath) => require("fs").readFileSync(filepath).toString()
const fput = (filepath, data) => require("fs").writeFileSync(filepath, data)

const filename = process.argv[2]
const out = fget(filename).replace(
  /=\[(\w+)\.join(.*?)\+"\.node"\]/g,
  '=["."+$1.join$2+".node"]'
)
fput(filename, out)

Hope this helps a bit