"Runtime.ImportModuleError: Error: Cannot find module '@libsql/linux-x64-gnu" when bundling with esbuild for aws lambda

[Mdev303]

I was switching my database from planetscale to turso but I'm getting the following error

 {
  "errorType": "Runtime.ImportModuleError",
  "errorMessage": "Error: Cannot find module '@libsql/linux-x64-gnu'\nRequire stack:\n- /var/task/index.js\n- /var/runtime/index.mjs",
  "trace": [
    "Runtime.ImportModuleError: Error: Cannot find module '@libsql/linux-x64-gnu'",
    "Require stack:",
    "- /var/task/index.js",
    "- /var/runtime/index.mjs",
    "    at _loadUserApp (file:///var/runtime/index.mjs:1061:17)",
    "    at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1093:21)",
    "    at async start (file:///var/runtime/index.mjs:1256:23)",
    "    at async file:///var/runtime/index.mjs:1262:1"
  ]
}

I use the aws cdk to build and deploy my aws lambda function internally it uses esbuild here is a simple reproduction step

the aws lambda function code
/nodejs/index.ts

import { drizzle } from 'drizzle-orm/libsql';
import { createClient } from '@libsql/client';
const client = createClient({ url: 'DATABASE_URL', authToken: 'DATABASE_AUTH_TOKEN' });

const db = drizzle(client);

export const handler = async (event) => {
  console.log('event', event);
}

the aws cdk code to deploy the function:

import {Construct} from 'constructs';
import {NodejsFunction} from 'aws-cdk-lib/aws-lambda-nodejs';
import {Architecture, Runtime} from 'aws-cdk-lib/aws-lambda';

export class SendUserToDynamo extends Construct {
  function: NodejsFunction;
  constructor(scope: Construct, id: string) {
    super(scope, id);

    // lambda function that triggers on aws lambda user created event
    this.function = new NodejsFunction(this, 'sendUserToDynamo', {
      entry: __dirname + '/nodejs/index.ts',
      handler: 'handler',
      architecture: Architecture.X86_64,
      runtime: Runtime.NODEJS_18_X,
    });
  }
}

Everything works when I'm using the PlanetScale driver, so the error must come from libsql and not the CDK build step. I tried using a Dockerfile without TypeScript and ESBuild, and it did work

[hugotox]

Similar error happens when deploying to vercel. Any advice?

Update
I reverted back to version 0.3.2 and the issue is gone

[raymclee]

same here
and confirm downgrading to 0.3.2 fixed it

[raymclee]

0.3.3 also works

[ebacksys]

bump

[thomascgray]

Confirming I'm seeing this too, also switching from Planetscale to Turso.
Interestingly the error doesn't happen in local env, only when deployed to Netlify. Running node 16.
Pulling @libsql/client down to 0.3.2 from 0.5.3 fixed things for me.

[jschuur]

Ran into this too and got it working with 0.3.3.

[bchilcott]

Having a similar issue, but I'm on Windows 11 and getting:

Cannot find module '@libsql/win32-x64-msvc'

Works with 0.3.3 for now, but no later versions.

[yspreen]

I'm experiencing this on an arm lambda: tursodatabase/libsql-js#70

edit: 0.3.3 does fix this.

[jschuur]

I ended up using the web/HTTP client to be able to run the latest version:

import { createClient } from '@libsql/client/web';

[yspreen]

it does say in the docs that you should use it for vercel functions and the like. the docs could be a bit clearer here. nothing mentions what to use for lambda

[thescientist13]

Just wanted to chime in that I also ran into this exact same issue deploying to AWS Lambda using Architect and on version 0.6.0 of @libsql/client. Including that it all worked fine locally. (using the Architect's sandbox mode for local dev)

The suggestion to use web worked for me. 👍

[patrickalima98]

The same problem here using versions 0.5.x and 0.6.x with Deno Deploy with NPM comparability layer. I tried to use it with esm.sh but I didn't have any success.

[dreamorosi]

If you want to use the native client that uses the libsql binary under the hood in Lambda, you'll have to build the function in a similar arch/environment or use a Lambda layer, also built under the same conditions (i.e. Docker).

A bundler like esbuild won't be able to include the correct version (i.e. @libsql-linux-arm64-gnu) unless it's itself running on that same hardware.

This worked for me:

import { Stack, type StackProps, CfnOutput } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { Architecture, Code, LayerVersion, Runtime } from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction, OutputFormat } from 'aws-cdk-lib/aws-lambda-nodejs';

export class LibsqlStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const libsqlLayer = new LayerVersion(this, 'LibsqlLayer', {
      compatibleArchitectures: [Architecture.ARM_64],
      compatibleRuntimes: [Runtime.NODEJS_20_X],
      code: Code.fromAsset('./layers/libsql/nodejs', {
        bundling: {
          image: Runtime.NODEJS_20_X.bundlingImage,
          environment: {
            npm_config_cache: "/tmp/npm_cache",
            npm_config_update_notifier: "false",
          },
          command: [
            'bash',
            '-xc',
            [
              'cd $(mktemp -d)',
              'cp /asset-input/package* .',
              'npm --prefix . i @libsql/client',
              'cp -r node_modules /asset-output/'
            ].join(' && '),
          ]
        }
      }),
    });

    const fn = new NodejsFunction(this, 'MyFunction', {
      runtime: Runtime.NODEJS_20_X,
      architecture: Architecture.ARM_64,
      entry: './src/index.ts',
      handler: 'handler',
      bundling: {
        minify: true,
        mainFields: ['module', 'main'],
        sourceMap: true,
        format: OutputFormat.ESM,
        externalModules: ['@libsql/client'],
      },
      layers: [libsqlLayer],
    });

    new CfnOutput(this, 'FunctionArn', {
      value: fn.functionArn,
    });
  }
}

then in my function I use @libsql/client as normal:

import { Logger } from "@aws-lambda-powertools/logger";
import { createClient } from "@libsql/client";

const logger = new Logger();
const db = createClient({
  url: "file:/tmp/sqlite.db",
});

export const handler = async () => {

  await db.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)");

  await db.execute({ sql: "INSERT INTO users (name) VALUES (?)", args:  ["Alice"]});

  const rows = await db.execute("SELECT * FROM users");

  logger.info("Rows: ", {rows});

  return {
    statusCode: 200,
    body: JSON.stringify("Hello, World!"),
  };
};

which when invoked prints this:

{
    "level": "INFO",
    "message": "Rows: ",
    "sampling_rate": 0,
    "service": "service_undefined",
    "timestamp": "2024-06-27T18:14:43.388Z",
    "xray_trace_id": "1-667dac12-2624c2867735f80f34586f58",
    "rows": {
        "columns": [
            "id",
            "name"
        ],
        "columnTypes": [
            "INTEGER",
            "TEXT"
        ],
        "rows": [
            [
                1,
                "Alice"
            ]
        ],
        "rowsAffected": 0,
        "lastInsertRowid": null
    }
}

The advantage of using a Lambda layer like that is that you can still continue using esbuild for the function(s) - the only caveat there is to make sure to specify externalModules: ['@libsql/client'], so that the dependency is not included in the bundle - this is because it comes from the Layer.

[NicoPowers]

If you want to use the native client that uses the libsql binary under the hood in Lambda, you'll have to build the function in a similar arch/environment or use a Lambda layer, also built under the same conditions (i.e. Docker).

A bundler like esbuild won't be able to include the correct version (i.e. @libsql-linux-arm64-gnu) unless it's itself running on that same hardware.

This worked for me:

import { Stack, type StackProps, CfnOutput } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { Architecture, Code, LayerVersion, Runtime } from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction, OutputFormat } from 'aws-cdk-lib/aws-lambda-nodejs';

export class LibsqlStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const libsqlLayer = new LayerVersion(this, 'LibsqlLayer', {
      compatibleArchitectures: [Architecture.ARM_64],
      compatibleRuntimes: [Runtime.NODEJS_20_X],
      code: Code.fromAsset('./layers/libsql/nodejs', {
        bundling: {
          image: Runtime.NODEJS_20_X.bundlingImage,
          environment: {
            npm_config_cache: "/tmp/npm_cache",
            npm_config_update_notifier: "false",
          },
          command: [
            'bash',
            '-xc',
            [
              'cd $(mktemp -d)',
              'cp /asset-input/package* .',
              'npm --prefix . i @libsql/client',
              'cp -r node_modules /asset-output/'
            ].join(' && '),
          ]
        }
      }),
    });

    const fn = new NodejsFunction(this, 'MyFunction', {
      runtime: Runtime.NODEJS_20_X,
      architecture: Architecture.ARM_64,
      entry: './src/index.ts',
      handler: 'handler',
      bundling: {
        minify: true,
        mainFields: ['module', 'main'],
        sourceMap: true,
        format: OutputFormat.ESM,
        externalModules: ['@libsql/client'],
      },
      layers: [libsqlLayer],
    });

    new CfnOutput(this, 'FunctionArn', {
      value: fn.functionArn,
    });
  }
}

then in my function I use @libsql/client as normal:

import { Logger } from "@aws-lambda-powertools/logger";
import { createClient } from "@libsql/client";

const logger = new Logger();
const db = createClient({
  url: "file:/tmp/sqlite.db",
});

export const handler = async () => {

  await db.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)");

  await db.execute({ sql: "INSERT INTO users (name) VALUES (?)", args:  ["Alice"]});

  const rows = await db.execute("SELECT * FROM users");

  logger.info("Rows: ", {rows});

  return {
    statusCode: 200,
    body: JSON.stringify("Hello, World!"),
  };
};

which when invoked prints this:

{
    "level": "INFO",
    "message": "Rows: ",
    "sampling_rate": 0,
    "service": "service_undefined",
    "timestamp": "2024-06-27T18:14:43.388Z",
    "xray_trace_id": "1-667dac12-2624c2867735f80f34586f58",
    "rows": {
        "columns": [
            "id",
            "name"
        ],
        "columnTypes": [
            "INTEGER",
            "TEXT"
        ],
        "rows": [
            [
                1,
                "Alice"
            ]
        ],
        "rowsAffected": 0,
        "lastInsertRowid": null
    }
}

The advantage of using a Lambda layer like that is that you can still continue using esbuild for the function(s) - the only caveat there is to make sure to specify externalModules: ['@libsql/client'], so that the dependency is not included in the bundle - this is because it comes from the Layer.

Hey @dreamorosi , thanks for this.

I am currently trying to get this to work with SST Ion that uses Pulumi under the hood, and it seems it won't bundle this automatically as CDK does.

How would I go about bundling this manually? Here you can see what Pulumi offers from an AWS lambda layer standpoint: https://www.pulumi.com/registry/packages/aws/api-docs/lambda/layerversion/#constructor-example

As you can see, there is no option to set the command or the bundling configuration. Same with the Lambda itself, as seen here: https://www.pulumi.com/registry/packages/aws/api-docs/lambda/function/#constructor-example