How to build this as a wasm extension

[srossross]

I see that there are wasm_mvp, wasm_threads and wasm_eh targets being built in CI however I get a number of errors trying to build locally

Are there instructions on how to build and test a wasm extension locally?

make [wasm_eh|wasm_threads]
...
CMake Error at /opt/homebrew/Cellar/cmake/3.29.6/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
  system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
  OPENSSL_INCLUDE_DIR)
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.29.6/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /opt/homebrew/Cellar/cmake/3.29.6/share/cmake/Modules/FindOpenSSL.cmake:686 (find_package_handle_standard_args)
  <DIR>/CMakeLists.txt:9 (find_package)
...

make wasm_mvp
mkdir -p build/wasm_mvp
emcmake cmake  -DDUCKDB_EXTENSION_CONFIGS='<DIR>/extension_config.cmake' -DWASM_LOADABLE_EXTENSIONS=1 -DBUILD_EXTENSIONS_ONLY=1 -DSKIP_EXTENSIONS="parquet;json"  -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=<HOME>/Documents/github.com/emscripten-core/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -Bbuild/wasm_mvp -DCMAKE_CXX_FLAGS="" -S "./duckdb/" -DDUCKDB_EXPLICIT_PLATFORM=wasm_mvp -DDUCKDB_CUSTOM_PLATFORM=wasm_mvp
configure: cmake -DDUCKDB_EXTENSION_CONFIGS=<DIR>/extension_config.cmake -DWASM_LOADABLE_EXTENSIONS=1 -DBUILD_EXTENSIONS_ONLY=1 -DSKIP_EXTENSIONS=parquet;json -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=<HOME>/Documents/github.com/emscripten-core/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -Bbuild/wasm_mvp -DCMAKE_CXX_FLAGS= -S ./duckdb/ -DDUCKDB_EXPLICIT_PLATFORM=wasm_mvp -DDUCKDB_CUSTOM_PLATFORM=wasm_mvp -DCMAKE_TOOLCHAIN_FILE=<HOME>/Documents/github.com/emscripten-core/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=<HOME>/Documents/github.com/emscripten-core/emsdk/node/18.20.3_64bit/bin/node
-- Running vcpkg install
Detecting compiler hash for triplet arm64-osx...
Compiler found: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
The following packages will be removed:
openssl is compatible with built-in CMake targets:

  find_package(OpenSSL REQUIRED)
  target_link_libraries(main PRIVATE OpenSSL::SSL)
  target_link_libraries(main PRIVATE OpenSSL::Crypto)

-- Running vcpkg install - done
CMake Error at /opt/homebrew/Cellar/cmake/3.29.6/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.29.6/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /opt/homebrew/Cellar/cmake/3.29.6/share/cmake/Modules/FindThreads.cmake:226 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  <DIR>/vcpkg/scripts/buildsystems/vcpkg.cmake:859 (_find_package)
  CMakeLists.txt:25 (find_package)


-- Configuring incomplete, errors occurred!
emcmake: error: 'cmake -DDUCKDB_EXTENSION_CONFIGS=<DIR>/extension_config.cmake -DWASM_LOADABLE_EXTENSIONS=1 -DBUILD_EXTENSIONS_ONLY=1 -DSKIP_EXTENSIONS=parquet;json -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=<HOME>/Documents/github.com/emscripten-core/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -Bbuild/wasm_mvp -DCMAKE_CXX_FLAGS= -S ./duckdb/ -DDUCKDB_EXPLICIT_PLATFORM=wasm_mvp -DDUCKDB_CUSTOM_PLATFORM=wasm_mvp -DCMAKE_TOOLCHAIN_FILE=<HOME>/Documents/github.com/emscripten-core/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=<HOME>/Documents/github.com/emscripten-core/emsdk/node/18.20.3_64bit/bin/node' failed (returned 1)
make: *** [wasm_mvp] Error 1

[carlopi]

Thanks, this does makes sense, I will need to come back on writing this properly, expecially on the testing part.

Part of the problem is that extension-template depends on VCPKG being properly set-up.

An easier to start point can be something like vss, you can try:

git clone https://github.com/duckdb/duckdb_vss
cd duckdb_vss
git submodule init
git submodule update
make wasm_eh

And see where this goes. I would expect this should work if you have emscripten on a recent enough version.

[srossross]

Yes, this built correctly for me, with no errors. It seems like my system is set up correctly

[srossross]

For testing I'm tying something like this, but I cant quite seem to get it working:

const duckdb = require("@duckdb/duckdb-wasm");
const path = require("path");
const Worker = require("web-worker");
const DUCKDB_DIST = path.dirname(require.resolve("@duckdb/duckdb-wasm"));

(async () => {
  try {
    const DUCKDB_CONFIG = await duckdb.selectBundle({
      mvp: {
        mainModule: path.resolve(DUCKDB_DIST, "./duckdb-mvp.wasm"),
        mainWorker: path.resolve(DUCKDB_DIST, "./duckdb-node-mvp.worker.cjs"),
      },
      eh: {
        mainModule: path.resolve(DUCKDB_DIST, "./duckdb-eh.wasm"),
        mainWorker: path.resolve(DUCKDB_DIST, "./duckdb-node-eh.worker.cjs"),
      },
    });

    const logger = new duckdb.ConsoleLogger();
    const worker = new Worker(DUCKDB_CONFIG.mainWorker);
    const db = new duckdb.AsyncDuckDB(logger, worker);
    await db.instantiate(DUCKDB_CONFIG.mainModule, DUCKDB_CONFIG.pthreadWorker);

    const conn = await db.connect();
    
    await conn.query(
      `
      INSTALL './build/wasm_mvp/extension/quack/quack.duckdb_extension'; 
      INSTALL './build/wasm_eh/extension/quack/quack.duckdb_extension'; 
      `
    );
    console.log("installed");
    await conn.query("LOAD 'quack'; ");
    console.log("loaded");
    const result = await conn.query(`select quack('Jane') as result;`);

    await conn.close();
    await db.terminate();
    await worker.terminate();
    console.log(
      "result:",
      result.toArray().map((row) => row.toJSON())
    );
  } catch (e) {
    console.error(e);
  }
})();

The above code will just hang at LOAD 'quack';

When I do should I be setting my custom_extension_repository if so to what? should I be setting allow_unsigned_extensions: true if so how?

[srossross]

Hi @carlopi, I tabled this for a while, but do you have an example or doc on how to do the above?

[srossross]

Hey, I had dropped this project for a while and just thought i would checkin and see if there are any updated docs to use? Happy to contribute if someone points me in the right direction