feat: refactor build and build duckdb from source files

Signed-off-by: Gordon Smith <GordonJSmith@gmail.com>

Author: GordonSmith

.github/copilot-instructions.md

@@ -24,7 +24,7 @@ This is a WebAssembly (WASM) monorepo providing JavaScript/TypeScript bindings f
 npm ci
 
 # 2. Install build dependencies (20-30 minutes - NEVER CANCEL)
-# Includes: emsdk install (~5 min), vcpkg install (~15-20 min), playwright (~5-10 min)
+# Includes: emsdk install (~5 min), vcpkg install (~15-20 min), playwright (~5-10 min), bundler test deps
 npm run install-build-deps
 
 # 3. Build C++ to WASM (30-45 minutes - NEVER CANCEL) 
@@ -34,7 +34,7 @@ npm run build-cpp
 # 4. Build TypeScript packages (5-10 minutes)
 npm run build-ws
 
-# 5. Full build command (45+ minutes total - NEVER CANCEL)
+# 5. Full build command equivalent to build-cpp + build-ws (45+ minutes total - NEVER CANCEL)
 # Equivalent to: build-cpp && build-ws
 npm run build
 ```
@@ -49,6 +49,20 @@ From actual runs in clean environment:
 - `npm run build-cpp`: 30-60 minutes (C++ compilation to WASM)
 - `npm run test`: 15-30 minutes (includes browser and node tests)
 
+## CMake / C++ (WASM) Build
+
+Prefer the npm scripts. They source `./emsdk/emsdk_env.sh` and use the repo's CMake preset(s):
+
+```bash
+npm run build-cpp
+```
+
+For fast iteration:
+
+```bash
+npm run build-cpp-watch
+```
+
 ### Quick Development (TypeScript Only)
 ```bash
 # For TypeScript-only changes when WASM files exist
@@ -57,32 +71,15 @@ npm run build-ws        # 5-10 minutes
 npm run lint           # 2-3 minutes
 ```
 
-## CMake Configuration
-
-### **CRITICAL**: ALWAYS Use VS Code CMake Extension
-
-**YOU MUST use the VS Code CMake extension for ALL CMake operations.** Manual cmake commands will fail because they don't properly set up the Emscripten environment.
+### Manual CMake (only when needed)
 
-**REQUIRED WORKFLOW** when modifying C++ code or vcpkg packages (patches, portfile.cmake):
+Manual CMake is supported, but you must source Emscripten first:
 
-1. **Remove vcpkg cache**: `rm -rf vcpkg/buildtrees/<package>`
-2. **Use VS Code Command Palette** (Ctrl/Cmd+Shift+P):
-   - **`CMake: Configure`** - Reconfigures CMake and reinstalls vcpkg packages with new patches
-   - **`CMake: Build`** - Builds the configured targets
-   - **`CMake: Clean Rebuild`** - Cleans and rebuilds everything
-
-**WHY THIS IS REQUIRED**:
-- Sets up Emscripten SDK environment variables correctly
-- Configures vcpkg toolchain properly
-- Handles WASM-specific build flags
-- Provides IntelliSense and debugging integration
-
-**DO NOT use manual cmake commands like:**
-- ❌ `cmake --preset vcpkg-emscripten-*` (will fail with "emcc compiler not found")
-- ❌ `cmake --build build --target <target>` (won't work without proper environment)
-- ❌ `ninja <target>` (missing environment and configuration)
-
-**EXCEPTION**: `npm run build-cpp` scripts are pre-configured with the correct environment and are safe to use for full builds.
+```bash
+source ./emsdk/emsdk_env.sh
+cmake -S . -B ./build --preset vcpkg-emscripten-MinSizeRel
+cmake --build ./build --parallel
+```
 
 ## Embind (C++ <-> JS Interop)
 
@@ -132,18 +129,6 @@ Reference: https://emscripten.org/docs/porting/connecting_cpp_and_javascript/emb
 ✘ [ERROR] Could not resolve "../../../build/packages/graphviz/src-cpp/graphvizlib.wasm"
 ```
 
-### DuckDB Exception
-The DuckDB package is special:
-```bash
-cd packages/duckdb
-npm run pack-duckdb    # Uses pre-built WASM from @duckdb/duckdb-wasm
-npm run build          # Can complete without C++ compilation
-```
-This works because DuckDB uses pre-compiled WASM binaries rather than building from C++ source.
-```
-ERROR: Could not resolve "../../../build/packages/graphviz/src-cpp/graphvizlib.wasm"
-```
-
 ## Testing
 
 ### Prerequisites
@@ -202,9 +187,10 @@ npm test
 ### 3. End-to-End Validation
 ```bash
 # Test the examples in the root directory
-node -e "
-const { Graphviz } = require('./packages/graphviz/dist/index.js');
-Graphviz.load().then(g => console.log(g.version()));
+node --input-type=module -e "
+import { Graphviz } from './packages/graphviz/dist/index.js';
+const g = await Graphviz.load();
+console.log(g.version());
 "
 ```
 
@@ -214,15 +200,10 @@ Graphviz.load().then(g => console.log(g.version()));
 - **Network restrictions**: Downloads may fail in restricted environments
 - **Docker alternative**: Use `npm run build-docker` if local builds fail
 - **SSL certificate issues**: May block package downloads
-- **Node.js version**: Requires Node.js 20+ for optimal compatibility
+- **Node.js version**: CI runs Node.js 22 and 24 (primary support); docs deploy currently uses Node.js 20
 
 ### Package-Specific Notes
 
-#### DuckDB Package
-- **Special build process**: Uses pre-built WASM from @duckdb/duckdb-wasm
-- **Can build without C++ compilation**: Run `npm run pack-duckdb` instead
-- **Build command**: Uses `pack-duckdb-eh` scripts to generate WASM files
-
 #### Other Packages (base91, expat, graphviz, llama, zstd)
 - **Require C++ compilation**: Must run `npm run build-cpp` first
 - **WASM dependencies**: TypeScript builds fail without WASM files
@@ -281,7 +262,7 @@ npm run build-docs          # Build VitePress documentation
 ### Advanced Build Options
 ```bash
 npm run build-cpp-watch     # Watch C++ files for changes
-npm run build-ts-watch      # Watch TypeScript files
+npm run build-watch      # Watch TypeScript files
 npm run build-dev           # Development builds
 ```
 

.github/instructions/README.md

@@ -1,5 +1,5 @@
 # Instructions Summary
-*AI Assistant instructions for hpcc-systems/hpcc-js-wasm - Updated July 2025*
+*AI Assistant instructions for hpcc-systems/hpcc-js-wasm - Updated December 2025*
 
 This directory contains comprehensive instructions for AI assistants working with the HPCC-JS-WASM repository.
 
@@ -29,6 +29,7 @@ This directory contains comprehensive instructions for AI assistants working wit
 - **TypeScript First**: Many operations can be done with TypeScript-only builds
 - **Test Expectations**: Fresh clone tests will fail without WASM builds (expected)
 - **Safe Operations**: Documentation changes, TypeScript modifications don't need full build
+- **Node.js**: CI runs Node.js 22 and 24; docs deploy currently uses Node.js 20
 
 ## Most Common Scenarios
 
@@ -42,7 +43,7 @@ npm run build-ws  # May fail without WASM, but can still work on TS
 ### Full Development Setup
 ```bash
 npm ci
-npm run install-build-deps  # Requires system tools
+npm run install-build-deps  # Installs emsdk, vcpkg, playwright, bundler test deps
 npm run build-cpp          # Compiles C++ to WASM
 npm run build-ws           # Builds TypeScript
 npm run test               # Full test suite

.github/instructions/main.md

@@ -1,5 +1,5 @@
 # Copilot Instructions for HPCC-JS-WASM
-*Last updated: July 2025 | For AI assistants working with hpcc-systems/hpcc-js-wasm*
+*Last updated: December 2025 | For AI assistants working with hpcc-systems/hpcc-js-wasm*
 
 This document provides guidance for AI assistants working with the HPCC-JS-WASM repository.
 
@@ -56,7 +56,7 @@ packages/[name]/
 npm ci
 
 # Install build dependencies (requires system tools)
-npm run install-build-deps  # Installs emsdk, vcpkg, playwright
+npm run install-build-deps  # Installs emsdk, vcpkg, playwright, bundler test deps
 
 # Build C++ to WASM (requires emscripten)
 npm run build-cpp
@@ -65,6 +65,10 @@ npm run build-cpp
 npm run build-ws
 ```
 
+Notes:
+- CI runs Node.js 22 and 24; docs deploy currently uses Node.js 20.
+- Browser tests typically require `npx playwright install --with-deps` on Ubuntu.
+
 ### Quick Development (TypeScript only)
 ```bash
 # Install dependencies

.github/instructions/troubleshooting.md

@@ -244,8 +244,8 @@ cd packages/base91  # Smallest package
 npm run build
 
 # 3. Check WASM loading
-node -e "
-const fs = require('fs');
+node --input-type=module -e "
+import fs from 'node:fs';
 const wasm = fs.readFileSync('build/packages/base91/src-cpp/base91lib.wasm');
 console.log('WASM size:', wasm.length);
 "
@@ -277,7 +277,7 @@ time npm run build-cpp
 time npm run build-ws
 
 # Use watch modes for development
-npm run build-ts-watch &
+npm run build-watch &
 npm run build-cpp-watch &
 ```
 
@@ -374,7 +374,7 @@ git reset --hard HEAD
 
 ### During Development
 1. **Test incrementally**: Build and test after each change
-2. **Watch for errors**: Use watch modes (`npm run build-ts-watch`)
+2. **Watch for errors**: Use watch modes (`npm run build-watch`)
 3. **Monitor memory**: Keep an eye on process memory usage
 
 ### Before Committing

.github/instructions/workflow.md

@@ -64,7 +64,7 @@ For comprehensive development including C++ changes:
 npm ci
 
 # 2. Install build tools (requires system dependencies)
-npm run install-build-deps
+npm run install-build-deps  # Installs emsdk, vcpkg, playwright, bundler test deps
 
 # 3. Build C++ to WASM
 npm run build-cpp

.github/workflows/deploy-docs.yml

@@ -53,7 +53,7 @@ jobs:
         uses: peaceiris/actions-gh-pages@v3
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
-          publish_dir: ./docs/.vitepress/dist
+          publish_dir: ./.vitepress/dist
           force_orphan: true
           # cname: example.com # if wanna deploy to custom domain
         env:

.gitignore

@@ -8,13 +8,15 @@ build/
 coverage/
 dist/
 dist-test/
-/docs/.vitepress/cache
-/docs/.vitepress/dist
+/.vitepress/cache
+/.vitepress/dist
 /docs/base91
 /docs/duckdb
 /docs/expat
 /docs/graphviz
+/docs/llama
 /docs/zstd
+/docs/packages
 /docs/README.md
 /emsdk*
 /lib-*

docs/.vitepress/config.js .vitepress/config.jsdocs/.vitepress/config.js renamed to .vitepress/config.js

@@ -1,12 +1,16 @@
-export default {
+import { resolve } from 'path';
+import { defineConfig } from 'vite'
+
+export default defineConfig({
     title: '@hpcc-js/wasm',
     description: 'HPCC Systems Wasm Libraries',
     base: '/hpcc-js-wasm/',
-    srcExclude: ['**/tests/**'],
+    srcExclude: ['build/', 'cmake/', 'docker/', 'emsdk/', 'vcpkg/', 'vcpkg-overlays/', '**/tests/**'],
+    ignoreDeadLinks: true,
 
     themeConfig: {
         repo: "hpcc-systems/hpcc-js-wasm",
-        docsDir: "docs",
+        docsDir: ".",
         docsBranch: "trunk",
         editLink: {
             pattern: 'https://github.com/hpcc-systems/hpcc-js-wasm/edit/trunk/docs/:path',
@@ -15,32 +19,32 @@ export default {
         lastUpdated: "Last Updated",
 
         nav: [
-            { text: 'Guide', link: '/getting-started' },
+            { text: 'Guide', link: './docs/getting-started' },
             { text: 'GitHub', link: 'https://github.com/hpcc-systems/hpcc-js-wasm' },
             { text: 'Changelog', link: 'https://github.com/hpcc-systems/hpcc-js-wasm/blob/trunk/CHANGELOG.md' },
         ],
         sidebar: [
             {
                 text: 'General',
                 items: [
-                    { text: 'Getting Started', link: '/getting-started' },
+                    { text: 'Getting Started', link: '/docs/getting-started' },
                 ]
             },
             {
                 text: 'WASM CLI',
                 items: [
-                    { text: 'Graphviz', link: '/graphviz-cli' },
+                    { text: 'Graphviz', link: '/docs/graphviz-cli' },
                 ]
             },
             {
                 text: 'WASM API',
                 items: [
-                    { text: 'Base91', link: '/base91/src/base91/classes/Base91' },
-                    { text: 'DuckDB', link: '/duckdb/src/duckdb/classes/DuckDB' },
-                    { text: 'Expat', link: '/expat/src/expat/classes/Expat' },
-                    { text: 'Graphviz', link: '/graphviz/src/graphviz/classes/Graphviz' },
-                    { text: 'Llama', link: '/llama/src/llama/classes/Llama' },
-                    { text: 'Zstd', link: '/zstd/src/zstd/classes/Zstd' },
+                    { text: 'Base91', link: '/packages/base91/README' },
+                    { text: 'DuckDB', link: '/packages/duckdb/README' },
+                    { text: 'Expat', link: '/packages/expat/README' },
+                    { text: 'Graphviz', link: '/packages/graphviz/README' },
+                    { text: 'Llama', link: '/packages/llama/README' },
+                    { text: 'Zstd', link: '/packages/zstd/README' },
                 ]
             }
 
@@ -50,4 +54,4 @@ export default {
             copyright: 'Copyright © 2019-present hpccsystems.com'
         }
     }
-}
+});

.vscode/tasks.json

@@ -14,8 +14,8 @@
         },
         {
             "type": "npm",
-            "label": "build-ts-watch",
-            "script": "build-ts-watch",
+            "label": "build-watch",
+            "script": "build-watch",
             "problemMatcher": [
                 "$tsc-watch"
             ],
@@ -72,7 +72,7 @@
             "label": "build",
             "dependsOn": [
                 "build-types-watch",
-                "build-ts-watch",
+                "build-watch",
                 "build-cpp-watch"
             ],
             "group": {