Rust: Unable to bootstrap MainWorker

[alshdavid]

I'm trying to bootstrap a MainWorker to embed Deno into my Rust application and getting

ext:deno_node/_process/process.ts:825: Uncaught Error: unreachable
thread 'main' panicked at cargo/registry/[...]/deno_runtime-0.148.0/worker.rs:565:57:
called `Option::unwrap()` on a `None` value

With my code looking like

#[tokio::main]
async fn main() {
  let pkg_dir = PathBuf::from("/path/to/pkg");
  let cache_dir = pkg_dir.join("tmp/cache");
  let origin_dir = pkg_dir.join("tmp/origin");
  let nm_dir = pkg_dir.join("node_modules");
  let main_module_path = pkg_dir.join("src/index.js");

  let main_module = Url::from_file_path(main_module_path.clone()).unwrap();

  let npm_resolver = npm::create_cli_npm_resolver(CliNpmResolverCreateOptions::Byonm(
    CliNpmResolverByonmCreateOptions{
      fs: Arc::new(deno_fs::RealFs),
      root_node_modules_dir: nm_dir,
    }))
    .await
    .unwrap()
    .into_npm_resolver();

  let permissions = PermissionsContainer::new(Permissions::allow_all());

  let bootstrap_options = BootstrapOptions{
    location: None,//Some(main_module.clone()),
    argv0: Some("target/debug/deno".to_string()),
    ..BootstrapOptions::default()
  };

  let worker_options = WorkerOptions{
    bootstrap: bootstrap_options.clone(),
    npm_resolver: Some(npm_resolver),
    cache_storage_dir: Some(cache_dir),
    origin_storage_dir: Some(origin_dir),
    ..WorkerOptions::default()
  };

  let mut deno_execution_env = MainWorker::from_options(
    main_module,
    permissions,
    worker_options,
  );

  deno_execution_env.bootstrap(bootstrap_options); // <-- panics
}

and the JavaScript

// "/path/to/pkg/src/index.js"
console.log('hi')

My project looks like:

Where the folders are pre-created

It panics here, in the v8 crate

pub fn bootstrap(&mut self, options: BootstrapOptions) {
  // Setup bootstrap options for ops.
  {
    let op_state = self.js_runtime.op_state();
    let mut state = op_state.borrow_mut();
    state.put(options.clone());
    if let Some(node_ipc_fd) = options.node_ipc_fd {
      state.put(deno_node::ChildPipeFd(node_ipc_fd));
    }
  }

  let scope = &mut self.js_runtime.handle_scope();
  let args = options.as_v8(scope);
  let bootstrap_fn = self.bootstrap_fn_global.take().unwrap();
  let bootstrap_fn = v8::Local::new(scope, bootstrap_fn);
  let undefined = v8::undefined(scope);
  bootstrap_fn.call(scope, undefined.into(), &[args]).unwrap(); // < panics here
}

[alshdavid]

In my case I need to embed a JavaScript engine with Node.js support within my Rust application and extend the standard library with my own packages - so my current plan is to surface then lean out the deno run command in the cli package.

I am stripping away parts of the CLI that I don't need and replacing workspace dependencies with their corresponding deno_* packages from crates.io.

This is where I am so far and this works:

#[tokio::main]
async fn main() {
    // Putting Deno in its own thread to allow for nested tokio runtime
    let deno_thread = std::thread::spawn(move || {
      let entry_script = PathBuf::from("/path/to/pkg/src/index.ts");
      deno_current_thread(run_program(entry_script));
    });

    deno_thread.join().unwrap();
}

/// Partially extracted "deno run" command
async fn run_program(script_path: PathBuf) {
  let run_flags = RunFlags {
    script: script_path.to_str().unwrap().to_string(),
    ..RunFlags::default()
  };

  let flags = Flags {
    unstable_config: UnstableConfig {
      byonm: true,
      ..UnstableConfig::default()
    },
    subcommand: DenoSubcommand::Run(run_flags.clone()),
    ..Flags::default()
  };

  let cli_options = CliOptions::from_flags(flags).unwrap();

  let factory = CliFactory {
    watcher_communicator: None,
    options: Arc::new(cli_options),
    services: Default::default(),
  };

  let cli_options = factory.cli_options();

  let main_module = cli_options.resolve_main_module().unwrap();

  let permissions = PermissionsContainer::new(Permissions::allow_all());

  let worker_factory = factory.create_cli_main_worker_factory().await.unwrap();

  let mut worker = worker_factory
    .create_main_worker(main_module, permissions)
    .await
    .unwrap();

  let exit_code = worker.run().await.unwrap();
}

/// Spawns a tokio task on current thread for Deno to evaluate JavaScript within.
///
/// Use channels to communicate with Deno
#[inline(always)]
fn deno_current_thread<F, R>(future: F) -> R
where
  F: std::future::Future<Output = R> + 'static,
  R: Send + 'static,
{
  let tokio_runtime = tokio::runtime::Builder::new_current_thread()
    .enable_io()
    .enable_time()
    .event_interval(61)
    .global_queue_interval(31)
    .max_io_events_per_tick(1024)
    .max_blocking_threads(32)
    .build()
    .unwrap();

  let future = async move {
    deno_core::unsync::spawn(async move { future.await }.boxed_local())
      .await
      .unwrap()
  };

  #[cfg(debug_assertions)]
  let future = Box::pin(unsafe { MaskFutureAsSend::new(future) });

  #[cfg(not(debug_assertions))]
  let future = unsafe { MaskFutureAsSend::new(future) };

  let join_handle = tokio_runtime.spawn(future);

  tokio_runtime.block_on(join_handle).unwrap().into_inner()
}

[bartlomieju]

If you get a panic in bootstrap_fn.call(scope, undefined.into(), &[args]).unwrap(); it means that the bootstrap function has thrown unexpectedly. You can remove the unwrap for the time being and the error should be properly surfaced.

[alshdavid]

Thanks 🙏 (I read your posts on embedding Deno earlier)

It appears that it's somehow related to v8 snapshots in the build.rs. Can't really find any documentation on how the build.rs builds them so I've just been poking around hoping that I'd learn something by breaking stuff.

I tried commenting bits out - like op_crate_libs.insert("deno.canvas", deno_canvas::get_declaration()); - but that results in a panic from the build script

WARNING: v8::OwnedIsolate for snapshot was leaked

I see that supabase also create a snapshot however when I plug theirs into the Deno CLI it also panics the build.

Haha, I feel like I lack so much context

[alshdavid]

If I have a new cargo project and install deno_runtime = "0.148.0", does the Deno v8 snapshot come with it or do I still need to include a build.rs to generate my own snapshot, manually creating the one in Deno cli?

I'm hoping to get support for TypeScript and the Node.js standard lib

[bartlomieju]

You do need to build the snapshot yourself. You should be good to go if you copy contents of cli/build.rs. There is some Deno specific stuff, but there's no way around it if you want TypeScript (type-checking) support.

[alshdavid]

@bartlomieju is it possible to initialize the snapshot at runtime without compiling one at build time - where the build-time snapshot could be an optimization implemented at a later date to offer faster start up times?

I have skipped the creation of a snapshot and am using the following code however it breaks with

console.log(fetch)

ReferenceError: fetch is not defined

use deno_core::futures::FutureExt;
use deno_core::unsync::MaskFutureAsSend;
use deno_core::PollEventLoopOptions;
use deno_runtime::deno_core::url::Url;
use deno_runtime::deno_core::FastString;
use deno_runtime::deno_http::DefaultHttpPropertyExtractor;
use deno_runtime::permissions::PermissionsContainer;
use std::rc::Rc;
use std::sync::Arc;
use deno_runtime::*;

const CODE: &str = r#"
  console.log(42)
"#;

fn main() {
  deno_current_thread(run_js());
}

pub async fn run_js() {
  let deno_fs = Arc::new(deno_fs::RealFs);
  PermissionsContainer::allow_all();

  let extensions = vec![
    deno_webidl::deno_webidl::init_ops(),
    deno_console::deno_console::init_ops(),
    deno_url::deno_url::init_ops(),
    deno_web::deno_web::init_ops::<PermissionsContainer>(
        Arc::new(deno_web::BlobStore::default()),
        None,
    ),
    deno_webgpu::deno_webgpu::init_ops(),
    deno_canvas::deno_canvas::init_ops(),
    deno_fetch::deno_fetch::init_ops::<PermissionsContainer>(deno_fetch::Options {
        user_agent: "Mach/0.0.0".to_string(),
        root_cert_store_provider: None,
        ..Default::default()
    }),
    deno_websocket::deno_websocket::init_ops::<PermissionsContainer>(
        "Mach/0.0.0".to_string(),
        None,
        None,
    ),
    deno_crypto::deno_crypto::init_ops(None),
    deno_broadcast_channel::deno_broadcast_channel::init_ops(
      deno_broadcast_channel::InMemoryBroadcastChannel::default(),
    ),
    deno_net::deno_net::init_ops::<PermissionsContainer>(None, None),
    deno_tls::deno_tls::init_ops(),
    deno_http::deno_http::init_ops::<DefaultHttpPropertyExtractor>(),
    deno_io::deno_io::init_ops(Some(Default::default())),
    deno_fs::deno_fs::init_ops::<PermissionsContainer>(deno_fs.clone()),
    deno_node::deno_node::init_ops::<PermissionsContainer>(None, deno_fs.clone()),
  ];

  let runtime_options = deno_core::RuntimeOptions {
    module_loader: Some(Rc::new(deno_core::FsModuleLoader)),
    extensions,
    ..Default::default()
  };

  let mut js_runtime = deno_core::JsRuntime::new(runtime_options);

  let exe_path = std::env::current_exe().unwrap().parent().unwrap().to_path_buf();
  let main_module = Url::from_file_path(exe_path).unwrap();
  
  let code: FastString = CODE.to_string().into(); 

  let mod_id = js_runtime.load_main_es_module_from_code(&main_module, code)
    .await
    .unwrap();

  let result = js_runtime.mod_evaluate(mod_id);

  js_runtime
    .run_event_loop(PollEventLoopOptions {
      wait_for_inspector: false,
      pump_v8_message_loop: false,
    })
    .await.unwrap();

  result.await.unwrap();
}

#[inline(always)]
fn deno_current_thread<F, R>(future: F) -> R
where
  F: std::future::Future<Output = R> + 'static,
  R: Send + 'static,
{
  let tokio_runtime = tokio::runtime::Builder::new_current_thread()
    .enable_io()
    .enable_time()
    .event_interval(61)
    .global_queue_interval(31)
    .max_io_events_per_tick(1024)
    .max_blocking_threads(32)
    .build()
    .unwrap();

  let future = async move {
    deno_core::unsync::spawn(async move { future.await }.boxed_local())
      .await
      .unwrap()
  };

  #[cfg(debug_assertions)]
  let future = Box::pin(unsafe { MaskFutureAsSend::new(future) });

  #[cfg(not(debug_assertions))]
  let future = unsafe { MaskFutureAsSend::new(future) };

  let join_handle = tokio_runtime.spawn(future);

  tokio_runtime.block_on(join_handle).unwrap().into_inner()
}

I have tried logging:

console.log(42);                         // works
console.log(globalThis);                 // fails
console.log(fetch);                      // fails
console.log(Deno);                       // works
console.log(await import('node:path'));  // fails

I thought that these two extensions would initialize support for fetch and the node.js standard library.

deno_fetch::deno_fetch::init_ops::<AppPermissions>(deno_fetch::Options {})
deno_node::deno_node::init_ops::<AppPermissions>(None, deno_fs.clone())

Are extensions only available if they are compiled into the snapshot or am I just missing something?

[bartlomieju]

Yes, it's possible to not use a snapshot - if you want to do that you need to replace ::init_ops calls for extensions with ::init_ops_and_esm(). Keep in mind that there's gonna be a significant performance penalty on startup without a snapshot.

[alshdavid]

Ok so I have started putting my experiments into a repo so it's easier to share than dumping source into the discussion.

The first thing I am trying to do is produce a basic snapshot.bin from the deno crates directly from main.rs (to start with, will move to a build.rs once I have it working:

https://github.com/alshdavid-labs/deno-experiments/blob/main/fail-snapshot-1
https://github.com/alshdavid-labs/deno-experiments/blob/main/fail-snapshot-1/src/create_snapshot.rs

It throws with:

cargo run
   Compiling deno1 v0.1.0 (/home/dalsh/Development/deno-sandbox/deno1/snapshot-1)
    Finished dev [unoptimized + debuginfo] target(s) in 7.41s
     Running `target/debug/deno1`
creating new snapshot
Creating a snapshot...
Error: illegal value for flag --lazy=false of type bool
To set or unset a boolean flag, use --flag or --no-flag.
The remaining arguments were ignored: --lazy-eval=false --lazy-streaming=false
Try --help for options
WARNING: v8::OwnedIsolate for snapshot was leaked
thread 'main' panicked at /home/dalsh/.local/rust/cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.269.0/runtime/jsruntime.rs:1854:9:
Failed to initialize JsRuntime for snapshotting: Following modules were not evaluated; make sure they are imported from other code:
   - ext:deno_webgpu/00_init.js
  - ext:deno_fetch/23_response.js
  - ext:deno_web/02_timers.js
  - ext:deno_web/09_file.js
  - ext:deno_webidl/00_webidl.js
  - ext:deno_url/01_urlpattern.js
  - ext:deno_web/04_global_interfaces.js
  - ext:deno_web/10_filereader.js
  - ext:deno_web/16_image_data.js
  - ext:deno_fetch/27_eventsource.js
  - ext:deno_web/08_text_encoding.js
  - ext:deno_cache/01_cache.js
  - ext:deno_url/00_url.js
  - ext:deno_web/15_performance.js
  - ext:deno_webgpu/02_surface.js
  - ext:deno_console/01_console.js
  - ext:deno_web/01_mimesniff.js
  - ext:deno_websocket/02_websocketstream.js
  - ext:deno_web/01_dom_exception.js
  - ext:deno_web/00_infra.js
  - ext:deno_web/05_base64.js
  - ext:deno_web/13_message_port.js
  - ext:deno_fetch/20_headers.js
  - ext:deno_fetch/23_request.js
  - ext:deno_web/02_structured_clone.js
  - ext:deno_fetch/22_body.js
  - ext:deno_websocket/01_websocket.js
  - ext:deno_web/02_event.js
  - ext:deno_fetch/22_http_client.js
  - ext:deno_fetch/26_fetch.js
  - ext:deno_web/06_streams.js
  - ext:deno_web/12_location.js
  - ext:deno_web/14_compression.js
  - ext:deno_web/03_abort_signal.js
  - ext:deno_fetch/21_formdata.js

[bartlomieju]

You need to create your own extension with at least one "entrypoint" where you import all the code from extension - in Deno it's done in runtime/js/99_main.js (which transitively imports all the necessary files from various extensions).

[alshdavid]

Ok I have made progress. Got my own custom Deno build integrated into my project with most of the Deno functionality included.

Things I've learned

1. You must have a snapshot to initialize the JsRuntime with
2. The extensions used by the JsRuntime must be the same (and same order) as the extensions used when creating the snapshot
   1. I haven't tested this but I believe you can have extra extensions added at runtime, you just need to match the base extensions.
3. MainWorker provided by the deno_runtime crate provides the default extensions used by Deno, including the Node.js standard library
4. The deno_* crates do not provide an implementation for resolving require calls - this can be extracted from the Deno cli. It would be nice if Deno provided this as a standalone crate as it appears this is the only functionality I need to extract from the cli.
   1. I see deno_npm, perhaps that's what I am looking for

Questions remaining:
I copy/pasted the snapshot built from the Deno CLI crate into my crate - so I gotta figure out how to build it myself - it looks straightforward enough. I think I can copy most of the build.rs and change the references to point to deno_runtime.

Because of how long it takes to generate a release snapshot, how many extra dependencies it requires and how manageable the Deno release schedule is, I am considering if I should have the snapshots generated externally from my main repo - perhaps in a separate alshdavid/deno_snapshots repo - where I release platform specific precompiled Deno v8 snapshots as Github releases.

Consumers (just me at this stage) who want to integrate Deno into their project could download the snapshots into their project. I could also publish a crate that bottles up the Deno dependencies (npm manager, web_worker initialization) to simplify integration.

That should make it trivial to embed Deno into an existing Rust project

[alshdavid]

Ok, so I have created a repo that publishes prebuilt Deno v8 snapshots from the cli (I hope this doesn't violate and licenses? it's all MIT from what I can tell).

I feel this is valuable because building the snapshot makes it tricky to embed Deno into projects where we cross-compile in CI/CD. Using prebuilt snapshot binaries means that's not a concern plus it speeds up compile time.

[alshdavid]

Ok well I've managed to embed Deno with the CLI functionality into my application by vendoring the deno_cli package into my project and replacing the starting point for the deno run command.

There's a lot of unused functionality, like linting, the test runner, docs, etc and it's quite difficult to untangle so I'll "tree shake" by hand later but for now I can run Deno like this:

fn main() {
  deno_embed::deno_current_thread(async move {
    let exit_code = deno_embed::run_script(
      deno_embed::DenoInitOptions{ // Combination of RunFlags and Flags
        script: "/home/dalsh/Development/alshdavid/mach/pkg2/index.js".to_string(),
        ..Default::default()
      })
      .await;

    println!("{:?}", exit_code);
  });
}

It's a bit hard to manage the dependencies of Deno because they are locked which causes collisions with my main application.

How I am using it

This is my wrapper of the deno_cli package

I had to vendor & rename SWC into my project to avoid collisions

[alshdavid]

Unfortunately it looks like custom_extensions are being dropped (or I don't know how to access them).

#[op2]
pub fn op_hello_world(
  #[global] f: v8::Global<v8::Function>,
) {
  println!("hi");
}

deno_core::extension!(
  my_extension_name,
  ops = [op_hello_world],
);

fn main() {
// ....
  let mut worker = worker_factory
    .create_custom_worker(
      main_module,
      permissions,
      vec![my_extension_name::init_ops_and_esm()],
      options.stdio,
    )
    .await?;
}

Where it's unavailable anywhere in the Deno namespace. I haven't had any luck finding it elsewhere.

Perhaps it relates to how ops are removed here

[bartlomieju]

You can access your ops in ext: files by importing ops like so:

import {
  op_name1,
  op_name2,
  op_name3,
} from "ext:core/ops";

[alshdavid]

As a sanity check, I made my change in the deno-cli package in an otherwise unmodified fork of this (denoland/deno) repo - I modified the eval command to insert my custom ext:

// cli/tools/run/mod.rs

// ... use statements
use deno_core::*;

#[op2(fast)]
pub fn op_custom_hello_world() {
  println!("hi from rust");
}

deno_core::extension!(
  my_extension_name,
  ops = [op_custom_hello_world],
  esm_entry_point = "ext:my_extension_name/100_custom.js",
  esm = ["100_custom.js"],
);

// ...unmodified

pub async fn eval_command(/* unmodified */) -> Result<i32, AnyError> {
  // ...unmodified

  // I changed "worker_factory.create_main_worker()"
  // to "create_custom_worker()" so I could pass in my extension:
  let mut worker = worker_factory
    .create_custom_worker(
      main_module,
      permissions,
      vec![my_extension_name::init_ops_and_esm()],
      Default::default(),
    )
    .await?;
}

Then I added the corresponding JS file

// cli/100_custom.js

import {
  op_custom_hello_world,
} from "ext:core/ops";

globalThis['op_custom_hello_world'] = op_custom_hello_world;

And ran cargo run -- run eval "console.log(globalThis['op_custom_hello_world'])"

This failed to build with:

Failed to initialize a JsRuntime: Uncaught SyntaxError: 
The requested module 'ext:core/ops' does not provide an export 
named 'op_custom_hello_world'
    at ext:my_extension_name/100_custom.js:2:3

I also tried importing the ext by name

import {
  my_extension_name,
} from "ext:core/ops";

globalThis['my_extension_name'] = my_extension_name;

However that failed with the same error as above

[alshdavid]

I was able to get access to it via:

// cli/100_custom.js
import { core } from "ext:core/mod.js";
const {
  op_custom_hello_world
} = core.ops

globalThis['op_custom_hello_world'] = op_custom_hello_world;

[alshdavid]

@bartlomieju hey, is Deno CLI open to PRs that organize/tidy/rearrange up some of the code?

In my quest to embed Deno (specifically deno run), I have run into challenges with how the internal crate imports reference each other (for example, you can't just delete the lsp, benchmark testing, etc folders and their entry points without that breaking the build because the entry points are not isolated from each other).

My motivation is that, in order to embed Deno into my project I have to vendor Deno CLI - it would would make it easier to keep my integration in sync with Deno if the dependencies were a little more hierarchical (ideally the complete cli functionality would be in its own crate).

Is there somewhere the Deno team chats about these kinds of changes (Discord, Slack or something)? I have lots of questions and suggestions I'd like to discuss with the team before dedicating time to them (no point in raising a PR if the team doesn't like the direction).

[bartlomieju]

Feel free to ping me in #dev channel on Discord