unable to require native module dynamically (for use in VSCode extension)

[TomasHubelbauer]

Hey, I want to use Keytar in a VS Code extension.

VS Code extensions do not allow for depending on native modules.

What extensions do to work around this so to download native dependencies after installation as a part of the activation procedure. I have drafted one such procedure substituting in arch, platform and modules to make sure I get the right binary for the runtime I'm in:

const version = '4.2.1';
const name = `keytar-v${version}-electron-v${process.versions.modules}-${process.platform}-${process.arch}`;
const tarGzFilePath = path.join(__dirname, name + '.tar.gz');
const tarFilePath = path.join(__dirname, name + '.tar.gz');
const nodeFilePath = path.join(__dirname, 'build/Release/keytar.node');
const response = await fetch('https://api.github.com/repos/atom/node-keytar/releases');
const data = await response.json();
const release = data.find(release => release.tag_name === 'v' + version);
const asset = release.assets.find(asset => asset.name === name + '.tar.gz');
await fs.writeFile(tarGzFilePath, await (await fetch(asset.browser_download_url)).buffer());
await tar.x({ f: tarGzFilePath, cwd: __dirname }, [name + '.tar']);
await tar.x({ f: tarFilePath, cwd: __dirname }, ['build/Release/keytar.node']);
const keytar = require(nodeFilePath);
await keytar.setPassword('test', 'test', 'test');
console.log(await keytar.findPassword('test'));

However (at least with keytar-v4.2.1-electron-v57-win32-x64), I'll get Module did not self-register. immediately when trying to use require on the downloaded file. I know the file is downloaded (otherwise I would get a file system exception) and I also know it is the right modules-platform-arch triplet as I substitute that dynamically based on the environment.

I checked the .node file and was able to verify it's a valid PE32+ executable built against x64 so I think I have the right file.

You can try this yourself.

What could be the problem?

[shiftkey]

@TomasHubelbauer as a sanity check, can you confirm this works in a standalone Electron app?

[TomasHubelbauer]

I've updated the demo.

When I run it, the Electron app starts up as expected and the dynamic dependency resolution process steps finish successfully:

Obtaining keytar-v4.2.1-electron-v57-win32-x64:
Downloaded Keytar GitHub releases.
Decoded releases.
Found the 4.2.1 release.
Found the keytar-v4.2.1-electron-v57-win32-x64 asset.
Downloaded the asset
Saved the asset
Extracted the asset .tar.gz.
Extracted the asset .tar.
Loaded the module
Set
undefined <-- not working

You can see it runs even setPassword and findPassword, so it doesn't crash immediately on require, but they don't work.

The app almost instantly crashes with an error afterwards:

#
# Fatal error in ../../v8/src/objects.cc, line 3550
# Check failed: receiver->IsJSFunction().
#
npm ERR! code ELIFECYCLE
npm ERR! errno 3221225477
npm ERR! electron-test@1.0.0 start: `electron .`
npm ERR! Exit status 3221225477

There are two call sites for IsJSFunction in objects.cc, I don't know which version of Chromium current Electron uses exactly to be able to trace the line and call stack precisely (the line 3550 is not it - IsFunction vs IsJSFunction), but this is the closest invocation to the reported line: CHECK(receiver->IsJSFunction()); in JSReceiver::GetCreationContext

[TomasHubelbauer]

@shiftkey you've added a needs-repro label, but I've posted a link to a repo with the repro. Were you not able to make it work on your end? Does it not reproduce?

[shiftkey]

@TomasHubelbauer that's on me to look at what you've provided. If I needed more from you I'd set it to needs-more-information.

[TomasHubelbauer]

My apologies, understood.

[TomasHubelbauer]

It's been a few releases so I've decided to check if this still happens with the latest version of Keytar and indeed it does, but while checking, I've found something which to me is very odd:

I no longer get the Module did not self-register error, instead, the require goes through and so do the setPassword and findPassword methods (but findPassword returns undefined so it's not working, just not crashing), but the code crashes as if after the process code has ran.

I've updated the repro repo to show this.

When I place process.exit(0) as a last line in the code, it no longer crashes (but still doesn't work, see above), but when I don't, it does. It feels as if Keytar was registering some sort of a destructor/exit callback (no idea what I'm saying, no experience with native) and my process.exit prevented that from running and throwing, instead causing it to leak.

I would even be okay with this as a workaround (the "leak" if that's what it is happens as the application exists so it wouldn't last long), but the findPassword method still returns undefined for me which means this workaround is not actually going to save me.

As a side-note to anyone who comes across this issue, microsoft/vscode#68738 shows in a comment how to hijack VS Code's bundled Keytar (which is static not dynamic) and since the VS Code API does not provide any secure credential storage mechanism (nor does it look like it will anytime soon) this is probably best one can do without going the route of building something themselves using node-keytar (because then this issue happens since you need to load it dynamically as VS Code extensions cannot bundle static native modules).

Basically you can just call require('keytar') in a VS Code extension codebase and rely on the fact that VS Code bundles it. When I tried doing that, it worked for me and I also show that in my repro repo.

[shiftkey]

@TomasHubelbauer apologies for the delay in replying. I had a look at your reproduction and realised it wasn't quite testing some important cases:

• it assumes the module loading worked fine
• it's still testing Electron 2, which is no longer supported

I ended up with some changes on top of your repro, which are here:

TomasHubelbauer/keytar-vscode-extension@master...shiftkey:improve-repro

Running npm start at the root now shows this on Windows:

$ npm start

> keytar-vscode-extension@1.0.0 start C:\Users\shiftkey\Documents\GitHub\keytar-vscode-extension
> node index.js

Using node 11.14.0
Downloaded Keytar GitHub releases
Decoded releases
Picked the latest 4.12.0 release
Obtaining keytar-v4.12.0-node-v67-win32-x64:
Found the keytar-v4.12.0-node-v67-win32-x64 asset
Downloaded the asset
Saved the asset
Extracted the asset .tar.gz
Extracted the asset .tar
Module exists: true
Loaded the module: [["getPassword",null],["setPassword",null],["deletePassword",null],["findPassword",null],["findCredentials",null]]
setPassword is 'undefined'


#
# Fatal error in , line 0
# Check failed: receiver->IsJSFunction().
#
#
#
#FailureMessage Object: 00000089AEEFF580npm ERR! code ELIFECYCLE
npm ERR! errno 3221225477
npm ERR! keytar-vscode-extension@1.0.0 start: `node index.js`
npm ERR! Exit status 3221225477
npm ERR!
npm ERR! Failed at the keytar-vscode-extension@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\shiftkey\AppData\Roaming\npm-cache\_logs\2019-07-23T16_49_46_427Z-debug.log

And this on macOS (with a different Node version):

$ npm start

> keytar-vscode-extension@1.0.0 start /Users/shiftkey/src/test/keytar-vscode-extension
> node index.js

Using node 10.16.0
Downloaded Keytar GitHub releases
Decoded releases
Picked the latest 4.12.0 release
Obtaining keytar-v4.12.0-node-v64-darwin-x64:
Found the keytar-v4.12.0-node-v64-darwin-x64 asset
Downloaded the asset
Saved the asset
Extracted the asset .tar.gz
Extracted the asset .tar
Loaded the module: [["getPassword",null],["setPassword",null],["deletePassword",null],["findPassword",null],["findCredentials",null]]
setPassword is 'undefined'


#
# Fatal error in , line 0
# Check failed: receiver->IsJSFunction().
#
#
#
#FailureMessage Object: 0x7ffeefbf4c70zsh: illegal hardware instruction  npm start

I think the runtime error you are seeing here is simply a result of trying to invoke undefined like a function, and we're back to not really understanding why the native module has the functions available but they're not defined.

[TomasHubelbauer]

Zero worries, thanks for taking the time to look at this!

we're back to not really understanding why the native module has the functions available but they're not defined.

If there is anything I can do to try and further research into this, please, let me know. :-)

[shiftkey]

If there is anything I can do to try and further research into this, please, let me know. :-)

I'm kind of stumped myself about the next steps, and I don't have the bandwidth to spelunk into Node's module loader logic and figure out why this isn't working for you.

Given VSCode includes keytar and there's a workaround for extensions to access it microsoft/vscode#68738 (comment), this will be a low-priority issue unless someone has additional insights into why this isn't working in this situation...