Compile LightGBM to WebAssembly output

Summary

My team builds a c++ library which has direct dependency on LightGBM tool. We were planning to sandbox the tool and for that to complete we want documentation on how to compile LightGBM into WebAssembly code.

Motivation

Since sandboxing or running c++ code in browser is one of the hot topics in the market, LightGBM providing official documentation on compiling to WebAssembly will be a huge win for everyone.

Description

Requesting for new documentation to be created which shows compilation of LightGBM to WebAssembly. We don't have a preference for tool to be used for compilation as long as output is in wasm format.

References

emscripten-core/emscripten#16592 -> similar issue while using emscripten tool for compiling c++ code

[jameslamb]

Thanks very much for this @shyam-lab ! I've added it to our list of feature requests in #2302 . Per this repo's policy, I'll close this issue for now until someone leaves a comment saying they'd like to work on this.

But to set the right expectation...I don't believe this will be worked on by LightGBM's maintainers any time soon. I don't know if any of the current maintainers have any experience with WebAssembly, and we are already struggling to keep up with the demand for changes to the existing C, C++, Python, and R code in this project.

If you do figure out how to do this, please post your solution here.

If you or anyone else reading this are interested in doing the work to add this support and in maintaining that documentation, we can have a conversation here about possibly including that in this repository.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.

[jameslamb]

Sorry, this was locked accidentally. Just unlocked it. We'd still love help with this feature!

[agriyakhetarpal]

We maintain a recipe for LightGBM with the Pyodide project, a port of CPython compiled with the Emscripten toolchain for WebAssembly. It's possible to use Pyodide with Node.js, and with projects that develop on top of it – such as PyScript, or via JupyterLite.

Please see the post-merge discussion at #6715 with @jameslamb around the proposition of adding a CI job that builds WASM wheels for LightGBM. This would aid us (quite a lot!) whenever we hope to update the LightGBM version in Pyodide (it's currently at version 4.3.0 right now).

For a complete list of packages and libraries available in Pyodide, please see: https://pyodide.org/en/stable/usage/packages-in-pyodide.html.

[jameslamb]

Great, thanks for that excellent write-up! Let's use this issue to track the work of possibly adding a WASM CI job here.

[agriyakhetarpal]

Thank you! Could this be re-opened, since it was closed when it was locked?

Also, for additional information, here's what would be needed for a WASM CI job:

• First, it's best to build directly from the Python source. For this, we provide a standards-compliant distribution builder at https://github.com/pyodide/pyodide-build, which can be used similar to other build frontends such as pip wheel and pypa/build, i.e., with the pyodide build invocation.
• The second part would be to add any missing dependencies to Pyodide in-tree (I don't think there are any, so far, but I'm not sure about new ones) and wait for a Pyodide release to come out with them (though, we have been planning to unvendor the recipes so that package version updates are not tied to Pyodide as a whole). If there are no new dependencies, we should be fine.
• Fix any issues related to bitness or mismatches in function signatures, etc.
• I expect that several tests won't work – for example, WASM does not support observability for floating-point exceptions yet, so warnings on mathematical operations might not be received. Other things include running tests without pytest-xdist (not supported yet), and disabling any functionality that uses multithreading or threading from the stdlib, since Emscripten does not support threads.
• To check for the WASM runtime, pytest decorators based on IS_WASM = sys.platform == 'emscripten' or platform.machine() in ["wasm32", "wasm64"] can be used for skipping/xfailing tests. We can use the "PYODIDE" environment variable at build time for build scripts.

I expect a significant portion of the test suite to pass, and a substantial portion of it to fail as well 😁