Installation instructions for Windows and macOS environments

[eblis]

I'm trying to install faust-streaming-rocksdb package on Windows and I'm getting various errors.

1. First error was related to pkg-config missing, for which I installed this lite version
2. Now I'm getting errors with my Visual Studio compiler, namely this:

The following error occurred when trying to handle this error:


  EnvCommandError

  Command <local user>\AppData\Local\pypoetry\Cache\virtualenvs\main-sfnLs_Sh-py3.9\Scripts\python.exe -m pip install --use-pep517 --disable-pip-version-check --prefix <local user>\AppData\Local\pypoetry\Cache\virtualenvs\main-sfnLs_Sh-py3.9 --no-deps <local user>\AppData\Local\pypoetry\Cache\artifacts\a3\db\2a\56fd9cba93b126c74d8376724d69cb1aa975a488d396619bbdee3836fa\faust-streaming-rocksdb-0.9.1.tar.gz errored with the following return code 1, and output:
  Processing <local user>\AppData\Local\pypoetry\cache\artifacts\a3\db\2a\56fd9cba93b126c74d8376724d69cb1aa975a488d396619bbdee3836fa\faust-streaming-rocksdb-0.9.1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: faust-streaming-rocksdb
    Building wheel for faust-streaming-rocksdb (pyproject.toml): started
    Building wheel for faust-streaming-rocksdb (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error

    Building wheel for faust-streaming-rocksdb (pyproject.toml) did not run successfully.
    exit code: 1

    [99 lines of output]
    <local user>\AppData\Local\Temp\pip-build-env-7s172fm3\overlay\Lib\site-packages\setuptools\config\setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
      warnings.warn(msg, warning_class)
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-cpython-39
    creating build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\errors.py -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\interfaces.py -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\merge_operators.py -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\__init__.py -> build\lib.win-amd64-cpython-39\rocksdb
    creating build\lib.win-amd64-cpython-39\rocksdb\tests
    copying rocksdb\tests\test_db.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
    copying rocksdb\tests\test_memtable.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
    copying rocksdb\tests\test_options.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
    copying rocksdb\tests\test_transactiondb_options.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
    copying rocksdb\tests\test_transaction_db.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
    copying rocksdb\tests\__init__.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
    running egg_info
    writing faust_streaming_rocksdb.egg-info\PKG-INFO
    writing dependency_links to faust_streaming_rocksdb.egg-info\dependency_links.txt
    writing requirements to faust_streaming_rocksdb.egg-info\requires.txt
    writing top-level names to faust_streaming_rocksdb.egg-info\top_level.txt
    reading manifest file 'faust_streaming_rocksdb.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    adding license file 'LICENSE'
    writing manifest file 'faust_streaming_rocksdb.egg-info\SOURCES.txt'
    <local user>\AppData\Local\Temp\pip-build-env-7s172fm3\overlay\Lib\site-packages\setuptools\command\build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'rocksdb.cpp' as data is deprecated, please list it in `packages`.
        !!


        ############################
        # Package would be ignored #
        ############################
        Python recognizes 'rocksdb.cpp' as an importable package,
        but it is not listed in the `packages` configuration of setuptools.

        'rocksdb.cpp' has been automatically added to the distribution only
        because it may contain data files, but this behavior is likely to change
        in future versions of setuptools (and therefore is considered deprecated).

        Please make sure that 'rocksdb.cpp' is included as a package by using
        the `packages` configuration field or the proper discovery methods
        (for example by using `find_namespace_packages(...)`/`find_namespace:`
        instead of `find_packages(...)`/`find:`).

        You can read more about "package discovery" and "data files" on setuptools
        documentation page.


    !!

      check.warn(importable)
    copying rocksdb\_rocksdb.cpp -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\_rocksdb.pyx -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\advanced_options.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\backup.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\cache.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\checkpoint.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\compaction_filter.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\comparator.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\concurrent_task_limiter.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\db.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\env.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\filter_policy.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\iterator.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\logger.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\memtablerep.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\merge_operator.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\options.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\slice_.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\slice_transform.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\snapshot.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\stackable_db.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\statistics.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\status.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\std_memory.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\table_factory.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\table_properties.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\transaction_db.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\types.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    copying rocksdb\universal_compaction.pxd -> build\lib.win-amd64-cpython-39\rocksdb
    creating build\lib.win-amd64-cpython-39\rocksdb\cpp
    copying rocksdb\cpp\comparator_wrapper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
    copying rocksdb\cpp\filter_policy_wrapper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
    copying rocksdb\cpp\memtable_factories.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
    copying rocksdb\cpp\merge_operator_wrapper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
    copying rocksdb\cpp\slice_transform_wrapper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
    copying rocksdb\cpp\utils.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
    copying rocksdb\cpp\write_batch_iter_helper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
    running build_ext
    building 'rocksdb._rocksdb' extension
    creating build\temp.win-amd64-cpython-39
    creating build\temp.win-amd64-cpython-39\Release
    creating build\temp.win-amd64-cpython-39\Release\rocksdb
    "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Irocksdb -I.\rocksdb -I<local user>\AppData\Local\pypoetry\Cache\virtualenvs\main-sfnLs_Sh-py3.9\include "-IC:\Program Files\Python39\include" "-IC:\Program Files\Python39\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\cppwinrt" /EHsc /Tprocksdb/_rocksdb.cpp /Fobuild\temp.win-amd64-cpython-39\Release\rocksdb/_rocksdb.obj -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing -fno-rtti
    cl : Command line error D8021 : invalid numeric argument '/Wextra'
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
    [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for faust-streaming-rocksdb
  Failed to build faust-streaming-rocksdb
  ERROR: Could not build wheels for faust-streaming-rocksdb, which is required to install pyproject.toml-based projects

[eblis]

Maybe it's possible to use vcpkg when installing on Windows ?

[wbarnha]

Sorry to see you're having this issue, I didn't expect this library to be used on Windows hence why it's incompatible. I'll see if I can extend the CI/CD configuration to support Windows somehow.

[wbarnha]

Looks like https://github.com/AustEcon/python-rocksdb/tree/develop was attempting to build python-rocksdb for Windows, could probably try where they left off and cross-reference with https://github.com/facebook/rocksdb/wiki/Building-on-Windows.

[mohammad-aqajani]

Same on macos apple M1, any solution?

[wbarnha]

I'm currently working on creating wheels for macOS and Windows. The plan currently is to get macOS wheels rolled out first, then Windows, since macOS is typically easier to get initially working.

@mohammad-aqajani you can test this project's wheel builds for macOS via pipx run cibuildwheel --platform macos --archs aarch64. You may experience some issues with delocate as I did in https://github.com/faust-streaming/python-rocksdb/actions/runs/3743356084/jobs/6355429481. I don't have immediate access to a macOS development environment, so adding support is a bit difficult on my end. If you'd like to assist, feel free to join the Faust Slack.

[wbarnha]

There are notes to support macOS on twmht/python-rocksdb#47

[martinffx]

Does this support Rocksdb 7 installed from homebrew?

[wbarnha]

This project doesn't have support for RocksDB 7 unfortunately, but I have been contemplating an upgrade.

I should note this project uses RocksDB 6, you may see errors similar to twmht/python-rocksdb#105 for example if you use versions of rocksdict later than 0.3.2 (which uses RocksDB 7) in tandem with this library.
Edit: rocksdb==0.3.7 is safe to use alongside this project now.