Unable to install Sycamore from the main branch due to a private submodule (opensearch-remote-processor)

[Adamooop]

Hello Aryn team,

I’ve been trying to install the latest Sycamore code from the main branch, but the installation fails because of a private submodule: opensearch-remote-processor.

Here are the specific issues I'm encountering:

• When running pip install --upgrade "git+https://github.com/aryn-ai/sycamore.git@main#egg=sycamore-ai[weaviate]", it attempts to clone opensearch-remote-processor via SSH, which is inaccessible to me.
• The build stops with an error indicating that I do not have the correct access rights to clone the submodule.

My goal is to use the latest Sycamore features, such as Map transformations and the updated DocSet functionality, which are not available in the PyPI release. However, without access to opensearch-remote-processor, the installation process fails.

Could you please:

1. Make the opensearch-remote-processor submodule public, or
2. Provide a version/branch of Sycamore that does not depend on private modules, or
3. Grant me access to the private repository if it’s legally permissible?

This issue blocks me from upgrading Sycamore to use the new features. Any advice or workaround would be appreciated.

System details:

• Windows 10 (Python 3.11)
• Sycamore-ai PyPI version is 0.1.32, which lacks the updated transformations.
• I tried git clone --recursive and also the direct pip install from GitHub, both failed with submodule permission errors.

Thank you for your help, and for all the work on Sycamore!

Best regards,

Stan, osm thome

[HenryL27]

Hey stan!

Thanks for checking this out. The opensearch remote processor repo is telling me that it is public already. My guess is that your git client doesn't have ssh keys set up? Can you try following this guide?

Alternatively you can clone non-recursively and a/ edit the .gitmodules file to point to https instead of ssh or b/ ignore the submodule as it's entirely unneeded for the sycamore library.

May I ask out of curiosity what new features you're looking for? docset.map has been around since v0.0.1