Skip to content

Commit d3617b9

Browse files
HarshaNalluruHarshaNalluru
committed
feat(connectors): add moss-connector-zeroentropy
Migrate a ZeroEntropy collection into a Moss index, following the moss-data-connector _template pattern: a ZeroEntropyConnector whose __iter__ pages through documents.get_info_list and yields DocumentInfo, plus the shared ingest() helper. - Default mapping (path -> id, content -> text, metadata -> metadata) so a whole collection migrates with no mapper; overridable via mapper=. - Fetches parsed text per document via get_info(include_content=True) and skips documents that never parsed (no content) rather than indexing empties. - Coerces ZeroEntropy's str|list[str] metadata to Moss's str-only values. - Unit tests (mocked, 9 passing) + skippable live integration test. - Publish workflow, package README, and a row in the connectors table. Verified against the installed zeroentropy 0.1.0a11 SDK: method names, params, and response fields all match.
1 parent 4027809 commit d3617b9

10 files changed

Lines changed: 1218 additions & 1 deletion

File tree

‎.github/workflows/publish-moss-connector-zeroentropy.yml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
name: Publish moss-connector-zeroentropy
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
workflow_dispatch:
8+
9+
concurrency:
10+
group: moss-connector-zeroentropy-release
11+
cancel-in-progress: false
12+
13+
jobs:
14+
determine-version:
15+
runs-on: ubuntu-latest
16+
outputs:
17+
version: ${{ steps.compute.outputs.version }}
18+
steps:
19+
- uses: actions/checkout@v4
20+
21+
- uses: actions/setup-python@v5
22+
with:
23+
python-version: "3.11"
24+
25+
- name: Read version from pyproject
26+
id: compute
27+
shell: python
28+
run: |
29+
import os, pathlib, re, sys
30+
31+
text = pathlib.Path("packages/moss-data-connector/moss-connector-zeroentropy/pyproject.toml").read_text(encoding="utf-8")
32+
match = re.search(r'(?m)^version\s*=\s*"([^"]+)"', text)
33+
if not match:
34+
print("Could not find version in pyproject", file=sys.stderr)
35+
sys.exit(1)
36+
version = match.group(1)
37+
38+
out = pathlib.Path(os.environ["GITHUB_OUTPUT"])
39+
with out.open("a", encoding="utf-8") as fh:
40+
fh.write(f"version={version}\n")
41+
42+
print(f"Publishing version: {version}")
43+
44+
build-and-publish:
45+
needs: determine-version
46+
strategy:
47+
fail-fast: false
48+
matrix:
49+
python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
50+
runs-on: ubuntu-latest
51+
permissions:
52+
contents: write
53+
env:
54+
VERSION: ${{ needs.determine-version.outputs.version }}
55+
56+
steps:
57+
- uses: actions/checkout@v4
58+
59+
- uses: actions/setup-python@v5
60+
with:
61+
python-version: ${{ matrix.python }}
62+
63+
- name: Install build tooling
64+
run: |
65+
python -m pip install --upgrade pip
66+
pip install build twine
67+
68+
- name: Build distributions
69+
working-directory: packages/moss-data-connector/moss-connector-zeroentropy
70+
run: |
71+
rm -rf dist
72+
python -m build
73+
74+
- name: Smoke test import
75+
shell: python
76+
run: |
77+
import glob, subprocess, sys
78+
79+
wheels = glob.glob("packages/moss-data-connector/moss-connector-zeroentropy/dist/*.whl")
80+
if not wheels:
81+
raise SystemExit("No wheel found in dist/")
82+
83+
wheel = wheels[0]
84+
subprocess.check_call([sys.executable, "-m", "pip", "install", "--force-reinstall", wheel])
85+
86+
import importlib
87+
module = importlib.import_module("moss_connector_zeroentropy")
88+
print("import ok; sample attrs:", dir(module)[:5])
89+
90+
- name: Publish to PyPI
91+
if: matrix.python == '3.11'
92+
env:
93+
TWINE_USERNAME: __token__
94+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
95+
run: |
96+
twine upload --skip-existing packages/moss-data-connector/moss-connector-zeroentropy/dist/*
97+
98+
- name: Tag release
99+
if: matrix.python == '3.11'
100+
env:
101+
VERSION: ${{ env.VERSION }}
102+
run: |
103+
git fetch --tags --force
104+
if git rev-parse "moss-connector-zeroentropy-v${VERSION}" >/dev/null 2>&1; then
105+
echo "Tag moss-connector-zeroentropy-v${VERSION} already exists"
106+
else
107+
git config user.name "github-actions"
108+
git config user.email "github-actions@users.noreply.github.com"
109+
git tag "moss-connector-zeroentropy-v${VERSION}"
110+
git push origin "moss-connector-zeroentropy-v${VERSION}"
111+
fi

‎packages/moss-data-connector/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@ moss-data-connector/
1111
├── moss-connector-mongodb/ # MongoDB source (requires pymongo)
1212
├── moss-connector-mysql/ # MySQL / MariaDB source (requires pymysql)
1313
├── moss-connector-supabase/ # Supabase source (requires supabase)
14-
└── moss-connector-dynamodb/ # Amazon DynamoDB source (requires boto3)
14+
├── moss-connector-dynamodb/ # Amazon DynamoDB source (requires boto3)
15+
└── moss-connector-zeroentropy/ # ZeroEntropy source (requires zeroentropy)
1516
```
1617

1718

@@ -42,6 +43,7 @@ Use `auto_id=True` when your mapper does not have a stable primary key and you w
4243
| [`moss-connector-mysql`](moss-connector-mysql) | MySQL | `pymysql` |
4344
| [`moss-connector-supabase`](moss-connector-supabase) | Supabase | `supabase` |
4445
| [`moss-connector-dynamodb`](moss-connector-dynamodb) | Amazon DynamoDB | `boto3` |
46+
| [`moss-connector-zeroentropy`](moss-connector-zeroentropy) | ZeroEntropy | `zeroentropy` |
4547

4648
## Adding a new connector
4749

‎packages/moss-data-connector/moss-connector-zeroentropy/README.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# moss-connector-zeroentropy
2+
3+
ZeroEntropy source connector for Moss. Uses the [ZeroEntropy Python SDK](https://github.com/zeroentropy-ai/zeroentropy-python) to read every document from a ZeroEntropy collection and copy it into a Moss index, so you can migrate off ZeroEntropy in a few lines.
4+
5+
## Install
6+
7+
```bash
8+
pip install moss-connector-zeroentropy
9+
```
10+
11+
This installs `zeroentropy` automatically.
12+
13+
## Usage
14+
15+
Migrating a whole collection needs no mapper: the default mapping uses each document's `path` as the Moss id, its parsed `content` as the searchable text, and copies its metadata.
16+
17+
```python
18+
import asyncio
19+
from moss_connector_zeroentropy import ZeroEntropyConnector, ingest
20+
21+
async def main():
22+
source = ZeroEntropyConnector(
23+
collection_name="my_collection",
24+
api_key="your-zeroentropy-key", # or set ZEROENTROPY_API_KEY
25+
)
26+
27+
result = await ingest(
28+
source,
29+
project_id="your_project_id",
30+
project_key="your_project_key",
31+
index_name="my_collection",
32+
)
33+
print(f"migrated {result.doc_count} documents")
34+
35+
asyncio.run(main())
36+
```
37+
38+
Set `ZEROENTROPY_API_KEY` in the environment and drop `api_key=` entirely.
39+
40+
### Custom mapping
41+
42+
Pass a `mapper` to control the `DocumentInfo`. It receives one row dict per document, holding the document's ZeroEntropy fields (`id`, `path`, `metadata`, `file_url`, `index_status`, `num_pages`, `size`) plus the fetched `content`:
43+
44+
```python
45+
source = ZeroEntropyConnector(
46+
collection_name="my_collection",
47+
mapper=lambda row: DocumentInfo(
48+
id=row["id"], # ZeroEntropy's UUID instead of the path
49+
text=row["content"],
50+
metadata={"path": row["path"], "pages": str(row["num_pages"])},
51+
),
52+
)
53+
```
54+
55+
Use `auto_id=True` on `ingest(...)` to have Moss generate UUID document ids instead.
56+
57+
## Migrating every collection
58+
59+
The SDK can list your collections, so migrating all of them is a loop:
60+
61+
```python
62+
from zeroentropy import ZeroEntropy
63+
64+
ze = ZeroEntropy(api_key="your-zeroentropy-key")
65+
for name in ze.collections.get_list().collection_names:
66+
source = ZeroEntropyConnector(collection_name=name, api_key="your-zeroentropy-key")
67+
await ingest(source, project_id="...", project_key="...", index_name=name)
68+
```
69+
70+
## Data requirements
71+
72+
The connector doesn't impose a schema — it hands each document to your `mapper` as a dict. The constraints come from `DocumentInfo`, not the connector.
73+
74+
`DocumentInfo` fields:
75+
76+
| Field | Type | Required? | Source in a ZeroEntropy document |
77+
|---|---|---|---|
78+
| `id` | `str` | yes | `path` (default) or `id` |
79+
| `text` | `str` | yes | `content` (the parsed document text) |
80+
| `metadata` | `Optional[Dict[str, str]]` | no | `metadata` |
81+
| `embedding` | `Optional[Sequence[float]]` | no | not exported by ZeroEntropy |
82+
83+
### Metadata values must be strings
84+
85+
ZeroEntropy metadata is typed `Dict[str, str | list[str]]` — a value can be a single string **or a list of strings**. `DocumentInfo.metadata` requires `Dict[str, str]`, so the default mapper coerces list values by joining them with `", "` (via `coerce_metadata`). If you write your own mapper, coerce list-valued metadata yourself:
86+
87+
```python
88+
# WILL FAIL — a list value
89+
metadata={"tags": row["metadata"]["tags"]}
90+
91+
# CORRECT
92+
metadata={"tags": ", ".join(row["metadata"]["tags"])}
93+
```
94+
95+
## Content and skipped documents
96+
97+
The list endpoint (`documents.get_info_list`) returns metadata only, so the connector fetches each document's text with `documents.get_info(..., include_content=True)` — one call per document. A document that never parsed (`index_status` of `parsing_failed`, `parsing`, etc.) has no `content`; the connector **skips it** rather than indexing an empty document into Moss.
98+
99+
To migrate metadata only (no per-document content fetch), pass `include_content=False` and a `mapper` that does not read `content`.
100+
101+
To restrict the migration to a subtree, pass `path_prefix="docs/2024/"`.
102+
103+
## Pagination
104+
105+
`documents.get_info_list` is an auto-paginating cursor, so the connector iterates every page transparently — there is no page-size knob to tune.
106+
107+
## Layout
108+
109+
```
110+
src/
111+
├── __init__.py # re-exports ZeroEntropyConnector, ingest, default_mapper, coerce_metadata
112+
├── connector.py # ZeroEntropyConnector + default_mapper + coerce_metadata
113+
└── ingest.py # ingest() - keep in sync with the other connector packages
114+
```
115+
116+
## Tests
117+
118+
```bash
119+
pip install -e ".[dev]"
120+
pytest tests/test_zeroentropy.py -v # mocked, no network needed
121+
pytest tests/test_integration_zeroentropy_moss.py -v -s # live ZeroEntropy + Moss
122+
```
123+
124+
The integration test requires `ZEROENTROPY_API_KEY`, `ZEROENTROPY_TEST_COLLECTION` (a pre-populated collection), `MOSS_PROJECT_ID`, and `MOSS_PROJECT_KEY`.

‎packages/moss-data-connector/moss-connector-zeroentropy/pyproject.toml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
[project]
2+
name = "moss-connector-zeroentropy"
3+
version = "0.0.1"
4+
description = "ZeroEntropy source connector for moss-connectors: migrate a ZeroEntropy collection into Moss."
5+
readme = "README.md"
6+
requires-python = ">=3.10,<3.15"
7+
license = { text = "BSD-2-Clause" }
8+
authors = [{ name = "InferEdge Inc.", email = "contact@moss.dev" }]
9+
keywords = ["moss", "connectors", "zeroentropy", "retrieval", "migration", "ingest", "etl"]
10+
classifiers = [
11+
"Development Status :: 3 - Alpha",
12+
"Intended Audience :: Developers",
13+
"License :: OSI Approved :: BSD License",
14+
"Programming Language :: Python :: 3",
15+
"Programming Language :: Python :: 3.10",
16+
"Programming Language :: Python :: 3.11",
17+
"Programming Language :: Python :: 3.12",
18+
"Programming Language :: Python :: 3.13",
19+
"Topic :: Database",
20+
]
21+
dependencies = [
22+
"moss>=1.1.1",
23+
"zeroentropy>=0.1.0a11",
24+
]
25+
26+
[project.optional-dependencies]
27+
dev = [
28+
"pytest>=8.0.0",
29+
"pytest-asyncio>=0.23.0",
30+
"python-dotenv>=1.0.0",
31+
"ruff>=0.5.0",
32+
]
33+
34+
[project.urls]
35+
Homepage = "https://github.com/usemoss/moss"
36+
Repository = "https://github.com/usemoss/moss"
37+
Source = "https://github.com/usemoss/moss/tree/main/packages/moss-data-connector/moss-connector-zeroentropy"
38+
39+
[build-system]
40+
requires = ["setuptools>=61.0"]
41+
build-backend = "setuptools.build_meta"
42+
43+
# Flat layout: src/ itself IS the package `moss_connector_zeroentropy`.
44+
[tool.setuptools]
45+
packages = ["moss_connector_zeroentropy"]
46+
package-dir = { "moss_connector_zeroentropy" = "src" }
47+
48+
[tool.ruff]
49+
line-length = 100
50+
target-version = "py310"
51+
52+
[tool.ruff.lint]
53+
select = ["E", "W", "F", "I", "B", "UP"]
54+
55+
[tool.pytest.ini_options]
56+
asyncio_mode = "auto"

‎packages/moss-data-connector/moss-connector-zeroentropy/src/__init__.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
from .connector import ZeroEntropyConnector, coerce_metadata, default_mapper
2+
from .ingest import ingest
3+
4+
__all__ = ["ZeroEntropyConnector", "coerce_metadata", "default_mapper", "ingest"]

0 commit comments

Comments
 (0)