feat: add moss-connector-postgres

Add PostgreSQL source connector for Moss that reads rows from a
PostgreSQL database via psycopg (v3) and yields DocumentInfo objects.

Supports regular Postgres, Neon, Supabase (direct URL), CockroachDB,
Amazon RDS, Timescale, and pgvector.

- PostgresConnector with DSN-based connection
- Uses psycopg.rows.dict_row for dict-keyed row output
- Unit tests with mocked psycopg connection (4 tests passing)
- Integration test boilerplate (skips without POSTGRES_DSN)

Closes #168

Author: JingliangGao

packages/moss-data-connector/README.md

@@ -11,7 +11,8 @@ moss-data-connector/
 ├── moss-connector-mongodb/     # MongoDB source (requires pymongo)
 ├── moss-connector-mysql/       # MySQL / MariaDB source (requires pymysql)
 ├── moss-connector-supabase/    # Supabase source (requires supabase)
-└── moss-connector-dynamodb/    # Amazon DynamoDB source (requires boto3)
+├── moss-connector-dynamodb/    # Amazon DynamoDB source (requires boto3)
+└── moss-connector-postgres/    # PostgreSQL source (requires psycopg)
 ```
 
 
@@ -35,15 +36,15 @@ Use `auto_id=True` when your mapper does not have a stable primary key and you w
 
 ## Available connectors
 
-| Package                                                    | Source        | Extra driver |
-| ---------------------------------------------------------- | ------------- | ------------ |
-| [`moss-connector-sqlite`](moss-connector-sqlite)           | SQLite        | —            |
-| [`moss-connector-mongodb`](moss-connector-mongodb)         | MongoDB       | `pymongo`    |
-| [`moss-connector-mysql`](moss-connector-mysql)             | MySQL         | `pymysql`    |
-| [`moss-connector-supabase`](moss-connector-supabase)       | Supabase      | `supabase`   |
-| [`moss-connector-dynamodb`](moss-connector-dynamodb)       | Amazon DynamoDB | `boto3`    |
+| Package                                                    | Source          | Extra driver |
+| ---------------------------------------------------------- | --------------- | ------------ |
+| [`moss-connector-sqlite`](moss-connector-sqlite)           | SQLite          | —            |
+| [`moss-connector-mongodb`](moss-connector-mongodb)         | MongoDB         | `pymongo`    |
+| [`moss-connector-mysql`](moss-connector-mysql)             | MySQL           | `pymysql`    |
+| [`moss-connector-supabase`](moss-connector-supabase)       | Supabase        | `supabase`   |
+| [`moss-connector-dynamodb`](moss-connector-dynamodb)       | Amazon DynamoDB | `boto3`      |
+| [`moss-connector-postgres`](moss-connector-postgres)       | PostgreSQL      | `psycopg`    |
 
 ## Adding a new connector
 
-See [`_template/README.md`](_template/README.md).
-
+See [`_template/README.md`](_template/README.md).

packages/moss-data-connector/moss-connector-postgres/README.md

@@ -0,0 +1,72 @@
+# moss-connector-postgres
+
+PostgreSQL source connector for Moss. Uses [psycopg](https://www.psycopg.org/psycopg3/) (v3) so it works against regular Postgres, Neon, Supabase (direct URL), CockroachDB, Amazon RDS, Timescale, and pgvector.
+
+## Install
+
+```bash
+pip install moss-connector-postgres
+```
+
+This installs `psycopg[binary]` automatically.
+
+## Usage
+
+```python
+import asyncio
+from moss import DocumentInfo
+from moss_connector_postgres import PostgresConnector, ingest
+
+async def main():
+    source = PostgresConnector(
+        dsn="postgresql://user:pass@localhost:5432/mydb",
+        query="SELECT id, title, body FROM articles",
+        mapper=lambda row: DocumentInfo(
+            id=str(row["id"]),
+            text=row["body"],
+            metadata={"title": row["title"]},
+        ),
+    )
+
+    result = await ingest(
+        source,
+        project_id="your_project_id",
+        project_key="your_project_key",
+        index_name="articles",
+    )
+    print(f"copied {result.doc_count} rows")
+
+asyncio.run(main())
+```
+
+### DSN formats
+
+The connector accepts any standard PostgreSQL connection string:
+
+```python
+# Local Postgres
+PostgresConnector(dsn="postgresql://user:pass@localhost:5432/mydb", ...)
+
+# Neon serverless
+PostgresConnector(dsn="postgresql://user:pass@ep-xxx.us-east-2.aws.neon.tech/neondb", ...)
+
+# Supabase (direct connection URL)
+PostgresConnector(dsn="postgresql://postgres:pass@db.xxx.supabase.co:5432/postgres", ...)
+```
+
+## Layout
+
+```
+src/
+├── __init__.py      # re-exports PostgresConnector and ingest
+├── connector.py     # PostgresConnector class
+└── ingest.py        # ingest() - keep in sync with the other connector packages
+```
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+pytest tests/test_postgres.py -v                         # mocked, no network or DB needed
+pytest tests/test_integration_postgres_moss.py -v -s     # live Postgres + Moss (requires POSTGRES_DSN, MOSS_PROJECT_ID, MOSS_PROJECT_KEY)
+```

packages/moss-data-connector/moss-connector-postgres/pyproject.toml

@@ -0,0 +1,55 @@
+[project]
+name = "moss-connector-postgres"
+version = "0.0.1"
+description = "PostgreSQL source connector for Moss. Uses psycopg (v3) so it works against Postgres, Neon, Supabase (direct URL), CockroachDB, Amazon RDS, Timescale, and pgvector."
+readme = "README.md"
+requires-python = ">=3.10,<3.15"
+license = { text = "BSD-2-Clause" }
+authors = [{ name = "InferEdge Inc.", email = "contact@moss.dev" }]
+keywords = ["moss", "connectors", "ingest", "postgres", "postgresql"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: BSD License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Database",
+]
+dependencies = [
+    "moss>=1.1.1",
+    "psycopg[binary]>=3.1",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-asyncio>=0.23.0",
+    "python-dotenv>=1.0.0",
+    "ruff>=0.5.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/usemoss/moss"
+Repository = "https://github.com/usemoss/moss"
+Source = "https://github.com/usemoss/moss/tree/main/packages/moss-data-connector/moss-connector-postgres"
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+packages = ["moss_connector_postgres"]
+package-dir = { "moss_connector_postgres" = "src" }
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "B", "UP"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"

packages/moss-data-connector/moss-connector-postgres/src/__init__.py

@@ -0,0 +1,9 @@
+"""PostgreSQL source connector for Moss.
+
+    from moss_connector_postgres import PostgresConnector, ingest
+"""
+
+from .connector import PostgresConnector
+from .ingest import ingest
+
+__all__ = ["PostgresConnector", "ingest"]

packages/moss-data-connector/moss-connector-postgres/src/connector.py

@@ -0,0 +1,47 @@
+"""PostgreSQL connector.
+
+Reads rows from a PostgreSQL database via ``psycopg`` (v3) and yields one
+``DocumentInfo`` per row. Uses ``DictRow`` factory so every row is a plain dict
+keyed by column name.
+
+Works against regular Postgres, Neon, Supabase (direct URL), CockroachDB,
+Amazon RDS, Timescale, and pgvector.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Iterator
+from typing import Any
+
+import psycopg
+import psycopg.rows
+from moss import DocumentInfo
+
+
+class PostgresConnector:
+    """Run a SELECT against a PostgreSQL database and yield one
+    ``DocumentInfo`` per row.
+
+    ``mapper`` turns a row (dict of column → value) into a ``DocumentInfo``;
+    the caller decides which columns become id / text / metadata / embedding.
+    """
+
+    def __init__(
+        self,
+        dsn: str,
+        query: str,
+        mapper: Callable[[dict[str, Any]], DocumentInfo],
+    ) -> None:
+        self.dsn = dsn
+        self.query = query
+        self.mapper = mapper
+
+    def __iter__(self) -> Iterator[DocumentInfo]:
+        conn = psycopg.connect(self.dsn, row_factory=psycopg.rows.dict_row)
+        try:
+            with conn.cursor() as cursor:
+                cursor.execute(self.query)
+                for row in cursor:
+                    yield self.mapper(row)
+        finally:
+            conn.close()

packages/moss-data-connector/moss-connector-postgres/src/ingest.py

@@ -0,0 +1,36 @@
+"""Copy rows into a Moss index."""
+
+from __future__ import annotations
+
+import uuid
+from collections.abc import Iterable
+
+from moss import DocumentInfo, MossClient, MutationResult
+
+
+def _replace_doc_id(doc: DocumentInfo) -> DocumentInfo:
+    return DocumentInfo(
+        id=str(uuid.uuid4()),
+        text=doc.text,
+        metadata=getattr(doc, "metadata", None),
+        embedding=getattr(doc, "embedding", None),
+    )
+
+
+async def ingest(
+    source: Iterable[DocumentInfo],
+    project_id: str,
+    project_key: str,
+    index_name: str,
+    model_id: str | None = None,
+    auto_id: bool = False,
+) -> MutationResult | None:
+    """Copy every `DocumentInfo` from `source` into a fresh Moss index."""
+    if auto_id:
+        docs = [_replace_doc_id(doc) for doc in source]
+    else:
+        docs = list(source)
+    if not docs:
+        return None
+    client = MossClient(project_id, project_key)
+    return await client.create_index(index_name, docs, model_id=model_id)

packages/moss-data-connector/moss-connector-postgres/tests/test_integration_postgres_moss.py

@@ -0,0 +1,126 @@
+"""End-to-end integration test against a real PostgreSQL database and Moss project.
+
+This test actually creates a temporary table in Postgres, ingests from it into a
+real Moss index, queries the index, and cleans up afterwards. It is SKIPPED
+unless POSTGRES_DSN, MOSS_PROJECT_ID, and MOSS_PROJECT_KEY are all set in the
+environment (or in a .env file at the repo root or package root).
+
+POSTGRES_DSN should be a connection string like:
+    postgresql://user:password@host:port/database
+
+Run it with:
+    cd packages/moss-data-connector/moss-connector-postgres
+    pytest tests/test_integration_postgres_moss.py -v -s
+"""
+
+from __future__ import annotations
+
+import os
+import uuid
+from pathlib import Path
+
+import pytest
+
+# Load .env from the package dir, then the repo root, if present.
+try:
+    from dotenv import load_dotenv
+
+    _here = Path(__file__).resolve()
+    for candidate in (
+        _here.parents[1] / ".env",                   # this package's own .env
+        _here.parents[2] / ".env",                   # moss-data-connector/.env
+        _here.parents[4] / ".env",                   # <repo>/.env
+    ):
+        if candidate.exists():
+            load_dotenv(candidate, override=False)
+except ImportError:
+    pass  # dotenv is optional; env vars can also be set directly.
+
+pytest.importorskip("psycopg")
+
+import psycopg  # noqa: E402
+import psycopg.rows  # noqa: E402
+from moss import DocumentInfo, MossClient, QueryOptions  # noqa: E402
+from moss_connector_postgres import PostgresConnector, ingest  # noqa: E402
+
+POSTGRES_DSN = os.getenv("POSTGRES_DSN")
+PROJECT_ID = os.getenv("MOSS_PROJECT_ID")
+PROJECT_KEY = os.getenv("MOSS_PROJECT_KEY")
+
+pytestmark = pytest.mark.skipif(
+    not (POSTGRES_DSN and PROJECT_ID and PROJECT_KEY),
+    reason="Set POSTGRES_DSN, MOSS_PROJECT_ID, and MOSS_PROJECT_KEY to run the real integration test.",
+)
+
+
+@pytest.fixture()
+def postgres_table():
+    """Create a temporary table with 5 recognisable rows, drop it after the test."""
+    table_name = f"moss_test_{uuid.uuid4().hex[:8]}"
+    conn = psycopg.connect(POSTGRES_DSN)
+    conn.autocommit = True
+    try:
+        with conn.cursor() as cur:
+            cur.execute(
+                f"CREATE TEMPORARY TABLE {table_name} "
+                "(id INT PRIMARY KEY, title VARCHAR(255), body TEXT)"
+            )
+            cur.executemany(
+                f"INSERT INTO {table_name} (id, title, body) VALUES (%s, %s, %s)",
+                [
+                    (1, "Refund policy", "Refunds take 3 to 5 business days."),
+                    (2, "Shipping time", "Orders ship within 24 hours."),
+                    (3, "Contact support", "Reach support 24/7 via live chat."),
+                    (4, "Password reset", "Click the link on the login page."),
+                    (5, "Order tracking", "Tracking number sent by email."),
+                ],
+            )
+        yield table_name
+    finally:
+        conn.close()
+
+
+async def test_postgres_ingest_end_to_end(postgres_table):
+    """Full round trip: Postgres → Moss index → query → delete."""
+    table_name = postgres_table
+    client = MossClient(PROJECT_ID, PROJECT_KEY)
+
+    # Unique index name per run so concurrent runs don't collide.
+    index_name = f"moss-postgres-e2e-{uuid.uuid4().hex[:8]}"
+
+    try:
+        connector = PostgresConnector(
+            dsn=POSTGRES_DSN,
+            query=f"SELECT id, title, body FROM {table_name}",
+            mapper=lambda r: DocumentInfo(
+                id=str(r["id"]),
+                text=r["body"],
+                metadata={"title": r["title"]},
+            ),
+        )
+
+        result = await ingest(connector, PROJECT_ID, PROJECT_KEY, index_name=index_name)
+        assert result is not None
+        assert result.doc_count == 5
+
+        # Query the live index. "refund" should pull back article 1.
+        await client.load_index(index_name)
+        result = await client.query(
+            index_name, "how long do refunds take", QueryOptions(top_k=3)
+        )
+
+        assert result.docs, "expected at least one document in the search result"
+        top_ids = [d.id for d in result.docs]
+        assert "1" in top_ids, f"refund-policy doc not in top 3: {top_ids}"
+
+        # Check the metadata survived the round trip.
+        refund_doc = next(d for d in result.docs if d.id == "1")
+        assert refund_doc.metadata is not None
+        assert refund_doc.metadata.get("title") == "Refund policy"
+
+    finally:
+        # Always try to clean up, even if an assertion above failed.
+        try:
+            await client.delete_index(index_name)
+        except Exception as exc:  # pragma: no cover, best-effort cleanup
+            print(f"warning: failed to delete test index {index_name}: {exc}")