feat: add moss-connector-postgres - #334
Closed
JingliangGao wants to merge 2 commits into
Closed
Conversation
Add PostgreSQL source connector for Moss that reads rows from a PostgreSQL database via psycopg (v3) and yields DocumentInfo objects. Supports regular Postgres, Neon, Supabase (direct URL), CockroachDB, Amazon RDS, Timescale, and pgvector. - PostgresConnector with DSN-based connection - Uses psycopg.rows.dict_row for dict-keyed row output - Unit tests with mocked psycopg connection (4 tests passing) - Integration test boilerplate (skips without POSTGRES_DSN) Closes #168
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new PostgreSQL source connector package under packages/moss-data-connector/ that reads rows via psycopg v3 and ingests them into a Moss index, plus associated docs/tests to match the existing connector ecosystem.
Changes:
- Introduces
moss-connector-postgres(connector + ingest helper) usingpsycopg.rows.dict_rowfor dict-keyed rows. - Adds unit tests (mocked psycopg + mocked MossClient) and a live integration test (skips without
POSTGRES_DSN+ Moss creds). - Updates
packages/moss-data-connector/README.mdto list the new connector.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/moss-data-connector/README.md | Adds Postgres connector to the connector directory listing/table (and touched formatting). |
| packages/moss-data-connector/moss-connector-postgres/src/connector.py | Implements PostgresConnector to run a query and yield DocumentInfo via a row mapper. |
| packages/moss-data-connector/moss-connector-postgres/src/ingest.py | Provides ingest() helper to create a Moss index from an iterable of DocumentInfo. |
| packages/moss-data-connector/moss-connector-postgres/src/init.py | Re-exports PostgresConnector and ingest as the package surface. |
| packages/moss-data-connector/moss-connector-postgres/tests/test_postgres.py | Adds mocked unit tests covering ingest, empty results, DSN forwarding, and auto_id. |
| packages/moss-data-connector/moss-connector-postgres/tests/test_integration_postgres_moss.py | Adds live end-to-end test (DB → Moss → query → cleanup) behind env var skip. |
| packages/moss-data-connector/moss-connector-postgres/README.md | Documents installation, usage, and test commands for the new connector. |
| packages/moss-data-connector/moss-connector-postgres/pyproject.toml | Defines the new package metadata/dependencies (including psycopg[binary]). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+60
to
+83
| def postgres_table(): | ||
| """Create a temporary table with 5 recognisable rows, drop it after the test.""" | ||
| table_name = f"moss_test_{uuid.uuid4().hex[:8]}" | ||
| conn = psycopg.connect(POSTGRES_DSN) | ||
| conn.autocommit = True | ||
| try: | ||
| with conn.cursor() as cur: | ||
| cur.execute( | ||
| f"CREATE TEMPORARY TABLE {table_name} " | ||
| "(id INT PRIMARY KEY, title VARCHAR(255), body TEXT)" | ||
| ) | ||
| cur.executemany( | ||
| f"INSERT INTO {table_name} (id, title, body) VALUES (%s, %s, %s)", | ||
| [ | ||
| (1, "Refund policy", "Refunds take 3 to 5 business days."), | ||
| (2, "Shipping time", "Orders ship within 24 hours."), | ||
| (3, "Contact support", "Reach support 24/7 via live chat."), | ||
| (4, "Password reset", "Click the link on the login page."), | ||
| (5, "Order tracking", "Tracking number sent by email."), | ||
| ], | ||
| ) | ||
| yield table_name | ||
| finally: | ||
| conn.close() |
Comment on lines
+51
to
+55
| [tool.ruff.lint] | ||
| select = ["E", "W", "F", "I", "B", "UP"] | ||
|
|
||
| [tool.pytest.ini_options] | ||
| asyncio_mode = "auto" No newline at end of file |
Comment on lines
10
to
+14
| ├── moss-connector-sqlite/ # SQLite source (stdlib, no driver) | ||
| ├── moss-connector-mongodb/ # MongoDB source (requires pymongo) | ||
| ├── moss-connector-mysql/ # MySQL / MariaDB source (requires pymysql) | ||
| ├── moss-connector-supabase/ # Supabase source (requires supabase) | ||
| └── moss-connector-dynamodb/ # Amazon DynamoDB source (requires boto3) | ||
| ├── moss-connector-dynamodb/ # Amazon DynamoDB source (requires boto3) |
Comment on lines
+39
to
+46
| | Package | Source | Extra driver | | ||
| | ---------------------------------------------------------- | --------------- | ------------ | | ||
| | [`moss-connector-sqlite`](moss-connector-sqlite) | SQLite | — | | ||
| | [`moss-connector-mongodb`](moss-connector-mongodb) | MongoDB | `pymongo` | | ||
| | [`moss-connector-mysql`](moss-connector-mysql) | MySQL | `pymysql` | | ||
| | [`moss-connector-supabase`](moss-connector-supabase) | Supabase | `supabase` | | ||
| | [`moss-connector-dynamodb`](moss-connector-dynamodb) | Amazon DynamoDB | `boto3` | | ||
| | [`moss-connector-postgres`](moss-connector-postgres) | PostgreSQL | `psycopg` | |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| try: | ||
| with conn.cursor() as cur: | ||
| cur.execute( | ||
| f"CREATE TEMPORARY TABLE {table_name} " |
There was a problem hiding this comment.
CONSIDER The test creates a temporary table on the fixture connection, then PostgresConnector opens a separate connection for SELECT, so Postgres will report the relation as missing when this integration test is enabled.
cur.execute(
f"CREATE TEMPORARY TABLE {table_name} "Use a regular uniquely named table and drop it in finally, or run the connector against the same connection/session. Prefer psycopg.sql.Identifier when formatting the table name.
Codex reviewThe connector implementation is small and mostly coherent, but the live Postgres integration test cannot pass as written because it seeds data in a session-local table that the connector never uses. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add PostgreSQL source connector for Moss that reads rows from a PostgreSQL database via psycopg (v3) and yields DocumentInfo objects.
Supports regular Postgres, Neon, Supabase (direct URL), CockroachDB, Amazon RDS, Timescale, and pgvector.
Closes #168
Pull Request Checklist
Please ensure that your PR meets the following requirements:
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
Fixes #168
Type of Change