Skip to content

CLAUDE.md

Latest commit

 

History

History
450 lines (338 loc) · 42.2 KB

CLAUDE.md

File metadata and controls

450 lines (338 loc) · 42.2 KB

Project: ponylang/postgres

Pure Pony PostgreSQL driver. Alpha-level. Version 0.2.2.

Building and Testing

make ssl=3.0.x                        # build and run all tests
make unit-tests ssl=3.0.x             # unit tests only (no postgres needed)
make integration-tests ssl=3.0.x      # integration tests (needs postgres)
make build-examples ssl=3.0.x         # compile examples
make start-pg-containers              # docker postgres:14.5 on ports 5432 (plain) and 5433 (SSL)
make stop-pg-containers               # stop docker containers

SSL version is mandatory. Tests run with --sequential. Integration tests require running PostgreSQL 14.5 containers with SCRAM-SHA-256 default auth and an MD5-only user (user: postgres, password: postgres, database: postgres; md5user: md5user, password: md5pass) — one plain on port 5432 and one with SSL on port 5433. Environment variables: POSTGRES_HOST, POSTGRES_PORT, POSTGRES_SSL_HOST, POSTGRES_SSL_PORT, POSTGRES_USERNAME, POSTGRES_PASSWORD, POSTGRES_DATABASE, POSTGRES_MD5_USERNAME, POSTGRES_MD5_PASSWORD.

Dependencies

  • ponylang/ssl 2.0.0 (MD5 password hashing, SCRAM-SHA-256 crypto primitives via ssl/crypto, SSL/TLS via ssl/net)
  • ponylang/lori 0.8.1 (TCP networking, STARTTLS support)

Managed via corral.

GitHub Labels

  • changelog - added, changelog - changed, changelog - fixed — PR-only labels. CI uses these to auto-generate CHANGELOG entries on merge. Never apply to issues.
  • bug, help wanted, good first issue, documentation, etc. — issue classification labels.

Architecture

Session State Machine

Session actor is the main entry point. Constructor takes ServerConnectInfo (auth, host, service, ssl_mode) and DatabaseConnectInfo (user, password, database). Implements lori.TCPConnectionActor and lori.ClientLifecycleEventReceiver. The stored ServerConnectInfo is accessible via server_connect_info() for use by _CancelSender. State transitions via _SessionState interface with concrete states:

_SessionUnopened  --connect (no SSL)-->  _SessionConnected
_SessionUnopened  --connect (SSL)-->     _SessionSSLNegotiating
_SessionUnopened  --fail-->              _SessionClosed
_SessionSSLNegotiating --'S'+TLS ok-->   _SessionConnected
_SessionSSLNegotiating --'N'/fail-->     _SessionClosed
_SessionConnected --MD5 auth ok-->       _SessionLoggedIn
_SessionConnected --MD5 auth fail-->     _SessionClosed
_SessionConnected --SASL challenge-->    _SessionSCRAMAuthenticating
_SessionSCRAMAuthenticating --auth ok--> _SessionLoggedIn
_SessionSCRAMAuthenticating --auth fail--> _SessionClosed
_SessionLoggedIn  --close-->             _SessionClosed

State behavior is composed via a trait hierarchy that mixes in capabilities and defaults:

  • _ConnectableState / _NotConnectableState — can/can't receive connection events
  • _ConnectedState / _UnconnectedState — has/doesn't have a live connection
  • _AuthenticableState / _NotAuthenticableState — can/can't authenticate
  • _AuthenticatedState / _NotAuthenticated — has/hasn't authenticated

_SessionSSLNegotiating is a standalone class (not using _ConnectedState) because it handles raw bytes — the server's SSL response is not a PostgreSQL protocol message, so _ResponseParser is not used. It mixes in _NotConnectableState, _NotAuthenticableState, and _NotAuthenticated.

_SessionSCRAMAuthenticating handles the multi-step SCRAM-SHA-256 exchange after _SessionConnected receives an AuthSASL challenge. It mixes in _ConnectedState (for on_received/TCP write) and _NotAuthenticated. Fields store the client nonce, client-first-bare, password, and expected server signature across the exchange steps.

This design makes illegal state transitions call _IllegalState() (panic) by default via the trait hierarchy, so only valid transitions need explicit implementation.

Query Execution Flow

  1. Client calls session.execute(query, ResultReceiver) where query is SimpleQuery, PreparedQuery, or NamedPreparedQuery; or session.prepare(name, sql, PrepareReceiver) to create a named statement; or session.close_statement(name) to destroy one; or session.copy_in(sql, CopyInReceiver) to start a COPY FROM STDIN operation; or session.copy_out(sql, CopyOutReceiver) to start a COPY TO STDOUT operation
  2. _SessionLoggedIn queues operations as _QueueItem — a union of _QueuedQuery (execute), _QueuedPrepare (prepare), _QueuedCloseStatement (close_statement), _QueuedCopyIn (copy_in), and _QueuedCopyOut (copy_out)
  3. The _QueryState sub-state machine manages operation lifecycle:
    • _QueryNotReady: initial state after auth, before the first ReadyForQuery arrives
    • _QueryReady: server is idle, try_run_query dispatches based on queue item type — SimpleQuery transitions to _SimpleQueryInFlight, PreparedQuery and NamedPreparedQuery transition to _ExtendedQueryInFlight, _QueuedPrepare transitions to _PrepareInFlight, _QueuedCloseStatement transitions to _CloseStatementInFlight, _QueuedCopyIn transitions to _CopyInInFlight, _QueuedCopyOut transitions to _CopyOutInFlight
    • _SimpleQueryInFlight: owns per-query accumulation data (_data_rows, _row_description), delivers results on CommandComplete
    • _ExtendedQueryInFlight: same data accumulation and result delivery as _SimpleQueryInFlight (duplicated because Pony traits can't have iso fields). Entered after sending Parse+Bind+Describe(portal)+Execute+Sync (unnamed) or Bind+Describe(portal)+Execute+Sync (named)
    • _PrepareInFlight: handles Parse+Describe(statement)+Sync cycle. Notifies PrepareReceiver on success/failure via ReadyForQuery
    • _CloseStatementInFlight: handles Close(statement)+Sync cycle. Fire-and-forget (no callback); errors silently absorbed
    • _CopyInInFlight: handles COPY FROM STDIN data transfer. Sends the COPY query via simple query protocol, receives CopyInResponse, then uses pull-based flow: calls pg_copy_ready on the CopyInReceiver to request data. Client calls send_copy_data (sends CopyData + pulls again), finish_copy (sends CopyDone), or abort_copy (sends CopyFail). Server responds with CommandComplete+ReadyForQuery on success, or ErrorResponse+ReadyForQuery on failure
    • _CopyOutInFlight: handles COPY TO STDOUT data reception. Sends the COPY query via simple query protocol, receives CopyOutResponse (silently consumed), then receives server-pushed CopyData messages (each delivered via pg_copy_data to the CopyOutReceiver), CopyDone (silently consumed), and finally CommandComplete (stores row count) + ReadyForQuery (delivers pg_copy_complete). On error, ErrorResponse delivers pg_copy_failed and the session remains usable
  4. Response data arrives: _RowDescriptionMessage sets column metadata, _DataRowMessage accumulates rows
  5. _CommandCompleteMessage triggers result delivery to receiver
  6. _ReadyForQueryMessage dequeues completed operation, transitions to _QueryReady

Only one operation is in-flight at a time. The queue serializes execution. query_queue, query_state, backend_pid, and backend_secret_key are non-underscore-prefixed fields on _SessionLoggedIn because other types in this package need cross-type access (Pony private fields are type-private). On shutdown, _SessionLoggedIn.on_shutdown calls query_state.drain_in_flight() to let the in-flight state handle its own queue item (skipping notification if on_error_response already notified the receiver), then drains remaining queued items with SessionClosed. This prevents double-notification when close() arrives between ErrorResponse and ReadyForQuery delivery.

Query cancellation: session.cancel() requests cancellation of the currently executing query by opening a separate TCP connection via _CancelSender and sending a CancelRequest. The cancel method on _SessionState follows the same "never illegal" contract as close — it is a no-op in all states except _SessionLoggedIn, where it fires only when a query is in flight (not in _QueryReady or _QueryNotReady). Cancellation is best-effort; the server may or may not honor it. If cancelled, the query's ResultReceiver receives pg_query_failed with an ErrorResponseMessage (SQLSTATE 57014).

Protocol Layer

Frontend (client → server):

  • _FrontendMessage primitive: startup(), password(), query(), parse(), bind(), describe_portal(), describe_statement(), execute_msg(), close_statement(), sync(), ssl_request(), cancel_request(), terminate(), sasl_initial_response(), sasl_response(), copy_data(), copy_done(), copy_fail() — builds raw byte arrays with big-endian wire format

Backend (server → client):

  • _ResponseParser primitive: incremental parser consuming from a Reader buffer. Returns one parsed message per call, None if incomplete, errors on junk.
  • _ResponseMessageParser primitive: routes parsed messages to the current session state's callbacks. Processes messages synchronously within a query cycle (looping until ReadyForQuery or buffer exhaustion), then yields via s._process_again() between cycles. This prevents behaviors like close() from interleaving between result delivery and query dequeuing. If a callback triggers shutdown during the loop, on_shutdown clears the read buffer, causing the next parse to return None and exit the loop.

Supported message types: AuthenticationOk, AuthenticationMD5Password, AuthenticationSASL, AuthenticationSASLContinue, AuthenticationSASLFinal, BackendKeyData, CommandComplete, CopyInResponse, CopyOutResponse, CopyData, CopyDone, DataRow, EmptyQueryResponse, ErrorResponse, NoticeResponse, NotificationResponse, ParameterStatus, ReadyForQuery, RowDescription, ParseComplete, BindComplete, NoData, CloseComplete, ParameterDescription, PortalSuspended. BackendKeyData is parsed and stored in _SessionLoggedIn (backend_pid, backend_secret_key) for future query cancellation. NotificationResponse is parsed into _NotificationResponseMessage and routed to _SessionLoggedIn.on_notification(), which delivers pg_notification to SessionStatusNotify. NoticeResponse is parsed into NoticeResponseMessage (using shared _ResponseFieldBuilder / _parse_response_fields with ErrorResponse) and routed via on_notice() to SessionStatusNotify.pg_notice(). Notices are delivered in all connected states (including during authentication) since PostgreSQL can send them at any time. ParameterStatus is parsed into _ParameterStatusMessage and routed via on_parameter_status() to SessionStatusNotify.pg_parameter_status(), which delivers a ParameterStatus val. Like notices, parameter status messages are delivered in all connected states. Extended query acknowledgment messages (ParseComplete, BindComplete, NoData, etc.) are parsed but silently consumed — they fall through the _ResponseMessageParser match without routing since the state machine tracks query lifecycle through data-carrying messages only.

Public API Types

  • Query — union type (SimpleQuery | PreparedQuery | NamedPreparedQuery)
  • SimpleQuery — val class wrapping a query string (simple query protocol)
  • PreparedQuery — val class wrapping a query string + Array[(String | None)] val params (extended query protocol, single statement only)
  • NamedPreparedQuery — val class wrapping a statement name + Array[(String | None)] val params (executes a previously prepared named statement)
  • Result trait — ResultSet (rows), SimpleResult (no rows), RowModifying (INSERT/UPDATE/DELETE with count)
  • Rows / Row / Field — result data. Field.value is FieldDataTypes union
  • FieldDataTypes = (Array[U8] val | Bool | F32 | F64 | I16 | I32 | I64 | None | String)
  • TransactionStatus — union type (TransactionIdle | TransactionInBlock | TransactionFailed). Reported via pg_transaction_status callback on every ReadyForQuery.
  • Notification — val class wrapping channel name, payload string, and notifying backend's process ID. Delivered via pg_notification callback.
  • NoticeResponseMessage — non-fatal PostgreSQL notice with all standard fields (same structure as ErrorResponseMessage). Delivered via pg_notice callback.
  • ParameterStatus — val class wrapping a runtime parameter name and value reported by the server. Delivered via pg_parameter_status callback during startup and after SET commands.
  • SessionStatusNotify interface (tag) — lifecycle callbacks (connected, connection_failed, authenticated, authentication_failed, transaction_status, notification, notice, parameter_status, shutdown)
  • ResultReceiver interface (tag) — pg_query_result(Session, Result), pg_query_failed(Session, Query, (ErrorResponseMessage | ClientQueryError))
  • PrepareReceiver interface (tag) — pg_statement_prepared(Session, name), pg_prepare_failed(Session, name, (ErrorResponseMessage | ClientQueryError))
  • CopyInReceiver interface (tag) — pg_copy_ready(Session), pg_copy_complete(Session, count), pg_copy_failed(Session, (ErrorResponseMessage | ClientQueryError)). Pull-based: session calls pg_copy_ready after copy_in and after each send_copy_data, letting the client control data flow
  • CopyOutReceiver interface (tag) — pg_copy_data(Session, Array[U8] val), pg_copy_complete(Session, count), pg_copy_failed(Session, (ErrorResponseMessage | ClientQueryError)). Push-based: server drives the flow, delivering data chunks via pg_copy_data and signaling completion via pg_copy_complete
  • ClientQueryError trait — SessionNeverOpened, SessionClosed, SessionNotAuthenticated, DataError
  • DatabaseConnectInfo — val class grouping database authentication parameters (user, password, database). Passed to Session.create() alongside ServerConnectInfo.
  • ServerConnectInfo — val class grouping connection parameters (auth, host, service, ssl_mode). Passed to Session.create() as the first parameter. Also used by _CancelSender.
  • SSLMode — union type (SSLDisabled | SSLRequired). SSLDisabled is the default (plaintext). SSLRequired wraps an SSLContext val for TLS negotiation.
  • ErrorResponseMessage — full PostgreSQL error with all standard fields
  • AuthenticationFailureReason = (InvalidAuthenticationSpecification | InvalidPassword | ServerVerificationFailed | UnsupportedAuthenticationMethod)

Type Conversion (PostgreSQL OID → Pony)

In _RowsBuilder._field_to_type():

  • 16 (bool) → Bool (checks for "t")
  • 17 (bytea) → Array[U8] val (hex-format decode: strips \x prefix, parses hex pairs)
  • 20 (int8) → I64
  • 21 (int2) → I16
  • 23 (int4) → I32
  • 700 (float4) → F32
  • 701 (float8) → F64
  • Everything else → String
  • NULL → None

Query Cancellation

_CancelSender actor — fire-and-forget actor that sends a CancelRequest on a separate TCP connection. PostgreSQL requires cancel requests on a different connection from the one executing the query. No response is expected on the cancel connection — the result (if any) arrives as an ErrorResponse on the original session connection. When the session uses SSLRequired, the cancel connection performs SSL negotiation before sending the CancelRequest — mirroring the main session's connection setup. If the server refuses SSL or the TLS handshake fails, the cancel is silently abandoned. Created by _SessionLoggedIn.cancel() using the session's ServerConnectInfo, backend_pid, and backend_secret_key. Design: discussion #88.

Mort Primitives

_IllegalState and _Unreachable in _mort.pony. Print file/line to stderr via FFI and exit. Issue URL: https://github.com/ponylang/postgres/issues.

Test Organization

Tests live in the main postgres/ package (private test classes), organized across multiple files by concern. The Main test actor in _test.pony is the single test registry that lists all tests.

_test.pony — Main test actor, _ConnectionTestConfiguration (shared env var helper), basic connection integration tests (Connect, ConnectFailure), basic auth integration tests (Authenticate, AuthenticateFailure), and shared notifies (_ConnectTestNotify, _AuthenticateTestNotify) reused by other test files.

_test_session.pony — Mock-server-based session behavior tests:

  • _TestHandlingJunkMessages — uses a local TCP listener that sends junk; verifies session shuts down
  • _TestUnansweredQueriesFailOnShutdown — uses a local TCP listener that auto-auths but never responds to queries; verifies queued queries get SessionClosed failures
  • _TestPrepareShutdownDrainsPrepareQueue — uses a local TCP listener that auto-auths but never becomes ready; verifies pending prepare operations get SessionClosed failures on shutdown
  • _TestTerminateSentOnClose — mock server fully authenticates and becomes ready; verifies that closing the session sends a Terminate message ('X') to the server
  • _TestZeroRowSelectReturnsResultSet — mock server sends RowDescription + CommandComplete("SELECT 0") with no DataRows; verifies ResultSet (not RowModifying) with zero rows
  • _TestByteaResultDecoding — mock server sends RowDescription (bytea column, OID 17) + DataRow with hex-encoded value + CommandComplete; verifies field value is Array[U8] val with correct decoded bytes
  • _TestEmptyByteaResultDecoding — mock server sends DataRow with empty bytea (\x); verifies empty Array[U8] val

_test_ssl.pony — SSL negotiation unit tests (mock servers) and SSL integration tests:

  • _TestSSLNegotiationRefused — mock server responds 'N' to SSLRequest; verifies pg_session_connection_failed fires
  • _TestSSLNegotiationJunkResponse — mock server responds with junk byte to SSLRequest; verifies session shuts down
  • _TestSSLNegotiationSuccess — mock server responds 'S', both sides upgrade to TLS, sends AuthOk+ReadyForQuery; verifies full SSL→auth flow
  • SSL integration tests: SSL/Connect, SSL/Authenticate, SSL/Query, SSL/Refused

_test_cancel.pony — Cancel query tests (mock servers + integration):

  • _TestCancelQueryInFlight — mock server accepts two connections; first authenticates with BackendKeyData(pid, key) and receives query; second receives CancelRequest and verifies 16-byte format with correct magic number, pid, and key
  • _TestSSLCancelQueryInFlight — same as _TestCancelQueryInFlight but with SSL on both connections; verifies that _CancelSender performs SSL negotiation before sending CancelRequest
  • Cancel integration tests: Cancel/Query, SSL/Cancel

_test_auth.pony — Authentication protocol unit tests (mock servers):

  • _TestSCRAMAuthenticationSuccess — mock server completes full SCRAM-SHA-256 handshake; verifies pg_session_authenticated fires
  • _TestSCRAMUnsupportedMechanism — mock server offers only unsupported SASL mechanisms; verifies pg_session_authentication_failed with UnsupportedAuthenticationMethod
  • _TestSCRAMServerVerificationFailed — mock server sends wrong signature in SASLFinal; verifies pg_session_authentication_failed with ServerVerificationFailed
  • _TestSCRAMErrorDuringAuth — mock server sends ErrorResponse 28P01 during SCRAM exchange; verifies pg_session_authentication_failed with InvalidPassword
  • _TestUnsupportedAuthentication — mock server sends unsupported auth type (cleartext password); verifies pg_session_authentication_failed with UnsupportedAuthenticationMethod

_test_md5.pony — MD5 integration tests: MD5/Authenticate, MD5/AuthenticateFailure, MD5/QueryResults

_test_query.pony — Query integration tests:

  • Simple query: Query/Results, Query/ByteaResults, Query/AfterAuthenticationFailure, Query/AfterConnectionFailure, Query/AfterSessionHasBeenClosed, Query/OfNonExistentTable, Query/CreateAndDropTable, Query/InsertAndDelete, Query/EmptyQuery, ZeroRowSelect, MultiStatementMixedResults
  • Prepared query: PreparedQuery/Results, PreparedQuery/NullParam, PreparedQuery/OfNonExistentTable, PreparedQuery/InsertAndDelete, PreparedQuery/MixedWithSimple
  • Named prepared statements: PreparedStatement/Prepare, PreparedStatement/PrepareAndExecute, PreparedStatement/PrepareAndExecuteMultiple, PreparedStatement/PrepareAndClose, PreparedStatement/PrepareFails, PreparedStatement/PrepareAfterClose, PreparedStatement/CloseNonexistent, PreparedStatement/PrepareDuplicateName, PreparedStatement/MixedWithSimpleAndPrepared
  • COPY IN: CopyIn/Insert, CopyIn/AbortRollback

_test_notification.pony — LISTEN/NOTIFY tests (mock servers + integration):

  • _TestNotificationDelivery — mock server authenticates, responds to query with CommandComplete + NotificationResponse + ReadyForQuery; verifies pg_notification fires with correct channel, payload, and pid
  • _TestNotificationDuringDataRows — mock server sends RowDescription + DataRow + NotificationResponse + DataRow + CommandComplete + ReadyForQuery; verifies both query result (2 rows) and notification are delivered
  • _TestListenNotify — integration test: full LISTEN/NOTIFY round-trip through a real PostgreSQL server

_test_transaction_status.pony — Transaction status tests (mock servers + integration):

  • _TestTransactionStatusOnAuthentication — mock server authenticates; verifies pg_transaction_status fires with TransactionIdle on initial ReadyForQuery
  • _TestTransactionStatusDuringTransaction — mock server tracks BEGIN/COMMIT; verifies status sequence idle→in-block→idle
  • _TestTransactionStatusOnFailedTransaction — mock server tracks BEGIN/error/ROLLBACK; verifies status sequence idle→in-block→failed→idle
  • Explicit transaction integration tests: Transaction/Commit, Transaction/RollbackAfterFailure

_test_copy_in.pony — COPY IN tests (mock servers + integration):

  • _TestCopyInSuccess — mock server authenticates, responds with CopyInResponse, accepts CopyData messages and CopyDone, responds with CommandComplete("COPY 2") + ReadyForQuery; verifies pull-based flow and pg_copy_complete(2)
  • _TestCopyInAbort — mock server authenticates, responds with CopyInResponse; client sends CopyFail in pg_copy_ready; server responds with ErrorResponse + ReadyForQuery; verifies pg_copy_failed with ErrorResponseMessage
  • _TestCopyInServerError — mock server responds to CopyData with ErrorResponse + ReadyForQuery; verifies pg_copy_failed and session remains usable (follow-up query succeeds)
  • _TestCopyInShutdownDrainsCopyQueue — uses _DoesntAnswerTestServer that authenticates but never becomes ready; verifies pending copy_in calls receive pg_copy_failed(SessionClosed) on shutdown
  • _TestCopyInAfterSessionClosed — integration test: connect, authenticate, close, call copy_in; verifies pg_copy_failed(SessionClosed)

_test_copy_out.pony — COPY OUT tests (mock servers + integration):

  • _TestCopyOutSuccess — mock server authenticates, responds with CopyOutResponse, sends two CopyData chunks, then CopyDone + CommandComplete("COPY 2") + ReadyForQuery; verifies 2 pg_copy_data calls and pg_copy_complete(2)
  • _TestCopyOutEmpty — mock server responds with CopyOutResponse, CopyDone, CommandComplete("COPY 0"), ReadyForQuery; verifies pg_copy_complete(0) and no pg_copy_data calls
  • _TestCopyOutServerError — mock server sends ErrorResponse during COPY OUT; verifies pg_copy_failed and session remains usable (follow-up query succeeds)
  • _TestCopyOutShutdownDrainsCopyQueue — uses _DoesntAnswerTestServer that authenticates but never becomes ready; verifies pending copy_out calls receive pg_copy_failed(SessionClosed) on shutdown
  • _TestCopyOutAfterSessionClosed — integration test: connect, authenticate, close, call copy_out; verifies pg_copy_failed(SessionClosed)
  • _TestCopyOutExport — integration test: create table, insert 3 rows, COPY TO STDOUT, verify received data contains 3 lines, drop table

_test_notice.pony — NoticeResponse tests (mock servers + integration):

  • _TestNoticeDelivery — mock server authenticates, responds to query with CommandComplete + NoticeResponse + ReadyForQuery; verifies pg_notice fires with correct severity, code, and message fields
  • _TestNoticeDuringDataRows — mock server sends RowDescription + DataRow + NoticeResponse + DataRow + CommandComplete + ReadyForQuery; verifies both query result (2 rows) and notice are delivered
  • _TestNoticeOnDropIfExists — integration test: executes DROP TABLE IF EXISTS on a nonexistent table; verifies pg_notice fires with severity "NOTICE" and code "00000"

_test_parameter_status.pony — ParameterStatus tests (mock servers + integration):

  • _TestParameterStatusDelivery — mock server authenticates, responds to query with CommandComplete + ParameterStatus + ReadyForQuery; verifies pg_parameter_status fires with correct name and value
  • _TestParameterStatusDuringDataRows — mock server sends RowDescription + DataRow + ParameterStatus + DataRow + CommandComplete + ReadyForQuery; verifies both query result (2 rows) and parameter status are delivered
  • _TestParameterStatusOnStartup — integration test: connects to real PostgreSQL; verifies pg_parameter_status fires with "server_version" during startup
  • _TestParameterStatusOnSet — integration test: executes SET application_name; verifies pg_parameter_status fires with the new value

_test_response_parser.pony — Parser unit tests (_TestResponseParser*) + test message builder classes (_Incoming*TestMessage) that construct raw protocol bytes for mock servers across all test files. _TestResponseParserNotificationResponseMessage verifies parsing into _NotificationResponseMessage with correct fields (including empty-payload edge case). _TestResponseParserNoticeResponseMessage verifies parsing into NoticeResponseMessage with correct severity, code, and message fields. _TestResponseParserParameterStatusMessage verifies parsing into _ParameterStatusMessage with correct name and value fields. _TestResponseParserMultipleMessagesAsyncThenAuth verifies buffer advancement across async message types (one parsed ParameterStatus, one parsed NoticeResponse, one parsed NotificationResponse) followed by AuthenticationOk. _TestResponseParserCopyOutResponseMessage verifies parsing 'H' into _CopyOutResponseMessage with correct format and column formats. _TestResponseParserCopyDataMessage verifies parsing 'd' into _CopyDataMessage with correct data payload. _TestResponseParserCopyDoneMessage verifies parsing 'c' into _CopyDoneMessage.

_test_frontend_message.pony — Frontend message unit tests (_TestFrontendMessage*).

_test_equality.pony — Example-based and PonyCheck property tests for Field/Row/Rows equality.

_test_scram.pony — SCRAM-SHA-256 computation unit tests (_TestScramSha256MessageBuilders, _TestScramSha256ComputeProof).

_test_mock_message_reader.pony — _MockMessageReader class that buffers TCP data and extracts complete PostgreSQL frontend messages. Two methods: read_startup_message() for startup-format messages (Int32 length + payload — StartupMessage, SSLRequest, CancelRequest) and read_message() for standard-format messages (Byte1 type + Int32 length + payload). Both return (Array[U8] val | None). Used by all mock servers that inspect incoming data to ensure state transitions happen on message boundaries, not TCP segment boundaries.

Test helpers: _ConnectionTestConfiguration reads env vars with defaults. Several test message builder classes (_Incoming*TestMessage) construct raw protocol bytes for unit tests. Mock server tests use ports in the 7669–7701 range and 9667–9668. Port 7680 is reserved by Windows (Update Delivery Optimization) and will fail to bind on WSL2 — do not use it.

Known Issues and TODOs in Code

  • _test_response_parser.pony:6 — TODO: chain-of-messages tests to verify correct buffer advancement across message sequences

Roadmap

SSL/TLS negotiation is implemented. Pass SSLRequired(sslctx) to Session.create() to enable. Design: discussion #76. SCRAM-SHA-256 authentication is implemented. It is the default PostgreSQL auth method since version 10. Design: discussion #83. Transaction status tracking is implemented. The pg_transaction_status callback on SessionStatusNotify fires on every ReadyForQuery with TransactionIdle, TransactionInBlock, or TransactionFailed. Design: discussion #102. LISTEN/NOTIFY is implemented. Notifications are parsed from NotificationResponse messages and delivered via pg_notification callback on SessionStatusNotify. Design: discussion #103. COPY FROM STDIN is implemented. Session.copy_in() initiates bulk data loading with a pull-based CopyInReceiver interface. The session calls pg_copy_ready after each send_copy_data, giving the client O(1) bounded memory flow control. Design: discussion #104. NoticeResponse handling is implemented. Non-fatal informational messages from the server are parsed into NoticeResponseMessage and delivered via pg_notice callback on SessionStatusNotify. Notices are delivered in all connected states including during authentication. ParameterStatus tracking is implemented. Runtime parameter name/value pairs from the server are parsed into _ParameterStatusMessage and delivered via pg_parameter_status callback on SessionStatusNotify as ParameterStatus values. Delivered in all connected states (during startup and after SET commands). COPY TO STDOUT is implemented. Session.copy_out() initiates bulk data export with a push-based CopyOutReceiver interface. The server drives the flow, delivering data chunks via pg_copy_data and signaling completion via pg_copy_complete. Full feature roadmap: discussion #72. CI uses stock postgres:14.5 for the non-SSL container (no md5user, SCRAM-SHA-256 default) and ghcr.io/ponylang/postgres-ci-pg-ssl:latest for the SSL container (SSL + md5user init script for backward-compat tests); built via build-ci-image.yml workflow dispatch or locally via .ci-dockerfiles/pg-ssl/build-and-push.bash. MD5 integration tests connect to the SSL container (without using SSL) because only that container has the md5user.

Supported PostgreSQL Features

SSL/TLS: Optional SSL negotiation via SSLRequired. The driver sends an SSLRequest before authentication. If the server accepts ('S'), the connection is upgraded to TLS via lori's start_tls(). If refused ('N'), connection fails. CVE-2021-23222 mitigated via expect(1) before SSLRequest.

Authentication: MD5 password and SCRAM-SHA-256. No SCRAM-SHA-256-PLUS (channel binding), Kerberos, GSS, or certificate auth. Design: discussion #83.

Protocol: Simple query protocol and extended query protocol (parameterized queries via unnamed and named prepared statements). Parameters are text-format only; type OIDs are inferred by the server. LISTEN/NOTIFY notifications are delivered via pg_notification callback. NoticeResponse messages (non-fatal informational feedback) are delivered via pg_notice callback. ParameterStatus messages (runtime parameter values) are delivered via pg_parameter_status callback. COPY FROM STDIN (bulk data loading) via Session.copy_in() with pull-based CopyInReceiver flow. COPY TO STDOUT (bulk data export) via Session.copy_out() with push-based CopyOutReceiver flow. No function calls. Design: discussion #104.

PostgreSQL Wire Protocol Reference

Message Structure

Every message (except StartupMessage) follows: Byte1(type) | Int32(length including self but not type byte) | payload. All integers are big-endian. Strings are null-terminated.

Extended Query Protocol (Prepared Statements)

Separates query processing into discrete steps with two objects:

  • Prepared Statement: parsed + analyzed query, not yet executable (no parameter values)
  • Portal: prepared statement + bound parameter values, ready to execute (like an open cursor)

Parse — Create Prepared Statement

Parse (P): Byte1('P') Int32(len) String(stmt_name) String(query) Int16(num_param_types) Int32[](param_type_oids)

  • Query must be a single statement. Parameters: $1, $2, ..., $n.
  • stmt_name = "" for unnamed. Named statements persist until Close or session end.
  • Unnamed statement destroyed by next Parse or simple Query.
  • Named statements MUST be Closed before redefinition.
  • OID 0 = let server infer type.

ParseComplete (1): Byte1('1') Int32(4)

Bind — Create Portal

Bind (B): Byte1('B') Int32(len) String(portal_name) String(stmt_name) Int16(num_param_fmt) Int16[](param_fmts) Int16(num_params) [Int32(val_len) Byte[](val)]* Int16(num_result_fmt) Int16[](result_fmts)

  • Format codes: 0=text, 1=binary. Shorthand: 0 entries=all text, 1 entry=applies to all, N entries=one per column.
  • Named portals persist until Close or transaction end. Unnamed destroyed by next Bind/Query/txn end.
  • ALL portals destroyed at transaction end.
  • Query planning typically happens at Bind time.

BindComplete (2): Byte1('2') Int32(4)

Describe — Request Metadata

Describe (D): Byte1('D') Int32(len) Byte1('S'|'P') String(name)

  • 'S' (statement): responds with ParameterDescription then RowDescription/NoData.
  • 'P' (portal): responds with RowDescription/NoData only.

ParameterDescription (t): Byte1('t') Int32(len) Int16(num_params) Int32[](param_oids)

NoData (n): Byte1('n') Int32(4) — for statements that return no rows.

Execute — Run Portal

Execute (E): Byte1('E') Int32(len) String(portal_name) Int32(max_rows)

  • max_rows 0 = no limit. Only applies to row-returning queries.
  • Responses: DataRow* + CommandComplete (done), DataRow* + PortalSuspended (more rows available), EmptyQueryResponse, or ErrorResponse.

PortalSuspended (s): Byte1('s') Int32(4)

Close — Destroy Statement/Portal

Close (C): Byte1('C') Int32(len) Byte1('S'|'P') String(name)

  • Closing a statement implicitly closes all its portals.
  • Not an error to close nonexistent objects.

CloseComplete (3): Byte1('3') Int32(4)

Sync — Synchronization Point

Sync (S): Byte1('S') Int32(4)

  • Commits (success) or rolls back (error) implicit transactions.
  • On error: backend discards messages until Sync, then sends ReadyForQuery.
  • Exactly one ReadyForQuery per Sync.

Flush — Force Output

Flush (H): Byte1('H') Int32(4)

Forces pending output without ending query cycle or producing ReadyForQuery.

Extended Query Typical Flow

Client: Parse → Bind → Describe(portal) → Execute → Sync
Server: ParseComplete → BindComplete → RowDescription → DataRow* → CommandComplete → ReadyForQuery

Equivalence: a simple Query is roughly Parse(unnamed) + Bind(unnamed, no params) + Describe(portal) + Execute(unnamed, 0) + Close(portal) + Sync.

Named vs Unnamed Lifetime

Object Unnamed Named
Statement Until next Parse/Query Until Close or session end
Portal Until next Bind/Query or txn end Until Close or txn end

Pipelining

Multiple extended query sequences can be sent without waiting. Each Sync is a segment boundary. On error in a segment, backend discards remaining messages until Sync, sends ReadyForQuery, then processes next segment independently.

Asynchronous Messages

Can arrive between any other messages (must always handle):

  • NoticeResponse (N): informational, same format as ErrorResponse
  • NotificationResponse (A): LISTEN/NOTIFY — Int32(pid) String(channel) String(payload)
  • ParameterStatus (S): runtime parameter changes

Parameter Encoding

  • Text (0): default. Human-readable via type's I/O functions. Integers as decimal ASCII, booleans as "t"/"f".
  • Binary (1): type-specific, big-endian for multi-byte, IEEE 754 for floats. May vary across PG versions for complex types.

Common Type OIDs

Type OID Type OID
bool 16 varchar 1043
bytea 17 date 1082
int8 20 time 1083
int2 21 timestamp 1114
int4 23 timestamptz 1184
text 25 interval 1186
json 114 numeric 1700
float4 700 uuid 2950
float8 701 jsonb 3802

Complete Message Type Bytes

Frontend: Q=Query, P=Parse, B=Bind, D=Describe, E=Execute, C=Close, S=Sync, H=Flush, X=Terminate, p=PasswordMessage/SASLInitialResponse/SASLResponse, d=CopyData, c=CopyDone, f=CopyFail

Backend: R=Auth, K=BackendKeyData, S=ParameterStatus, Z=ReadyForQuery, T=RowDescription, D=DataRow, C=CommandComplete, G=CopyInResponse, H=CopyOutResponse, d=CopyData, c=CopyDone, I=EmptyQueryResponse, 1=ParseComplete, 2=BindComplete, 3=CloseComplete, t=ParameterDescription, n=NoData, s=PortalSuspended, E=ErrorResponse, N=NoticeResponse, A=NotificationResponse

File Layout

postgres/                         # Main package (54 files)
  postgres.pony                   # Package-level docstring (user-facing API overview)
  notification.pony                # Notification val class (channel, payload, pid)
  parameter_status.pony           # ParameterStatus val class (name, value)
  session.pony                    # Session actor + state machine traits + query sub-state machine
  database_connect_info.pony       # DatabaseConnectInfo val class (user, password, database)
  server_connect_info.pony        # ServerConnectInfo val class (auth, host, service, ssl_mode)
  ssl_mode.pony                   # SSLDisabled, SSLRequired, SSLMode types
  simple_query.pony               # SimpleQuery class
  prepared_query.pony             # PreparedQuery class
  named_prepared_query.pony       # NamedPreparedQuery class
  query.pony                      # Query union type
  result.pony                     # Result, ResultSet, SimpleResult, RowModifying
  result_receiver.pony            # ResultReceiver interface
  prepare_receiver.pony           # PrepareReceiver interface
  copy_in_receiver.pony           # CopyInReceiver interface (pull-based COPY IN callbacks)
  copy_out_receiver.pony          # CopyOutReceiver interface (push-based COPY OUT callbacks)
  session_status_notify.pony      # SessionStatusNotify interface
  transaction_status.pony         # TransactionStatus union type (TransactionIdle, TransactionInBlock, TransactionFailed)
  query_error.pony                # ClientQueryError types
  error_response_message.pony     # ErrorResponseMessage + _ResponseFieldBuilder (shared with NoticeResponseMessage)
  notice_response_message.pony    # NoticeResponseMessage (non-fatal server notices)
  field.pony                      # Field class
  field_data_types.pony           # FieldDataTypes union
  row.pony                        # Row class
  rows.pony                       # Rows, RowIterator, _RowsBuilder
  _cancel_sender.pony              # Fire-and-forget cancel request actor
  _frontend_message.pony          # Client-to-server messages
  _backend_messages.pony          # Server-to-client message types
  _message_type.pony              # Protocol message type codes
  _response_parser.pony           # Wire protocol parser
  _response_message_parser.pony   # Routes parsed messages to session state
  _authentication_request_type.pony
  _authentication_failure_reason.pony
  _md5_password.pony              # MD5 password construction
  _scram_sha256.pony              # SCRAM-SHA-256 computation primitive
  _mort.pony                      # Panic primitives
  _test.pony                      # Main test actor + shared test config + basic connect/auth integration tests
  _test_session.pony              # Mock-server session behavior tests (junk, shutdown, terminate, zero-row)
  _test_ssl.pony                  # SSL negotiation unit tests + SSL integration tests
  _test_cancel.pony               # Cancel query tests (mock servers + integration)
  _test_auth.pony                 # Authentication protocol unit tests (SCRAM, unsupported auth)
  _test_md5.pony                  # MD5 integration tests
  _test_query.pony                # Query integration tests
  _test_mock_message_reader.pony  # _MockMessageReader: TCP buffering for mock servers
  _test_response_parser.pony      # Parser unit tests + test message builders
  _test_frontend_message.pony     # Frontend message unit tests
  _test_equality.pony             # Equality tests for Field/Row/Rows (example + PonyCheck property)
  _test_scram.pony                # SCRAM-SHA-256 computation tests
  _test_notification.pony         # LISTEN/NOTIFY tests (mock servers)
  _test_notice.pony               # NoticeResponse tests (mock servers + integration)
  _test_parameter_status.pony     # ParameterStatus tests (mock servers + integration)
  _test_transaction_status.pony   # Transaction status tests (mock servers + integration)
  _test_copy_in.pony              # COPY IN tests (mock servers + integration)
  _test_copy_out.pony             # COPY OUT tests (mock servers + integration)
assets/test-cert.pem              # Self-signed test certificate for SSL unit tests
assets/test-key.pem               # Private key for SSL unit tests
examples/README.md                # Examples overview
examples/bytea/bytea-example.pony # Binary data with bytea columns
examples/query/query-example.pony # Simple query with result inspection
examples/ssl-query/ssl-query-example.pony # SSL-encrypted query with SSLRequired
examples/prepared-query/prepared-query-example.pony # PreparedQuery with params and NULL
examples/named-prepared-query/named-prepared-query-example.pony # Named prepared statements with reuse
examples/crud/crud-example.pony   # Multi-query CRUD workflow
examples/cancel/cancel-example.pony # Query cancellation with pg_sleep
examples/copy-in/copy-in-example.pony # Bulk data loading with COPY FROM STDIN
examples/copy-out/copy-out-example.pony # Bulk data export with COPY TO STDOUT
examples/notice/notice-example.pony # NoticeResponse handling with DROP IF EXISTS
examples/parameter-status/parameter-status-example.pony # ParameterStatus tracking with SET
examples/transaction-status/transaction-status-example.pony # Transaction status tracking with BEGIN/COMMIT
.ci-dockerfiles/pg-ssl/           # Dockerfile + init scripts for SSL-enabled PostgreSQL CI container (SCRAM-SHA-256 default + MD5 user)