Postgres filter: implement (equivalent) Postgres HBA in Envoy

[ahachete]

Postgres filter: implement (equivalent) Postgres HBA in Envoy: enabling offloading Postgres' HBA (Host-Based Authentication) to Envoy, leveraging Envoy's Postgres filter.

Postgres uses a mechanism to accept or reject connections called HBA ("Host-Based Authentication"), that is controlled by configuration file pg_hba.conf. This file can be reloaded anytime to update the configuration, and supports accepting or rejecting connections based on connection metadata such as the source IP/CIDR, username, database, authentication method and whether the connection is SSL/is not.

What if Envoy's Postgres filter would implement this functionality? It would allow offloading these checks off of Postgres, potentially saving Postgres from "DoS attacks" or basically too many illegal connections. It is also necessary in essence when fronting Postgres with Envoy, as a) SSL status of the original downstream connection would be unknown to Postgres, making it impossible to filter based on SSL enforcement or not; b) we also don't preserve the source IP (there's some separate research on this area).

The Postgres filter could capture this connection metadata (user, database, auth method, source host/port, destination host/port, downstream SSL or not) easily, on connection establishment, and use this metadata to allow/reject the connection. Please note that unlike current's dynamic metadata, we don't need any SQL parsing here to generate the connection metadata, it is easily extracted from the connection message.

Ideally, we could pass this metadata to the RBAC filter, and accept/reject there the connection. But RBAC filter knows nothing about PG wire protocol, and thus may only drop the connection --which will have the side effect of not returning an appropriate error downstream, as a normal Postgres would do under this circumstance. And alternative would be to implement a Postgres filter "submodule", called HBA, to emulate this Postgres' HBA functionality within the Envoy filter, without relying on the RBAC filter.

Supporting HBA on Envoy's side, apart from solving the mentioned problems (lack of visibility about downstream SSL/not SSL and source IP information) could extend HBA to more advanced filtering mechanism, potentially combining with other technologies, enabling for example accepting/rejecting connections based on pod labels, to name one.

There are several possible options, subject to discussion:

• Gather the extra metadata and pass to RBAC filter. Easiest to implement, but denying the connection is not the same experience that current Postgres users get when rejected by HBA reasons (they get a nice appropriate error message, indicating even possible causes).
• Don't use RBAC module for this use case and rebuild the relevant functionality part on the Postgres filter. Shouldn't bee too hard, but definitely requires some effort and in some sense is a bit reinventing a wheel.
• Raise a broader discussion to see if it is possible / makes sense to have the RBAC filter piggy back to Postgres filter. This may or may not require some bigger architectural changes, as filters normally operate in a forward chain.

[alyssawilk]

cc @cpakulski @dio

[fabriziomello]

Moving to here a discussion we started by email this week to keep this issue updated:

First @cpakulski answer after @mattklein123 ask for more information:

My understanding of this problem is that postgres filter would produce some metadata, which is later on evaluated by RBAC filter with allow/deny result.

For DENY result, the best option would be to somehow go back to the postgres filter to reply to the client with postgres specific message, instead of just closing the connection.
That is the meaning of "chain with no backwards communication possible:.".
We looked at RBAC code, but could not find a mechanism to go back to the filter which created the metadata which violated RBAC rules

Then @mattklein123 said:

Yeah I don't think there is going to be a good way to go backwards. I think you could either implement it all in the single postgres filter as you mention, or have the postgres filter inserted twice (on either side of the rbac filter) and via config and/or filter state communicate with each other. In that scenario you could use dynamic metadata from RBAC to do something in the 2nd postgres filter instance.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[ahachete]

Based on the comments above, I'd be more inclined to implement HBA inside of the existing Postgres filter. After all, metadata that could be used for Postgres current HBA is very simple (source IP address and CIDR matching), as other information is gathered from the Postgres filter itself (username, database, authentication mechanism). There may be some possibility to share some functions/code, however (like CIDR matching).

[yangminzhu]

Based on the comments above, I'd be more inclined to implement HBA inside of the existing Postgres filter. After all, metadata that could be used for Postgres current HBA is very simple (source IP address and CIDR matching), as other information is gathered from the Postgres filter itself (username, database, authentication mechanism). There may be some possibility to share some functions/code, however (like CIDR matching).

Is the main difficulty of option 1 (Gather the extra metadata and pass to RBAC filter) related to the custom deny response in RBAC filter? How flexible the error response need to be and where does these information come from?

One variant is to improve the RBAC filter to allow returning a custom error message which could even be based on the dynamic metadata itself. The Postgres filter can write all the information (including potential error message) into the dynamic metadata and delegate the access control to RBAC filter.

[ahachete]

Based on the comments above, I'd be more inclined to implement HBA inside of the existing Postgres filter. After all, metadata that could be used for Postgres current HBA is very simple (source IP address and CIDR matching), as other information is gathered from the Postgres filter itself (username, database, authentication mechanism). There may be some possibility to share some functions/code, however (like CIDR matching).

Is the main difficulty of option 1 (Gather the extra metadata and pass to RBAC filter) related to the custom deny response in RBAC filter? How flexible the error response need to be and where does these information come from?

One variant is to improve the RBAC filter to allow returning a custom error message which could even be based on the dynamic metadata itself. The Postgres filter can write all the information (including potential error message) into the dynamic metadata and delegate the access control to RBAC filter.

The problem is that RBAC doesn't know (nor it should) how to send Postgres protocol messages, this is not standard HTTP ;)

[sfozz]

As a half-way house could the Postgres proxy reject connections where the client parameter sslmode=disabled is set, as currently this just results in the client being able to authenticate with the upstream over an un-encrypted connection?

Another thought would be to push this into the starttls transport socket, if possible, such that the connection is accepted but not progressed unless starttls is initiated?