Expose postgres filter metrics with labels for prometheus format

[teoincontatto]

Title: Expose postgres filter metrics with labels for prometheus format

Description:

Currently postgres filter metrics are exposed without any label when queried with prometheus format:

$ curl -s localhost:8001/stats/prometheus | grep '^envoy_postgres'
envoy_postgres_ingress_postgres_errors{} 0
envoy_postgres_ingress_postgres_errors_error{} 0
envoy_postgres_ingress_postgres_errors_fatal{} 0
envoy_postgres_ingress_postgres_errors_panic{} 0
envoy_postgres_ingress_postgres_errors_unknown{} 0
envoy_postgres_ingress_postgres_messages{} 690221
envoy_postgres_ingress_postgres_messages_backend{} 517666
envoy_postgres_ingress_postgres_messages_frontend{} 172555
envoy_postgres_ingress_postgres_messages_unknown{} 0
envoy_postgres_ingress_postgres_notices{} 0
envoy_postgres_ingress_postgres_notices_debug{} 0
envoy_postgres_ingress_postgres_notices_info{} 0
envoy_postgres_ingress_postgres_notices_log{} 0
envoy_postgres_ingress_postgres_notices_notice{} 0
envoy_postgres_ingress_postgres_notices_unknown{} 0
envoy_postgres_ingress_postgres_notices_warning{} 0
envoy_postgres_ingress_postgres_sessions{} 15687
envoy_postgres_ingress_postgres_sessions_encrypted{} 0
envoy_postgres_ingress_postgres_sessions_unencrypted{} 15687
envoy_postgres_ingress_postgres_statements{} 109806
envoy_postgres_ingress_postgres_statements_delete{} 0
envoy_postgres_ingress_postgres_statements_insert{} 15686
envoy_postgres_ingress_postgres_statements_other{} 31375
envoy_postgres_ingress_postgres_statements_select{} 15687
envoy_postgres_ingress_postgres_statements_update{} 47058
envoy_postgres_ingress_postgres_transactions{} 109806
envoy_postgres_ingress_postgres_transactions_commit{} 109806
envoy_postgres_ingress_postgres_transactions_rollback{} 0

Stats are absolutely global for all database and instances of postgres connected through the filter and all listener addresses that have the filter attached.
Labels to be added are to be discussed but here is list of minimal labels that could be added (if possible):

• listener address: To reveal the client entry point of the stats.
• cluster name: To reveal the postgres instance of the stats.
• database: Postgres connections are bounded to just only one database. It will allow to have stats per database instead of globally to an instance.

For example:

$ curl -s localhost:8001/stats/prometheus | grep '^envoy_postgres'
envoy_postgres_ingress_postgres_errors{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_errors_error{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_errors_fatal{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_errors_panic{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_errors_unknown{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_messages{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 690221
envoy_postgres_ingress_postgres_messages_backend{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 517666
envoy_postgres_ingress_postgres_messages_frontend{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 172555
envoy_postgres_ingress_postgres_messages_unknown{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_notices{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_notices_debug{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_notices_info{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_notices_log{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_notices_notice{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_notices_unknown{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_notices_warning{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_sessions{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 15687
envoy_postgres_ingress_postgres_sessions_encrypted{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_sessions_unencrypted{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 15687
envoy_postgres_ingress_postgres_statements{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 109806
envoy_postgres_ingress_postgres_statements_delete{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0
envoy_postgres_ingress_postgres_statements_insert{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 15686
envoy_postgres_ingress_postgres_statements_other{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 31375
envoy_postgres_ingress_postgres_statements_select{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 15687
envoy_postgres_ingress_postgres_statements_update{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 47058
envoy_postgres_ingress_postgres_transactions{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 109806
envoy_postgres_ingress_postgres_transactions_commit{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 109806
envoy_postgres_ingress_postgres_transactions_rollback{envoy_listener_address="0.0.0.0_7432",envoy_cluster_name="postgres_cluster",datname="postgres"} 0

This will involve that stats memory will increase dynamically but IMO it is worth to have such data included. Possibly there could be some configuration to work out in order to disable stats and only allow or deny some databases (maybe allowing regexp or some wildcard system to ease thing a bit in the configuration) so that users can get rid or reduce the impact of the feature in case it is not worth for them.

[cpakulski]

Makes sense.
Connection info probably may be extracted from connection object which filter has access to.
database is passed in the first Postgres message when connection is opened.

As per implementation, it will add a dynamic aspect to stats. I am not sure if internally it is easy/possible to return global stats for /stats and stats with labels for /stats/prometheus.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.