Expose HTTP endpoint SQL queries, queries count and execution time via Prometheus

[tatiana]

Context

Since the 0.7.0 release (#1906), Marquez supports pushing metrics to Prometheus.

This task proposes extending the current capability to give visibility to Marquez's SQL queries. Some of the questions we'd like to be answered:

• What queries is Marquez running?
• How long does each query take?
• How many times does a specific query run?

By identifying potential bottlenecks in Marquez queries and the database, this extension could facilitate the provisioning of adequate resources. This, in turn, could lead to improved performance and efficiency of the database and Marquez itself.

Implementation

If possible, we could give visibility of frequency (count) and duration (gauge) for all queries Marquez runs. There is a possibility this could be done close to jdbi: https://metrics.dropwizard.io/4.2.0/manual/jdbi.html

If this is not possible, we could add the instrumentation to specific write and read endpoints, covering at least the SQL queries triggered by the following endpoints:

• POST api/v1/lineage (*)
• GET api/v1/namespaces/{namespace} (*)
• GET api/v1/namespaces
• GET api/v1/namespaces/{namespace}/jobs/{job}
• GET api/v1/namespaces/{namespace}/datasets
• GET api/v1/column-lineage

The most critical are (*)

[boring-cyborg]

Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template!

[tatiana]

As of Marquez 0.47.0, the /metrics endpoint already exposes the following information:

• query identifier in the code (it includes the file path / class identifier and the method e.g. marquez/db/JobDao.java:findAll is named marquez_db_JobDao_findAll)
• query duration
• query count

Example of the information made available in this endpoint:

Confirm if we need further details.

[tatiana]

At the moment, we can see the data of interest using the Java method, but we desire this feature to allow us to see it from an HTTP endpoint perspective as well (e.g., POST api/v1/lineage).

[tatiana]

@mobuchowski 's suggest: add the HTTP verb and the endpoint path
E.g. marquez_api_post_v1Lineage_db_JobDao

We need to investigate if and how this could be accomplished, and if there are better ways

[tatiana]

@JDarDagran is implementing this in #2850

[tatiana]

Closed by: #2850