Consider adding prometheus metrics for DMCI /insert, /update and maybe /validate endpoints

[magnarem]

The https://github.com/rycus86/prometheus_flask_exporter, supports exporting flask metrics out of the box.
This could be beneficial to monitor HTTP error codes for the DMCI endpoints, if insert and update calls fails etc, and then we can get alerts if DMCI have a lot of failed inserts for example.

[magnarem]

Also what my thought was for this issue, when we have a working prometheus exporter, is to see if we also can create some
custom metrics for the worker in dmci.

The worker returns a list of failed distributors, so we should have some metrics like:

# failed file_dist <counter>
# failed csw_dist <counter>
# failed solr_dist <counter>

From the failed array in https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/api/worker.py#L171.

Maybe the code have to be rewritten a bit, since we dont get this array back into the /update and /insert endpoints,

So the lines at https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/api/app.py#L74
and https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/api/app.py#L69

need to get the failed array back. so the code

msg, code = self._insert_update_method_post("insert", request) need to be changed to
msg, code, failed = self._insert_update_method_post("insert", request)
and if there are elements in the failed array, then it should be possible to add it as a custom metric using the prometheus_flask_exporter API.

[charlienegri]

do I read correctly that the only place in practice where this 'failed' array would be not null is in this if block? https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/api/app.py#L159
because it would have to be returned by _distributor_wrapper at L157, something like
err, failed = self._distributor_wrapper(worker)
and then it's relevant only for that next if block

[magnarem]

Yes. You are right. you will have to return the failed array from the _distributor_wrapper and then if err and failed is not empty then add to failed metrics.

I think if failed is not empty, the elements there are the name of the distributors that failed.

[charlienegri]

ok this probably means creating custom metrics counters defined via prometheus_client and incrementing them in post_insert and post_update based on failed, but I have no idea of that is out of the box compatible with the GunicornPromethusMetrics wrapper or then there's gonna have to have some extra custom collection registry layer...

[magnarem]

I am not sure myself how this is working.. but It should be possible somehow. I bet other projects using flask app also want to expose custom prometheus metrics for their business-logic and not just reponse codes from api endpoints

[charlienegri]

anyway, for now still no default metrics exposure..
from the pod /proc/net/tcp I see nothing about port 9200
I think 'expose' in the dockerfile does nothing here, and we have to expose the port in the tjeneste repo and update here too https://gitlab.met.no/tjenester/s-enda/-/blob/dev/base/dmci/statefulset.yaml?ref_type=heads#L128

[charlienegri]

giving it a try https://gitlab.met.no/tjenester/s-enda/-/merge_requests/2928