Alternative for "internal monitoring" of project health

[ChristopherFoerster]

Up to 4.20 CCM provided a dedicated monitoring page in the settings that checked for each model in a project:

• is there a backup plan
• is there a backup
• is there a diagram cache
• is there a complexity badge
• is there a description

Do you intend to replace the dropped functionality again via Grafana or similar?

The feature was quite valuable to check correct configuration of models and backups and indirectly as well the linking to git. Would be great if there is a revival planned. Otherwise we would implement it using REST API and some more advanced data like last backup etc. and put the data into PowerBI.

[MoritzWeber0]

Hey @ChristopherFoerster,

the monitoring overview wasn't properly parallelized and took minutes to load on larger instances, eventually running into a gateway timeout. In addition, there was an increased load on the server, sometimes leading to Pod restarts.

As part of our performance initiative, we had to remove it for better stability. There are plans to add a reduced overview page again, probably as Grafana dashboard to keep the monitoring in one place. So far, nothing like that is implemented.

If you have the need for the monitoring page, iterate over all projects and models in the project via the REST API for now. Then, fetch the pipelines, diagram cache, complexity badge and description for each model. This was more or less the logic of the monitoring page. If you have a script, it might be helpful to share it there - so that others can reuse it.

[ChristopherFoerster]

@MoritzWeber0in this case we will create a python script to collect the data with database upload (for PowerBI).

For Badge/Cache i see no issue. For pipelines the API does not show nightly runs as well. Is there a way to enable this in the future or via optional parameter? Otherwise I only see that something is configured but not if automation is successful and still working. For us backup is one of the most important features for operation.
/api/v1/projects/{project_slug}/models/{model_slug}/backups/pipelines/{pipeline_id}/runs

[MoritzWeber0]

Currently, there is no proper way to see the status of nightly runs. This was also not possible in the old monitoring dashboard, which didn't consider nightly runs.

The issue with nightly runs is that we just create a CronJob in Kubernetes and have no further control over it. As a longer term goal we'll switch to our own solution with full control.

However: There is an alerting rule for Grafana, which sends an alert / notification on nightly run failure: https://github.com/DSD-DBS/capella-collab-manager/blob/main/helm/config/grafana/alerting/rules.yaml#L50

[MoritzWeber0]

Quick update: We're working on making the status of nightly job runs available via the API. The change will be introduced in #2389.