Migrate existing Google Cloud alerts from click-ops to git-ops model

[spiffxp]

Discussed in k8s-infra meeting 2020-02-03

We have some slack alerting setup today, but it's been configured by humans clicking around on the Google Cloud website (aka "click-ops"). It would be ideal if we could drive that configuration automatically via files checked into git (aka "git-ops").

This is likely similar to or overlaps with making a gitops-driven workflow for Google Cloud Monitoring dashboards (#1376)

/wg k8s-infra
/sig release
/area release-eng
FYI @kubernetes/release-engineering since #k8s-infra-alerts contains container image promoter alerts
/priority important-longterm

[spiffxp]

/help

[k8s-ci-robot]

@spiffxp:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rikatz]

I'm with low bandwidth now, but If we have some time (not urgent) I can take a look into this to see how to manage the alerts and dashboards with Gitops :)

[rikatz]

/assign

[rikatz]

So far:

• Pulumi needs some coding, and also some account, discarded for now
• Made a simple program to create alert, and also to fetch it and the channel so @hasheddan can take a look into how viable is this on Crossplane -> https://gist.github.com/rikatz/d9dc76d27b8371d590924508bd2be6c0
• Will make some attempts with terraform as well, tomorrow

My thoughts on this specific part: I really like the idea of using crossplane (k8s objects) to manage our cloud env, but I guess a lot of folks are familiar already with Terraform (although I agree with Justin, migration between versions sometimes is...annoying...)

Will create some simple .tf tomorrow with the same approach, trying to create notification channels and alert policies, and seeing how this reflects on stack driver.

[rikatz]

@ameukam will work on this, using @thockin tests to monitor certificates renew and expiration as an example.

[rikatz]

#1877 <- Created a PR with a really simple Terraform that adds an uptime check and the current alert policy.

We can improve this, like adding latency/uptime alerting (like for cs.k8s.io and others), etc.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[ameukam]

/remove-lifecycle stale

[spiffxp]

A good first step would be understanding how to export whatever existing alerts we have as part of audit/audit-gcp.sh

[spiffxp]

https://github.com/GoogleCloudPlatform/oss-test-infra/tree/master/prow/oss/terraform/modules/alerts good prior art to start from

[spiffxp]

/milestone v1.23
I think it would be really handy to use this at a bare minimum for uptime checks on the apps we run on aaa

[ameukam]

/milestone clear

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale
/lifecycle frozen
/milestone clear