Kubernetes CI Policy: all prow.k8s.io jobs must have testgrid alerts configured

[spiffxp]

Part of #18551

Why this is important:

• In order to ensure effective use of community resources, we expect them to be spent on jobs that provide useful signal, and are actively maintained
• Configuring testgrid alerts requires setting an e-mail address, which gives us a point of contact to escalate to if a job is deemed an ineffective use of community resources
• We'll use this to implement a policy where we reserve the right to remove/disable jobs that are deemed an ineffective use of resources (e.g. perma-failing for O(weeks)) if the point of contact is unresponsive

TODO:

• come up with test to enforce this policy (logging only)
• come up with report to identify jobs and likely candidate sig owners
  • eg: if a job is on "sig-foo"'s dashboard, it's likely they want to own it
  • who to contact for jobs that aren't sigs?
• send out notice to all sigs, give a deadline of N weeks
• any jobs that don't have e-mail addresses configured should be removed
• flip test to failing once all jobs meet the policy

Other thoughts / notes:

• need to parse both prowjob config and testgrid config for this
  • there are some testgrids not populated by prow, we can probably ignore these?
  • there are some prowjobs that don't have all their testgrid config in annotations

/sig testing

[spiffxp]

Some prior art to serve as starting points:

• go run ./experiment/prowjob-report/main.go --config ./config/prow/config.yaml --job-config ./config/jobs --format csv > jobs.csv which I periodically reimport into this spreadsheet
  • the owner_dash column is a guess on who should own the job based on which sig/wg-prefixed testgrid dashboard the job lives under
  • could run with --format json and pipe through to jq
  • there are some gaps; the big one I'm aware of is that it misses testgrid info that is solely in testgrid config
• This test (which does more than its name lets on) should serve as a good starting point for how to fail or log jobs/tabs that don't have alerts configured:

  test-infra/config/tests/testgrids/config_test.go

  Lines 506 to 544 in 2293860

  	func TestReleaseBlockingJobsMustHaveTestgridDescriptions(t *testing.T) {
  	// TODO(spiffxp): start with master, enforce for all release branches
  	re := regexp.MustCompile("^sig-release-master-(blocking|informing)$")
  	for _, dashboard := range cfg.Dashboards {
  	if !re.MatchString(dashboard.Name) {
  	continue
  	}
  	suffix := re.FindStringSubmatch(dashboard.Name)[1]
  	for _, dashboardtab := range dashboard.DashboardTab {
  	intro := fmt.Sprintf("dashboard_tab %v/%v is release-%v", dashboard.Name, dashboardtab.Name, suffix)
  	if dashboardtab.Name == "" {
  	t.Errorf("%v: - Must have a name", intro)
  	}
  	if dashboardtab.TestGroupName == "" {
  	t.Errorf("%v: - Must have a test_group_name", intro)
  	}
  	if dashboardtab.Description == "" {
  	t.Errorf("%v: - Must have a description", intro)
  	}
  	// TODO(spiffxp): enforce for informing as well
  	if suffix == "informing" {
  	if !strings.HasPrefix(dashboardtab.Description, "OWNER: ") {
  	t.Logf("NOTICE: %v: - Must have a description that starts with OWNER: ", intro)
  	}
  	if dashboardtab.AlertOptions == nil {
  	t.Logf("NOTICE: %v: - Must have alert_options (ensure informing dashboard is listed first in testgrid-dashboards)", intro)
  	} else if dashboardtab.AlertOptions.AlertMailToAddresses == "" {
  	t.Logf("NOTICE: %v: - Must have alert_options.alert_mail_to_addresses", intro)
  	}
  	} else {
  	if dashboardtab.AlertOptions == nil {
  	t.Errorf("%v: - Must have alert_options (ensure blocking dashboard is listed first in testgrid-dashboards)", intro)
  	} else if dashboardtab.AlertOptions.AlertMailToAddresses == "" {
  	t.Errorf("%v: - Must have alert_options.alert_mail_to_addresses", intro)
  	}
  	}
  	}
  	}
  	}

[spiffxp]

/help

[RobertKielty]

@spiffxp, myself, @ScrapCodes and @rayandas will do work together on the test. Coordinating with them both now.

[RobertKielty]

So @rayandas @ScrapCodes and myself met up to do exploratory work on this and as result we learned a little bazel!

To run the test above we need to invoke bazel as follows :

cd TEST_INFRA_REPO_ROOT
bazel test //config/tests/testgrids:go_default_test --config--test_output=all  

Worth noting that we must use bazel to run this test and not the go test runner.

The bazel build file is used to pull in test-grid runtime configuration using the
following dependencies :

        "@com_github_googlecloudplatform_testgrid//config:go_default_library",
        "@com_github_googlecloudplatform_testgrid//pb/config:go_default_library",

and also to pass in parameters to the test

        "--config=$(location testconf.pb)",
        "--prow-config=$(location //config/prow:config.yaml)",
        "--job-config=config/jobs",
 

[RobertKielty]

/assign
/remove help-wanted

[RobertKielty]

To run the specific test use

bazel test //config/tests/testgrids:go_default_test \
--test_output=all \
--test_filter=TestReleaseBlockingJobsShouldHaveTestgridDescriptions