Skip to content

Commit 7eef9f3

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #125 from ShwethaKumbla/watcher
implement k8s resource watch and trigger action based on the events
2 parents df2ab4b + db17e9f commit 7eef9f3

File tree

8 files changed

+583
-3
lines changed

8 files changed

+583
-3
lines changed

‎docs/design/k8s-resource-watcher.md

+164
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# Watcher for K8S Objects
2+
3+
K8S object watchers are great functionality provided by k8s to get efficient change notifications on resources.
4+
5+
The events supported by these watchers are
6+
1. ADD
7+
2. MODIFY/UPDATE
8+
3. DELETE
9+
10+
## Motivation
11+
Users must implement watchers such a way that whenever any events recieved some action/job has to be triggered. Developer has to write lots of code to do this and its sometimes difficult to manage.
12+
13+
This can be achieved by using the k8s client built in function but understanding which packages to import or which core type needs to be used might be a complex for the developer.
14+
15+
The below design would make developer life easier. They have to just register their actions for respective events. To stay informed about when these events get triggered just use Watch(), which resides inside klient/k8s/resources package.
16+
17+
## Proposal
18+
Watch function accepts a `object ObjectList` as a argument. ObjectList type is used to inject the resource type in which Watch has to be applied.
19+
20+
`klient/k8s/resources/resources.go`
21+
```go=
22+
import (
23+
"sigs.k8s.io/controller-runtime/pkg/client"
24+
"k8s.io/apimachinery/pkg/watch"
25+
)
26+
27+
func (r *Resources) Watch(object k8s.ObjectList, opts ...ListOption) *watcher.EventHandlerFuncs {
28+
listOptions := &metav1.ListOptions{}
29+
30+
for _, fn := range opts {
31+
fn(listOptions)
32+
}
33+
34+
o := &cr.ListOptions{Raw: listOptions}
35+
36+
return &watcher.EventHandlerFuncs{
37+
ListOptions: o,
38+
K8sObject: object,
39+
Cfg: r.GetConfig(),
40+
}
41+
}
42+
```
43+
44+
Watch() in resources.go will return the `watcher` type which helps to call `Start()`. InvokeEventHandler accepts `EventHandlerFuncs` which carries the user registerd function sets.
45+
46+
file : klient/k8s/resources/watch.go
47+
48+
```go=
49+
// Start triggers the registered methods based on the event recieved for particular k8s resources.
50+
func (watcher watch.Interface)Start(ctx context.Context) {
51+
...
52+
go func() {
53+
for {
54+
select {
55+
case <-ctx.Done():
56+
if ctx.Err() != nil {
57+
return
58+
}
59+
case event := <-e.watcher.ResultChan():
60+
// retrieve the event type
61+
eventType := event.Type
62+
63+
switch eventType {
64+
case watch.Added:
65+
// calls AddFunc if it's not nil.
66+
if e.addFunc != nil {
67+
e.addFunc(event.Object)
68+
}
69+
case watch.Modified:
70+
// calls UpdateFunc if it's not nil.
71+
if e.updateFunc != nil {
72+
e.updateFunc(event.Object)
73+
}
74+
case watch.Deleted:
75+
// calls DeleteFunc if it's not nil.
76+
if e.deleteFunc != nil {
77+
e.deleteFunc(event.Object)
78+
}
79+
}
80+
}
81+
}
82+
}()
83+
...
84+
}
85+
86+
// EventHandlerFuncs is an adaptor to let you easily specify as many or
87+
// as few of functions to invoke while getting notification from watcher
88+
type EventHandlerFuncs struct {
89+
addFunc func(obj interface{})
90+
updateFunc func(newObj interface{})
91+
deleteFunc func(obj interface{})
92+
watcher watch.Interface
93+
ListOptions *cr.ListOptions
94+
K8sObject k8s.ObjectList
95+
Cfg *rest.Config
96+
}
97+
98+
// EventHandler can handle notifications for events that happen to a resource.
99+
// Start will be waiting for the events notification which is responsible
100+
// for invoking the registered user defined functions.
101+
// Stop used to stop the watcher.
102+
type EventHandler interface {
103+
Start(ctx context.Context)
104+
Stop()
105+
}
106+
107+
```
108+
109+
`Start()` is invoked in a goroutine so that whenever watched resource changes the states it will call the registered user defined functions.
110+
`Stop()` should be explicitly invoked by the user after the watch once the feature is done to ensure no unwanted go routine thread leackage.
111+
112+
If any error while Start() one can retry it for number of times.
113+
114+
This example shows how to use klient/resources/resources.go Watch() func and how to register the user defined functions.
115+
116+
```go=
117+
118+
import (
119+
"sigs.k8s.io/e2e-framework/klient/conf"
120+
"sigs.k8s.io/e2e-framework/klient/k8s/resources"
121+
)
122+
123+
func main() {
124+
...
125+
cfg, _ := conf.New(conf.ResolveKubeConfigFile())
126+
cl, err := cfg.NewClient()
127+
if err != nil {
128+
t.Fatal(err)
129+
}
130+
131+
dep := appsv1.Deployment{
132+
ObjectMeta: metav1.ObjectMeta{Name: "watch-dep", Namespace: cfg.Namespace()},
133+
}
134+
135+
// watch for the deployment and triger action based on the event recieved.
136+
cl.Resources().Watch(&appsv1.DeploymentList{}, resources.WithFieldSelector(labels.FormatLabels(map[string]string{"metadata.name": dep.Name}))).
137+
WithAddFunc(onAdd).WithDeleteFunc(onDelete).Start(ctx)
138+
139+
...
140+
}
141+
142+
// onAdd is the function executed when the kubernetes watch notifies the
143+
// presence of a new kubernetes deployment in the cluster
144+
func onAdd(obj interface{}) {
145+
dep := obj.(*appsv1.Deployment)
146+
_, ok := dep.GetLabels()[K8S_LABEL_AWS_REGION]
147+
if ok {
148+
fmt.Printf("It has the label!")
149+
}
150+
}
151+
152+
// onDelete is the function executed when the kubernetes watch notifies
153+
// delete event on deployment
154+
func onDelete(obj interface{}) {
155+
dep := obj.(*appsv1.Deployment)
156+
_, ok := dep.GetLabels()[K8S_LABEL_AWS_REGION]
157+
if ok {
158+
fmt.Printf("It has the label!")
159+
}
160+
}
161+
162+
```
163+
164+
The e2e flow of how to use watch is demonsrated in the examples/ folder.

‎docs/design/test-harness-framework.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -469,7 +469,7 @@ The framework should automatically inject these filter values into the environme
469469
### Skipping features
470470
The test framework should provide the ability to explicitly exclude features during a test run. This could be done with the following flags:
471471

472-
* `--skip-feature` - a regular expression that skips features with matching names
472+
* `--skip-features` - a regular expression that skips features with matching names
473473
* `--skip-assessment` - a regular expression that skips assessment with matching name
474474
* `--skip-lables` - a comma-separated list of key/value pairs used to skip features with matching lables
475475

‎examples/watch_resources/README.md

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Watching for Resource Changes
2+
3+
The test harness supports several methods for querying Kubernetes object types and watching for resource states. This example shows how to watch particular resource and how to register the functions to act upon based on the events recieved.
4+
5+
6+
# Watch for the deployment and triger action based on the event
7+
8+
Watch has to run as goroutine to get the different events based on the k8s resource state changes.
9+
```go
10+
func TestWatchForResources(t *testing.T) {
11+
...
12+
dep := appsv1.Deployment{
13+
ObjectMeta: metav1.ObjectMeta{Name: "watch-dep", Namespace: cfg.Namespace()},
14+
}
15+
16+
// watch for the deployment and triger action based on the event recieved.
17+
cl.Resources().Watch(&appsv1.DeploymentList{}, &client.ListOptions{
18+
FieldSelector: fields.OneTermEqualSelector("metadata.name", dep.Name),
19+
Namespace: dep.Namespace}, cl.RESTConfig()).WithAddFunc(onAdd).WithDeleteFunc(onDelete).Start(ctx)
20+
...
21+
}
22+
```
23+
24+
# Function/Action definition and registering these actions
25+
26+
```go
27+
// onAdd is the function executed when the kubernetes watch notifies the
28+
// presence of a new kubernetes deployment in the cluster
29+
func onAdd(obj interface{}) {
30+
dep := obj.(*appsv1.Deployment)
31+
depName := dep.GetName()
32+
fmt.Printf("Dep name recieved is %s", depName)
33+
if depName == "watch-dep" {
34+
fmt.Println("Dep name matches with actual name!")
35+
}
36+
}
37+
38+
// onDelete is the function executed when the kubernetes watch notifies
39+
// delete event on deployment
40+
func onDelete(obj interface{}) {
41+
dep := obj.(*appsv1.Deployment)
42+
depName := dep.GetName()
43+
if depName == "watch-dep" {
44+
fmt.Println("Deployment deleted successfully!")
45+
}
46+
}
47+
```
48+
49+
The above functions can be registered using Register functions(WithAddFunc(), WithDeleteFunc(), WithUpdateFunc()) defined under klient/k8s/watcher/watch.go as shown in the example.
50+
51+
# How to stop the watcher
52+
Create a global EventHandlerFuncs variable to store the watcher object and call Stop() as shown in example TestWatchForResourcesWithStop() test.
53+
54+
Note: User should explicitly invoke the Stop() after the watch once the feature is done to ensure no unwanted go routine thread leackage.

‎examples/watch_resources/main_test.go

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
/*
2+
Copyright 2022 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package watch_resources
18+
19+
import (
20+
"os"
21+
"testing"
22+
23+
"sigs.k8s.io/e2e-framework/pkg/env"
24+
"sigs.k8s.io/e2e-framework/pkg/envconf"
25+
"sigs.k8s.io/e2e-framework/pkg/envfuncs"
26+
)
27+
28+
var testenv env.Environment
29+
30+
func TestMain(m *testing.M) {
31+
testenv = env.New()
32+
kindClusterName := envconf.RandomName("watch-for-resources", 16)
33+
namespace := envconf.RandomName("watch-ns", 16)
34+
testenv.Setup(
35+
envfuncs.CreateKindCluster(kindClusterName),
36+
envfuncs.CreateNamespace(namespace),
37+
)
38+
testenv.Finish(
39+
envfuncs.DeleteNamespace(namespace),
40+
envfuncs.DestroyKindCluster(kindClusterName),
41+
)
42+
os.Exit(testenv.Run(m))
43+
}

0 commit comments

Comments
 (0)