Add functionality to trace API calls

[gandro]

To aid in debugging (but also learnability regarding the Hubble API), it would be nice to have a flag/mode for the CLI to track the API calls emitted by the CLI in a given run.

Example of a similar feature from kubectl:

$ kubectl --v=8 get pods -n kube-system
I1210 16:43:37.292636    9562 loader.go:379] Config loaded from file:  /home/vagrant/.kube/config
I1210 16:43:37.310335    9562 round_trippers.go:422] GET https://192.168.33.11:6443/api/v1/namespaces/kube-system/pods?limit=500
I1210 16:43:37.310379    9562 round_trippers.go:429] Request Headers:
I1210 16:43:37.310385    9562 round_trippers.go:433]     Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
I1210 16:43:37.310390    9562 round_trippers.go:433]     User-Agent: kubectl/v1.20.0 (linux/amd64) kubernetes/3321f00
I1210 16:43:37.345064    9562 round_trippers.go:448] Response Status: 200 OK in 34 milliseconds
I1210 16:43:37.345093    9562 round_trippers.go:451] Response Headers:
I1210 16:43:37.345102    9562 round_trippers.go:454]     Cache-Control: no-cache, private
I1210 16:43:37.345259    9562 round_trippers.go:454]     Content-Type: application/json
I1210 16:43:37.345635    9562 round_trippers.go:454]     X-Kubernetes-Pf-Flowschema-Uid: fba42907-d6e4-4b79-b35a-5964a353086d
I1210 16:43:37.346165    9562 round_trippers.go:454]     X-Kubernetes-Pf-Prioritylevel-Uid: a4104551-f5eb-4385-b6e5-696678b32669
I1210 16:43:37.346185    9562 round_trippers.go:454]     Date: Thu, 10 Dec 2020 16:43:37 GMT
I1210 16:43:37.353313    9562 request.go:1107] Response Body: {"kind":"Table","apiVersion":"meta.k8s.io/v1","metadata":{"resourceVersion":"110001"},"columnDefinitions":[{"name":"Name","type":"string","format":"name","description":"Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names","priority":0},{"name":"Ready","type":"string","format":"","description":"The aggregate readiness state of this pod for accepting traffic.","priority":0},{"name":"Status","type":"string","format":"","description":"The aggregate status of the containers in this pod.","priority":0},{"name":"Restarts","type":"integer","format":"","description":"The number of times the containers in this pod have been restarted.","priority":0},{"name":"Age","type":"string","format":"","description":"CreationTimestamp is [truncated 5536 chars]
NAME                       READY   STATUS    RESTARTS   AGE
coredns-6954c77b9b-gjdmp   1/1     Running   0          25h

[slayer321]

Hey @gandro, I did like to work on this issue.
I will need some pointers to start working on this issue. I tried to search how the "--v" flag is Implemented in Kubectl, but I could not find it. Though I found that client-go has some Implementation of it ..... but will need some example how can I start Implementing it

[gandro]

Hey @slayer321

The client-go library is using HTTP, while Hubble is using gRPC, so the implementation will likely be very different. Since this issue was created, we have gained some features which are similar to tracing, notably --print-raw-filters, which prints the request to be sent.

In any case, I think what needs to be done is to check what options exist on grpc-go and if it provides an option to print out every request sent via the connection. Maybe the interceptor logic can be used, but I'm not sure, so I cannot provide an example either. I think this issue would ideally be tackled by someone which a good understanding of grpc-go.

[rohithsai1904]

hello @rolinh @gandro Can you please assign the issue to me? I would like to work on it.

[kaworu]

@slayer321 have you already started working on this? Just to clarify so that we don't have many people on this issue.

[slayer321]

No, I'm not working on it.

[rohithsai1904]

Hey @gandro @kaworu , I have started knowing about the Cilium, Hubble and worked on the Isovalent Lab to get a hands-on. I have no experience in this domain and i have no clue to start working on the issue. Can you please let me know how to tackle the issue?

[rohithsai1904]

Hey @gandro @kaworu , I have started knowing about the Cilium, Hubble and worked on the Isovalent Lab to get a hands-on. I have no experience in this domain and i have no clue to start working on the issue. Can you please let me know how to tackle the issue?

Although I have got an idea based on what @gandro has said, I want a deeper look on how can I tackle it. And can you please tell me the requirements also? Thanks

[gandro]

This issue is quite old. As mentioned before, my main requirement for me was being able to see what kind of request is being sent to the API, and this has mostly been addressed via --print-raw-filters. The remaining goal of this issue is to provide some form of debug output for any grpc request issued by the CLI.

Notably, since this might not be clear from the title, this issue is not about the ability to trace API calls by the system observed by Hubble. This issue is only about adding a debug flag to the CLI which allows you to observe what the Hubble CLI itself is doing when talking to the backend.

I'm unfortunately not familiar with the facilities of go-rpc. Investigating the available options here is part of the work that needs to be done, i.e. this is not an issue with very clear instructions on what needs to be done, it will require some design work by the person working on it to figure out how exactly the feature should work. It might well be that the answer in the end is "getting a trace of the invoked methods is not possible with go-grpc", in which case we would simply close the issue.