policy-reporter presents incomplete results

[mikebryant]

Version: ghcr.io/kyverno/policy-reporter:2.19.0

We're getting inconsistent UI behaviour on startup - where it's showing us only part of the cluster (which can give people the impression they have no violations to worry about, when they actually do)

We're seeing these logs (from the policy-reporter Deployment):

2024-06-28 09:40:59	INFO	UI configured
2024-06-28 09:40:59	INFO	sqlite connection created
2024-06-28 09:40:59	DEBUG	preparing database
2024-06-28 09:40:59	DEBUG	failed to load config	{"error": "no such table: policy_report_config"}
2024-06-28 09:40:59	DEBUG	database schema upgrade started
2024-06-28 09:40:59	INFO	REST api enabled
2024-06-28 09:40:59	INFO	metrics enabled
2024-06-28 09:40:59	DEBUG	register new result listener
2024-06-28 09:40:59	INFO	start client	{"worker": 5}
2024-06-28 09:40:59	WARN	informers not synced yet, waiting for k8s client to be up
github.com/kyverno/policy-reporter/pkg/api.(*httpServer).RegisterLifecycleHandler.ReadyHandler.func2
	/app/pkg/api/handler.go:40
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2166
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2683
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:3137
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:2039
2024-06-28 09:40:59	INFO	informer sync completed
``

But a call to `/v1/namespaces` only shows some Namespaces (and it slowly increases over time). Similarly `/v1/policy-reports | jq '.count'` shows a slowly increasing count, until it eventually stabilies at the same value as `kubectl get policyreport -A | wc -l`

We would have expected it to not respond with incomplete data, and to not report itself as `ready`, until it's actually ready for use.

[mikebryant]

Sigh, and I realise having written this we're actually not on the latest helm chart, so I'll go and upgrade that and see if that happens to fix things

[mikebryant]

Upgraded to latest chart (2.24.0), which gives image version ghcr.io/kyverno/policy-reporter:2.20.0 - same issue

[fjogeleit]

So you mean its not showing all namespaces at the beginning but after some time the list is complete? Or the list is never fully complete?

In general means ready that the listener starts working, all information including namespaces comes from PolicyReports and PolicyReporter needs to process all existing PolicyReports before all data is available.

I never have an indicator when all at start time existing reports are processed because they come one by one from the listener. Also depending on cluster, amount of policies and resources, the time to process everything is very different.

[mikebryant]

After some time the list is complete.

It's being particularly noticeable right now on our sbx environment, because it only has one replica and it keeps being restarted by the VPA controller when the resource estimate changes, and each time the UI stops being accurate for 5-10 minutes.

The issue I have is I want to be able to point our users at this as a tool to look at the state of their apps - but it's tricky when the data may be inaccurate, but presented as if it's accurate. So for my use-case, I'd prefer it not be available until it's ready, rather than responding with missing data.

I've seen other projects have similar requirements, e.g. kubernetes/kubernetes#113763 & https://github.com/istio/istio/blob/master/pkg/kube/kclient/client.go#L190-L203

[fjogeleit]

So you would prefer that policy reporter needs 5 to 10 minutes to be marked as running?

I am not sure if your example hits this situation. HasSynced is already used as indicator for readiness:

https://github.com/kyverno/policy-reporter/blob/main/pkg/api/handler.go#L32

But this dosn't mean that everything is processed. The API is based on an internal Database, which will be filled after the sync by processing each PR one by one. The API does not get its data directly from the K8s API. And this process takes time depending on the amount of reports and results.

I have no way to know or to check if all initial reports are processed at least once.

[mikebryant]

So you would prefer that policy reporter needs 5 to 10 minutes to be marked as running?

Tbh, yes - I'd like the option for that at least. My perspective is unavailable and inaccurate are both bad. (Though I suppose available but with a warning saying it's incomplete would also work)

I am not sure if your example hits this situation. HasSynced is already used as indicator for readiness:

🤔 Hmm, I know that things like istio do this somehow (e.g. you don't want to start allowing traffic before you know you've processed all AuthorizationPolicy)

I'm happy to take a look at trying to achieve this.

I guess my question is - is this something you'd be interested in accepting as a PR? (Possibly as an option?)

[fjogeleit]

As an Option sure, maybe one way could be to use an external DB, which is already possible and improve the behavior after a restart. So it keeps already existing data, only updates changes and remove no longer existing information (which is the hardest part).

I need to focus on the UI for now, so if you would like to take a look, I am happy about each contribution.