ClusterLoader - 80% of logs are useless

[krzysied]

Currently 80% of the logs gather by cluster loader looks like this:

I0719 22:58:11.023759  199702 phase_latency.go:76] PodStartupLatency: failed to find schedule time for test-ahw1lj-13/big-deployment-2-788bf9864f-jktzx
I0719 22:58:11.023786  199702 phase_latency.go:76] PodStartupLatency: failed to find schedule time for test-ahw1lj-37/small-deployment-199-66b4cc59b7-kqz5b
I0719 22:58:11.023795  199702 phase_latency.go:76] PodStartupLatency: failed to find schedule time for test-ahw1lj-41/medium-deployment-3-5fb6668747-gkbrn

[krzysied]

/area clusterloader

[wojtek-t]

I guess the problem is that this info is being gathered from events, which has 1h TTLs.

If we really want that information, we should switch to watching events (which will be costly to apiserver btw).

[krzysied]

I was thinking about listing events every 30 minutes...

[krzysied]

Or we can turn off this logging - I'm not sure if anyone ever used this info...

[wojtek-t]

That may work too, though it will have non-negligible performance impact too (fortunately events are stored in separate etcd, so at least on etcd-level we should be fine).

[krzysied]

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[wojtek-t]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale