What would you like to be added:
We should look into allowing to fetch only PartialObjectMetadata for these configmap/secrets objects.

Why is this needed:
ConfigMap objects can contain up to 1M of config or binary data. The metrics generated for the configmap is only relying on metadata.
Using PartialObjectMetadata will reduce memory consumption of KSM and require less network traffic.

For Secrets I think this will add the benefit that the secret data never is requested over the wire.

Describe the solution you'd like
Probably this metadatalister can be used instead of the ListWatch we use for other objects.
https://github.com/kubernetes/client-go/blob/71959c526d543a5e4c3ca6fb808f535c2726483f/metadata/metadatalister/lister.go

Additional context