Allow override of "reconciling object" tracing span in Controller reconcile loop

[tgrushka]

Would you like to work on this feature?

no

What problem are you trying to solve?

Every single reconcile, and every single tracing::info! or higher log call, this is added to my logs:

22:31:17.769 INFO reconciling object{object.ref=fully.qualified.domain.name.full-name.namespace object.reason=object updated}:reconcile: then finally my log

Of course it respects the tracing format I've specified, but this hard-coded span makes the logs very long and hard to read.

I am visually impaired. I want to control my own log format. I tried to create a new Span, but can't, because it's not Send.

There ought to be an option in the Controller::new(...).with_config(Config { ... }) struct with which I can configure my own Span format or something like that.

I found the culprit buried deep in the bowels of kube-rs where it cannot be overridden at https://github.com/kube-rs/kube/blob/main/kube-runtime/src/controller/mod.rs#L394

Please, is there any way to suppress / remove this and still extract the reconcile reason? I guess I can use a filter, but then I can't get the reconcile reason, right?

Thank you.

Describe the solution you'd like

I'd like to be able to:

• ideally: configure my own Some(Span) (or None) in the Controller::new(...).with_config(config) option (how? I'm not a tracing expert), or
• allow suppression of the default Span altoegether from the reconcile (via the Config or with a call to .without_span() or similar), or
• be able to specify the format, or
• allow a reason argument to reconcile and be able to suppress Span altogether, or
• something else -- please, my logs are just too verbose because it shows up in every single log I print in my controller!

Describe alternatives you've considered

Tracing Filter. I guess I'll have to do that for now. Looks like there are only a couple reasons for reconcile anyway, but would be nice to have this info.

Documentation, Adoption, Migration Strategy

Hopefully it should only add a new Config option, which should just default to what it is now and let the user override.

EDIT: At least, in the interim before a solution, can we have an example tracing Filter in the docs or README or something on how to suppress this span?

Target crate for feature

kube-runtime

[clux]

Hey, thanks for this issue. I can sympathise with our logs not necessarily being designed with the best accessibility in mind.

This line has become a bit of an incidental expectation for me though, so I wonder if having it as configurable is the way forward... 🤔

I.e. right now it is using info_span, but we could potentially make it a raw span with a default Level configurable via the runtime::Config using some kind of setter such as Config::reconcile_trace_level. That way you could set Level::DEBUG and not be annoyed by default. EDIT: upon converations this may not be possible because the level might need to be compile time constant.

However, if this is broadly seen as an annoying message (rather than generally helpful), we could consider doing what we do elsewhere in the codebase and never inject info level spans and just downgrade this to warning (which seems to be more common in the ecosystem around libraries, at least from what i've seen from rustls/hyper etc). EDIT: it's been noted that web frameworks do inject info spans for their main operations so maybe that's a more fair comparison

I'm not sure which one would be the best solution, given that the message is a bit of an incidental expectation atm. Will ask around.

---

In the mean time, you can work around with:

RUST_LOG=info,kube=warn cargo run X

if you have a default tracing_subscriber::fmt::init(); style setup (or something similar that uses tracing's EnvFilter). We do have more examples on advanced tracing use in controller-rs's telemetry.rs or in the kube.rs observability doc.