[NEW] Use LTTng for generic server logging

[PingXie]

          I like lttgn. giving its low overhead, I think we should consider retiring (but not removing just yet) serverLog going forward. we should introduce lttng as a proper logging mechanism that supports level logging. I don't think we need to mimic the existing latency logging paradigm.

@pizhenwei @artikell

Originally posted by @PingXie in #2070 (comment)

[madolson]

We support systems that aren't linux, how can we retire serverLog?

[pizhenwei]

We support systems that aren't linux, how can we retire serverLog?

what about trying to write log to lttng first, then fallback to the legacy file based log on failure?

[PingXie]

We support systems that aren't linux, how can we retire serverLog?

what about trying to write log to lttng first, then fallback to the legacy file based log on failure?

I think, ideally, we will keep the same entry point like serverLog and based on the build time option, we use either LTTng or the current one. I don't think we want to fall back on failure at runtime. this is as good as "deprecation" to me as the change will be limited to serverLog, hopefully.

[madolson]

based on the build time option

Build time options are annoying to keep track of, also means more configurations we have to test. I don't really like bifurcated behavior for something which is generally not that latency sensitive. Then we have to be aware the nuances of the different implementations.

[PingXie]

based on the build time option

Build time options are annoying to keep track of, also means more configurations we have to test. I don't really like bifurcated behavior for something which is generally not that latency sensitive. Then we have to be aware the nuances of the different implementations.

I am open to other ways to enable this behavior.

that being said, there is a significant amount value of adopting this modern logging infrastructure. today we can't write much/often into the log because every write involves at least 3 syscalls. this leads to a situation where it is very challenging to understand the server behavior in a production environment without a significant amount of performance impact.

imo, I would like us to get a point where (more) detailed logging is not frowned upon automatically and if needed for a short period of time, even more detailed logging can be enabled with say 10% of performance drop, as opposed to 90% (exaggeration).

I am sure we can sort out the engineering problems once we are aligned on the value/use cases. we can discuss this more in one of the core meetings.

[madolson]

imo, I would like us to get a point where (more) detailed logging is not frowned upon automatically and if needed for a short period of time, even more detailed logging can be enabled with say 10% of performance drop, as opposed to 90% (exaggeration).

I don't think we are that adverse to logging. The problem is we can't log on the data path. Doing a get/set operation is always going to be an order of magnitude faster than writing to a disk, so we will never really log that. What's left could probably do better on logging.

I am sure we can sort out the engineering problems once we are aligned on the value/use cases. we can discuss this more in one of the core meetings.

I agree yeah. There are some places where logging is a problem.

[PingXie]

I don't think we are that adverse to logging. The problem is we can't log on the data path.

there is a lot of insight to get on the performance sensitive path (including but not limited to data path) but we can't afford them. for instance, cluster bus activity should ideally be logged as much as possible but the cost is too high (#2076)

[madolson]

for instance, cluster bus activity should ideally be logged as much as possible but the cost is too high (#2076)

In that trace it was only 3% during a high burst event. The actual resolution gave a slightly more nuanced opinion, #2010, it called the logs spam because they weren't strictly that useful. Generally my dislike of logs is that it becomes a wall that eats a way at an LLM or a human context window.

I still stand behind if you think it's worth having that context, we should keep the log, I'll pay the 3% for it during bursty failover events.

[PingXie]

3% is not a small number imo. Then we are taking about adding more nodes. So 3% could go higher. I am not so concerned about other areas than clustering but more often than not i wish i had a more detailed log for cluster activities, which is also evident in the few recent clustering related issues. Furthermore i am sure we will be adding more complexity into the engine down the road so it makes sense to take advantage of the much more performant logging facility. Sure i agree that we need to consider other OSes to not overburden them with logging and for that I think we could use the trace level carefully (or adding proper component/provider support).

[madolson]

3% is not a small number imo.

It's for when 1/3 of a 1000 cluster does a failover. So, it's a 3% spike over a rather short period of time, it's not in steady state, which I'm mostly OK with it. It also wouldn't be too hard to optimize just that (by doing a buffered write) to disk.

Furthermore i am sure we will be adding more complexity into the engine down the road so it makes sense to take advantage of the much more performant logging facility.

I just really don't like taking dependencies.