Report on the number of plans processed to NewRelic

[diegoeche]

Thanks again for maintaining this, and the answers from my last question. #320 🥳 🥳 following your advice, we got something working using log4j and a newrelic agent for log-forwarding.

As a followup, we are looking to not only log the "errors" of lineage harvesting, but also the "successes". We are basically trying to figure out the rate on which Spline is generating lineage.

  def harvest(result: Try[Duration]): HarvestResult = {
    logDebug(s"Harvesting lineage from ${ctx.logicalPlan.getClass}")

I guess I could change our debugging level to use Debug, and maybe use log4j to make it apply only for the spline agent package.
But as usual you guys always think of a better approach :)

We also thought about recompiling spline and change that line to use INFO (which sounds like a bad idea for future maintenance).

[wajda]

Why not utilizing custom dispatchers for that?

spline.lineageDispatcher=composite

spline.lineageDispatcher.composite.dispatchers=http,myMetricsCapturingDispatcher

spline.lineageDispatcher.myMetricsCapturingDispatcher.className=x.y.z.MyMetricsCapturingDispatcher

With that config the custom MyMetricsCapturingDispatcher would be called in addition to the http one.

Although at the moment, the dispatchers are only called on success events, but after #32 is implemented the failed executions will also go through the dispatchers.