Programmatic init and REQUIRED mode

[wajda]

Problem description

In #489 we decided to remove REQUIRED mode. There were a few reasons for this:

• it was not playing well with the codeless init as the listener was called too late - when the main job has been already done and there is no way to prevent it from done without lineage tracking in case of the agent init failure.
• even when init is successful the lineage dispatcher is called after work anyway, again leading to inability to actually stop the job in case of lineage dispatching failure.
  In other words REQUIRED mode was not strong enough to be called that way, and in many cases behaved like the BEST_EFFORT due to limitations of Spark listener API. That why we replaced both modes with just ENABLED.

But... there still are cases when fast failure is desired and is technically possible. Like, for instance, when using programmatic initialization method by explicitly calling enableLineageTracking(). In that case a user might logically expect the program to throw an exception if lineage tracking cannot be enabled for whatever reason. Unfortunately, after the aforementioned change the above method always returns with no error, and there is no way for a user to enforce the exception throwing, or to verify the agent initialization status.

Questions to the community

1. Do you think the described aspect is important enough to bother about it?
2. If yes then how what behavior would be more correct/useful?:
   a) should the enableLineageTracking() always throw in case of failure, or
   b) should this behavior be optional, and be controlled by, say a boolean method parameter (e.g. enableLineageTracking(..., silentOnError = true) to swallow errors, otherwise it would throw by default)
   c) should we add another method (say isLineageTrackingEnabled()) to let a user to check the agent status

[wajda]

@cerveada