Have a factory based creation of alibi explainers (based on config)

[sakoush]

alibi detect is considering init based config or something similar, which will be good to apply for explain at some point as well.

Currently implementation details of alibi explainers are leaked in mlserver which we are trying to keep at minimum. Ideally mlserver should depend on interfaces and not concrete implementations.

For example If we can create the explainer from config / factory on the alibi side, mlserver doesnt need to depend on specific impl of explainers.

I am using a hack now to have a loose dependency injection by having a config map on mlserver side. However any change to alibi implementation details will require an (unnecessary) change to mlserver.

[jklaise]

Definitely, there will be a lot of overlap with alibi-detect work so we should strive to do to it the same way.

The way I picture it now is roughly:

• New class (or function) e.g. ExplainerFactory(config: dict) -> Explainer taking in a config and returning an explainer
• The config will have (as a minimum) the following fields:
  • name (e.g. name: AnchorImage) - identifies the explainer class to be initialized
  • init_kwargs (e.g. init_kwargs: {predictor: ?, image_shape: [28,28,1], segmentation_fn: ?, ... })
• Nice to have - use pydantic models to validate the config (this would solve simple things like Allow AnchorImage image_shape init parameter to be consumed as any Sequence #493 where a list will be parsed into a tuple)
  • Will need to think how this can be done effectively, e.g. conditional on name the init_kwargs will have a different spec.

Questions and challenges:

• How to specify predictor in the config (and other "complex" data types, e.g. numpy array as an in-memory training set, custom functions)? There could be a few approaches depending on the goals:
  • The config is actually treated as a light DSL where some strings map to functions/objects defined in the application runtime, e.g. https://thinc.ai/docs/usage-config#registry. This requires the alibi application to "register" complex types (like a model) using a registry function in alibi which then can be referenced via a string key
  • The config specifies the location of the resource (e.g. white-box artefact or black-box endpoint) - this would limit the config-driven explainer initialization to mostly deployment use cases as pure alibi apps would like to point to the resources already loaded in memory
  • alibi may want to allow functionality for "deferring" initialization of complex types, e.g. __init__ can be called creating the explainer, then a method like reset_predictor can be called to pass in an in-memory predictor. This way the config doesn't have any reference to the complex types like model which can be seen as both a pro or a con.
• Should the config be extended to fit and explain methods? What are the benefits?
  • We certainly would like to use pydantic models to validate fit and especially explain arguments within alibi which reduces the validation burden on the application layer
  • Does it make sense to "initialize" an explainer followed by calling a fit method if the config is filled out?
  • Does it make sense to "initialize" an explainer with pre-configured explain kwargs or will it cause confusion?

[sakoush]

We might want also to consider:

• If we are creating a schema for init_kwargs, explain_kwargs, we might want something like ExplainerConfigValidator exposed so that clients (e.g. mlserver) can validate the config without having to init the actual explainer (because it might be expensive to init)? It could be initially part of the error codes returned when trying to init the explainer but it feels that having it separate could be a nice separation of concerns?

I realised that there this point is pretty much what I have in this #488

[jklaise]

@sakoush we could certainly have that separate from the initialization (e.g. as a class-method), however I don't think there is much difference exposing that functionality to the client vs just running it as the first thing upon initialization (unless we also want to validate explain_kwargs pre-initialization?).