ENH: Allow users to drop some pipeline steps

[scott-huberty]

Somewhat related to #215 -

I am wondering if it is feasible to allow users to disable some pipeline steps. Using the filtering step as an example, I think an API that follows #215 would look like:

config = ll.Config().load_default()
config["filtering"] = None

Or:

filtering:

Deleting a (currently mandatory) step will probably break a lot of our code across the pipeline, rejection, and dashboard modules, so a PR should be made to see how feasible it is to implement this proposal

[christian-oreilly]

Agreed. As you said, it is also related to the more general issue of #215. I think it would be good to refactor the pipeline a bit more like a "pipeline", where steps can be inserted or removed. Unfortunately, the "pipelines" I have seen generally add a lot of clutter because they try to abstract what is a step in the pipeline and end up requiring a lot of boilerplate code to add these steps, which makes the code hard to work with. Brainstorm would be an example of this. If we go down that road, let's keep code readability and simplicity at the forefront.

[scott-huberty]

Unfortunately, the "pipelines" I have seen generally add a lot of clutter because they try to abstract what is a step in the pipeline and end up requiring a lot of boilerplate code to add these steps, which makes the code hard to work with. Brainstorm would be an example of this. If we go down that road, let's keep code readability and simplicity at the forefront.

I couldn't agree with you more, I have had the same experience. We have to be careful here and I'm okay with us taking the time needed to get it right.