Provide more granular configuration options to choose DAO implementations

[aravindanr]

Currently the DAO (MetadataDAO, QueueDAO, ExecutionDAO) implementations are chosen by one property called db. This prevents users from picking different implementations for different purposes. For instance, executions could be stored in Cassandra, but queue could be MySQL.

One approach is to provide a configuration property for each DAO interface.

[marcocrasso]

@aravindanr
I'd like to contribute with this, however I had a couple of questions after diving at 3.0 source code and #2016. Based on available DAOs and implementations found in this repo, I sketched this:

Based on "One approach is to provide a configuration property for each DAO interface." and the fact that there are seven DAOs, but we can ignore IndexDAO here, using a granularity level of 6 would mean going from:

conductor.db.type=[memory | dynomite | redis* | postgres | mysql]

to:

conductor.db.metadata.type=[ cassandra | memory | dynomite | redis* | postgres | mysql ]
conductor.db.eventhandler.type= [ cassandra | memory |dynomite | redis* | postgres | mysql ]
conductor.db.polldata.type=[ cassandra | memory | dynomite | redis* | postgres | mysql ]
conductor.db.execution.type=[ cassandra | memory| dynomite | redis* | postgres | mysql ]
conductor.db.queue.type=[  memory | dynomite | redis* | postgres | mysql ]
conductor.db.ratelimiting.type=[ memory | dynomite | redis* | postgres | mysql ]

where "redis*" eq redis_standalone | redis_cluster | redis_sentinel

Are the DAO implementations fully independent from each other? Is achievable this approach?

Or, alternatively the granularity level should be 3: MetadataDAO, QueueDAO, ExecutionDAO

conductor.db.metadata.type=[ cassandra | memory | dynomite | redis* | postgres | mysql ]
conductor.db.polldata.type=[ cassandra | memory | dynomite | redis* | postgres | mysql ]
conductor.db.execution.type=[ cassandra | memory| dynomite | redis* | postgres | mysql ]

in which case eventhandler, polldata, and ratelimiting should inherit the configuration due to the intrinsic dependencies.

Any suggestion here will allow me to move forward. I am can extend current support using ConditionalOnExpression on each Bean declaration. Regards

[aravindanr]

@marcocrasso Thanks for volunteering. The idea behind this is to not eliminate conductor.db.type, but allow overrides for it. For instance, in #2016, the ask was to use MySQL for definitions & executions but Redis for queues. In the current setup, that is not possible.

The implementation that i am aiming for is something like,

conductor.db.type=mysql
com.netflix.conductor.dao.QueueDAO=<fully qualified class name of a class that implements QueueDAO>

The effect of the above configuration should be, for all DAO implementations except QueueDAO, the MySQL implementations are chosen. It is important that another layer of grouping is not introduced. With this override, users have greater flexibility in choosing DAO implementations, while somewhat retaining the conciseness that a grouping like db.type offers.

The reason for using the fully qualified class names in the key and value is, in the future, the DAO contracts can be published as an SPI module and implementations are loaded during runtime using ServiceLoader.

Hope that clarifies the intention.

[marcocrasso]

@aravindanr
I've refactored all x-persistence modules to implement these as Java SPI modules, added one new module named service-loader which loads SPI Service Providers as spring-boot beans, updated documentation and added docker-compose-* with backend combinations, like MySQL +Cassandra.

I have forked the repo. Right now, me and our team is testing the Cassandra + MySQL combination. I decided to fork it, since the refactoring was big. Once tested and confirmed everything is working as expected, my next step would be to submit a PR. I assigned the issue to myself, hope that is ok.

Changes

Main work was about replacing the DI mechanism used for backend, spring-boot Conditional beans with Java SPI. This requires to load beans into spring-boot context differently (not more relying on ConditionalBeans, and to assume that two or more implementations can co-exist in the class loader without causing beans collisions), and to accommodate those modules providing DAOs implementations (i.e. the x-persistence ones) to Java SPI impositions, such default no args constructor. I addressed the backward compatibility goal too.

This class holds the new beans mechanism service-loader

By backward compatibility I meant supporting beans configuration using the previous “one-property” approach.

Hard Constraints

I auto-imposed as a hard constraint NOT TO TOUCH core module. My rationale was that we are replacing how beans are loaded into spring-boot context, and nothing else. Therefore, core must keep working without noticing these if and only if required beans can be auto-wired.

Documentation:

Overview of backend as Java SPI

https://github.com/marcocrasso/conductor/tree/iss2017_granular_spi_module_configuration/service-loader#service-provider-loader

Migration guide

https://github.com/marcocrasso/conductor/tree/iss2017_granular_spi_module_configuration/service-loader#persistence-module-migration-guide how to move an existence *-persistence module to the new Java SPI-based approach.

Updated Oficial docs

https://github.com/marcocrasso/conductor/blob/iss2017_granular_spi_module_configuration/docs/docs/extend.md I extended official project documentation in section “extend” to explain how to add a new backend implementation.

Running on Docker

https://github.com/marcocrasso/conductor/tree/iss2017_granular_spi_module_configuration/docker#readme completely updated Docker steps to show how to use the system with Cassandra, MySQL, Redis, or combinations; all on top of docker-compose.

Testing

All tests, both individual and integrations, are passing. Further testing is ongoing.