Extract common code from connectors.

[Claudenw]

We currently have 5 connectors (3 sink and 2 source). They share the same basic strategies but not the same code.

I would like to extract the test cases into commons/testFixtures and then make sure that all the sinks implement the commons suite of sink tests and all the sources implement the commons suite of source tests.

In this way we will be able to verify that the connectors are consistent.

Once we have common tests we can create common implementations that use an interface to communicate with each individual storage layer. Something similar exists in the common source but it is poorly designed as it depends on generics to define methods across classes that all resolve to being able to read the underlying storage. I think the abstract methods can be gathered together into a couple of interfaces (one for source and one for sink). The specific implementation would then just need to implement the interface to provide a complete implementation.

All items like compression, grouping, file name templating, will be implemented in the core code so that there is only one implementation. (This should help reduce testing time as we won't have 5 packages testing the compression code.)

So the two major tasks for this proposal are:

1. Create a set of common test suites.
   • Extract common unit tests from source test cases
   • Extract common integration tests from source integration tests
   • Extract common unit tests from sink test cases
   • Extract common integration test from sink test cases.
2. Create a set of common implementations
   • Extract common implementations from each of the source classes.
   • Extract common implementations from each o the sink classes.

[RyanSkraba]

This sounds like the right thing to do. Having consistent connector features and operations (where the underlying store allows it) is a good thing for users and usability, and we currently take this on faith.