Testing

[quintona]

It should be possible to execute a test in local mode that combines both cascading and Hive. I have done a bit of research and it seems that this isn't possible in the hive world (http://stackoverflow.com/questions/16719541/integration-testing-hive-jobs). This therefore means, essentially a vagrant based approach, where a local minimal cluster is used as part of the test.

[quintona]

Comment copied from the private repo, by @fs111: Speaking of vagrant, we bascially have everything you need over here: https://github.com/Cascading/vagrant-cascading-hadoop-cluster You would only have to add hive to the set-up and your are done. For simplicity, I would go with the single-node setup: https://github.com/Cascading/vagrant-cascading-hadoop-cluster#single-node-setup

[quintona]

I used this vagrant box when I was originally testing your implementation. However I later discovered that this box was very different in many ways to CDH4.6 and therefore much of testing, and packaging tooling was invalidated. I actually wrote a whole SBT plugin to do build fat jars in a way that would work on this cluster, only to find that sbt assembly worked perfectly on our cluster. So, we will maintain packer, puppet and vagrant scripts on our github page for this purpose, and integrate these into specs2, however it should still be possible to test cascading-hive independently of the layers we introduce...

[fs111]

In Cascading we base everything on apache hadoop, since that is the base for all downstream distributions. CDH 4.x is a very different distro indeed. They ported parts of hadoop 1.x onto hadoop 2.x and their jars are very different as well.

That being said, CDH 5 is compatible with Cascading 2.5, as you can see on our compatibility page: http://www.cascading.org/support/compatibility/