Hydra pipeline #22
Description
Review the pipeline processing system in Hydra, writing up the purpose and needs of this system. Consider if/how this system could be run in isolation from the rest of the Hydra stack in order to manage work from multiple sites. Consider how AWS services such as SQS, SNS, Elastic Beanstalk, Container Service, Lambda, and/or Simple Workflow could be leveraged to manage the execution of pipeline processing tasks. Write up a recommendation for further investigation with the services and techniques that appear to be the best fit
Goal:
A plan for further investigation of ways to handle pipeline processing in a distributed environment