I've found that EFS is expensive when running this pipeline within AWS. I would love another alternative where each batch job writes to S3, and then downstream jobs pull the necessary files. I've tried using mount-s3 instead of EFS (I mounted an S3 location to the default EFS location), but the script tries to create symlinks which is not compatible with mount-s3. If this could be changed, mount-s3 could also be a great solution.
Let me know if there are any plans for these options in the future! Thanks!
I've found that EFS is expensive when running this pipeline within AWS. I would love another alternative where each batch job writes to S3, and then downstream jobs pull the necessary files. I've tried using mount-s3 instead of EFS (I mounted an S3 location to the default EFS location), but the script tries to create symlinks which is not compatible with mount-s3. If this could be changed, mount-s3 could also be a great solution.
Let me know if there are any plans for these options in the future! Thanks!