nutch-aws EMR issues

[malhaar2]

Hi,

We have setup nutch on amazon EMR using below tutorials

https://github.com/eleflow/nutch-aws

The problem is we are not able to copy crawl file from EMR to bucket.

Can anyone please help how to check the directory for crawl file in EMR

[maggias]

Having the same problem, did you find a solution?

[paulomagalhaes]

I believe this is because the aws-cli api changed a lot since I wrote this. For instance, to upload files to a bucket it used to be:
aws s3 put-object <file> s3://<bucket>
but now it is:
aws s3 cp <file> s3://<bucket>

[emanuelecasadio]

I am working on a fork that is using the new AWS CLI. You can find the work in progress here: https://github.com/emanuelecasadio/nutch-aws. At the end I will open a Pull Request on this repo. :)

At this moment, the cluster creation part and the ssh connection are working correctly. The first two operation are completed succesfully, but the third (on a total of five) fails. I still have to verify if this is a problem of my version of the Makefile.

EDIT: The fork is fully working now but I'm still testing it.