List big buckets fix

[zakalibit]

When trying to list a very large bucket ~1M objects s3cmd runs out of memory.

[fviard]

Thank you for your contribution.
The problem that you are trying to fix is a real one, I agree, but there is a practical problem with your PR:
With your change, the ls command will alternatively list "prefixes" (ie "folders") and "files".
In the current case it will create a regression, but also if it was for a new command, the behavior would still be strange.

It is probably why the ls command was not modified to use directly the "streaming list".

[zakalibit]

Thank you for looking, I would not call that behaviour strange, maybe unexpected. It is still consumable, it is way stranger to run out of memory and coredump :)
Maybe we can look at few other options:

• streaming command line option, or automatic one depending on the size of the bucket (and maybe free memory on the system)
• write prefixes and file data to tmp files, then print from there, instead of gathering in memory, still can run out of disk space, but probably can be monitored, so it would stop at the right time

Then out of curiosity I looked at aws cli code and it seems that it does stream and interleaves folders with files output
https://github.com/aws/aws-cli/blob/be17675af6b1e26fea2840dd7ac9006f30805d7a/awscli/customizations/s3/subcommands.py#L456