MemoryError on RaspberryPi

[timmalich]

I didn't found an issue for this already:

I guess you are loading the whole file into ram before uploading. On small devices like raspberry you can run easily into an out of memory exception. At least that seams to be the issue on my device. I've attached the trace output:

---

2016-05-30 19:25:16,713 uploadr.py:532 - uploadImage() Upload failed
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 515, in uploadImage
res = getResponse(url)
File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 102, in getResponse
data = flickr.unmarshal(minidom.parse(urllib2.urlopen(url)))
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1194, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/usr/lib/python2.7/httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 895, in _send_output
msg += message_body

MemoryError

BTW: Thanks for the good and easy tool. 👍 :)

[richq]

Wow, yeah, this is what happens. Who knew? The uploadImage method has an open().read() there to suck up all the file, then encodes using the multipart protocol. But the whole file is read in. Maybe I should do some fancy size limit thing based on machine RAM. Flickr itself I think has a ~1Gb limit for big movie files.

The solution to this seems tricky and would require using another 3rd party lib https://stackoverflow.com/questions/15973204 Not sure what to do here :-S

[timmalich]

I guess at least fancy memory thing would be nice. I didn't got the stackoverflow hint either.
Furthermore I'm now facing the issue that the script stops after a while. I guess it's related to the same memory bug.
It seams to stop one special movie but I it doesn't gimme a nice error msg. It just says:

---

2016-06-01 18:43:14,469 uploadr.py:498 - uploadImage() Uploading image /FOO/GOPR0315.MP4 with tags "#2015/2015 FOO.MP4" "2015" "2015 FOOO"

Killed

The error.log doesn't provide a timestamp so I only can guess that the last error message has something to do with it:

---

Upload failed
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 514, in uploadImage
url = buildRequest(api.upload, d, (photo,))
File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 158, in buildRequest
contenttype, body = encodeMultipartFormdata(fields, files)
File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 147, in encodeMultipartFormdata
body = crlf.join(L)

MemoryError

[en5ads]

Does this help?
https://toolbelt.readthedocs.io/en/latest/uploading-data.html#streaming-multipart-data-encoder