sufficiently large event files cause EOFerror on pickle load

[mmljcollier]

Given a sufficiently large event file, attempting to load the eventID pickle causes a crash. The workaround is to forcibly rotate logfiles (I moved all mine to /var/logs/shotgunEventDaemon/old) and restart.

(Note that I am testing locally and we haven't gone into production yet, so the first traceback is for 'you haven't configured any email alerts' and can be safely disregarded. It's the actual crash I'm interested in. If it helps, I promise to configure email alerts when everything's sorted ;)

Some sort of automated log rotation or other functionality to avoid that on pickle load would be great. Thanks for your attention and support.

mcm004:src jcollier$ sudo ./shotgunEventDaemon.py foreground
Password:
INFO:engine:Using Shotgun version 3.0.17
INFO:engine:Loading plugin at /usr/local/shotgun/shotgunEvents/plugins/logArgs.py
INFO:engine:Loading plugin at /usr/local/shotgun/shotgunEvents/plugins/statusFinalShotOnApproval.py
Traceback (most recent call last):
File "./shotgunEventDaemon.py", line 983, in emit
smtp.connect(self.mailhost, port)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/smtplib.py", line 310, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/smtplib.py", line 285, in _get_socket
return socket.create_connection((host, port), timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno 8] nodename nor servname provided, or not known

Logged from file shotgunEventDaemon.py, line 323
CRITICAL:engine:Crash!!!!! Unexpected error (<type 'exceptions.EOFError'>) in main loop.

Traceback (most recent call last):
File "./shotgunEventDaemon.py", line 316, in start
self._loadEventIdData()
File "./shotgunEventDaemon.py", line 340, in _loadEventIdData
self._eventIdData = pickle.load(fh)
EOFError

[cman122887]

Thanks for posting this. I was getting the same thing and was trying to figure out what was wrong with my event file. Yeah +1: auto log rotation

edit: actually im still getting the error after manually rotating my log files. still looking into why

[yawpitch]

This problem is likely due to the way the file is being opened, both on read and write. In all of the calls to open(eventIdFile) on lines 338, 354, and 484 of the current master branch the default "text" mode is used, instead of specifying "binary" mode. End result is an eventFile that is not cross-platform, and any eventIdFile written on Linux will cause an EOFError when read on Windows, due to the difference in newline characters ('\n', '\r\n' respectively). The two read calls should read open(eventFile, "rb") and the one write call should read open(eventFile, "wb") for cross-platform compatibility.

This is exacerbated by the single call to pickle.dump (on like 485 of the current master) being made using the default pickle protocol 0 ... that adds a LOT of newlines to the quasi-human-readable output of the very old protocol. With the fixes above and file IO being in binary mode performance and reliability would be improved by changing this call to pickle.dump(self._eventIdData, fh, protocol=pickle.HIGHEST_PROTOCOL) ... this has been a stable change since Python 2.3, and since this code is only meant to support >= 2.4 such an update should be non-problematic, with the caveat that the resulting file would NOT be even partially human readable.

[tk421storm]

For me, this error was occuring because I had an event id file as the last event, but it was empty. Changing the read/write mode to binary as michaelmovies suggested did not fix this error. For me, deleting the empty eventID file was the only thing that fixed the problem. This should probably be handled somehow - it needs to check if the file is empty, and if so, skip checking for an eventID.

For me, this error was occuring because I had an event id file as the last event, but it was empty. Changing the read/write mode to binary as michaelmovies suggested did not fix this error. For me, deleting the empty eventID file was the only thing that fixed the problem. This should probably be handled somehow - it needs to check if the file is empty, and if so, skip checking for an eventID.

Fix proposed in #64