Add Logic/Message for Files too large to be previewed well

[djbrooke]

We're testing a few larger files on Harvard Dataverse and they are not loading in the previewers:

• https://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/DZBFRG/XIYRIK
• https://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/HOKKOM/1CILTK

I linked two PDFs above, but there are files of different types that this also happens for.

I was hoping we could edit the existing error message to include some words about the file being too large, but it never loads enough to view it.

We could consider a pre-load check on file size before attempting to load a preview, and instead show a message that the file is too large to preview, or something else.

Suggestions welcome!

[qqmyers]

Hmm - both ~work for me. The first showed up after ~2 minutes. The second took more like 5 minutes and displayed some lines of the file before showing the script paused and a 'Paused before potential out-of-memory crash' message in the console.

My guess, if you're not seeing the error with an HTTP status code is that the download works and it is then some memory limit on the local machine/browser that is the issue. (might be worth checking the browser console though). If that's the case, it may be hard to set a good limit for all previewers - they could vary in how much memory they require for a given size file (e.g. a video player will stream).

Options I can see:

• Set a limit in Dataverse for when the preview gets embedded - could be low so that the page loads quickly and would still allow someone to try the external tool button for larger files.
• Set a limit in the external tools manifest when you register, so Dataverse will refuse to launch a tool on a file larger than the limit.
• Add some 'progress' info for previewers - for any that use the common code to download the file bytes, it would be possible to show download progress anyway. Showing progress after the download is complete would be previewer specific and, for third-party ones, could be hard to get (if they don't already show it in their view). This would stop the error, but would let people know if they might want to keep waiting or not.
• See if there's a way to catch out-of-memory/script paused type of errors. (In my case, there was a display in the main window, but that could have been because I had the console open as well - seems like you didn't see any indication unless it was just very slow)
• Making a limit in the previewers overall, or per previewer, would also be possible, but - that catches things too late to not show an empty preview window and would probably have to be conservative and therefore would stop some people from previewing when it would work for them. That said, I think because the previewer code retrieves metadata, I think the file size is available to do this.

In any case, problems like this could be used to drive previewer advances - there are certainly spreadsheet viewers that incrementally load data rather than retrieving and parsing the whole file to start. If there are enough GB spreadsheets, I'm sure they could be handled with some additional work.

[TaniaSchlatter]

I ran into this when trying to preview this file. I'm very interested in a progress indicator for previews. People will wait if they know something is coming.