feat/Guard against excessive memory usage when partitioning PDFs

[flash1293]

Is your feature request related to a problem? Please describe.
Using the OCR strategy when partitioning PDFs, processing of some PDF files will allocate a large amount of memory that isn't available in all environments (e.g. when running via Google cloud run with limited resources).

For example, the following 23MB PDF causes memory usage of >10GB when partitioning: https://drive.google.com/file/d/1lr-Pwh3QTVfdY4F6R-fk4tVU9FNSK27p/view?usp=sharing

Describe the solution you'd like

Unstructured should employ sensitive defaults to avoid this kind of situations (e.g. a max size of a page when rendered in memory). This could also be configurable as optional argument on the partitioning method.

In cases where this isn't feasible, the partitioning method should raise a descriptive exception so the caller can handle the situation gracefully instead of crashing the process.

The most important aspect is giving a way to limit the amount of memory unstructured will use during partitioning.

Describe alternatives you've considered

Alternatively, the partitioning can be run in a separate memory-limited process which is controlled by another process. In case the partitioning process runs out of memory, the orchestration process can handle the situation.

[MthwRobinson]

@christinestraub - I know you've added some code (like batching pages for PDF to image conversion) to limit excessive memory usage. Was there anything else you had in mind to help with that? Think this would be a good topic to follow up on.

[ImadSaddik]

Any update ?