Restore batch processing and make it optional

[andybroomfield]

Could we consider restoring the batch processing? For many small PDFs I don't think the additional plugins should increase the process time, and it is a lot more understandable. This could either be an option on the form or on the process pipeline to import via batch, or maybe something on the import view to process via the UI (like search api).

[rupertj]

I'm in favour of this. I deliberately didn't rip out all the batch code, because I thought it'd still be useful under some circumstances.

What makes it tricky is letting people know which option is appropriate to use. Batch UI fails for us in these scenarios:

1. The PDF takes too long to parse.
2. The AI request takes too long to return.

I've only seen the first case on a 600 page PDF. OF course, we don't know how many pages a PDF has until it's parsed, so we can't make the decision to use the batch or not based on the number of pages. I did think about having something like "Try the batch first and if it fails, fall back to background processing", but batch UI doesn't seem to have a nice failure mode for requests that time out.

Then for case 2, we have no idea how long the AI's going to take. I'm pretty sure it scales with the length of the text we send, but we have no way to know how long that is until we're well into the import process already.

But for the moment, we could offer the choice, and kick the user off into the batch based on that choice. Doing it on the upload form would make sense to me, and feels like the simplest way to implement it.