Should ExecutionPlan spawn tasks in execute function

[westonpace]

A contributor of ours (lance) hit a bit of a snag implementing a foreign table provider for datafusion python. It turns out the issue is that we spawn a background thread in ExecutionPlan::execute. The foreign table provider code re-attaches the tokio runtime when it polls the returned stream but it doesn't attach it for the call to execute. I think there are two ways this could be resolved and wanted to see if anyone had strong opinions one way or the other.

1. I could modify my execution plan implementation (in Lance) so that the background thread is not spawned until it is polled for the first time.

2. We could enter the foreign runtime before calling the execute function in datafusion

I guess the more general question is:

Is there an expectation that the ExecutionPlan streams do no background work until first polled?

[westonpace]

This may just be a failure in my ability to read the manual. I now see this in the docs:

Most ExecutionPlans should not do any work before the first RecordBatch is requested from the stream.

That would seem to suggest that spawning background threads in execute is a bad idea.