Create async version of curl_init_pooled

[eryi]

HHVM Version

3.18.1

Standalone code, or other way to reproduce the problem

<?hh

\HH\curl_create_pool('mypool', 1, 100000);

$awaitables = array_map(async ($index) ==> {
  $curl = curl_init_pooled('mypool', "http://google.com");
  print "REQUESTING $index\n";
  await HH\Asio\curl_exec($curl);
  print "FINISHED $index\n";
}, range(1,5));

\HH\Asio\join(\HH\Asio\v($awaitables));

Expected result

REQUESTING 1
FINISHED 1
REQUESTING 2
FINISHED 2
REQUESTING 3
FINISHED 3
REQUESTING 4
FINISHED 4
REQUESTING 5
FINISHED 5

Actual result

REQUESTING 1
(wait for 100000 ms here)
FINISHED 1

Fatal error: Uncaught exception 'RuntimeException' with message 'Timeout reached waiting for an available pooled curl connection!'

[eryi]

I'm guessing that the 2nd curl_init_pooled is blocking the 1st curl_exec from completing. Is there an awaitable version of curl_init_pooled?

[mofarrell]

It looks like your guess is correct. The code looks like it politely gives way to other threads, but it is not async aware. https://github.com/facebook/hhvm/blob/master/hphp/runtime/ext/curl/curl-pool.cpp#L79

Making a similar version that uses some of the async rescheduling constructs could be interesting.

[fredemmott]

There have been some discussions about if HH\Asio\curl_exec() is:

• sound
• actually useful: given the thread overhead, curl_multi_exec() + await HH\Asio\usleep() can actually be more efficient

It might be better to deprecate curl_multi_await(), and remove HH\Asio\curl_exec()'s usage of it.

I'm not sure who the expert is here - probably one of @dneiter, @KendallHopkins, @periodic1236, or @FBNeal

[fredemmott]

So, that expected result isn't going to happen, regardless of what's done for async - e.g. this has the huge wait between printing 1 and 2:

<?hh

\HH\curl_create_pool('mypool', 1, 100000);

$curls = [];
var_dump(0);
$curls[] = \HH\curl_init_pooled('mypool', 'http://example.com');
var_dump(1);
$curls[] = \HH\curl_init_pooled('mypool', 'http://example.com');
var_dump(2);

The pool size limits the amount of available entries in the pool.

The actual pool already interoperates fine with async, as you can see by icnreasing the pool size above one in your example.

[eryi]

Reading your last comment, I'm not sure if we are on the same page. Allow me to clarify.

\HH\curl_init_pooled blocks the main thread if the pool is at its limit. When \HH\Asio\curl_exec from a different main thread completes, \HH\curl_init_pooled in the original thread returns. The underlying assumption is that many different main threads share a single pool.

In my example, a single main thread creates sufficient entries to exhaust the pool. When \HH\curl_init_pooled blocks in that main thread, none of the other \HH\Asio\curl_exec can complete, causing the \HH\curl_init_pooled to block forever.

The behavior I described is the expected behavior of \HH\curl_init_pooled and this is not a bug. However, I wouldn't consider blocking forever "interoperates fine with async."

Having an async version of \HH\curl_init_pooled will allow me to easily curl a large list of urls from a single thread while limiting the number of simultaneous connections. I can't think of any way to implement this since the semaphore functions are not async as well.

[turadg]

Regarding this earlier comment,

It might be better to deprecate curl_multi_await(), and remove HH\Asio\curl_exec()'s usage of it.

Latest from the HackLang slack,

The known issues were resolved in 3.28 with 93c31ec