Skip to content

fix: Reduce the chance of CurlEventPublisher exhausting file descriptors on high throughput PHP-FPM servers #216

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 25, 2025

Conversation

Watercycle
Copy link
Contributor

@Watercycle Watercycle commented Mar 13, 2025

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions

Related issues

Describe the solution you've provided

TLDR: --connect-timeout vs --max-time (docs)


Consider the following code:

function curl()
{
    shell_exec("/usr/bin/env curl -X POST --connect-timeout 3 -H 'Content-Type: application/json' -H 'Accept: application/json' https://reqres.in/api/users?delay=30" . " >> /dev/null 2>&1 &");
}

+

while true; do   echo "$(date +"%H:%M:%S") - Open files: $(lsof | wc -l)";   sleep 0.5; done

However long the request takes is how long file descriptors hang open for.

Web apps like Laravel frequently use PHP-FPM as part of their server infrastructure where each request is a new process. As a result, on high throughput servers, EventProcessor.php::__destruct gets invoked quite frequently. Having many child worker processes seems to also mean new sub-processes consuming many file descriptors. While the Relay Proxy is great at allowing the PHP SDK to quickly read from data stores like Redis, the impressions event processing system ends up quickly bottle-necking if there's any slow-down in publishing events.

Describe alternatives you've considered

I think ideally:

  1. CurlEventPublisher should use the actual curl client or something like Guzzle for making async requests instead of using shell_exec.
  2. That CurlEventPublisher should possibly explore batch requesting strategies that won't choke out technologies like PHP-FPM (e.g. using the file system + a separate worker thread). With more consideration for making sure these processes terminate - like logging warnings when it happens.

I ultimately stuck with a simple change to minimize the chance of it being a breaking change.

Additional context

This is related to an issue I filed a bit ago: https://support.launchdarkly.com/hc/en-us/requests/87721

See https://curl.se/docs/manpage.html - this helps make sure background processes aren't piling up.
@Watercycle Watercycle requested a review from a team as a code owner March 13, 2025 19:34
@Watercycle Watercycle changed the title bug: Reduce the chance of CurlEventPublisher exhausting file descriptors on high throughput PHP-FPM servers Reduce the chance of CurlEventPublisher exhausting file descriptors on high throughput PHP-FPM servers Mar 13, 2025
@Watercycle Watercycle changed the title Reduce the chance of CurlEventPublisher exhausting file descriptors on high throughput PHP-FPM servers fix: Reduce the chance of CurlEventPublisher exhausting file descriptors on high throughput PHP-FPM servers Mar 13, 2025
@@ -80,7 +80,7 @@ private function createCurlArgs(string $payload): string
{
$scheme = $this->_ssl ? "https://" : "http://";
$args = " -X POST";
$args.= " --connect-timeout " . $this->_connectTimeout;
$args.= " --max-time " . $this->_connectTimeout;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LDClient config already allows specifying both a connect_timeout and a max length timeout parameter. We use that timeout parameter in the existing Guzzle client.

Can you update this so that we still support the connect time parameter as before, and also add the max time flag hooked up to that value?

@keelerm84 keelerm84 merged commit 1c969be into launchdarkly:main Mar 25, 2025
7 checks passed
@keelerm84
Copy link
Member

Thank you @Watercycle for the contribution. I went ahead and made that minor modification. I appreciate your time and effort on this work.

abarker-launchdarkly pushed a commit that referenced this pull request Mar 25, 2025
🤖 I have created a release *beep* *boop*
---


##
[6.5.2](6.5.1...6.5.2)
(2025-03-25)


### Bug Fixes

* Honor `timeout` configuration for CurlEventPublisher
([#216](#216))
([1c969be](1c969be))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants