ReactPHP Implementation for Concurrent API Requests

[theihasan]

📝 Why is PR is required?

This PR implements an asynchronous, non-blocking approach to API requests using ReactPHP, which significantly improves the performance and efficiency of job data fetching. The previous implementation used synchronous HTTP requests, which led to slow performance and inefficient resource utilization when fetching job data across multiple categories and countries.

✅ Checklist

• My code follows the project's contribution guidelines.
• I have performed a self-review of my code.
• I have added tests that prove my fix is effective or that my feature works.
• I have tested this code locally.

🛠️ Changes

• Feature (non-breaking change that adds functionality)
• Refactor (code improvement without changing external behavior)

📌 Related Issues

None

📸 Screenshots

None

📣 Additional Comments

Overview

This PR introduces ReactPHP to handle concurrent API requests in the GetJobData job. The implementation uses ReactPHP's event loop, promises, and HTTP client to make non-blocking API requests, which allows for better performance and resource utilization when fetching job data from external APIs.

Key Changes

1. Asynchronous HTTP Requests : Replaced Laravel's synchronous HTTP client with ReactPHP's asynchronous Browser for making API requests.
2. Controlled Concurrency : Implemented a mechanism to limit the number of concurrent requests (default: 5) to prevent overwhelming the API and respect rate limits.
3. Event Loop Management : Added proper event loop handling to process batches of requests efficiently and ensure all promises are resolved.
4. Error Handling : Implemented comprehensive error handling for API responses, including rate limit detection, JSON parsing errors, and general request failures.

Benefits

1. Improved Performance : Asynchronous requests significantly reduce the total time needed to fetch job data across multiple categories and countries.
2. Better Resource Utilization : Non-blocking I/O allows the server to handle more requests with fewer resources.
3. Scalability : The system can now handle a larger number of API requests without performance degradation.
4. Rate Limit Management : Better control over API request rates with the ability to process requests in batches.
5. Detailed Monitoring : Added comprehensive logging throughout the request lifecycle for better debugging and monitoring.

Technical Implementation

The implementation uses:

• React\EventLoop\Loop for managing the event loop
• React\Http\Browser for making HTTP requests
• React\Promise\all for handling multiple promises
  The code processes API requests in batches, limiting the number of concurrent requests to avoid overwhelming the API. Each batch is processed through the event loop before moving to the next batch. After all batches are processed, any remaining promises are resolved.

Added Dependencies

• react/event-loop : ^1.5
• react/http : ^1.11

Testing

The implementation has been tested with multiple job categories and countries, confirming:

• Successful API requests with proper headers
• Correct handling of rate limits
• Proper processing of API responses
• Accurate updating of API key usage information

Next Steps

1. Consider implementing retry mechanisms for failed requests
2. Add more granular control over concurrency based on API provider capabilities
3. Implement circuit breaker pattern for better fault tolerance

Related Issues

None