Make Concurrent Requests in PHP

Here’s how you can perform concurrent HTTP requests in PHP using curl_multi_exec, which allows you to manage multiple cURL sessions concurrently.

Here’s the PHP code:

<?php

function sendRequest($query)
{
    $url = "https://piloterr.com/api/v2/website/crawler";
    $xApiKey = "YOUR-X-API-KEY";
    $fullUrl = $url . "?x_api_key=" . urlencode($xApiKey) . "&query=" . urlencode($query);
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $fullUrl);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $response = curl_exec($ch);
    if (curl_errno($ch)) {
        echo "Request failed: " . curl_error($ch) . "\n";
    } else {
        echo "Response from $query:\n$response\n";
    }
    curl_close($ch);
}
function scrapeConcurrently($urls)
{
    $multiHandle = curl_multi_init();
    $curlHandles = [];
    foreach ($urls as $url) {
        $ch = curl_init();
        $xApiKey = "YOUR-X-API-KEY";
        $fullUrl = "https://piloterr.com/api/v2/website/crawler?x_api_key=" . urlencode($xApiKey) . "&query=" . urlencode($url);
        curl_setopt($ch, CURLOPT_URL, $fullUrl);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_multi_add_handle($multiHandle, $ch);
        $curlHandles[$url] = $ch;
    }
    $active = null;
    do {
        $mrc = curl_multi_exec($multiHandle, $active);
    } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    while ($active && $mrc == CURLM_OK) {
        if (curl_multi_select($multiHandle) != -1) {
            do {
                $mrc = curl_multi_exec($multiHandle, $active);
            } while ($mrc == CURLM_CALL_MULTI_PERFORM);
        }
    }
    foreach ($curlHandles as $url => $ch) {
        $response = curl_multi_getcontent($ch);
        if (curl_errno($ch)) {
            echo "Request to $url failed: " . curl_error($ch) . "\n";
        } else {
            echo "Response from $url:\n$response\n";
        }
        curl_multi_remove_handle($multiHandle, $ch);
        curl_close($ch);
    }
    curl_multi_close($multiHandle);
    echo "Process Ended\n";
}
$urlsToScrape = [
    "https://www.piloterr.com",
    "https://www.piloterr.com/blog"
];
scrapeConcurrently($urlsToScrape);
?>

Explanation:

1. curl_multi_exec: This function is used to manage multiple cURL handles simultaneously. It allows you to send multiple requests concurrently.

2. curl_multi_init and curl_multi_add_handle: These functions are used to initialize and add individual cURL sessions to a multi-cURL handle.

3. curl_multi_exec loop: The loop runs the requests concurrently and waits for them to finish.

4. curl_multi_getcontent: Retrieves the response for each individual request once it’s done.

How to Use:

• Replace "YOUR-X-API-KEY" with your actual API key.

• Update the $urlsToScrape array with the URLs you want to scrape.

Benefits:

• This approach allows you to perform multiple HTTP requests concurrently in PHP, which can significantly speed up your scraping operations.

• It’s easy to adapt this solution to handle more requests or to include additional options for each cURL session.

This PHP code provides an efficient way to handle concurrent HTTP requests, making it suitable for scenarios where performance is a key consideration.