Make Concurrent Requests in Go

Here’s how you can perform concurrent HTTP requests in Go, similar to the examples shown in Ruby, Python, and Node.js. We will use goroutines and the sync.WaitGroup to handle concurrent requests.

Here’s the Go code:

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "sync"
)
func sendRequest(query string, wg *sync.WaitGroup) {
    defer wg.Done()
    url := "https://piloterr.com/api/v2/website/crawler"
    xApiKey := "YOUR-X-API-KEY"
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        log.Fatalf("Failed to create request: %v", err)
    }
    q := req.URL.Query()
    q.Add("x_api_key", xApiKey)
    q.Add("query", query)
    req.URL.RawQuery = q.Encode()
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Printf("HTTP Request failed: %v", err)
        return
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        log.Printf("Failed to read response body: %v", err)
        return
    }
    fmt.Printf("Response from %s:\n%s\n", query, body)
}
func main() {
    var wg sync.WaitGroup
    urlsToScrape := []string{
        "https://www.piloterr.com",
        "https://www.piloterr.com/blog",
    }
    for _, url := range urlsToScrape {
        wg.Add(1)
        go sendRequest(url, &wg)
    }
    wg.Wait()
    fmt.Println("Process Ended")
}

Explanation:

1. Goroutines: Go’s built-in concurrency mechanism is used here. Each request is handled within its own goroutine, allowing them to run concurrently.

2. sync.WaitGroup: This is used to manage and synchronize the completion of all the goroutines. Each request adds to the WaitGroup, and once the request is complete, the Done() method is called to signal completion.

3. http.NewRequest: Used to construct the HTTP GET request with query parameters for the API key and the target URL.

How to Use:

• Replace "YOUR-X-API-KEY" with your actual API key.

• Modify the urlsToScrape array to include the URLs you want to scrape.

Benefits:

• Go’s concurrency model is lightweight and highly efficient for handling a large number of requests.

• This approach is easily scalable, allowing you to add more URLs or adjust the concurrency level as needed.

This Go example provides a robust solution for making concurrent HTTP requests, ideal for scenarios where high-performance scraping is needed.