🐥 Using errgroup with SetLimit in Golang

In a previous post, I discussed how you can use sync.WaitGroup with buffered channels to control concurrency.

Today, I learned that there is an easier way to do this. If you are using the errgroup package, it's as easy as just calling the SetLimit function to control the concurrency.

package main

import (
    "context"
    "fmt"
    "net/http"

    "golang.org/x/sync/errgroup"
)

func main() {

    urls := []string{
        "https://www.easyjet.com/",
        "https://www.skyscanner.de/",
        "https://www.ryanair.com",
        "https://wizzair.com/",
        "https://www.swiss.com/",
    }

    ctx := context.Background()
    g, _ := errgroup.WithContext(ctx)

    // See https://pkg.go.dev/golang.org/x/sync/errgroup#Group.SetLimit
    g.SetLimit(3)

    for _, url := range urls {
        url := url // https://golang.org/doc/faq#closures_and_goroutines
        g.Go(func() error {
            fmt.Printf("%s: checking\n", url)
            res, err := http.Get(url)
            if err != nil {
                return err
            }

            defer res.Body.Close()

            return nil
        })
    }

    if err := g.Wait(); err != nil {
        fmt.Printf("Error: %v", err)
        return
    }

    fmt.Println("Successfully fetched all URLs.")
}

If you want to learn more about errgroup, there's a lot of information about it on:

• Learning Go Concurrency Patterns Using Errgroup Package
• errgroup - GoDoc

If you want to do the same in Python, you can find more information here.