In this post, I'll show you how you can combine the elegance of a WaitGroup with the buffering of channels. This way, you can use a sync.WaitGroup
but still control the concurrency.
A sync.WaitGroup
is a very nice concept in Go and is simple to use. However, the stock sync.WaitGroup
just launches all the
goroutines at once without letting you have any control over the concurrency.
Buffered channels on the other hand allow you to control the concurrency, but the syntax is a bit hard and more verbose than using a sync.WaitGroup
.
In this following code sample, we show a way to combine the best of both by creating a custom WaitGroup
which implements a
goroutine pool and allows us to control the concurrency.
1package waitgroup
2
3import (
4 "sync"
5)
6
7// WaitGroup implements a simple goruntine pool.
8type WaitGroup struct {
9 size int
10 pool chan byte
11 waitGroup sync.WaitGroup
12}
13
14// NewWaitGroup creates a waitgroup with a specific size (the maximum number of
15// goroutines to run at the same time). If you use -1 as the size, all items
16// will run concurrently (just like a normal sync.WaitGroup)
17func NewWaitGroup(size int) *WaitGroup {
18 wg := &WaitGroup{
19 size: size,
20 }
21 if size > 0 {
22 wg.pool = make(chan byte, size)
23 }
24 return wg
25}
26
27// BlockAdd pushes ‘one' into the group. Blocks if the group is full.
28func (wg *WaitGroup) BlockAdd() {
29 if wg.size > 0 {
30 wg.pool <- 1
31 }
32 wg.waitGroup.Add(1)
33}
34
35// Done pops ‘one' out the group.
36func (wg *WaitGroup) Done() {
37 if wg.size > 0 {
38 <-wg.pool
39 }
40 wg.waitGroup.Done()
41}
42
43// Wait waiting the group empty
44func (wg *WaitGroup) Wait() {
45 wg.waitGroup.Wait()
46}
Using it is just like a normal sync.WaitGroup
. The only difference is the initialisation. When you use waitgroup.NewWaitGroup
, you have the option to specify it's size.
Any int
which is bigger than 0
will limit the number of concurrent goroutines. If you specify -1
or 0
, all goroutines will run at once (just like a plain sync.WaitGroup
).
1package main
2
3import (
4 "fmt"
5 "github.com/pieterclaerhout/go-waitgroup"
6)
7
8func main() {
9
10 urls := []string{
11 "https://www.easyjet.com/",
12 "https://www.skyscanner.de/",
13 "https://www.ryanair.com",
14 "https://wizzair.com/",
15 "https://www.swiss.com/",
16 }
17
18 wg := waitgroup.NewWaitGroup(3)
19
20 for _, url := range urls {
21 wg.BlockAdd()
22 go func(url string) {
23 fmt.Println("%s: checking", url)
24 res, err := http.Get(url)
25 if err != nil {
26 fmt.Println("Error: %v")
27 } else {
28 defer res.Body.Close()
29 fmt.Println("%s: result: %v", err)
30 }
31 wg.Done()
32 }(url)
33 }
34
35 wg.Wait()
36 fmt.Println("Finished")
37
38}
You can import the package from github.com/pieterclaerhout/go-waitgroup.
If this post was enjoyable or useful for you, please share it! If you have comments, questions, or feedback, you can email my personal email. To get new posts, subscribe use the RSS feed.