Getting started

worker-pool runs jobs on a pool of goroutines fed by one bounded queue. It has three parts:

The queue. Delegate adds a job, and waits while the queue is full.
Workers. AddWorker starts a goroutine that takes jobs one at a time and calls a func with each.
Stop. Stop runs every job still queued on the workers left, and waits for them to exit.

Install

worker-pool needs Go 1.26 or newer.

go get github.com/vardius/worker-pool/v3

Your first pool

This program prints the SHA-256 of every file it is given, hashing one file per CPU at once. A job is the index of a file, so each worker writes its own result and nothing needs a lock:

// Command checksum prints the SHA-256 of every file named on its command line, hashing one file per CPU at once.
//
//  go run ./examples/checksum go.mod README.md
package main

import (
  "context"
  "crypto/sha256"
  "encoding/hex"
  "fmt"
  "io"
  "log"
  "os"
  "os/signal"
  "runtime"

  "github.com/vardius/worker-pool/v3"
)

// A sum is a file's SHA-256, or the error that kept it from being read.
type sum struct {
  hex string
  err error
}

// checksums hashes the files on a pool of workers; sums[i] is for paths[i]. Once ctx is done,
// it queues no more files, waits for the ones already queued, and returns ctx's error.
func checksums(ctx context.Context, paths []string, workers int) ([]sum, error) {
  sums := make([]sum, len(paths))

  // A job is the index of a path. Each one is run once, by one worker, so the writes to sums need no lock.
  pool := workerpool.New[int](workers)
  for range workers {
    pool.AddWorker(func(i int) { sums[i].hex, sums[i].err = hashFile(paths[i]) })
  }
  for i := range paths {
    if err := pool.Delegate(ctx, i); err != nil {
      pool.Stop()
      return nil, err
    }
  }
  pool.Stop() // returns once every queued file is hashed

  return sums, nil
}

func hashFile(path string) (string, error) {
  f, err := os.Open(path)
  if err != nil {
    return "", err
  }
  defer f.Close()

  h := sha256.New()
  if _, err := io.Copy(h, f); err != nil {
    return "", fmt.Errorf("read %s: %w", path, err)
  }
  return hex.EncodeToString(h.Sum(nil)), nil
}

func main() {
  log.SetFlags(0)
  ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
  defer stop()
  context.AfterFunc(ctx, stop) // after the first Ctrl-C, a second one ends the program at once

  paths := os.Args[1:]
  sums, err := checksums(ctx, paths, runtime.NumCPU())
  if err != nil {
    log.Fatal(err)
  }

  failed := false
  for i, s := range sums {
    if s.err != nil {
      log.Print(s.err)
      failed = true
      continue
    }
    fmt.Printf("%s  %s\n", s.hex, paths[i])
  }
  if failed {
    os.Exit(1)
  }
}

Run it

From a clone of the repository, hash a few files:
Terminal window
```
go run ./examples/checksum go.mod README.md LICENSE.md
```
It prints one line per file, in the order you gave them, whichever worker hashed it.
Add a file that doesn’t exist. The others are still hashed, and the error is printed on its own line:
Terminal window
```
go run ./examples/checksum go.mod no-such-file
```
```
open no-such-file: no such file or directory
```

What just happened

workerpool.New[int](workers) made a pool whose jobs are ints, with room for one queued job per worker.
AddWorker started one worker per CPU, each calling the same func.
Delegate queued every index. Whenever the queue was full, it waited for a worker to take a job. Press Ctrl-C while files are still being queued and its context is done: Delegate returns the context’s error, the files already queued are hashed, and the program exits with “context canceled”. Once every file is queued, the run finishes what it has; a second Ctrl-C ends it at once.
Stop ran what was still queued and returned once every worker had exited. Only then were the sums read, so they needed no lock.

Next steps

The queueWhat Delegate does when the queue is full, the context is done, or the pool is stopped.

WorkersAdd and remove workers at any time, each with its own func.

StoppingFinish the queue, wait for the workers, and shut down cleanly.

BenchmarksWhat a job costs, and how v3 compares with v2.