Testing

Tests are written against behaviour you can name: “a stale version conflicts”, “a code works once”, “another user gets a 404”. Each layer has its own kind of test, and the fast ones need no Docker.

Commands

Command	Runs
`go test -short ./...`	Everything except tests that start containers. No Docker needed.
`make test`	`go test -race -shuffle=on ./...`: everything, in random order, with the race detector, including the MySQL and MongoDB containers.
`make check`	What CI runs, except the container and end-to-end tests: formatting, `go fix`, vet and staticcheck for every store build and the end-to-end tests, the short tests with the race detector, the proto lint and generated-file check, the web app’s checks and the Helm chart’s.
`make e2e`	The end-to-end suite, against a stack started with `make up`.
`make e2e-web`	The Playwright suite, against the same stack.

CI runs the short tests once per store build (memory, persistence_mysql and persistence_mongodb), and the container tests in a job per database.

Aggregates: Given, When, Then

An aggregate test says what happened before, what is tried now, and which events must result, or which error and no events:

func TestScenario(t *testing.T) {
  id := uuid.New()
  estest.Given(t, &counter{}, opened{ID: id}, incremented{By: 4}).
    When(func(c *counter) error { return c.increment(6) }).
    Then(incremented{By: 6})

  estest.Given(t, &counter{}, opened{ID: id}, incremented{By: 4}).
    When(func(c *counter) error { return c.increment(7) }).
    ThenError(errLimit)
}

The helper is a few lines in estest:

// Given replays history into a, a new aggregate.
func Given[A es.Aggregate](t testing.TB, a A, history ...any) *Scenario[A] {
  t.Helper()
  if err := es.Replay(a, history...); err != nil {
    t.Fatalf("replay history: %v", err)
  }
  return &Scenario[A]{t: t, a: a}
}

// When runs the action under test on the aggregate.
func (s *Scenario[A]) When(action func(A) error) *Scenario[A] {
  s.err = action(s.a)
  return s
}

// Then checks that the action succeeded and raised exactly want, in order.
func (s *Scenario[A]) Then(want ...any) A {
  s.t.Helper()
  if s.err != nil {
    s.t.Fatalf("action failed: %v", s.err)
  }
  got := es.Pending(s.a)
  if len(got) != len(want) || (len(want) > 0 && !reflect.DeepEqual(got, want)) {
    s.t.Fatalf("raised events:\n got %#v\nwant %#v", got, want)
  }
  return s.a
}

// ThenError checks that the action failed with an error matching target
// (errors.Is) and raised nothing.
func (s *Scenario[A]) ThenError(target error) {
  s.t.Helper()
  if !errors.Is(s.err, target) {
    s.t.Fatalf("error = %v, want %v", s.err, target)
  }
  if got := es.Pending(s.a); len(got) > 0 {
    s.t.Fatalf("a failed action raised %#v", got)
  }
}

Write one per rule, for both outcomes. They touch no store, so there is no setup to share and nothing to clean up.

Repositories and concurrency

Repository tests run on the memory backend and check what the aggregate test can’t: versions, conflicts and rollbacks.

func TestRepositoryConflict(t *testing.T) {
  repo, db, _ := newRepository(nil)
  id := uuid.New()
  c := &counter{}
  _ = es.Raise(c, opened{ID: id})
  if _, err := save(t.Context(), db, repo, c); err != nil {
    t.Fatal(err)
  }

  first, _ := repo.Load(t.Context(), id)
  second, _ := repo.Load(t.Context(), id)
  _ = first.increment(1)
  _ = second.increment(1)
  if _, err := save(t.Context(), db, repo, first); err != nil {
    t.Fatal(err)
  }
  _, err := save(t.Context(), db, repo, second)
  if !errors.Is(err, es.ErrConcurrency) || !errors.Is(err, apperr.ErrConflict) {
    t.Fatalf("second writer: err = %v, want ErrConcurrency, an ErrConflict", err)
  }
}

Contract suites

Every store interface has one suite, and every backend runs it. The suite is where a rule lives; a backend passes it or isn’t done.

// OpenStore returns an empty store and the unit of work its transactions
// come from. Each call must return a store that shares no data with others.
type OpenStore func(t *testing.T) (es.Store, uow.UnitOfWork)

// TestStore checks the rules every es.Store must follow. Each backend's
// tests call it:
//
//  func TestEventStore(t *testing.T) {
//    estest.TestStore(t, func(t *testing.T) (es.Store, uow.UnitOfWork) {
//      db := memdb.New()
//      return memdb.NewEventStore(db), db
//    })
//  }
func TestStore(t *testing.T, open OpenStore) {
  t.Run("AppendThenLoad", func(t *testing.T) { testAppendThenLoad(t, open) })
  t.Run("AppendContinuesStream", func(t *testing.T) { testAppendContinuesStream(t, open) })
  t.Run("StaleVersionConflicts", func(t *testing.T) { testVersionConflicts(t, open, 1) })
  t.Run("FutureVersionConflicts", func(t *testing.T) { testVersionConflicts(t, open, 3) })
  t.Run("ConcurrentAppendsOneWins", func(t *testing.T) { testConcurrentAppends(t, open) })
  t.Run("RollbackDiscardsAppend", func(t *testing.T) { testRollback(t, open) })
  t.Run("UnknownStreamIsEmpty", func(t *testing.T) { testUnknownStream(t, open) })
  t.Run("StreamsAreIndependent", func(t *testing.T) { testIndependentStreams(t, open) })
}

func TestEventStore(t *testing.T) {
  estest.TestStore(t, func(*testing.T) (es.Store, uow.UnitOfWork) {
    db := memdb.New()
    return memdb.NewEventStore(db), db
  })
}

func TestOutbox(t *testing.T) {
  outboxtest.TestStore(t, func(*testing.T) (outbox.Store, uow.UnitOfWork) {
    db := memdb.New()
    return memdb.NewOutbox(db), db
  })
}

The MySQL and MongoDB runs use testcontainers: one container per test binary, a fresh database per test. They skip under -short.

Time without sleeping

Code that reads the time takes a Now func() time.Time, and tests move it forward:

func TestRelayRetriesUntilDelivered(t *testing.T) {
  f := newFixture(t)
  var calls []int
  var delivered []uuid.UUID
  f.handle = func(_ context.Context, m outbox.Message) error {
    calls = append(calls, m.Attempts)
    if len(calls) < 3 {
      return errors.New("smtp: connection refused")
    }
    delivered = append(delivered, m.ID)
    return nil
  }
  m := f.add(t, "mail")

  f.drain(t) // fails, retry in 1s
  f.drain(t) // not due yet
  f.now = f.now.Add(outbox.Backoff(1))
  f.drain(t) // fails, retry in 2s
  f.now = f.now.Add(outbox.Backoff(2))
  f.drain(t) // succeeds
  f.now = f.now.Add(time.Hour)
  f.drain(t) // nothing left

  if !slices.Equal(calls, []int{1, 2, 3}) {
    t.Errorf("handler saw attempts %v, want [1 2 3]", calls)
  }
  if !slices.Equal(delivered, []uuid.UUID{m.ID}) {
    t.Errorf("delivered %v, want exactly %s", delivered, m.ID)
  }
}

Code that waits on timers runs inside testing/synctest, where time only moves when every goroutine is blocked. The test finishes instantly and never flakes on a slow machine:

func TestRelayRunPollsUntilCancelled(t *testing.T) {
  synctest.Test(t, func(t *testing.T) {
    db := memdb.New()
    store := memdb.NewOutbox(db)
    var delivered atomic.Int32
    relay := &outbox.Relay{
      Store:    store,
      Handlers: map[string]outbox.Handler{"mail": func(context.Context, outbox.Message) error { delivered.Add(1); return nil }},
      Config:   outbox.RelayConfig{Interval: time.Second},
      Logger:   slog.New(slog.NewTextHandler(io.Discard, nil)),
    }

    ctx, cancel := context.WithCancel(t.Context())
    done := make(chan error)
    go func() { done <- relay.Run(ctx) }()

    time.Sleep(500 * time.Millisecond)
    m, _ := outbox.NewMessage("mail", struct{}{}, time.Now())
    if err := db.Do(ctx, func(tx uow.Tx) error { return store.Add(ctx, tx, m) }); err != nil {
      t.Fatal(err)
    }
    time.Sleep(time.Second)
    synctest.Wait()
    if delivered.Load() != 1 {
      t.Errorf("delivered %d messages after one poll, want 1", delivered.Load())
    }

    cancel()
    if err := <-done; err != nil {
      t.Errorf("Run returned %v after cancel, want nil", err)
    }
  })
}

HTTP: the authorization matrix

Each service has a table-driven test, TestAuthorizationMatrix, that calls every route as each kind of caller and checks the status each gets: anonymous, the owner, another user, a token without the route’s scope, and an OAuth app’s own client-credentials token. auth also calls its routes with a third-party app’s token for a user, which may not manage sessions or apps. A route with no row fails the test: in user, through TestEveryRouteHasARow. This is the test that would have caught the 2022 layout’s account-takeover bug.

TestOpenAPIDocumentsEveryRoute checks the router against api/<service>.openapi.yaml: every route is documented and every documented route exists. The tests’ HTTP rig also checks every response against its schema in that document.

End to end

make e2e, which runs go test -tags e2e ./e2e/..., drives the running stack through the gateway. CI runs it against the memory, MySQL and MongoDB builds, each started with make up STORE=<store>.

Playwright drives the web app on the memory build: it signs in through the Mailpit API, creates an OAuth app, runs the client-credentials flow and revokes a session.