Designing Safe Concurrent Systems in Go

Go makes concurrency easy to start and hard to get right. The race detector catches obvious bugs, but subtle concurrency issues can hide for months. Here’s how I design concurrent systems that don’t break.

The Decision: Mutex vs Channel

The Go proverb says “share memory by communicating.” But sometimes a mutex is simpler.

Use channels when:

• Transferring ownership of data between goroutines
• Coordinating multiple goroutines (fan-out, pipeline)
• Signaling events (done, ready, shutdown)

Use mutexes when:

• Protecting a shared data structure from concurrent access
• Simple read/write synchronization
• The critical section is small and fast

// Mutex: simple shared counter
type Counter struct {
    mu    sync.Mutex
    value int64
}

func (c *Counter) Inc() {
    c.mu.Lock()
    c.value++
    c.mu.Unlock()
}

// Channel: transferring work between stages
jobs := make(chan Job, 100)
go producer(jobs)
go consumer(jobs)

Don’t use channels where a mutex is clearer. A channel-based counter is clever but confusing.

RWMutex for Read-Heavy Workloads

If reads vastly outnumber writes, sync.RWMutex allows concurrent readers:

type ConfigStore struct {
    mu     sync.RWMutex
    config map[string]string
}

func (s *ConfigStore) Get(key string) (string, bool) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    val, ok := s.config[key]
    return val, ok
}

func (s *ConfigStore) Set(key, value string) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.config[key] = value
}

Multiple goroutines can call Get simultaneously. Set blocks until all readers finish and holds exclusive access.

Atomic Operations for Simple Values

For simple counters and flags, sync/atomic avoids mutex overhead entirely:

type Stats struct {
    requestCount  atomic.Int64
    errorCount    atomic.Int64
    isHealthy     atomic.Bool
}

func (s *Stats) RecordRequest(err error) {
    s.requestCount.Add(1)
    if err != nil {
        s.errorCount.Add(1)
    }
}

func (s *Stats) GetErrorRate() float64 {
    total := s.requestCount.Load()
    if total == 0 {
        return 0
    }
    return float64(s.errorCount.Load()) / float64(total)
}

The Race Detector

Run tests with -race. Always. In CI. No exceptions.

go test -race ./...

The race detector instruments memory accesses and reports concurrent unsynchronized access. It has ~10x overhead, so don’t use it in production — but it should be in every CI pipeline.

Common Race Conditions

Race #1: Shared slice append

// RACE: append is not thread-safe
var results []Result
var wg sync.WaitGroup
for _, item := range items {
    wg.Add(1)
    go func(item Item) {
        defer wg.Done()
        results = append(results, process(item)) // RACE!
    }(item)
}

Fix: use a mutex or collect via channel:

var mu sync.Mutex
var results []Result
var wg sync.WaitGroup
for _, item := range items {
    wg.Add(1)
    go func(item Item) {
        defer wg.Done()
        result := process(item)
        mu.Lock()
        results = append(results, result)
        mu.Unlock()
    }(item)
}

Race #2: Map concurrent access

// RACE: maps are not safe for concurrent use
cache := map[string]int{}
go func() { cache["a"] = 1 }()
go func() { _ = cache["a"] }()

Fix: use sync.Map or a mutex-protected map:

var cache sync.Map
cache.Store("a", 1)
val, ok := cache.Load("a")

sync.Map is optimized for two patterns: keys written once and read many times, or disjoint key sets per goroutine. For other patterns, a mutex-protected map is faster.

Race #3: Closure variable capture

// BUG: all goroutines share the same loop variable
for _, item := range items {
    go func() {
        process(item) // Always processes the last item!
    }()
}

// Fix: pass as parameter
for _, item := range items {
    go func(item Item) {
        process(item)
    }(item)
}

// Note: Go 1.22+ fixes this — loop variables are per-iteration

Immutability as Safety

The safest concurrent data is data that never changes:

type Config struct {
    DatabaseURL string
    MaxConns    int
    Features    map[string]bool
}

// Store an immutable config behind an atomic pointer
var currentConfig atomic.Pointer[Config]

func UpdateConfig(new *Config) {
    // Create a deep copy — the old config is still safe to read
    currentConfig.Store(new)
}

func GetConfig() *Config {
    return currentConfig.Load()
}

Readers always get a consistent snapshot. Writers create a new Config and atomically swap the pointer. No locks needed.

Testing Concurrent Code

Use -count to run tests multiple times — races are non-deterministic:

go test -race -count=100 ./...

For specific race scenarios, use runtime.Gosched() to increase the chance of interleaving:

func TestConcurrentAccess(t *testing.T) {
    store := NewStore()
    var wg sync.WaitGroup

    for i := 0; i < 100; i++ {
        wg.Add(2)
        go func(val int) {
            defer wg.Done()
            store.Set("key", val)
        }(i)
        go func() {
            defer wg.Done()
            runtime.Gosched() // Encourage interleaving
            store.Get("key")
        }()
    }

    wg.Wait()
}

Concurrency safety isn’t about clever tricks. It’s about clear ownership: who can read this data? Who can write it? How is access synchronized? Answer those questions for every shared variable, and your concurrent code will be correct.