Design a Notification System

Problem introduction

Every product ends up sending notifications, and almost every team first does it the easy way: the order service calls the SMS gateway directly. That works until the gateway is slow, a campaign floods it, a retry sends the same OTP three times, or someone gets a sale SMS at 3 a.m. This problem is about the shared service that fixes all of that, and its core lesson applies to any background work: accept the request quickly, do the slow and unreliable part later, and make every step safe to repeat.

Problem statement

Nusrat ordered a phone from a shop in Dhaka. Over the next two days, several systems need to tell her things: the order service ("your order has shipped"), the courier ("arriving today"), the payment service ("refund done"), and the marketing team ("Eid sale, 30% off"). Some of these she must get (an OTP for login), some she has turned off (marketing email), and none should wake her up at night unless it's urgent.

What we are building: one notification service that any internal service can call with "send template X to user Y", and that delivers it by push, SMS or email according to the user's preferences, reliably, without duplicates, and without slowing the caller down.

Who uses it: internal services (order, payment, auth, chat, courier) and the marketing team's campaign tool on one side; tens of millions of end users on the other; and support staff who need to answer "did the customer get the SMS?".

What it solves: every team stops writing its own retries, templates, opt-out rules and provider integrations; users get a consistent experience; and there's one place to see what was sent.

What makes it technically interesting:

• Unreliable, slow providers: SMS gateways time out, push services throttle, email providers reject addresses. Calls take hundreds of milliseconds and fail in different ways.
• Duplicates: callers retry, queues redeliver, workers crash halfway. A duplicate OTP is confusing; a duplicate "payment received" is alarming.
• Bursts: a campaign to 25 million users must not delay a login OTP.
• Rules per user: opt-outs, quiet hours, daily limits, language, which channels they even have.
• Cost: SMS costs money per message; push and email are nearly free. The design decides which channel to use for what.

Requirements

Functional requirements

1. Send: an internal service sends {user, template, params, idempotency key}; the service delivers on the template's channels (push, SMS, email).
2. Preferences: users can opt out of marketing per channel and set quiet hours; transactional messages (OTP, order and payment updates) are always allowed.
3. Templates: messages come from versioned templates with parameters and translations (Bangla and English).
4. Reliability: transient failures are retried; messages that can't be delivered go to a dead-letter queue for inspection and replay.
5. No duplicates from retries: the same idempotency key never produces a second notification.
6. Campaigns: marketing can send to a segment of millions, at a rate the providers accept.
7. Tracking: the status of every message (queued, sent, delivered, failed) can be looked up.
8. Out of scope: the in-app inbox UI, A/B testing of message text, and building our own SMS or push infrastructure (we use providers).

Non-functional requirements

Accept latency, durability, no-duplicates and isolation drive the design. Delivery latency is relaxed except for OTPs, which is why OTPs get their own path.

Capacity estimation

Assumptions (said out loud, round numbers):

• 50 million users; about 25 million reachable for marketing (opted in, have the channel).
• Per day: 10 million order and delivery updates, 5 million OTPs, 35 million app pushes (chat, ride status), and two marketing campaigns of 25 million each: about 100 million messages a day.
• Channel mix: 65 million push, 25 million email, 10 million SMS (SMS is reserved for OTPs and important updates because it costs money).
• A campaign goes out over about an hour; normal traffic peaks at 3× the average.

Traffic

Storage

Workers (Little's law): the number of calls in flight = rate × time per call. At 10,000 sends a second and ~200 ms per provider call, about 2,000 calls are in flight at any moment. With asynchronous I/O, 20 worker instances handling 100 concurrent calls each cover the peak. The real limit is usually not our CPU but the providers' rate limits: an SMS account might accept only a few hundred messages a second, so workers must throttle to it.

What the numbers say: the volume is moderate for a queue but too slow and unreliable to do inside an API call. Storage is dominated by the status log. The hard parts are not size but behaviour: duplicates, bursts, slow providers and per-user rules.

API design

• 202 Accepted, not 200 OK: the API promises the notification is durably queued, not that it was delivered. That's what lets it answer in milliseconds.
• Idempotency key: chosen by the caller and tied to the business event (order-BD-7790-shipped), not random per attempt. A retried call with the same key returns the first result and sends nothing new.
• Authentication: service-to-service (mTLS or signed tokens); each caller has a quota, and only the marketing tool may create campaigns or send marketing templates.
• Callers send template names, not text: wording, translation and legal footers live in one place, and a compromised service can't send arbitrary text to every user.
• Webhooks are signed by the provider and verified; they update the status from "sent" to "delivered" or "failed".

Database design

Access patterns: preferences and contacts are read on every request (read-heavy, small, per user). Templates are read constantly and change rarely. Status records are written for every message and updated a few times (write-heavy, looked up by notification or by user). Dedup keys are checked once per request and expire after a day.

-- Preferences and contacts (PostgreSQL, cached in memory with a short TTL).
CREATE TABLE user_prefs (
  user_id       BIGINT PRIMARY KEY,
  locale        TEXT NOT NULL DEFAULT 'bn',
  tz            TEXT NOT NULL DEFAULT 'Asia/Dhaka',
  quiet_from    TIME,                       -- marketing only
  quiet_to      TIME,
  optout_push   BOOLEAN NOT NULL DEFAULT false,   -- marketing opt-outs per channel
  optout_sms    BOOLEAN NOT NULL DEFAULT false,
  optout_email  BOOLEAN NOT NULL DEFAULT false
);
CREATE TABLE contacts (
  user_id   BIGINT NOT NULL,
  channel   TEXT   NOT NULL,                -- push, sms, email
  address   TEXT   NOT NULL,                -- device token, phone, email
  verified  BOOLEAN NOT NULL,
  active    BOOLEAN NOT NULL DEFAULT true,  -- false after the provider says "invalid"
  PRIMARY KEY (user_id, channel, address)
);

-- Templates: versioned, one row per language and channel.
CREATE TABLE templates (
  name     TEXT, version INT, locale TEXT, channel TEXT,
  kind     TEXT NOT NULL,                   -- transactional | marketing
  body     TEXT NOT NULL,
  PRIMARY KEY (name, version, locale, channel)
);

-- Status of every message (Cassandra / DynamoDB style: partition by user, newest first).
--   notifications_by_user: PRIMARY KEY ((user_id), created_at DESC, message_key)
--     columns: channel, template, status, attempts, provider, provider_msg_id, updated_at
--   by_provider_msg: PRIMARY KEY (provider, provider_msg_id) -> user_id, message_key  (for webhooks)

-- Dedup: Redis  SET dedup:<idempotency key> <result> NX EX 86400

• Why PostgreSQL for preferences: small (10 GB), relational, updated by users through settings pages, and needs constraints. Reads are served from a cache, so the database sees little load.
• Why a wide-column store for status: about 100 million inserts plus updates a day, simple key lookups, natural TTL for old records, and easy horizontal scaling. At a smaller scale, a partitioned PostgreSQL table is perfectly fine; switch when write volume or retention makes it painful.
• Why Redis for dedup: an atomic "set if not exists, with expiry" in under a millisecond is exactly the operation needed, and keys only need to live a day. Redis isn't the source of truth for anything permanent, so losing it is an inconvenience, not data loss (see Failures).
• Partitioning: status by user_id (so "this user's notifications" is one partition); queues partitioned by user ID inside each topic, so one user's messages stay in order.

Visual architecture

We deliberately don't have: a database write per message on the API path (only the queue write is required for durability), one microservice per channel on day one (one codebase, separate queues and worker pools), or our own push or SMS infrastructure.

Request lifecycle

Accepting a request

1. Order service → LB → Notify API: POST /v1/notifications with Idempotency-Key: evt-2.
2. API checks the caller's quota, the template and the parameters (a missing parameter is a 400, never a message with a blank in it).
3. SET dedup:evt-2 NX EX 86400 in Redis. If the key already exists, return the stored result with 200: this is a retry.
4. Read preferences and contacts (from cache). Decide channels: the template says push + SMS; Nusrat has no device token, so push is skipped.
5. Marketing only: check opt-outs, quiet hours (delay to the end of the window) and the daily limit.
6. Render the template in the user's language and enqueue one message per channel (evt-2:sms), keyed by user so a user's messages stay in order.
7. Return 202 with what was queued and skipped. Total: a few milliseconds, no provider call.

Sending a message (the worker side)

1. An SMS worker takes evt-2:sms from the queue (it's leased, not deleted, until the worker acknowledges it).
2. It checks the status store: if this message key is already sent, it acknowledges and stops. This catches redeliveries after a crash.
3. It marks the message sending and calls the gateway with a timeout (e.g. 5 s), passing the message key as the provider's idempotency key where the provider supports one.
4. Success: record sent with the provider's message ID, acknowledge the queue message.
5. Transient failure (timeout, 5xx, throttled): schedule a retry after base × 2^(attempt-1) plus random jitter, acknowledge the original. After the maximum attempts, move it to the dead-letter queue.
6. Permanent failure (invalid number, unknown address): don't retry. Dead-letter it and mark the contact inactive so we stop trying that address.
7. Later, the provider's webhook reports delivered (or failed); the status is updated through by_provider_msg.

Component deep dive

Duplicates: where they come from and what stops each one

Exactly-once delivery to a phone is not something a broker setting gives you: the last step is an external call that can succeed while our acknowledgement fails. What we can do is make every step idempotent and shrink the unsafe window to "crashed in the few milliseconds between the provider's answer and our status write". That's an honest answer interviewers like.

A queue per channel, and a lane for urgent messages

Separate queues give each channel its own speed, its own worker count tuned to its provider's rate limit, and its own failure domain. The same idea applies to priority: OTPs and security alerts go to a high-priority topic with dedicated workers, so a 25-million-message campaign sitting in the normal queue never delays a login code. In Kafka terms: one topic per channel (and per priority), partitioned by user ID.

Retries, backoff and the dead-letter queue

Retrying immediately makes an overloaded provider worse: every worker hammers it at once. Exponential backoff spaces the attempts out (1, 2, 4, 8... in our demo seconds; minutes in production), and jitter (a random part of the wait) stops thousands of messages that failed together from retrying together. After a maximum number of attempts, or after a permanent error, the message goes to a dead-letter queue: it isn't lost, an alert fires if the DLQ grows, and after the cause is fixed someone can replay it. A circuit breaker sits on top: if a provider fails most calls for a while, stop calling it for a short time (or switch to a second provider) instead of piling up retries.

Delayed retries need somewhere to wait. Options: a "retry" topic per delay level (sms-retry-1m, sms-retry-10m) that workers read only when the delay has passed, or a scheduled-jobs table/sorted set keyed by due time. Our core uses a Due time on each message, which is the sorted-set idea in miniature.

Preferences, quiet hours and limits

Two classes of messages with different rules:

The rules run at accept time (opt-outs, limits) and are rechecked at send time for messages that waited (a user may opt out while a campaign is queued). Preferences are cached for a short time (seconds to a minute); the recheck makes that staleness harmless.

Campaigns: fan-out without flooding

A campaign to 25 million users is not one API call with 25 million recipients. A scheduler reads the segment in pages, creates requests in batches of, say, 10,000, with idempotency keys like camp-7:<user_id> (so a restarted scheduler doesn't double-send), and feeds them at a controlled rate into the normal-priority queues. The total rate is capped below the providers' limits and below what would delay other traffic. Campaigns can be paused and resumed because progress is just "how far through the segment we are".

Provider failover

Each channel can have two providers. Workers track error rate and latency per provider; when the primary's circuit breaker opens, traffic goes to the secondary. For SMS this also helps with cost and with operator-specific delivery problems. The message key travels with the message, so status and webhooks still line up whichever provider sent it.

The core, runnable

This program is the heart of the design, small enough to read in one sitting: templates with parameters, users with contacts, opt-outs and quiet hours, an API-side Send that deduplicates and applies the rules atomically, one queue per channel, and workers that call providers, retry transient failures with exponential backoff and dead-letter the rest. A simulated clock makes the timing exact, and fake providers fail on cue.

notify/
├── go.mod
├── model.go           Channel, Kind, Template, User, Request, Message, Render, Clock
├── provider.go        Outcome, Provider, FakeProvider (scripted failures)
├── service.go         Service: Send (dedup, prefs, quiet hours, limits), Work/Step (retry, DLQ)
├── notify_test.go     duplicates under -race, backoff timing, DLQ, quiet hours, opt-outs, 4 workers
└── cmd/demo/main.go   order updates, a duplicate, a flaky gateway, a campaign at night

// model.go
package notify

import (
	"errors"
	"fmt"
	"strings"
)

// Channel is one way to reach a person. Each channel has its own queue and its own provider,
// so a slow SMS gateway never holds up push or email.
type Channel string

const (
	Push  Channel = "push"
	SMS   Channel = "sms"
	Email Channel = "email"
)

// Kind decides which rules apply. Users can't opt out of transactional messages (an order
// update, a one-time code), but marketing respects opt-outs, quiet hours and a daily limit.
type Kind int

const (
	Transactional Kind = iota
	Marketing
)

// Template is a message shape: which channels it goes to and its text with {placeholders}.
type Template struct {
	Kind     Kind
	Channels []Channel
	Text     string
}

// User is what the preference service knows about a person.
type User struct {
	Contacts  map[Channel]string // device token, phone number, email address
	OptOut    map[Channel]bool   // marketing opt-outs per channel
	QuietFrom int                // quiet hours for marketing, e.g. 22 to 8 (0, 0 = none)
	QuietTo   int
}

// Request is what another service sends us: "tell this user about this event".
// ID is the idempotency key: the same ID twice means the same notification.
type Request struct {
	ID, User, Template string
	Params             map[string]string
}

// Message is one notification on one channel, waiting in that channel's queue.
type Message struct {
	Key     string // request ID + channel, e.g. "evt-1:sms"
	Channel Channel
	To      string
	Body    string
	Kind    Kind
	Attempt int   // attempts made so far
	Due     int64 // simulated seconds; not sent before this
	seq     int   // enqueue order, breaks ties between equal Due times
}

var (
	ErrDuplicate       = errors.New("duplicate request, already accepted")
	ErrUnknownUser     = errors.New("unknown user")
	ErrUnknownTemplate = errors.New("unknown template")
	ErrRateLimited     = errors.New("marketing limit reached for today")
)

// Render fills {name} placeholders; a missing value is an error, never a blank in a message.
func Render(text string, params map[string]string) (string, error) {
	var b strings.Builder
	for {
		i := strings.IndexByte(text, '{')
		if i < 0 {
			b.WriteString(text)
			return b.String(), nil
		}
		j := strings.IndexByte(text[i:], '}')
		if j < 0 {
			return "", fmt.Errorf("unclosed placeholder in %q", text)
		}
		name := text[i+1 : i+j]
		v, ok := params[name]
		if !ok {
			return "", fmt.Errorf("missing parameter %q", name)
		}
		b.WriteString(text[:i])
		b.WriteString(v)
		text = text[i+j+1:]
	}
}

// Clock formats simulated seconds as "D1 21:59:50".
func Clock(t int64) string {
	s := t % 86400
	return fmt.Sprintf("D%d %02d:%02d:%02d", t/86400+1, s/3600, s/60%60, s%60)
}

// provider.go
package notify

import "sync"

// Outcome is what a provider (APNs/FCM, an SMS gateway, an email service) said.
type Outcome int

const (
	OK        Outcome = iota
	Transient         // timeout, 5xx, throttled: worth retrying
	Permanent         // invalid number, unknown address: retrying won't help
)

func (o Outcome) String() string {
	switch o {
	case OK:
		return "delivered"
	case Transient:
		return "failed (timeout)"
	default:
		return "rejected (bad address)"
	}
}

// Provider sends one message on one channel.
type Provider interface {
	Send(m Message) Outcome
}

// FakeProvider plays back scripted outcomes per address (attempt 1, 2, ...); after the
// script runs out, sends succeed. It also counts sends per message key.
type FakeProvider struct {
	mu     sync.Mutex
	Script map[string][]Outcome
	Sent   map[string]int
}

func NewFakeProvider(script map[string][]Outcome) *FakeProvider {
	if script == nil {
		script = map[string][]Outcome{}
	}
	return &FakeProvider{Script: script, Sent: map[string]int{}}
}

func (p *FakeProvider) Send(m Message) Outcome {
	p.mu.Lock()
	defer p.mu.Unlock()
	p.Sent[m.Key]++
	if s := p.Script[m.To]; m.Attempt < len(s) {
		return s[m.Attempt]
	}
	return OK
}

// service.go
// Package notify is the core of a notification system: requests are deduplicated, checked
// against preferences, rendered from templates and put on one queue per channel; workers send
// them through providers, retry transient failures with exponential backoff, and move
// messages that can't be delivered to a dead-letter queue.
package notify

import (
	"strconv"
	"sync"
)

// MaxAttempts is how many times a message is tried before it goes to the dead-letter queue.
const MaxAttempts = 4

// Backoff is the wait before retry number n (1, 2, 4, ... seconds). Production adds jitter
// and uses minutes, not seconds; the demo keeps it exact so every language prints the same.
func Backoff(attempt int) int64 { return int64(1) << (attempt - 1) }

// Event is one delivery attempt, for logs and the demo.
type Event struct {
	Time    int64
	Msg     Message
	Outcome Outcome
	RetryAt int64 // set when the message was put back on the queue
	Dead    bool  // set when it went to the dead-letter queue
}

// Service accepts requests and runs the per-channel queues.
type Service struct {
	mu                 sync.Mutex
	users              map[string]User
	templates          map[string]Template
	providers          map[Channel]Provider
	seen               map[string]bool // accepted request IDs (production: a store with a TTL)
	marketing          map[string]int  // "user/day" -> marketing messages accepted
	MaxMarketingPerDay int
	queues             map[Channel][]Message
	seq                int
	Dead               []Message
	Delivered          int
}

func NewService(users map[string]User, templates map[string]Template, providers map[Channel]Provider) *Service {
	return &Service{
		users: users, templates: templates, providers: providers,
		seen: map[string]bool{}, marketing: map[string]int{}, MaxMarketingPerDay: 1,
		queues: map[Channel][]Message{},
	}
}

func inQuiet(u User, now int64) bool {
	if u.QuietFrom == u.QuietTo {
		return false
	}
	h := int(now % 86400 / 3600)
	if u.QuietFrom < u.QuietTo {
		return h >= u.QuietFrom && h < u.QuietTo
	}
	return h >= u.QuietFrom || h < u.QuietTo
}

// quietEnd is the next time the clock reads QuietTo:00.
func quietEnd(u User, now int64) int64 {
	day := now / 86400 * 86400
	end := day + int64(u.QuietTo)*3600
	if end <= now {
		end += 86400
	}
	return end
}

// Accepted says what happened to a request: channels queued, channels skipped (with the
// reason), and when sending starts (later than now during quiet hours).
type Accepted struct {
	Queued  []Channel
	Skipped []string
	Due     int64
}

// Send accepts one request. The whole check-and-accept runs under one lock, so two copies of
// a request racing each other can't both get through.
func (s *Service) Send(r Request, now int64) (Accepted, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	if s.seen[r.ID] {
		return Accepted{}, ErrDuplicate
	}
	u, ok := s.users[r.User]
	if !ok {
		return Accepted{}, ErrUnknownUser
	}
	t, ok := s.templates[r.Template]
	if !ok {
		return Accepted{}, ErrUnknownTemplate
	}
	body, err := Render(t.Text, r.Params)
	if err != nil {
		return Accepted{}, err
	}
	var a Accepted
	dayKey := r.User + "/" + strconv.FormatInt(now/86400, 10)
	if t.Kind == Marketing && s.marketing[dayKey] >= s.MaxMarketingPerDay {
		return Accepted{}, ErrRateLimited
	}
	a.Due = now
	if t.Kind == Marketing && inQuiet(u, now) {
		a.Due = quietEnd(u, now)
	}
	for _, ch := range t.Channels {
		to, ok := u.Contacts[ch]
		switch {
		case !ok:
			a.Skipped = append(a.Skipped, string(ch)+": no address")
		case t.Kind == Marketing && u.OptOut[ch]:
			a.Skipped = append(a.Skipped, string(ch)+": opted out")
		default:
			s.seq++
			s.queues[ch] = append(s.queues[ch], Message{Key: r.ID + ":" + string(ch), Channel: ch, To: to, Body: body, Kind: t.Kind, Due: a.Due, seq: s.seq})
			a.Queued = append(a.Queued, ch)
		}
	}
	s.seen[r.ID] = true
	if t.Kind == Marketing && len(a.Queued) > 0 {
		s.marketing[dayKey]++
	}
	return a, nil
}

// take removes the earliest due message (on one channel, or on any if ch is ""), if one is due.
func (s *Service) take(ch Channel, now int64) (Message, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()
	bestCh, best := Channel(""), -1
	for _, c := range []Channel{Push, SMS, Email} {
		if ch != "" && c != ch {
			continue
		}
		for i, m := range s.queues[c] {
			if m.Due > now {
				continue
			}
			if best < 0 || m.Due < s.queues[bestCh][best].Due ||
				(m.Due == s.queues[bestCh][best].Due && m.seq < s.queues[bestCh][best].seq) {
				bestCh, best = c, i
			}
		}
	}
	if best < 0 {
		return Message{}, false
	}
	q := s.queues[bestCh]
	m := q[best]
	s.queues[bestCh] = append(q[:best:best], q[best+1:]...)
	return m, true
}

// deliver calls the provider outside the lock (it's network I/O), then records the result.
func (s *Service) deliver(m Message, now int64) Event {
	out := s.providers[m.Channel].Send(m)
	m.Attempt++
	ev := Event{Time: now, Msg: m, Outcome: out}
	s.mu.Lock()
	defer s.mu.Unlock()
	switch {
	case out == OK:
		s.Delivered++
	case out == Transient && m.Attempt < MaxAttempts:
		m.Due = now + Backoff(m.Attempt)
		s.queues[m.Channel] = append(s.queues[m.Channel], m)
		ev.RetryAt = m.Due
	default:
		s.Dead = append(s.Dead, m)
		ev.Dead = true
	}
	return ev
}

// Work is one worker step on one channel's queue: send the earliest due message, if any.
func (s *Service) Work(ch Channel, now int64) (Event, bool) {
	m, ok := s.take(ch, now)
	if !ok {
		return Event{}, false
	}
	return s.deliver(m, now), true
}

// Step sends the earliest due message on any channel (the demo's single worker).
func (s *Service) Step(now int64) (Event, bool) { return s.Work("", now) }

// NextDue is the earliest time any queued message becomes due.
func (s *Service) NextDue() (int64, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()
	var next int64
	found := false
	for _, q := range s.queues {
		for _, m := range q {
			if !found || m.Due < next {
				next, found = m.Due, true
			}
		}
	}
	return next, found
}

// Pending is how many messages are still queued.
func (s *Service) Pending() int {
	s.mu.Lock()
	defer s.mu.Unlock()
	n := 0
	for _, q := range s.queues {
		n += len(q)
	}
	return n
}

// cmd/demo/main.go
// Command demo sends a few order updates and a marketing campaign through the notification
// core, with a flaky SMS gateway, a bad email address, quiet hours and duplicate requests.
package main

import (
	"fmt"
	"strings"

	"notify"
)

func main() {
	users := map[string]notify.User{
		"rafi": {Contacts: map[notify.Channel]string{notify.Push: "tok-rafi", notify.SMS: "+8801711000001", notify.Email: "rafi@example.com"},
			QuietFrom: 22, QuietTo: 8},
		"nusrat": {Contacts: map[notify.Channel]string{notify.SMS: "+8801811000002", notify.Email: "nusrat@example.com"},
			OptOut: map[notify.Channel]bool{notify.Email: true}},
		"tanvir": {Contacts: map[notify.Channel]string{notify.Email: "tanvir@bad-domain"}},
	}
	templates := map[string]notify.Template{
		"order_shipped": {Kind: notify.Transactional, Channels: []notify.Channel{notify.Push, notify.SMS}, Text: "Order {order} is on the way, arriving {day}"},
		"eid_sale":      {Kind: notify.Marketing, Channels: []notify.Channel{notify.Push, notify.Email}, Text: "Eid sale: {pct}% off until Friday"},
	}
	providers := map[notify.Channel]notify.Provider{
		notify.Push:  notify.NewFakeProvider(nil),
		notify.SMS:   notify.NewFakeProvider(map[string][]notify.Outcome{"+8801811000002": {notify.Transient, notify.Transient}}),
		notify.Email: notify.NewFakeProvider(map[string][]notify.Outcome{"tanvir@bad-domain": {notify.Permanent}}),
	}
	s := notify.NewService(users, templates, providers)

	send := func(now int64, r notify.Request) {
		a, err := s.Send(r, now)
		head := fmt.Sprintf("  %-16s %-13s %-6s ->", r.ID, r.Template, r.User)
		if err != nil {
			fmt.Printf("%s error: %v\n", head, err)
			return
		}
		parts := []string{}
		for _, ch := range a.Queued {
			parts = append(parts, string(ch))
		}
		line := "queued " + strings.Join(parts, ", ")
		if a.Due > now {
			line += " for " + notify.Clock(a.Due) + " (quiet hours)"
		}
		if len(a.Queued) == 0 {
			line = "nothing to send"
		}
		if len(a.Skipped) > 0 {
			line += " (skipped " + strings.Join(a.Skipped, "; ") + ")"
		}
		fmt.Printf("%s %s\n", head, line)
	}
	run := func(until int64) {
		for {
			next, ok := s.NextDue()
			if !ok || next > until {
				return
			}
			ev, _ := s.Step(next)
			line := ev.Outcome.String()
			switch {
			case ev.RetryAt > 0:
				line += ", retry at " + notify.Clock(ev.RetryAt)
			case ev.Dead:
				line += ", moved to dead letters"
			default:
				line += ": " + ev.Msg.Body
			}
			fmt.Printf("  %s %-5s %-19s try %d: %s\n", notify.Clock(ev.Time), ev.Msg.Channel, ev.Msg.Key, ev.Msg.Attempt, line)
		}
	}

	t1 := int64(21*3600 + 59*60 + 50)
	fmt.Println("D1 21:59:50, the order service sends three requests:")
	send(t1, notify.Request{ID: "evt-1", User: "rafi", Template: "order_shipped", Params: map[string]string{"order": "BD-7781", "day": "Sunday"}})
	send(t1, notify.Request{ID: "evt-1", User: "rafi", Template: "order_shipped", Params: map[string]string{"order": "BD-7781", "day": "Sunday"}})
	send(t1, notify.Request{ID: "evt-2", User: "nusrat", Template: "order_shipped", Params: map[string]string{"order": "BD-7790", "day": "Monday"}})
	fmt.Println("workers:")
	run(t1 + 10)

	t2 := int64(22*3600 + 5)
	fmt.Println("D1 22:00:05, the marketing team starts a campaign:")
	send(t2, notify.Request{ID: "camp-7:rafi", User: "rafi", Template: "eid_sale", Params: map[string]string{"pct": "30"}})
	send(t2, notify.Request{ID: "camp-7:nusrat", User: "nusrat", Template: "eid_sale", Params: map[string]string{"pct": "30"}})
	send(t2, notify.Request{ID: "camp-7:tanvir", User: "tanvir", Template: "eid_sale", Params: map[string]string{"pct": "30"}})
	send(t2, notify.Request{ID: "camp-8:rafi", User: "rafi", Template: "eid_sale", Params: map[string]string{"pct": "40"}})
	send(t2, notify.Request{ID: "camp-9:tanvir", User: "tanvir", Template: "eid_sale", Params: map[string]string{}})
	fmt.Println("workers:")
	run(2 * 86400)
	fmt.Printf("delivered %d, dead letters %d, still queued %d\n", s.Delivered, len(s.Dead), s.Pending())
}

Running the demo:

$ go run ./cmd/demo
D1 21:59:50, the order service sends three requests:
  evt-1            order_shipped rafi   -> queued push, sms
  evt-1            order_shipped rafi   -> error: duplicate request, already accepted
  evt-2            order_shipped nusrat -> queued sms (skipped push: no address)
workers:
  D1 21:59:50 push  evt-1:push          try 1: delivered: Order BD-7781 is on the way, arriving Sunday
  D1 21:59:50 sms   evt-1:sms           try 1: delivered: Order BD-7781 is on the way, arriving Sunday
  D1 21:59:50 sms   evt-2:sms           try 1: failed (timeout), retry at D1 21:59:51
  D1 21:59:51 sms   evt-2:sms           try 2: failed (timeout), retry at D1 21:59:53
  D1 21:59:53 sms   evt-2:sms           try 3: delivered: Order BD-7790 is on the way, arriving Monday
D1 22:00:05, the marketing team starts a campaign:
  camp-7:rafi      eid_sale      rafi   -> queued push, email for D2 08:00:00 (quiet hours)
  camp-7:nusrat    eid_sale      nusrat -> nothing to send (skipped push: no address; email: opted out)
  camp-7:tanvir    eid_sale      tanvir -> queued email (skipped push: no address)
  camp-8:rafi      eid_sale      rafi   -> error: marketing limit reached for today
  camp-9:tanvir    eid_sale      tanvir -> error: missing parameter "pct"
workers:
  D1 22:00:05 email camp-7:tanvir:email try 1: rejected (bad address), moved to dead letters
  D2 08:00:00 push  camp-7:rafi:push    try 1: delivered: Eid sale: 30% off until Friday
  D2 08:00:00 email camp-7:rafi:email   try 1: delivered: Eid sale: 30% off until Friday
delivered 5, dead letters 1, still queued 0

Reading the output:

• The duplicate: the second evt-1 is rejected as already accepted. Rafi gets one push and one SMS, not two of each.
• Skipped channels: Nusrat has no device token, so evt-2 goes by SMS only, and the response says why push was skipped. The caller learns exactly what happened.
• The flaky gateway: Nusrat's SMS times out at 21:59:50, is retried 1 s later, times out again, waits 2 s, and is delivered on try 3 at 21:59:53. Rafi's messages, queued at the same moment, went out first and weren't held up.
• Quiet hours: at 22:00:05 Rafi is in quiet hours (22 to 8), so the campaign is queued for D2 08:00:00 and delivered then. The order updates an hour earlier weren't delayed, because quiet hours apply to marketing only.
• Opt-out: Nusrat opted out of marketing email and has no push token, so the campaign has nothing to send to her. That's a successful request with zero messages, not an error.
• Permanent failure: Tanvir's email address is bad. The provider rejects it on try 1 and it goes straight to dead letters; retrying would only waste calls.
• Limits and validation: a second campaign to Rafi the same day hits the daily marketing limit; a request missing pct is refused instead of sending "Eid sale: % off".
• Totals: 5 delivered, 1 dead letter, nothing left queued.

The tests check what matters most: 50 identical requests at once produce exactly one queued message (-race), retries happen at 0, 1, 3 and 7 s and then go to the DLQ, permanent failures are not retried, quiet hours delay only marketing and wrap around midnight, opt-outs don't block transactional messages, a failed request doesn't burn its idempotency key, and four workers draining one queue send each message exactly once. Releasing the lock between the duplicate check and the accept makes the duplicate test fail (4 copies got through in one run), so the test really guards that step.

$ go test -race -v ./...
=== RUN   TestDuplicateRequestsQueueOnce
--- PASS: TestDuplicateRequestsQueueOnce (0.00s)
=== RUN   TestRetriesBackOffThenDeadLetter
--- PASS: TestRetriesBackOffThenDeadLetter (0.00s)
=== RUN   TestPermanentFailureIsNotRetried
--- PASS: TestPermanentFailureIsNotRetried (0.00s)
=== RUN   TestQuietHoursDelayMarketingOnly
--- PASS: TestQuietHoursDelayMarketingOnly (0.00s)
=== RUN   TestOptOutAppliesToMarketingOnly
--- PASS: TestOptOutAppliesToMarketingOnly (0.00s)
=== RUN   TestMarketingLimitAndValidation
--- PASS: TestMarketingLimitAndValidation (0.00s)
=== RUN   TestConcurrentWorkersSendEachMessageOnce
--- PASS: TestConcurrentWorkersSendEachMessageOnce (0.02s)
PASS
ok  	notify	1.034s
?   	notify/cmd/demo	[no test files]

The same core in five more languages

Single-threaded versions with the same classes and the same output. The worker loop is sequential here; the Go version is the one that runs workers concurrently. What differs:

• Python: messages are frozen dataclasses, and a retry creates a changed copy with replace(), so a message on the queue is never edited in place.
• TypeScript: Outcome is a const object instead of an enum, because Node runs .ts files by stripping types and enums aren't just types. SendRequest and Attempt avoid clashing with the DOM's global Request and Event.
• Rust: Channel, Kind and Outcome are enums matched exhaustively, so adding a channel forces every match to handle it. render uses let ... else for early returns.
• Java: records for immutable messages; send and take are synchronized and the provider call happens outside the lock, as in Go.
• C++: a std::mutex with std::lock_guard around the same critical sections; std::optional for "no message due".

"""A notification system's core: requests are deduplicated, checked against preferences,
rendered from templates and put on one queue per channel; workers send them through
providers, retry transient failures with exponential backoff, and move messages that can't be
delivered to a dead-letter queue."""
from dataclasses import dataclass, field, replace
from enum import Enum

PUSH, SMS, EMAIL = "push", "sms", "email"
CHANNELS = [PUSH, SMS, EMAIL]
TRANSACTIONAL, MARKETING = 0, 1
MAX_ATTEMPTS = 4


class Outcome(Enum):
    OK = "delivered"
    TRANSIENT = "failed (timeout)"
    PERMANENT = "rejected (bad address)"


@dataclass
class Template:
    kind: int
    channels: list[str]
    text: str


@dataclass
class User:
    contacts: dict[str, str]
    opt_out: set[str] = field(default_factory=set)  # marketing opt-outs per channel
    quiet_from: int = 0  # quiet hours for marketing, e.g. 22 to 8 (0, 0 = none)
    quiet_to: int = 0


@dataclass
class Request:
    id: str  # the idempotency key
    user: str
    template: str
    params: dict[str, str]


@dataclass(frozen=True)
class Message:
    key: str  # request ID + channel, e.g. "evt-1:sms"
    channel: str
    to: str
    body: str
    kind: int
    attempt: int  # attempts made so far
    due: int  # simulated seconds; not sent before this
    seq: int  # enqueue order, breaks ties


@dataclass
class Accepted:
    queued: list[str] = field(default_factory=list)
    skipped: list[str] = field(default_factory=list)
    due: int = 0


@dataclass
class Event:
    time: int
    msg: Message
    outcome: Outcome
    retry_at: int = 0
    dead: bool = False


class Rejected(Exception):
    pass


def render(text: str, params: dict[str, str]) -> str:
    """Fills {name} placeholders; a missing value is an error, never a blank in a message."""
    out = []
    while True:
        i = text.find("{")
        if i < 0:
            return "".join(out) + text
        j = text.find("}", i)
        if j < 0:
            raise Rejected(f"unclosed placeholder in {text!r}")
        name = text[i + 1 : j]
        if name not in params:
            raise Rejected(f'missing parameter "{name}"')
        out += [text[:i], params[name]]
        text = text[j + 1 :]


def clock(t: int) -> str:
    s = t % 86400
    return f"D{t // 86400 + 1} {s // 3600:02d}:{s // 60 % 60:02d}:{s % 60:02d}"


def backoff(attempt: int) -> int:
    """Wait before retry n: 1, 2, 4... seconds (production: minutes, with jitter)."""
    return 1 << (attempt - 1)


class FakeProvider:
    """Plays back scripted outcomes per address; after the script runs out, sends succeed."""

    def __init__(self, script: dict[str, list[Outcome]] | None = None):
        self.script = script or {}
        self.sent: dict[str, int] = {}

    def send(self, m: Message) -> Outcome:
        self.sent[m.key] = self.sent.get(m.key, 0) + 1
        s = self.script.get(m.to, [])
        return s[m.attempt] if m.attempt < len(s) else Outcome.OK


def in_quiet(u: User, now: int) -> bool:
    if u.quiet_from == u.quiet_to:
        return False
    h = now % 86400 // 3600
    if u.quiet_from < u.quiet_to:
        return u.quiet_from <= h < u.quiet_to
    return h >= u.quiet_from or h < u.quiet_to


def quiet_end(u: User, now: int) -> int:
    end = now // 86400 * 86400 + u.quiet_to * 3600
    return end + 86400 if end <= now else end


class Service:
    def __init__(self, users: dict[str, User], templates: dict[str, Template], providers: dict[str, FakeProvider]):
        self.users, self.templates, self.providers = users, templates, providers
        self.seen: set[str] = set()  # accepted request IDs (production: a store with a TTL)
        self.marketing: dict[str, int] = {}  # "user/day" -> marketing messages accepted
        self.max_marketing_per_day = 1
        self.queues: dict[str, list[Message]] = {c: [] for c in CHANNELS}
        self.seq = 0
        self.dead: list[Message] = []
        self.delivered = 0

    def send(self, r: Request, now: int) -> Accepted:
        if r.id in self.seen:
            raise Rejected("duplicate request, already accepted")
        u = self.users.get(r.user)
        if u is None:
            raise Rejected("unknown user")
        t = self.templates.get(r.template)
        if t is None:
            raise Rejected("unknown template")
        body = render(t.text, r.params)
        day_key = f"{r.user}/{now // 86400}"
        if t.kind == MARKETING and self.marketing.get(day_key, 0) >= self.max_marketing_per_day:
            raise Rejected("marketing limit reached for today")
        a = Accepted(due=now)
        if t.kind == MARKETING and in_quiet(u, now):
            a.due = quiet_end(u, now)
        for ch in t.channels:
            if ch not in u.contacts:
                a.skipped.append(f"{ch}: no address")
            elif t.kind == MARKETING and ch in u.opt_out:
                a.skipped.append(f"{ch}: opted out")
            else:
                self.seq += 1
                self.queues[ch].append(Message(f"{r.id}:{ch}", ch, u.contacts[ch], body, t.kind, 0, a.due, self.seq))
                a.queued.append(ch)
        self.seen.add(r.id)
        if t.kind == MARKETING and a.queued:
            self.marketing[day_key] = self.marketing.get(day_key, 0) + 1
        return a

    def _take(self, now: int) -> Message | None:
        due = [m for c in CHANNELS for m in self.queues[c] if m.due <= now]
        if not due:
            return None
        m = min(due, key=lambda m: (m.due, m.seq))
        self.queues[m.channel].remove(m)
        return m

    def step(self, now: int) -> Event | None:
        m = self._take(now)
        if m is None:
            return None
        out = self.providers[m.channel].send(m)
        m = replace(m, attempt=m.attempt + 1)
        ev = Event(now, m, out)
        if out == Outcome.OK:
            self.delivered += 1
        elif out == Outcome.TRANSIENT and m.attempt < MAX_ATTEMPTS:
            m = replace(m, due=now + backoff(m.attempt))
            self.queues[m.channel].append(m)
            ev.retry_at = m.due
        else:
            self.dead.append(m)
            ev.dead = True
        return ev

    def next_due(self) -> int | None:
        dues = [m.due for q in self.queues.values() for m in q]
        return min(dues) if dues else None

    def pending(self) -> int:
        return sum(len(q) for q in self.queues.values())


def main() -> None:
    users = {
        "rafi": User({PUSH: "tok-rafi", SMS: "+8801711000001", EMAIL: "rafi@example.com"}, quiet_from=22, quiet_to=8),
        "nusrat": User({SMS: "+8801811000002", EMAIL: "nusrat@example.com"}, opt_out={EMAIL}),
        "tanvir": User({EMAIL: "tanvir@bad-domain"}),
    }
    templates = {
        "order_shipped": Template(TRANSACTIONAL, [PUSH, SMS], "Order {order} is on the way, arriving {day}"),
        "eid_sale": Template(MARKETING, [PUSH, EMAIL], "Eid sale: {pct}% off until Friday"),
    }
    providers = {
        PUSH: FakeProvider(),
        SMS: FakeProvider({"+8801811000002": [Outcome.TRANSIENT, Outcome.TRANSIENT]}),
        EMAIL: FakeProvider({"tanvir@bad-domain": [Outcome.PERMANENT]}),
    }
    s = Service(users, templates, providers)

    def send(now: int, r: Request) -> None:
        head = f"  {r.id:<16} {r.template:<13} {r.user:<6} ->"
        try:
            a = s.send(r, now)
        except Rejected as e:
            print(f"{head} error: {e}")
            return
        line = "queued " + ", ".join(a.queued)
        if a.due > now:
            line += f" for {clock(a.due)} (quiet hours)"
        if not a.queued:
            line = "nothing to send"
        if a.skipped:
            line += " (skipped " + "; ".join(a.skipped) + ")"
        print(f"{head} {line}")

    def run(until: int) -> None:
        while (nxt := s.next_due()) is not None and nxt <= until:
            ev = s.step(nxt)
            assert ev is not None
            line = ev.outcome.value
            if ev.retry_at:
                line += f", retry at {clock(ev.retry_at)}"
            elif ev.dead:
                line += ", moved to dead letters"
            else:
                line += f": {ev.msg.body}"
            print(f"  {clock(ev.time)} {ev.msg.channel:<5} {ev.msg.key:<19} try {ev.msg.attempt}: {line}")

    t1 = 21 * 3600 + 59 * 60 + 50
    print("D1 21:59:50, the order service sends three requests:")
    send(t1, Request("evt-1", "rafi", "order_shipped", {"order": "BD-7781", "day": "Sunday"}))
    send(t1, Request("evt-1", "rafi", "order_shipped", {"order": "BD-7781", "day": "Sunday"}))
    send(t1, Request("evt-2", "nusrat", "order_shipped", {"order": "BD-7790", "day": "Monday"}))
    print("workers:")
    run(t1 + 10)

    t2 = 22 * 3600 + 5
    print("D1 22:00:05, the marketing team starts a campaign:")
    send(t2, Request("camp-7:rafi", "rafi", "eid_sale", {"pct": "30"}))
    send(t2, Request("camp-7:nusrat", "nusrat", "eid_sale", {"pct": "30"}))
    send(t2, Request("camp-7:tanvir", "tanvir", "eid_sale", {"pct": "30"}))
    send(t2, Request("camp-8:rafi", "rafi", "eid_sale", {"pct": "40"}))
    send(t2, Request("camp-9:tanvir", "tanvir", "eid_sale", {}))
    print("workers:")
    run(2 * 86400)
    print(f"delivered {s.delivered}, dead letters {len(s.dead)}, still queued {s.pending()}")


if __name__ == "__main__":
    main()

// A notification system's core: requests are deduplicated, checked against preferences,
// rendered from templates and put on one queue per channel; workers send them through
// providers, retry transient failures with exponential backoff, and move messages that can't
// be delivered to a dead-letter queue.

type Channel = "push" | "sms" | "email";
const CHANNELS: Channel[] = ["push", "sms", "email"];
type Kind = "transactional" | "marketing";
const MAX_ATTEMPTS = 4;

// A const object instead of an enum: Node runs this file by stripping types, and enums aren't types.
const Outcome = { OK: "delivered", Transient: "failed (timeout)", Permanent: "rejected (bad address)" } as const;
type Outcome = (typeof Outcome)[keyof typeof Outcome];

type Template = { kind: Kind; channels: Channel[]; text: string };
type User = {
  contacts: Partial<Record<Channel, string>>;
  optOut?: Set<Channel>; // marketing opt-outs per channel
  quietFrom?: number; // quiet hours for marketing, e.g. 22 to 8
  quietTo?: number;
};
// SendRequest and Attempt, not Request and Event: those names are DOM globals.
type SendRequest = { id: string; user: string; template: string; params: Record<string, string> };
type Message = {
  readonly key: string; // request ID + channel, e.g. "evt-1:sms"
  readonly channel: Channel;
  readonly to: string;
  readonly body: string;
  readonly kind: Kind;
  readonly attempt: number; // attempts made so far
  readonly due: number; // simulated seconds; not sent before this
  readonly seq: number; // enqueue order, breaks ties
};
type Accepted = { queued: Channel[]; skipped: string[]; due: number };
type Attempt = { time: number; msg: Message; outcome: Outcome; retryAt: number; dead: boolean };

class Rejected extends Error {}

/** Fills {name} placeholders; a missing value is an error, never a blank in a message. */
function render(text: string, params: Record<string, string>): string {
  let out = "";
  for (;;) {
    const i = text.indexOf("{");
    if (i < 0) return out + text;
    const j = text.indexOf("}", i);
    if (j < 0) throw new Rejected(`unclosed placeholder in ${JSON.stringify(text)}`);
    const name = text.slice(i + 1, j);
    if (!(name in params)) throw new Rejected(`missing parameter "${name}"`);
    out += text.slice(0, i) + params[name];
    text = text.slice(j + 1);
  }
}

const pad2 = (n: number) => String(n).padStart(2, "0");
function clock(t: number): string {
  const s = t % 86400;
  return `D${Math.floor(t / 86400) + 1} ${pad2(Math.floor(s / 3600))}:${pad2(Math.floor(s / 60) % 60)}:${pad2(s % 60)}`;
}

/** Wait before retry n: 1, 2, 4... seconds (production: minutes, with jitter). */
const backoff = (attempt: number) => 2 ** (attempt - 1);

/** Plays back scripted outcomes per address; after the script runs out, sends succeed. */
class FakeProvider {
  sent = new Map<string, number>();
  readonly script: Map<string, Outcome[]>;
  constructor(script: Record<string, Outcome[]> = {}) {
    this.script = new Map(Object.entries(script));
  }
  send(m: Message): Outcome {
    this.sent.set(m.key, (this.sent.get(m.key) ?? 0) + 1);
    const s = this.script.get(m.to) ?? [];
    return m.attempt < s.length ? s[m.attempt]! : Outcome.OK;
  }
}

function inQuiet(u: User, now: number): boolean {
  const from = u.quietFrom ?? 0;
  const to = u.quietTo ?? 0;
  if (from === to) return false;
  const h = Math.floor((now % 86400) / 3600);
  return from < to ? h >= from && h < to : h >= from || h < to;
}

function quietEnd(u: User, now: number): number {
  const end = Math.floor(now / 86400) * 86400 + (u.quietTo ?? 0) * 3600;
  return end <= now ? end + 86400 : end;
}

class Service {
  seen = new Set<string>(); // accepted request IDs (production: a store with a TTL)
  marketing = new Map<string, number>(); // "user/day" -> marketing messages accepted
  maxMarketingPerDay = 1;
  queues = new Map<Channel, Message[]>(CHANNELS.map((c) => [c, []]));
  seq = 0;
  dead: Message[] = [];
  delivered = 0;
  readonly users: Map<string, User>;
  readonly templates: Map<string, Template>;
  readonly providers: Map<Channel, FakeProvider>;

  constructor(users: Record<string, User>, templates: Record<string, Template>, providers: Record<Channel, FakeProvider>) {
    this.users = new Map(Object.entries(users));
    this.templates = new Map(Object.entries(templates));
    this.providers = new Map(Object.entries(providers) as [Channel, FakeProvider][]);
  }

  send(r: SendRequest, now: number): Accepted {
    if (this.seen.has(r.id)) throw new Rejected("duplicate request, already accepted");
    const u = this.users.get(r.user);
    if (!u) throw new Rejected("unknown user");
    const t = this.templates.get(r.template);
    if (!t) throw new Rejected("unknown template");
    const body = render(t.text, r.params);
    const dayKey = `${r.user}/${Math.floor(now / 86400)}`;
    if (t.kind === "marketing" && (this.marketing.get(dayKey) ?? 0) >= this.maxMarketingPerDay)
      throw new Rejected("marketing limit reached for today");
    const a: Accepted = { queued: [], skipped: [], due: now };
    if (t.kind === "marketing" && inQuiet(u, now)) a.due = quietEnd(u, now);
    for (const ch of t.channels) {
      const to = u.contacts[ch];
      if (to === undefined) a.skipped.push(`${ch}: no address`);
      else if (t.kind === "marketing" && u.optOut?.has(ch)) a.skipped.push(`${ch}: opted out`);
      else {
        this.queues.get(ch)!.push({ key: `${r.id}:${ch}`, channel: ch, to, body, kind: t.kind, attempt: 0, due: a.due, seq: ++this.seq });
        a.queued.push(ch);
      }
    }
    this.seen.add(r.id);
    if (t.kind === "marketing" && a.queued.length > 0) this.marketing.set(dayKey, (this.marketing.get(dayKey) ?? 0) + 1);
    return a;
  }

  private take(now: number): Message | undefined {
    let best: Message | undefined;
    for (const c of CHANNELS)
      for (const m of this.queues.get(c)!)
        if (m.due <= now && (!best || m.due < best.due || (m.due === best.due && m.seq < best.seq))) best = m;
    if (best) {
      const q = this.queues.get(best.channel)!;
      q.splice(q.indexOf(best), 1);
    }
    return best;
  }

  step(now: number): Attempt | undefined {
    const taken = this.take(now);
    if (!taken) return undefined;
    const outcome = this.providers.get(taken.channel)!.send(taken);
    const m: Message = { ...taken, attempt: taken.attempt + 1 };
    const ev: Attempt = { time: now, msg: m, outcome, retryAt: 0, dead: false };
    if (outcome === Outcome.OK) this.delivered++;
    else if (outcome === Outcome.Transient && m.attempt < MAX_ATTEMPTS) {
      ev.retryAt = now + backoff(m.attempt);
      this.queues.get(m.channel)!.push({ ...m, due: ev.retryAt });
    } else {
      this.dead.push(m);
      ev.dead = true;
    }
    return ev;
  }

  nextDue(): number | undefined {
    let next: number | undefined;
    for (const q of this.queues.values()) for (const m of q) if (next === undefined || m.due < next) next = m.due;
    return next;
  }

  pending(): number {
    let n = 0;
    for (const q of this.queues.values()) n += q.length;
    return n;
  }
}

const s = new Service(
  {
    rafi: { contacts: { push: "tok-rafi", sms: "+8801711000001", email: "rafi@example.com" }, quietFrom: 22, quietTo: 8 },
    nusrat: { contacts: { sms: "+8801811000002", email: "nusrat@example.com" }, optOut: new Set<Channel>(["email"]) },
    tanvir: { contacts: { email: "tanvir@bad-domain" } },
  },
  {
    order_shipped: { kind: "transactional", channels: ["push", "sms"], text: "Order {order} is on the way, arriving {day}" },
    eid_sale: { kind: "marketing", channels: ["push", "email"], text: "Eid sale: {pct}% off until Friday" },
  },
  {
    push: new FakeProvider(),
    sms: new FakeProvider({ "+8801811000002": [Outcome.Transient, Outcome.Transient] }),
    email: new FakeProvider({ "tanvir@bad-domain": [Outcome.Permanent] }),
  },
);

function send(now: number, r: SendRequest): void {
  const head = `  ${r.id.padEnd(16)} ${r.template.padEnd(13)} ${r.user.padEnd(6)} ->`;
  let a: Accepted;
  try {
    a = s.send(r, now);
  } catch (e) {
    if (!(e instanceof Rejected)) throw e;
    console.log(`${head} error: ${e.message}`);
    return;
  }
  let line = "queued " + a.queued.join(", ");
  if (a.due > now) line += ` for ${clock(a.due)} (quiet hours)`;
  if (a.queued.length === 0) line = "nothing to send";
  if (a.skipped.length > 0) line += ` (skipped ${a.skipped.join("; ")})`;
  console.log(`${head} ${line}`);
}

function run(until: number): void {
  for (let next = s.nextDue(); next !== undefined && next <= until; next = s.nextDue()) {
    const ev = s.step(next)!;
    let line: string = ev.outcome;
    if (ev.retryAt) line += `, retry at ${clock(ev.retryAt)}`;
    else if (ev.dead) line += ", moved to dead letters";
    else line += `: ${ev.msg.body}`;
    console.log(`  ${clock(ev.time)} ${ev.msg.channel.padEnd(5)} ${ev.msg.key.padEnd(19)} try ${ev.msg.attempt}: ${line}`);
  }
}

const t1 = 21 * 3600 + 59 * 60 + 50;
console.log("D1 21:59:50, the order service sends three requests:");
send(t1, { id: "evt-1", user: "rafi", template: "order_shipped", params: { order: "BD-7781", day: "Sunday" } });
send(t1, { id: "evt-1", user: "rafi", template: "order_shipped", params: { order: "BD-7781", day: "Sunday" } });
send(t1, { id: "evt-2", user: "nusrat", template: "order_shipped", params: { order: "BD-7790", day: "Monday" } });
console.log("workers:");
run(t1 + 10);

const t2 = 22 * 3600 + 5;
console.log("D1 22:00:05, the marketing team starts a campaign:");
send(t2, { id: "camp-7:rafi", user: "rafi", template: "eid_sale", params: { pct: "30" } });
send(t2, { id: "camp-7:nusrat", user: "nusrat", template: "eid_sale", params: { pct: "30" } });
send(t2, { id: "camp-7:tanvir", user: "tanvir", template: "eid_sale", params: { pct: "30" } });
send(t2, { id: "camp-8:rafi", user: "rafi", template: "eid_sale", params: { pct: "40" } });
send(t2, { id: "camp-9:tanvir", user: "tanvir", template: "eid_sale", params: {} });
console.log("workers:");
run(2 * 86400);
console.log(`delivered ${s.delivered}, dead letters ${s.dead.length}, still queued ${s.pending()}`);