Case Study · Messaging · 2022 — 2023

Notifications — async fan-out that doesn't drop messages

Event-driven delivery that fans out to SMS and email asynchronously, with retries on failure.

Problem

Notifications were sent synchronously inside request handlers, so a slow SMS provider would stall the user's request, and a provider outage would silently drop messages. Critical alerts — OTPs, payment confirmations — were occasionally never delivered, with no record of why.

Context & constraints

A high-throughput product sending OTPs, receipts, and alerts over both SMS (Arkesel) and email. Providers fail intermittently and rate-limit; delivery must survive that without blocking the user-facing request that triggered it.

Async deliverysending must never block the originating request

At-least-oncea transient failure must not lose the message

Observabilityevery send attempt is logged with its outcome

Architecture

Request path

App event

emit

→

Queue

BullMQ

→

Worker

consume

→

Provider

SMS / email

→

Delivery log

recorded

Failure recovery & consistency

Send fails

provider error

→

Backoff

exponential

→

Retry

n attempts

→

Dead letter

if exhausted

Implementation

Producer

·Thin emit() API called from request handlers
·Returns immediately after enqueue
·Per-message channel + priority metadata

Worker

·BullMQ consumers with concurrency control
·Exponential backoff on transient failures
·Dead-letter queue for exhausted retries

Channels

·SMS adapter (Arkesel) with rate limiting
·Email adapter (Nodemailer / SMTP)
·Pluggable interface for new providers

Key decisions

Queue between event and delivery

Decoupling the emit from the send means a slow or down provider can never stall the user's request. The request enqueues and returns; the worker handles delivery on its own timeline.

Exponential backoff + dead-letter queue

Transient provider failures are retried with growing delays so we don't hammer a struggling provider. Messages that exhaust retries land in a dead-letter queue for inspection, never silently lost.

Pluggable channel adapters

Each channel implements a common send interface, so adding a new provider or swapping one out is a localized change that doesn't touch the producer or queue.

Tradeoffs

Chose

·Async queue-based delivery

·At-least-once with idempotent consumers

·Dead-letter queue over silent drops

Gave up

·Exactly-once delivery (chose simpler at-least-once)

·Instant synchronous confirmation to the caller

·Lower infra footprint (Redis + workers required)

Outcome

user requests blocked by provider latency

100%

send attempts logged with outcome

↓

dropped critical messages via retry + DLQ