Distributed Task Queue

Distributed Task Queue with templating, preferences, retries, and provider fallbacks.

Requirements

Functional Requirements

• Enqueue/send notifications across channels
• Templates, localization, user preferences

Non-functional Requirements

• High delivery success rate
• Resilience to provider issues with retries and DLQs

High-Level Design

• API → bus → router → channel workers → providers

Capacity & Sizing

• Notifications/sec, retries, concurrency per provider

Key Components

• API, Router, Channel workers, Provider adapters, Preferences store

Architecture

Data Model

• templates (template_id PK, name, locale, body)
• notifications (id PK, user_id, template_id, status, ts)
• deliveries (id PK, notif_id, channel, status, attempts)

APIs

• POST /api/notifications { userId, templateId, data }
• PUT /api/preferences/:userId
• GET /api/notifications/:id

Hot Path

1. Enqueue → route → deliver → ack

Caching & TTL

• Cache templates for minutes; preferences with short TTL

Scaling

• Partition by channel
• Autoscale workers
• Provider rate limits and pools

Trade-offs

• Throughput vs per-user ordering
• Personalization vs cacheability
• Provider lock-in vs abstraction

Failure Modes & Mitigations

• Provider outage → fallback
• Queue backlog → rate limit upstream
• Poison messages → DLQ handling

Observability

• Delivery success rate
• Worker lag
• Provider error codes