Message Queue & Async Processing

Decouple services, buffer load spikes, and enable async workflows

Queue Basics

0
Queue Depth
0
Produced
0
Consumed
0
Msg/sec

Message Queue Guarantees

Queues act as a buffer between producers and consumers. When consumers are slower than producers, the queue absorbs the spike. This prevents cascading failures under load.

FIFO orderingAt-least-once deliveryDurability via persistenceHorizontal consumer scaling

Producer-Consumer Pattern

0
Queue Depth
0
Consumer Lag

When producers outpace consumers, queue depth grows. Tune consumer count or processing speed. Consumer lag = number of messages waiting. High lag means consumers are falling behind — scale out or optimize processing.

Scale consumers independentlyBackpressure signals to producersCircuit breaker on overflow

Dead Letter Queue (DLQ)

0
Total Processed
0
Success
0
In DLQ
0
Retries

Why DLQ?

When a message fails processing repeatedly (network error, bad data, bug), retrying forever blocks the queue. After max retries, move it to the DLQ for inspection and manual replay. DLQ = safety net for poison messages.

AWS SQS DLQKafka dead-letter topicsRabbitMQ x-dead-letter

Publish / Subscribe

Delivery Guarantees

At-Most-Once

Send and forget. May lose messages. Fast, no overhead. Used for logs, metrics where loss is acceptable.

At-Least-Once

Retry until acknowledged. May deliver duplicates. Consumer must be idempotent. Most common guarantee.

Exactly-Once

Complex — requires distributed transactions or idempotency keys. Kafka with transactions, or deduplication layer.