Queues

Some work doesn’t belong in a web request. Sending emails, processing uploads, generating reports, syncing with external APIs — these tasks are too slow, too unreliable, or too resource-intensive to run while a user waits. Background jobs solve this by moving work out of the request cycle and into a separate processing system. The Basic Architecture ┌ │ └ ─ ─ ─ W ─ ─ e ─ ─ b ─ ─ ─ │ │ └ ─ A ─ ─ ─ p ─ ─ ─ p ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ ┘ ─ ─ ─ ─ ─ ─ ▶ ─ ┌ │ └ ▶ ─ ─ ┌ │ └ ─ R ─ ─ ─ ─ e ─ ─ Q ─ ─ s ─ ─ u ─ ─ u ─ ─ e ─ ─ l ─ ─ u ─ ─ t ─ ─ e ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ ┐ │ ┘ ─ ◀ ─ ─ ─ ─ ─ ─ ▶ ─ ┌ │ └ ─ ─ ─ ─ ─ W ─ ─ ─ o ─ ─ ─ r ─ │ ┘ ─ k ─ │ ─ e ─ ─ r ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ Producer: Web app enqueues jobs Queue: Stores jobs until workers are ready Workers: Process jobs independently Results: Optional storage for job outcomes Choosing a Queue Backend Redis (with Sidekiq, Bull, Celery) 1 2 3 4 5 6 7 8 9 # Celery with Redis from celery import Celery app = Celery('tasks', broker='redis://localhost:6379/0') @app.task def send_email(user_id, template): user = get_user(user_id) email_service.send(user.email, template) Pros: Fast, simple, good ecosystem Cons: Not durable by default (can lose jobs on crash) ...

Synchronous request-response is simple: client asks, server answers, everyone waits. But some operations don’t fit that model. Sending emails, processing images, generating reports — these take time, and your users shouldn’t wait. Message queues decouple the “request” from the “work,” letting you respond immediately while processing happens in the background. The Basic Pattern 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # API endpoint - returns immediately @app.post("/orders") def create_order(order_data): order = db.create_order(order_data) # Queue async work instead of doing it now queue.enqueue('process_order', order.id) return {"order_id": order.id, "status": "processing"} # Worker - processes queue in background def process_order(order_id): order = db.get_order(order_id) charge_payment(order) send_confirmation_email(order) notify_warehouse(order) update_inventory(order) The user gets a response in milliseconds. The actual work happens whenever the worker gets to it. ...