Async

Synchronous processing is a lie. At some point, your request-response cycle will hit a wall: sending emails, processing images, charging credit cards, generating reports. The solution: message queues. Here’s how to use them without creating distributed system nightmares. Why Queues? W U W U i s i s t e t e h r h r o u → q → t u A e A q P u P u I └ e I W e ─ : o u → ─ → r e k : [ 3 [ e S + Q ↓ r e u s n s e d e u p c e r E o o m n T c a d a e i s s s l k s ] o s f ] a → s w → y [ a n P i R c r t e o i s c n p e g o s n s ─ s ─ e I ─ m ─ ( a ─ 2 g ─ 0 e ─ 0 ] ─ m ─ s → ─ ) ─ [ ─ C ─ h ─ a ─ r ─ g ─ e ─ ─ C ─ a ─ r ─ d ┘ ] → R e s p o n s e Benefits: ...

Your API is slow because it’s doing too much synchronously. Here’s when to reach for a message queue, and how to implement it without overcomplicating everything. When You Need a Queue Signs you need async processing: API response time dominated by side effects (emails, webhooks, analytics) Downstream service failures cascade to user-facing errors Traffic spikes overwhelm dependent services You need to retry failed operations automatically Work needs to happen on a schedule or with delay Signs you don’t: ...

Some work doesn’t belong in a web request. Sending emails, processing uploads, generating reports, syncing with external APIs — these tasks are too slow, too unreliable, or too resource-intensive to run while a user waits. Background jobs solve this by moving work out of the request cycle and into a separate processing system. The Basic Architecture ┌ │ └ ─ ─ ─ W ─ ─ e ─ ─ b ─ ─ ─ │ │ └ ─ A ─ ─ ─ p ─ ─ ─ p ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ ┘ ─ ─ ─ ─ ─ ─ ▶ ─ ┌ │ └ ▶ ─ ─ ┌ │ └ ─ R ─ ─ ─ ─ e ─ ─ Q ─ ─ s ─ ─ u ─ ─ u ─ ─ e ─ ─ l ─ ─ u ─ ─ t ─ ─ e ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ ┐ │ ┘ ─ ◀ ─ ─ ─ ─ ─ ─ ▶ ─ ┌ │ └ ─ ─ ─ ─ ─ W ─ ─ ─ o ─ ─ ─ r ─ │ ┘ ─ k ─ │ ─ e ─ ─ r ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ Producer: Web app enqueues jobs Queue: Stores jobs until workers are ready Workers: Process jobs independently Results: Optional storage for job outcomes Choosing a Queue Backend Redis (with Sidekiq, Bull, Celery) 1 2 3 4 5 6 7 8 9 # Celery with Redis from celery import Celery app = Celery('tasks', broker='redis://localhost:6379/0') @app.task def send_email(user_id, template): user = get_user(user_id) email_service.send(user.email, template) Pros: Fast, simple, good ecosystem Cons: Not durable by default (can lose jobs on crash) ...

Async Python lets you handle thousands of concurrent I/O operations with a single thread. No threads, no processes, no GIL headaches. But it requires thinking differently about how code executes. These patterns help you write async code that’s both correct and efficient. The Basics 1 2 3 4 5 6 7 8 9 10 11 12 13 14 import asyncio async def fetch_data(url: str) -> dict: # This is a coroutine - it can be paused and resumed async with aiohttp.ClientSession() as session: async with session.get(url) as response: return await response.json() # Running coroutines async def main(): data = await fetch_data("https://api.example.com/data") print(data) asyncio.run(main()) await pauses the coroutine until the result is ready, letting other coroutines run. ...

When Service A needs to tell Service B something happened, the simplest approach is a direct HTTP call. But what happens when Service B is slow? Or down? Or overwhelmed? Message queues decouple your services, letting them communicate reliably even when things go wrong. Why Queues? Without a queue: U s e r R e q u e s t → A P I → P ( ( a i i y f f m e s d n l o t o w w n S , , e r u r v s e i e q c r u e e w s → a t i E t f m s a a ) i i l l s ) S e r v i c e → R e s p o n s e With a queue: ...