Async

Some work doesn’t belong in a web request. Sending emails, processing uploads, generating reports, syncing with external APIs — these tasks are too slow, too unreliable, or too resource-intensive to run while a user waits. Background jobs solve this by moving work out of the request cycle and into a separate processing system. The Basic Architecture ┌ │ └ ─ ─ ─ W ─ ─ e ─ ─ b ─ ─ ─ │ │ └ ─ A ─ ─ ─ p ─ ─ ─ p ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ ┘ ─ ─ ─ ─ ─ ─ ▶ ─ ┌ │ └ ▶ ─ ─ ┌ │ └ ─ R ─ ─ ─ ─ e ─ ─ Q ─ ─ s ─ ─ u ─ ─ u ─ ─ e ─ ─ l ─ ─ u ─ ─ t ─ ─ e ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ ┐ │ ┘ ─ ◀ ─ ─ ─ ─ ─ ─ ▶ ─ ┌ │ └ ─ ─ ─ ─ ─ W ─ ─ ─ o ─ ─ ─ r ─ │ ┘ ─ k ─ │ ─ e ─ ─ r ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ Producer: Web app enqueues jobs Queue: Stores jobs until workers are ready Workers: Process jobs independently Results: Optional storage for job outcomes Choosing a Queue Backend Redis (with Sidekiq, Bull, Celery) 1 2 3 4 5 6 7 8 9 # Celery with Redis from celery import Celery app = Celery('tasks', broker='redis://localhost:6379/0') @app.task def send_email(user_id, template): user = get_user(user_id) email_service.send(user.email, template) Pros: Fast, simple, good ecosystem Cons: Not durable by default (can lose jobs on crash) ...

Async Python lets you handle thousands of concurrent I/O operations with a single thread. No threads, no processes, no GIL headaches. But it requires thinking differently about how code executes. These patterns help you write async code that’s both correct and efficient. The Basics 1 2 3 4 5 6 7 8 9 10 11 12 13 14 import asyncio async def fetch_data(url: str) -> dict: # This is a coroutine - it can be paused and resumed async with aiohttp.ClientSession() as session: async with session.get(url) as response: return await response.json() # Running coroutines async def main(): data = await fetch_data("https://api.example.com/data") print(data) asyncio.run(main()) await pauses the coroutine until the result is ready, letting other coroutines run. ...

When Service A needs to tell Service B something happened, the simplest approach is a direct HTTP call. But what happens when Service B is slow? Or down? Or overwhelmed? Message queues decouple your services, letting them communicate reliably even when things go wrong. Why Queues? Without a queue: U s e r R e q u e s t → A P I → P ( ( a i i y f f m e s d n l o t o w w n S , , e r u r v s e i e q c r u e e w s → a t i E t f m s a a ) i i l l s ) S e r v i c e → R e s p o n s e With a queue: ...