Webhooks are the internet’s way of saying “hey, something happened.” Simple in concept, surprisingly tricky in practice.
The challenge: HTTP is unreliable, servers go down, networks flake out, and your webhook payload might arrive zero times, once, or five times. Building reliable webhook infrastructure means handling all of these gracefully.
The Sender Side Retry with Exponential Backoff First attempt fails? Try again. But not immediately.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 RETRY_DELAYS = [60, 300, 900, 3600, 14400, 43200] # seconds def deliver_webhook(event, attempt=0): try: response = requests.post( event.webhook_url, json=event.payload, timeout=30, headers={ "Content-Type": "application/json", "X-Webhook-ID": event.id, "X-Webhook-Timestamp": str(int(time.time())), "X-Webhook-Signature": sign_payload(event.payload) } ) if response.status_code >= 200 and response.status_code < 300: mark_delivered(event) return if response.status_code >= 500: # Server error, retry schedule_retry(event, attempt) else: # Client error (4xx), don't retry mark_failed(event, response.status_code) except requests.Timeout: schedule_retry(event, attempt) except requests.ConnectionError: schedule_retry(event, attempt) def schedule_retry(event, attempt): if attempt >= len(RETRY_DELAYS): mark_failed(event, "max_retries") return delay = RETRY_DELAYS[attempt] # Add jitter to prevent thundering herd jitter = random.uniform(0, delay * 0.1) queue_at = time.time() + delay + jitter enqueue(event, attempt + 1, queue_at) Key decisions:
...