How do you implement webhook systems for event-driven integrations?

Webhooks provide real-time event notification between systems via HTTP callbacks. Architecture: event producer detects a state change, serializes the event data, sends an HTTP POST to registered webhook URLs, and handles responses. Implementation for webhook provider: implement a webhook registration API (URL, events to subscribe to, secret for signature verification), maintain a registry of active subscriptions, implement a delivery pipeline (serialize event, compute HMAC signature, send HTTP POST, handle response), and implement retry logic for failed deliveries. Delivery reliability: retry failed deliveries with exponential backoff (1s, 10s, 60s, 300s, 3600s), implement maximum retry attempts (typically 5-10), track delivery status per subscription, and implement circuit breaker for consistently failing endpoints (disable after N consecutive failures, re-enable after recovery period). Security: sign payloads using HMAC-SHA256 (include the signature in a header like X-Webhook-Signature), provide a shared secret per subscription, and consumers verify the signature before processing (prevents spoofed webhook deliveries). Consumer implementation: expose an HTTPS endpoint for receiving webhooks, verify the signature immediately (reject if invalid), respond with 200 quickly (process asynchronously), implement idempotency using a delivery ID (prevent duplicate processing), and store the raw event for debugging. Ordering and consistency: webhooks may arrive out of order (network timing, retries), include a timestamp and sequence number, consumers should handle out-of-order delivery (use timestamps to resolve conflicts), and include a resource URL for consumers to fetch the current state if the webhook payload might be stale. Monitoring: track delivery success rates per subscriber, alert on high failure rates, monitor webhook latency (time from event to delivery), and provide a delivery log UI for subscriber debugging.