What are the design patterns for building resilient automated workflows?

Resilient workflow design patterns: (1) Retry with backoff: exponential backoff starting at 1s, doubling to max 60s, with jitter to prevent thundering herd. (2) Circuit breaker: track failure rate per dependency, open circuit at 50% failure rate over 10 requests, half-open after 30s to test recovery. (3) Timeout: set per-activity timeouts slightly above P99 latency, workflow-level timeout as the sum of critical path activities plus buffer. (4) Bulkhead: isolate workflow queues by priority tier so high-priority workflows aren't blocked by bulk operations. (5) Idempotency: use deterministic IDs for every operation, implement idempotency keys at API boundaries, and design state transitions to be re-executable. (6) Dead letter queue: capture permanently failed work items with full context for investigation. (7) Heartbeat: long-running activities send periodic heartbeats; missing heartbeats trigger timeout and reassignment. (8) Poison pill detection: identify and isolate work items that consistently cause failures. (9) Graceful degradation: define fallback behavior for non-critical steps. Combine patterns: a production workflow typically uses retries within a circuit breaker, with idempotent activities, bulkhead isolation, and dead letter queues.