How do you implement observability for distributed cloud systems?

Observability provides insight into system behavior through three pillars: metrics, logs, and traces. Metrics: numerical measurements sampled over time (request rate, error rate, latency percentiles, CPU usage, memory usage). Implement using Prometheus (pull-based collection), StatsD/Datadog (push-based), or CloudWatch. Define SLIs (Service Level Indicators), SLOs (Service Level Objectives), and error budgets for each service. Logs: structured event records in JSON format with consistent fields (timestamp, level, service, requestid, userid, message). Implement centralized log aggregation (ELK stack: Elasticsearch, Logstash, Kibana; or Grafana Loki for cost-effective storage). Correlation: propagate request IDs across service boundaries for log correlation. Traces: distributed execution paths showing how requests traverse multiple services. Implement using OpenTelemetry (vendor-neutral instrumentation standard) with backends like Jaeger, Zipkin, Tempo, or Datadog APT. Auto-instrument HTTP clients, database drivers, and message consumers for comprehensive trace coverage. Implementation: adopt OpenTelemetry as the instrumentation standard (SDKs for all major languages), deploy collectors for data aggregation and routing, and implement sampling strategies (head-based or tail-based) to manage data volume. Dashboards: create golden signal dashboards per service (latency, traffic, errors, saturation), implement service maps showing dependencies and health, and create on-call dashboards optimized for incident response. Alerting: implement SLO-based alerting (alert when error budget consumption rate threatens the SLO), use multi-window burn rates to balance sensitivity and noise, and require actionable runbooks for every alert. Cost management: implement retention policies (detailed data for 7-14 days, aggregated for months), sample high-volume traces, and compress logs before storage.