How do you implement effective monitoring and alerting for production systems?

Production monitoring provides visibility into system health and enables rapid incident response. Four golden signals: latency (response time distribution: P50, P95, P99), traffic (request rate, active users), errors (error rate by type and endpoint), and saturation (resource utilization: CPU, memory, disk, network). Implementation: instrument application code with metrics (counters for requests/errors, histograms for latency), collect infrastructure metrics (node exporter for servers, cAdvisor for containers), and aggregate in a time-series database (Prometheus, InfluxDB, Datadog). Dashboards: create hierarchical dashboards: top-level overview (system health at a glance), service-level (per-service golden signals), and debugging (detailed metrics for investigation). Use Grafana or Datadog for visualization. Keep dashboards focused (one purpose per dashboard) and actionable. Alerting: implement SLO-based alerting (alert when error budget consumption rate threatens the SLO rather than on individual failures), use multi-window burn rates (fast burn triggers immediate alert, slow burn triggers ticket), require every alert to have a documented runbook, and implement notification routing (critical to PagerDuty, warning to Slack, info to dashboard). Alert design: reduce noise (alert only on actionable conditions), avoid alert fatigue (consolidate related alerts, adjust thresholds based on data), implement alert suppression during maintenance windows, and review alert effectiveness monthly (tune or remove alerts that never fire or always fire). Incident response: implement on-call rotation (PagerDuty, Opsgenie), define severity levels with response time expectations, maintain incident communication channels (dedicated Slack channel per incident), and conduct blameless post-mortems for significant incidents.