How do you implement data quality frameworks for reliable analytics?
Data quality frameworks ensure data is accurate, complete, consistent, timely, and valid throughout the pipeline. Dimensions: accuracy (data correctly represents real-world values), completeness (all required data is present), consistency (same data agrees across systems), timeliness (data is available within expected latency), uniqueness (no unintended duplicates), and validity (data conforms to expected format and business rules). Implementation with dbt tests: schema tests (notnull, unique, acceptedvalues, relationships for foreign keys), custom SQL tests (complex business rules, cross-table validations), and dbt-expectations package (statistical tests, distribution checks). Great Expectations: define expectation suites per dataset (expectcolumnvaluestonotbenull, expectcolumnmeantobebetween, expectcolumnvaluestomatchregex), run validations after each pipeline run, and generate data quality reports. Quality monitoring: implement quality scores per dataset (weighted average of dimension-specific checks), track quality trends over time, alert on quality degradation, and maintain SLAs per dataset with quality thresholds. Data observability: tools like Monte Carlo, Anomalo, or Elementary provide automated anomaly detection on data pipelines, identifying volume changes, schema changes, distribution shifts, and freshness issues without manually defining rules for every table. Root cause analysis: when quality issues are detected, trace lineage to identify the source (was it a source system issue, transformation bug, or ingestion failure), implement automated incident creation, and maintain a quality issue log for pattern analysis. Remediation: implement quarantine patterns (isolate bad data, process good data), automated fixes for known patterns (date format standardization, null filling with defaults), and human review workflows for complex issues. Prevention: implement schema contracts between producers and consumers, validate data at ingestion points, and shift quality checks left (earlier in the pipeline).