How do you implement automated data labeling and annotation for ML training?

Data labeling is a bottleneck for ML model development. Automation strategies reduce cost and time. Active learning: train an initial model on a small labeled set, use the model to identify the most informative unlabeled examples (high uncertainty, high disagreement between models), label only these examples, retrain, and repeat. This achieves equivalent model performance with 50-80% fewer labeled examples. Semi-supervised learning: train on a small labeled set, use the model to pseudo-label unlabeled data (only accept high-confidence predictions), add pseudo-labeled examples to the training set, and retrain iteratively. Useful when unlabeled data is abundant. LLM-assisted labeling: use GPT-4 or Claude to label data with structured prompts (provide labeling guidelines, examples, and output schema), validate LLM labels against human labels on a sample, and use LLM labels as initial annotations for human review (correct rather than create from scratch). This can reduce annotation time by 60-70%. Weak supervision: define labeling functions (heuristic rules that noisily label data), aggregate multiple noisy labels using a model (Snort/Snorkel framework), and train on the aggregated probabilistic labels. Scales well when domain experts can express rules. Annotation tools: Label Studio (open-source, multi-modal), Prodigy (active learning built-in, spaCy integration), Scale AI (managed workforce), and Amazon SageMaker Ground Truth (AWS-integrated). Quality control: implement inter-annotator agreement measurement (Cohen's kappa, Fleiss' kappa), require consensus from multiple annotators for ambiguous cases, implement gold standard questions (pre-labeled examples to check annotator quality), and provide clear annotation guidelines with examples. Data management: version labeled datasets alongside code, track labeling progress and quality metrics, implement label review workflows, and maintain annotation guidelines as living documents.