How do you implement automated infrastructure provisioning for AI workloads?
AI infrastructure requires GPU compute, high-bandwidth storage, and specialized networking. Architecture: compute layer (GPU instances for training and inference), storage layer (high-throughput storage for training data, model artifacts), networking (high-bandwidth interconnects for distributed training), and orchestration (job scheduling and resource management). GPU infrastructure: NVIDIA A100/H100 for training (multi-GPU with NVLink for fast inter-GPU communication), NVIDIA L4/T4 for inference (cost-effective for serving), and cloud GPU instances (AWS p4d/p5, GCP a3, Azure ND) for elastic scaling. Kubernetes for ML: deploy with GPU operator (manages NVIDIA drivers and device plugins), implement resource quotas per team (GPU limits, memory limits), use priority classes for job preemption (training jobs preempt inference during off-peak), and implement node pools with different GPU types for different workload requirements. Storage: high-throughput shared storage for training data (FSx for Lustre on AWS, Filestore on GCP, or cloud object storage with caching), model registry storage (S3/GCS for model artifacts), and fast local NVMe for dataset caching during training. Orchestration: use Kubernetes with custom schedulers for GPU-aware scheduling, implement job queuing (Volcano, Kueue) for training job management, and auto-scaling based on job queue depth. Infrastructure as Code: define GPU clusters with Terraform/Pulumi, implement auto-scaling policies for inference serving, and maintain separate compute pools for training (batch, preemptible) and inference (stable, low-latency). Cost optimization: use spot/preemptible instances for training (implement checkpointing for interruption recovery), right-size GPU instances based on model requirements, implement GPU sharing for inference (multiple models on one GPU), and schedule training during off-peak hours for lower spot pricing. Monitoring: track GPU utilization (target 80% during training), memory usage, job queue wait times, and cost per training run.