How do you optimize embedding models for domain-specific RAG?

Domain-specific embedding optimization improves retrieval accuracy for specialized content. Approaches: (1) Fine-tune an existing embedding model on domain data using contrastive learning (train the model to produce similar embeddings for related domain concepts and dissimilar embeddings for unrelated ones). Training data: create positive pairs (query + relevant document), hard negatives (query + superficially similar but irrelevant document), and use mining techniques to find challenging negatives. Frameworks: sentence-transformers makes fine-tuning accessible with MultipleNegativesRankingLoss or ContrastiveLoss. (2) Matryoshka embeddings: train models to produce useful embeddings at multiple dimensions (truncate 768-dim embedding to 256-dim with minimal quality loss), enabling flexible accuracy/speed trade-offs. (3) Domain vocabulary extension: ensure the tokenizer handles domain terms correctly (medical, legal, financial terminology), potentially extending the tokenizer vocabulary and continuing pre-training. Evaluation: create a domain-specific benchmark with query-relevant document pairs, measure NDCG@10, recall@10, and MRR before and after optimization, and compare against general-purpose embedding models. Data requirements: fine-tuning typically needs 1000-10000 query-document pairs, with quality mattering more than quantity. Synthetic data generation: use an LLM to generate queries from domain documents (given this passage, what question would it answer), creating training pairs at scale. A/B testing: deploy optimized embeddings alongside general models, route a percentage of traffic to each, and compare retrieval quality metrics. Continuous improvement: as new domain content is added, periodically re-evaluate and potentially re-fine-tune embedding models to maintain optimal performance.