How do you handle multi-modal RAG with images, tables, and text?

Multi-modal RAG processes diverse content types: use ColPali or ColQwen for document-image retrieval that understands visual layout, apply table parsers (Camelot, Tabula, or LLM-based extraction) to convert tables to structured markdown, and embed images using CLIP or SigLIP alongside text embeddings. Architecture: during indexing, segment documents into text chunks, table chunks (as markdown), and image descriptions (via LLaVA or GPT-4V captioning). Store each modality with its embedding and metadata indicating content type. At retrieval time, convert the query to both text and image embeddings, search across all modalities, and pass multi-modal context to the LLM. For tables specifically, convert to markdown format which LLMs handle well, and include the table caption and surrounding context. For charts and diagrams, generate detailed text descriptions during indexing. Key challenge: cross-modal relevance scoring requires calibrating similarity scores across different embedding spaces. Solution: train a lightweight fusion model or use separate retrieval per modality with RRF combination.