What is context compression in RAG and when should you use it?

Context compression reduces the size of retrieved content before sending it to the LLM, improving efficiency and focus. Extractive compression selects only the most relevant sentences or passages from each retrieved chunk. Abstractive compression uses a smaller LLM to summarize each chunk, retaining key information in fewer tokens. LLMLingua and similar tools identify and remove low-information tokens. Use context compression when retrieved chunks contain significant irrelevant content alongside relevant information, when multiple chunks have overlapping content, when the total retrieved context approaches or exceeds the LLM context window, and when you need to reduce LLM API costs by minimizing input tokens. The trade-off is additional latency from the compression step and potential loss of important details. Test compressed versus uncompressed performance on your evaluation set. Compression is most valuable when chunks are large or when retrieval precision is moderate, providing maximum benefit when there is significant noise in retrieved results.