Retrieval-augmented generation (RAG) helps large language models incorporate external knowledge but suffers from two core flaws: long input contexts slow down inference, and the retrieval and generation stages are optimized separately with disjoint objectives. CLaRa attacks this tension by compressing retrieved documents into a continuous latent space, bypassing the need for verbose text while preserving semantic information needed for generation.

To build semantically rich and retrievable compressed vectors, CLaRa introduces SCP, a data synthesis framework that uses question-answering and paraphrase supervision to produce key-preserving representations. The reranker and generator are then trained end-to-end using a single language modeling loss, with gradients flowing through both modules via a differentiable top‑k estimator. This joint optimization theoretically aligns retrieval relevance with final answer quality, eliminating the mismatch between traditional reranking metrics and generation success.

The practical outcome is striking: CLaRa achieves state‑of‑the‑art compression and reranking performance even at a text compression rate of 16×, outperforming text‑based fine‑tuned baselines across multiple QA benchmarks. For builders, this demonstrates that moving retrieval and generation into a shared continuous space—and optimizing them together—can dramatically reduce context length without sacrificing fidelity, offering a path toward more efficient and coherent RAG systems.