Sentence Transformers v6.0 introduces MultiVectorEncoder, a fourth model type for ColBERT-style late interaction retrieval. Instead of pooling a text into one vector, the model keeps one projected token embedding per token and defers interaction to scoring time: MaxSim sums, over query tokens, the highest cosine similarity against any document token. Documents are encoded independently and indexed offline, placing late interaction between a bi-encoder and a cross-encoder.

The tradeoff is retrieval quality for index size. Token-level matching preserves exact identifiers and multi-requirement queries that a single vector has to average away, and it is state of the art for visual document retrieval because page images are scored directly, with no OCR step. On 4,874 Natural Questions passages, LateOn produced 608,414 token vectors, roughly 124.8 per passage and about 42x the storage of a MiniLM dense index (62 KiB per passage). Compression changes that picture: the same vectors take 92 MB as a fast-plaid/PLAID index versus 311.5 MB of float32, landing near a 4096-dimensional dense index.

Loading matches the familiar API: MultiVectorEncoder(“lightonai/LateOn”) works for PyLate checkpoints, Stanford-NLP ColBERT checkpoints, bare transformers, and, with repository revisions, colpali-engine checkpoints. Queries and documents are asymmetric: encode_query and encode_document apply different prefixes, length caps, query expansion, and skiplists, and return one 2D (num_tokens, embedding_dim) tensor per input. model.similarity computes the all-pairs MaxSim matrix. Scores are close together and magnitude scales with query token count, so MeanMaxSim gives a bounded average cosine similarity instead.

The post covers three deployment patterns. Exhaustive MaxSim is exact and practical for a few thousand documents: 4,874 passages are searched in about 120 ms. Retrieve and rerank avoids a late-interaction index entirely, using a fast bi-encoder to narrow the corpus and the multi-vector model to rescore candidates, cheaper per pair than a cross-encoder. For scale, the post demonstrates fast-plaid (PLAID, approximate), Qdrant, Weaviate, and Vespa; at this corpus size the three databases reproduce exhaustive MaxSim scores to four decimals, while fast-plaid’s scores drift slightly. Qdrant and Vespa both recommend using late interaction for reranking rather than full scans at scale.

ColPali-style visual document retrieval runs through the same API: one page produced 755 token vectors against 25 for the query, and supported models span 252M to 8.8B parameters. Audio and video retrieval are zero-shot for vidore/colqwen-omni-v0.1, with no transcription step; sampling video at 0.5 fps cut VRAM from 20.8 GB to 12.5 GB. MaxSim decomposes exactly, so image heatmaps and text token maps show why each document ranked, and special tokens contributed 22.7% of one score. HierarchicalTokenPooling clusters document token vectors to shrink the index, cutting vector count 1.99x at pool_factor 2 while retaining 100.6% of unpooled retrieval performance on BEIR on average, and 99.0% at factor 3.

For inference, fp16 with Flash Attention reached 2.44x the throughput of fp32 with no measurable retrieval loss, though non-attend query expansion models like ColBERTv2 must use sdpa instead; OpenVINO and int8 quantization help on CPU at about 0.4% accuracy cost. MultiVectorNanoBEIREvaluator runs the 13 NanoBEIR subsets without extra preparation, with the post cautioning that it is a small proxy. Using it on same-backbone LateOn and DenseOn models, late interaction wins 9 of 13 datasets and the mean (0.6868 vs 0.6764 NDCG@10), loses on ArguAna, FiQA2018, SCIDOCS, and SciFact, and keeps a similar gap on full BEIR (57.22 vs 56.20). PyLate and colpali-engine users get a migration table, with one-way save compatibility: MultiVectorEncoder output is not loadable by those libraries.