A bi-encoder in Sentence Transformers refers to a model architecture where two input sentences are encoded independently into dense vector representations (embeddings) using the same or separate neural networks. The goal is to map semantically similar sentences closer together in the embedding space, enabling tasks like semantic similarity comparison or retrieval. Unlike architectures that process sentence pairs jointly (e.g., cross-encoders), bi-encoders handle each sentence separately, making them computationally efficient for large-scale applications.
Bi-encoders work by processing each sentence through a shared or dual encoder network. For example, a model like all-MiniLM-L6-v2 uses a single transformer-based encoder to generate embeddings for both sentences independently. During training, pairs of sentences (e.g., a query and a relevant document) are fed into the encoder, and the model optimizes a contrastive loss function (like cosine similarity loss or triplet loss) to ensure that related sentences have embeddings with high similarity scores. This approach avoids cross-attention between sentences, reducing computation during inference. Once trained, the embeddings of new sentences can be precomputed and stored, allowing fast similarity searches using metrics like cosine similarity or dot product.
Bi-encoders are widely used in retrieval tasks, such as finding relevant documents for a search query or detecting paraphrases. For instance, in a FAQ retrieval system, a bi-encoder can precompute embeddings for all FAQ entries. When a user submits a query, the system encodes it and compares it against precomputed embeddings to find the closest matches efficiently. This efficiency makes bi-encoders suitable for real-time applications, though they may trade off some accuracy compared to cross-encoders, which analyze sentence pairs interactively. Popular bi-encoder models in Sentence Transformers include multi-qa-MiniLM-L6-dot-v1, optimized for retrieval, and paraphrase-MiniLM-L3-v2, designed for paraphrase detection.