Lexical search systems rely on statistical ranking models that determine how well a document matches a query based on word frequency and distribution. The most common models are TF-IDF (Term Frequency–Inverse Document Frequency) and BM25 (Best Matching 25). TF-IDF scores documents by multiplying how often a term appears in a document (TF) by how unique that term is across all documents (IDF). This ensures that common words like “the” or “and” receive low scores, while rare, meaningful terms like “embedding” or “Milvus” are weighted higher. TF-IDF is simple, interpretable, and efficient, making it suitable for many applications.
BM25 extends TF-IDF by introducing normalization factors that account for document length and diminishing term frequency returns. It includes two main parameters: k1, which controls the influence of term frequency, and b, which adjusts for document length. This makes BM25 more robust for heterogeneous datasets where document sizes vary widely—for example, mixing short technical notes with long research papers. Tuning these parameters helps balance precision and recall depending on the use case. Many modern search engines default to BM25 due to its strong empirical performance across domains.
Beyond these, newer Lexical ranking methods incorporate learning-to-rank (LTR) approaches that combine multiple features—such as BM25 scores, click data, or recency—using machine learning models. In hybrid systems that integrate with Milvus, Lexical ranking models like BM25 can serve as the first-stage ranker, providing a shortlist of candidates that are then semantically re-ranked using vector similarity. This design leverages BM25’s explainability and Milvus’s semantic strength. The combination allows developers to achieve reliable, interpretable, and context-aware search results optimized for both keyword precision and conceptual understanding.