Haystack performs document ranking primarily through a combination of embedding models and BM25 retrieval techniques. When a query is presented, Haystack first generates embeddings for both the documents and the query using a pre-trained model. This process transforms text into high-dimensional vectors that capture the semantic meaning of the sentences. By computing the similarity between the query embedding and the document embeddings, Haystack can rank documents based on how closely they relate to the input query.
In addition, Haystack uses a traditional information retrieval method called BM25, which is a probabilistic retrieval function. BM25 evaluates the relevance of documents based on factors like term frequency (how often a term appears in a document), inverse document frequency (how rare a term is across the entire dataset), and the length of the documents. This model allows it to normalize the document's terms to ensure that longer documents do not skew results unfairly. Developers can adjust BM25 parameters to fine-tune how strictly they want their document ranking to adhere to these factors.
Ultimately, Haystack can combine these methods effectively to provide a more refined ranking of documents. The document ranking process is flexible; developers can choose to prioritize either the semantic similarity from embeddings or the lexical relevance provided by BM25, based on their application needs. This adaptability allows for optimized search and retrieval experiences, catering to various use cases such as chatbots or advanced search functionalities.